missing messages when DC ships bad utf8 #20

New Issue

mappu04 · 2018-11-18T07:09:03Z

mappu04 commented

2018-11-18 07:09:03 +00:00

We're dealing with raw bytes almost everywhere. It's assumed these are UTF-8 strings.

When DC sends us a message that's not valid UTF-8 (e.g. today, a pasted em-dash in Windows-1252 encoding(?)) TG silently(?) drops the message.

We should either

better handle error messages returned from the message-submission API, or
filter out invalid characters before submitting them to TG

We're dealing with raw bytes almost everywhere. It's assumed these are UTF-8 strings. When DC sends us a message that's not valid UTF-8 (e.g. today, a pasted em-dash in Windows-1252 encoding(?)) TG silently(?) drops the message. We should either - better handle error messages returned from the message-submission API, or - filter out invalid characters before submitting them to TG

mappu04 added the

bug

label 2018-11-18 07:09:39 +00:00

mappu04 commented

2019-04-01 05:43:47 +00:00

Attempt to parse all incoming messages as UTF-8. If they fail, replace invalid characters with U+FFFD REPLACEMENT CHARACTER

mappu04 commented

2019-05-17 06:07:44 +00:00

2019/04/07 13:02:08 Delivering PM to telegram user: Bad Request: strings must be encoded in UTF-8

``` 2019/04/07 13:02:08 Delivering PM to telegram user: Bad Request: strings must be encoded in UTF-8 ```

mappu04 commented

2020-03-26 00:05:24 +00:00

2020/03/26 11:12:28 Delivering public message to group chat: Bad Request: text must be encoded in UTF-8

``` 2020/03/26 11:12:28 Delivering public message to group chat: Bad Request: text must be encoded in UTF-8 ```

Sign in to join this conversation.

No Label

bug

wishlist

No Milestone

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: code.ivysaur.me/nmdc-telegramfrontend#20