Comments and answers for "JSONFragmenter UTF-8 Encoding Issue" https://knowledge.safe.com/questions/89147/jsonfragmenter-utf-8-encoding-issue.html The latest comments and answers for the question "JSONFragmenter UTF-8 Encoding Issue" Comment by lynn_bryant on lynn_bryant's answer https://knowledge.safe.com/comments/89269/view.html

Thanks David. Looks like I found a bug!

 Thu, 28 Mar 2019 14:18:15 GMT lynn_bryant Comment by lynn_bryant on lynn_bryant's answer https://knowledge.safe.com/comments/89268/view.html

Thanks Holly!

 Thu, 28 Mar 2019 14:17:43 GMT lynn_bryant Answer by hollyatsafe https://knowledge.safe.com/answers/89237/view.html

Hi @lynn_bryant,

For the Text File Reader this is a warning rather than an error so with the link provided you should be able to continue this translation without any problems. However I can reproduce the warning in FME Desktop and can see it is complaining about the File Name rather than the data itself so I do not believe this should have any impact on your workflow. You can confirm this by saving the data as a text file first without any of those characters and this warning is no longer in the log. Having said that I do believe this is a bug because those characters are valid in UTF-8 encoding so have filed FMEENGINE-59626 to get this corrected.

With regard to the JSONFragmenter warning this is because the HTTP response is not actually UTF-8 but is actually windows-1252 encoding so although most of the characters will be the same, some might be incorrect. The JSONFragmenter should be able to handle this so I have filed FMEENGINE-59627. In the meantime the workaround to remove this warning from the log is by using the HTTPCaller to download the file to an attribute as this should handle the encoding correctly (this will also remove the need for the Text Reader that was causing problems as well).


EDIT: Both these issues have been fixed for 2019.1.

Wed, 27 Mar 2019 23:56:12 GMT hollyatsafe Comment by lynn_bryant https://knowledge.safe.com/comments/89224/view.html

These are the errors reported out from FME Server:

Text File Reader: Renamed attribute 'ATT928hoi71!789$k2F' to 'ATT928hoi71_789_k2F' to remove the following invalid characters '&-@$!~()'
JSONFragmenter(JSONQueryFactory): A sequence of bytes was found which is invalid in the utf-8 encoding.
 Wed, 27 Mar 2019 20:21:05 GMT lynn_bryant Comment by ebygomm on ebygomm's comment https://knowledge.safe.com/comments/89213/view.html

I cannot recreate the error

 Wed, 27 Mar 2019 18:12:29 GMT ebygomm Comment by lynn_bryant https://knowledge.safe.com/comments/89209/view.html

Here are the settings being used

 Wed, 27 Mar 2019 17:41:05 GMT lynn_bryant Comment by ebygomm on ebygomm's comment https://knowledge.safe.com/comments/89208/view.html

What are your settings in the JSONfragmenter?

Wed, 27 Mar 2019 17:37:26 GMT ebygomm Comment by lynn_bryant https://knowledge.safe.com/comments/89207/view.html < div class = " fr-view clearfix " > < p > JDH——谢谢你your response. This is the public link being used: https://poweroutage.us/api/getcountyoutageinfo/ATT928hoi71!789$k2F

 Wed, 27 Mar 2019 17:31:13 GMT lynn_bryant Answer by david_r https://knowledge.safe.com/answers/89165/view.html

Are you 100% certain that the file is in fact encoded as utf-8?

If you can open the file in e.g. Notepad++ you can look in the Encoding menu to verify the encoding, it should look like this:

Without changing anything in the Encoding menu, verify that the special characters are properly displayed.

If either of those two checks fail, there's an encoding problem in the input file.

 Wed, 27 Mar 2019 07:37:54 GMT david_r Comment by jdh https://knowledge.safe.com/comments/89154/view.html

Can you provide a sample? I can't reproduce the problem.

 Tue, 26 Mar 2019 21:37:00 GMT jdh