Python requests post utf 8 encoding
I'm trying to post a snippet of text containing fancy unicode symbols to a web service using the requests library. I'm using Python 3.5.
I get an UnicodeEncodeError, but I can't figure out what I'm doing wrong on my side, the docs for requests only talk about unicode in GET requests from what I see.
WORKAROUND: I remove all unicode characters from the text from the "emoticon" block, U+1F600 - U+1F64F and Symbols And Pictographs" block, U+1F300 - U+1F5FF according to this answer with the following code, since I don't need emoticons and pictures for the analysis:
UPDATE The creator of the web service has fixed this now and updated the documentation. All you have to do is to send an encoded string, in Python 3: Apr-25-2019, 05:25 PM (This post was last modified: Apr-25-2019, 06:00 PM by MaverinCode.) I'm trying to send a unicode string, which contains chinese characters, from a python program to a web server. The code for this is: import json import requests #get json data = json.dumps({"Input": {"Row": [{"BusinessCard": "那是一句话"}]}}, ensure_ascii=False) #encode json to utf-8 encoded_data = data.encode('utf-8') r = requests.post("http://some-server.com/service", data=encoded_data, headers={'Content-Type': 'application/json; charset=UTF-8'}, auth=requests.auth.HTTPBasicAuth("user", "password")) The encoded_data variable: b'{"Input": {"Row": [{"BusinessCard": "\xe9\x82\xa3\xe6\x98\xaf\xe4\xb8\x80\xe5\x8f\xa5\xe8\xaf\x9d"}]}}' My problem is, if send this to the server, the server doesn't recognize the utf-8 encoded data as chinese characters. The server just sees '"\xe9\x82\xa3...' and can't handle them. But I'm forced to encode the data to utf-8 or the post method will give me the this unicode error:
If I send the request from Postman, it works fine. The postman request is similiar, there is only the header for basic authentication and the content type header. Also, if I send from the python program a test request to a simple spring boot server with a rest controller, the controller/spring recognizes the encoded values right as chinese characters. How can I send the json data in the body like Postman does, so that the server understands the chinese characters? I searched a few hours but couldn't find any answer to a similar case like this. I would be really thankful for some help. Greetings Ok, got some help from Stack Overflow. I set the ensure_ascii parameter to 'True' and it works. But I would be thankful for some further explanation if someone knows more. The documentation states: "If ensure_ascii is true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If ensure_ascii is false, these characters will be output as-is." Why have servers no trouble to encode escaped unicode signs which have been encoded to utf-8 but trouble with none escape unicode signs which have been also been encode to utf-8? JaiM Posts: 1 Threads: 0 Joined: Nov 2020 Reputation: 0 Nov-08-2020, 06:45 AMVery Useful when we try to send data with Chinese Characters via REST API. Thank you for your post. |