I read data from salesforce and put this into a pandas dataframe. When I try to print the result I get an unicode encode error.
First I read data from source and put the result into a pandas dataframe.
# query to executesql_code = """ Select Id, ResponseShortText FROM SurveyQuestionResponse """# get datadf_xyz_raw = pd.DataFrame(sf.query_all(query = sql_code)["records"])
When I print the result a get this error.
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 229: surrogates not allowed
I try to encode and decode the data.
df_xyz_raw ["ResponseShortText"] = df_xyz_raw ["ResponseShortText"].str.encode('utf-8', errors='ignore').str.decode('utf-8')
This works but only because of errors=ignore.
I think it is an emoji which is not part of utf-8? Does that mean that the source system is using another unicode format in comparison to utf-8 in my python environment?
Is their any way to handle these character and print them?