Quantcast
Channel: Active questions tagged utf-8 - Stack Overflow
Viewing all articles
Browse latest Browse all 1053

How to correctly represent a supplementary unicode char in python3 (3.6.1+) by using \u or \U escape within string

$
0
0

Recently I'm learing python and has encountered a problem with unicode escape literal in python 3.

It seems that like Java, the \u escape is interpreted as UTF-16 code point which Java uses, but here comes the problem:

For example, if I try to put a 3 bytes utf-8 char like "♬" (https://unicode-table.com/en/266C/) or even supplementary unicode char like "𠜎" (https://unicode-table.com/en/2070E/) by the format of \uXXXX or \UXXXXXXXX in a normal string as followed:

print('\u00E2\u99AC')  # UTF-8, messy code for sureprint('\U00E299AC')    # UTF-8, with 8 bytes \U, (unicode error) for sureprint('\u266C')        # UTF-16 BE, music note appeares# from which I suppose \u and \U function the same way they should do in Java# (may be a little different since they function like macro in Java, and can be useed in comments)# However, while print('\u266C') gives me '♬','\u266C' == '♬' is equal to false# which is true in Java semantics.# Further more, print('\UD841DF0E') didn't give me '𠜎' : (unicode error) 'unicodeescape' codec can't decode bytes in position 0-9: illegal Unicode character# which I suppose it should be, so it appears to me that I may get it wrong# Here again : print('\uD841\uDF0E')  # Error, 'utf-8' codec can't encode characters in position 0-1: surrogates not allowedprint('\xD8\x41\xDF\x0E')  # also tried this, messy code# maybe UTF-16 LE?print('\u41D8\u0EDF')  # messy codeprint('\U41D80EDF')  # error

So, I could see that python "doesn't support supplementary escape literal", and its behavior is also weird.

Well, I already know that the correct way to decode and encode such characters:

s_decoded = '\\xe2\\x99\\xac'.encode().decode('unicode-escape')\               .encode('latin-1').decode('utf-8')print(b'\xf0\xa0\x9c\x8e'.decode('utf-8'))print(b'\xd8\x41\xdf\x0e'.decode('utf-16 be'))assert s_decoded == '♬'

But still don't get how to do it right using \u & \U escape literal. Hopefully someone could point it out what I'm doing wrong and how it differs from Java's way, thanks!

By the way, my environment is PyCharm win, python 3.6.1, source code is encoded as UTF-8


Viewing all articles
Browse latest Browse all 1053

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>