℞ 19: Open file with specific encoding While setting the default Unicode encoding for IO is sensible, sometimes the default encoding is not correct. In this case, specify the encoding for a filehandle manually in the mode option to open…
℞ 18: Make all I/O and args default to utf8 The core rule of Unicode handling in Perl is “always encode and decode at the edges of your program”. If you’ve configured everything such that all incoming and outgoing data…
℞ 17: Make file I/O default to utf8 If you’ve ever had the misfortune of seeing the Unicode warning “wide character in print”, you may have realized that something forgot to set the appropriate Unicode-capable encoding on a filehandle somewhere…
℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR. While it may be most common for modern…
℞ 15: Declare STD{IN,OUT,ERR} to be UTF-8 Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR. As documented in perldoc perlrun, the PERL_UNICODE environment variable or…
℞ 14: Decode program arguments as locale encoding While it may be most common in modern operating systems for your command-line arguments to be encoded as UTF-8, @ARGV may use other encodings. If you have configured your system with a…
℞ 13: Decode program arguments as utf8 While the standard Perl Unicode preamble makes Perl’s filehandles use UTF-8 encoding by default, filehandles aren’t the only sources and sinks of data. The command-line arguments to your programs, available through @ARGV, may…
℞ 12: Explicit encode/decode While the standard Perl Unicode preamble makes Perl’s filehandles use UTF-8 encoding by default, filehandles aren’t the only sources and sinks of data. On rare occasions, such as a database read, you may be given encoded…
℞ 11: Names of CJK codepoints CJK refers to Chinese, Japanese, and Korean. In the context of Unicode, it usually refers to the Han ideographs used in the modern Chinese and Japanese writing systems. As you can expect, pictoral languages…
℞ 10: Custom named characters As several other recipes demonstrate, the charnames pragma offers considerable power to use and manipulate Unicode characters by their names. Its :alias option allows you to give your own lexically scoped nicknames to existing characters,…
℞ 9: Unicode named sequences Unicode includes the feature of named character sequences, which combine multiple Unicode characters behind a single name. The charnames pragma allows the use of these named sequences in literals, just as it allows the use…
℞ 8: Unicode named characters Use the \N{charname} notation to get the character by that name for use in interpolated literals (double-quoted strings and regexes). In v5.16, there is an implicit use charnames qw(:full :short); But prior to v5.16, you…
℞ 7: Get character number by name Unicode allows you to refer to characters by number or by name. Computers don’t care, but humans do. When you have a character name, you can translate it to its number with the…
℞ 6: Get character name by number Unicode allows you to refer to characters by number or by name. Computers don’t care, but humans do. When you have a character number, you can translate it to its name with the…
℞ 5: Unicode literals by character number In an interpolated literal, whether a double-quoted string or a regex, you may specify a character by its number using the \x{HHHHHH} escape. String: "\x{3a3}" Regex: /\x{3a3}/ String: "\x{1d45b}" Regex: /\x{1d45b}/ # even…
℞ 4: Characters and their numbers Do you need to translate a codepoint to a character or a character to its codepoint? The ord and chr functions work transparently on all codepoints, not just on ASCII alone—nor in fact, not…
℞ 3: Declare source in UTF-8 for identifiers and literals Without the all-critical use utf8 declaration, putting UTF‑8 in your literals and identifiers won’t work right. If you used the standard Perl Unicode preamble, this already happened. If you did,…
℞ 2: Fine-tuning Unicode warnings It’s easy to get Unicode wrong, especially when handling user input and dealing with multiple encodings. Perl is happy to help you detect unexpected conditions of your data. Perl is also happy to let you…
To handle Unicode effectively, always decompose on the way in, then recompose on the way out.
Tom Christiansen demonstrates the bare minimum feature set necessary to work with Unicode effectively in Perl 5.
To get in touch, send an email to perl.com-editor@perl.org, or submit an issue to perladvent/perldotcom on GitHub.
This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.
Perl.com and the authors make no representations with respect to the accuracy or completeness of the contents of all work on this website and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. The information published on this website may not be suitable for every situation. All work on this website is provided with the understanding that Perl.com and the authors are not engaged in rendering professional services. Neither Perl.com nor the authors shall be liable for damages arising herefrom.
