Latest recipes tagged "unicode"

No more UnicodeDecodeErrors when printing (Python)

2013-12-17T23:02:37-08:00

Python recipe 578788 by Ádám Szieberth (decode, error, print, terminal, unicode). Revision 2.

I hate getting UnicodeDecodeErrors when I use print() for bugtracing or for some other reason. I decided to make a module which spares me the headache.

Repair common Unicode mistakes after they've been made (obsoleted by ftfy package) (Python)

2016-09-28T02:33:32-07:00

Python recipe 578243 by Rob Speer (gremlins, mojibake, obsolete, unicode, utf8). Revision 4.

Something you will find all over the place, in real-world text, is text that's encoded as UTF-8, decoded in some ugly format like Latin-1 or even Windows codepage 1252, and mistakenly encoded as UTF-8 again.

This causes your perfectly good Unicode-aware code to end up with garbage text because someone else (or maybe "someone else") made a mistake.

This function looks for the evidence of that having happened and fixes it. It determines whether it should replace nonsense sequences of single-byte characters that were really meant to be UTF-8 characters, and if so, turns them into the correctly-encoded Unicode character that they were meant to represent.

Remove UTF-8 Byte Order Mark (BOM) from text files (Bash)

2011-10-18T06:37:52-07:00

Bash recipe 577912 by Graham Poulter (text, unicode, utf8).

Shell script to removes UTF-8 Byte Order Mark (BOM) from text files, where present.

Correctly reading CSV files in arbitrary encodings (Python)

2011-07-25T07:29:08-07:00

Python recipe 577778 by Devin Jeanpierre (csv, unicode). Revision 4.

Recipe for using unicode files (i.e. files opened with codecs.open) with the csv module.

Safe unicode representation (Python)

2011-03-17T23:28:40-07:00

Python recipe 577614 by Sridhar Ratnakumar (unicode).

Safely convert any given string type (text or binary) to unicode. You won't get UnicodeDecodeError error, at the cost of ignoring those errors during conversion, which is useful for debugging and logging. This recipe requires the six package.

Decorator for writing polymorphic functions (Python)

2010-09-21T06:08:48-07:00

Python recipe 577393 by Baptiste Carvello (polymorphism, unicode). Revision 2.

Python 3 makes a clean separation between unicode text strings (str) and byte strings (bytes). However, for some tasks (notably networking), it makes sense to apply the same process to str and bytes, usually relying on the byte string beeing encoded with an ASCII compatible encoding.

In this context, a polymorphic function is one which will operate on unicode strings (str) or bytes objects (bytes) depending on the type of the arguments. The common difficulty is that string constants used in the function also have to be of the right type. This decorator helps by allowing to use a different set of constants depending on the type of the argument.

Rename non-ASCII filenames to readable ASCII, i.e. replace accented characters, etc (Python)

2011-11-03T17:50:55-07:00

Python recipe 577226 by ccpizza (conversion, renaming, unicode). Revision 3.

The script converts any accented characters in filenames to their ASCII equivalents. e.g.:

Example:

â > a
ä > a
à > a
á > a
é > e
í > i
ó > o
ú > u
ñ > n
ü > u
...

Before-and-after example:

01_Antonín_Dvořák_Allegro.mp3   >>>  01_Antonin_Dvorak_Allegro.mp3

Usage:

Running the script without arguments will rename all files in the current folder. 
!!!WARNING!!! ***No*** backups are created.

Simple reverse converter of unicode code points string (Python)

2009-09-22T20:02:32-07:00

Python recipe 576909 by Ryan (code, points, prefix, reverse, str, string, u, unicode). Revision 4.

It's a simple recipe to convert a str type string with pure unicode code point (e.g string = "\u5982\u679c\u7231" ) to an unicode type string. Actually, this method has the same effect with 'u' prefix. But differently, it allows you to pass a variable of code points string as well as a constant one.

safe print (Python)

2009-01-02T15:40:30-08:00

Python recipe 576602 by Trent Mick (encode, safe, unicode).

A replacement for "print" that will safely handle unicode conversion.