Text Files Coding and End of Line Characters

Character Sets

Text files use character sets. A character set contains punctuation marks, numerals, uppercase and lowercase letters, and all other printable characters. Each character in the set is identified by a number.

Most character sets in use are supersets of the U.S. ASCII character set, whose 95 printable characters have numeric values from 32 through 126. The major groups of character sets are:

Windows Character Set (single-byte coding scheme)

Legacy Windows single-byte code pages, often called ANSI code pages, use 8 bits per character and can assign at most 256 (2^8) values. Different code pages provide different characters for particular languages; one code page does not cover every language. Modern Windows also supports Unicode encodings.

Multibyte character set

Many East Asian writing systems require more characters than a single-byte encoding can represent. Legacy double-byte code pages use one or two bytes per character. Other multibyte encodings use different rules; for example, UTF-8 uses one to four bytes per Unicode scalar value. The encoding must be known to decode the bytes correctly.

Unicode Character Set

Unicode assigns code points to characters across writing systems, punctuation, symbols, and other text elements. UTF-8, UTF-16, and UTF-32 encode Unicode text using different code-unit sizes. UTF-16 uses one 16-bit code unit for a character in the Basic Multilingual Plane or a pair of 16-bit code units for a supplementary character. Unicode is not limited to 65,536 characters, and an encoding must be chosen correctly when reading or writing a text file.

OEM Character Set

The OEM character set is typically used in full-screen MS-DOS sessions for screen display. Printable characters 32 through 126 are usually the same in the OEM, U.S. ASCII, and Windows character sets. The other characters in the OEM character set (0 through 31 and 128 through 255) correspond to the characters that can be displayed in a full-screen MS-DOS session. These characters are generally different from the Windows characters.

Symbol Character Set

The Symbol character set contains special characters typically used to represent mathematical and scientific formulas.

End of Line Characters

Text files use different line endings. MS-DOS, Windows, and OS/2 traditionally use CR/LF; UNIX and modern macOS use LF. Classic Mac OS used CR.

See Also

Converting Contents of Files