| 1 |
{-# LANGUAGE CPP #-} |
| 2 |
#if __GLASGOW_HASKELL__ < 7100 |
| 3 |
{-# LANGUAGE DeriveDataTypeable #-} |
| 4 |
#endif |
| 5 |
-- | |
| 6 |
-- Module : Data.Unicode.Types |
| 7 |
-- Copyright : (c) 2016 Harendra Kumar |
| 8 |
-- |
| 9 |
-- License : BSD-3-Clause |
| 10 |
-- Maintainer : [email protected] |
| 11 |
-- Stability : experimental |
| 12 |
-- Portability : GHC |
| 13 |
-- |
| 14 |
-- Character set normalization functions for Unicode. The documentation and |
| 15 |
-- API in this module is largely borrowed from @text-icu@. |
| 16 |
|
| 17 |
module Data.Unicode.Types |
| 18 |
( |
| 19 |
NormalizationMode(..) |
| 20 |
) where |
| 21 |
|
| 22 |
import Data.Typeable (Typeable) |
| 23 |
|
| 24 |
-- | |
| 25 |
-- Normalization transforms Unicode text into an equivalent |
| 26 |
-- composed or decomposed form, allowing for easier sorting and |
| 27 |
-- searching of text. Standard normalization forms are described in |
| 28 |
-- <https://unicode.org/reports/tr15/>, |
| 29 |
-- Unicode Standard Annex #15: Unicode Normalization Forms. |
| 30 |
-- |
| 31 |
-- Characters with accents or other adornments can be encoded in |
| 32 |
-- several different ways in Unicode. For example, take the character A-acute. |
| 33 |
-- In Unicode, this can be encoded as a single character (the |
| 34 |
-- \"composed\" form): |
| 35 |
-- |
| 36 |
-- @ |
| 37 |
-- 00C1 LATIN CAPITAL LETTER A WITH ACUTE |
| 38 |
-- @ |
| 39 |
-- |
| 40 |
-- or as two separate characters (the \"decomposed\" form): |
| 41 |
-- |
| 42 |
-- @ |
| 43 |
-- 0041 LATIN CAPITAL LETTER A |
| 44 |
-- 0301 COMBINING ACUTE ACCENT |
| 45 |
-- @ |
| 46 |
-- |
| 47 |
-- To a user of your program, however, both of these sequences should |
| 48 |
-- be treated as the same \"user-level\" character \"A with acute |
| 49 |
-- accent\". When you are searching or comparing text, you must |
| 50 |
-- ensure that these two sequences are treated equivalently. In |
| 51 |
-- addition, you must handle characters with more than one accent. |
| 52 |
-- Sometimes the order of a character's combining accents is |
| 53 |
-- significant, while in other cases accent sequences in different |
| 54 |
-- orders are really equivalent. |
| 55 |
-- |
| 56 |
-- Similarly, the string \"ffi\" can be encoded as three separate letters: |
| 57 |
-- |
| 58 |
-- @ |
| 59 |
-- 0066 LATIN SMALL LETTER F |
| 60 |
-- 0066 LATIN SMALL LETTER F |
| 61 |
-- 0069 LATIN SMALL LETTER I |
| 62 |
-- @ |
| 63 |
-- |
| 64 |
-- or as the single character |
| 65 |
-- |
| 66 |
-- @ |
| 67 |
-- FB03 LATIN SMALL LIGATURE FFI |
| 68 |
-- @ |
| 69 |
-- |
| 70 |
-- The \"ffi\" ligature is not a distinct semantic character, and |
| 71 |
-- strictly speaking it shouldn't be in Unicode at all, but it was |
| 72 |
-- included for compatibility with existing character sets that |
| 73 |
-- already provided it. The Unicode standard identifies such |
| 74 |
-- characters by giving them \"compatibility\" decompositions into the |
| 75 |
-- corresponding semantic characters. When sorting and searching, you |
| 76 |
-- will often want to use these mappings. |
| 77 |
-- |
| 78 |
-- Normalization helps solve these problems by transforming text into |
| 79 |
-- the canonical composed and decomposed forms as shown in the first |
| 80 |
-- example above. In addition, you can have it perform compatibility |
| 81 |
-- decompositions so that you can treat compatibility characters the |
| 82 |
-- same as their equivalents. Finally, normalization rearranges accents |
| 83 |
-- into the proper canonical order, so that you do not have to worry |
| 84 |
-- about accent rearrangement on your own. |
| 85 |
-- |
| 86 |
-- The W3C generally recommends to exchange texts in 'NFC'. Note also |
| 87 |
-- that most legacy character encodings use only precomposed forms and |
| 88 |
-- often do not encode any combining marks by themselves. For |
| 89 |
-- conversion to such character encodings the Unicode text needs to be |
| 90 |
-- normalized to 'NFC'. For more usage examples, see the Unicode |
| 91 |
-- Standard Annex. |
| 92 |
-- |
| 93 |
data NormalizationMode |
| 94 |
= NFD -- ^ Canonical decomposition. |
| 95 |
| NFKD -- ^ Compatibility decomposition. |
| 96 |
| NFC -- ^ Canonical decomposition followed by canonical composition. |
| 97 |
| NFKC -- ^ Compatibility decomposition followed by canonical composition. |
| 98 |
deriving (Eq, Show, Enum, Typeable) |