Debian Codegraph @

docs/encoding.txt

raw
1 =========================
2 Fiona and String Encoding
3 =========================
4
5 Reading
6 -------
7
8 With Fiona, all 'str' type record attributes are unicode strings. The source
9 data is encoded in some way. It might be a standard encoding (ISO-8859-1 or
10 UTF-8) or it might be a format-specific encoding. How do we get from encoded
11 strings to Python unicode? ::
12
13 encoded File | (decode?) OGR (encode?) | (decode) Fiona
14
15 E_f R E_i
16
17 The internal encoding `E_i` is used by the ``FeatureBuilder`` class to create
18 Fiona's record dicts. `E_f` is the encoding of the data file. `R` is ``True``
19 if OGR is recoding record attribute values to UTF-8 (a recent feature that
20 isn't implemented for all format drivers, hence the question marks in the
21 sketch above), else ``False``.
22
23 The value of E_i is determined like this::
24
25 E_i = (R and 'utf-8') or E_f
26
27 In the real world of sloppy data, we may not know the exact encoding of the
28 data file. Fiona's best guess at it is this::
29
30 E_f = E_u or (R and E_o) or (S and 'iso-8859-1') or E_p
31
32 `E_u`, here, is any encoding provided by the programmer (through the
33 ``Collection`` constructor). `E_o` is an encoding detected by OGR (which
34 doesn't provide an API to get the detected encoding). `S` is ``True`` if the
35 file is a Shapefile (because that's the format default). `E_p` is
36 locale.getpreferredencoding().
37
38 Bottom line: if you know that your data file has an encoding other than
39 ISO-8859-1, specify it. If you don't know what the encoding is, you can let the
40 format driver try to figure it out (Requires GDAL 1.9.1+).
41
42 Writing
43 -------
44
45 On the writing side::
46
47 Fiona (encode) | (decode?) OGR (encode?) | encoded File
48
49 E_i R E_f
50
51 We derive `E_i` from `R` and `E_f` again as above. `E_f` is::
52
53 E_f = E_u or (S and 'iso-8859-1') or E_p
54
55 Appending
56 ---------
57
58 The diagram is the same as above, but `E_f` is as in the Reading section.
59