| 1 |
========================= |
| 2 |
Fiona and String Encoding |
| 3 |
========================= |
| 4 |
|
| 5 |
Reading |
| 6 |
------- |
| 7 |
|
| 8 |
With Fiona, all 'str' type record attributes are unicode strings. The source |
| 9 |
data is encoded in some way. It might be a standard encoding (ISO-8859-1 or |
| 10 |
UTF-8) or it might be a format-specific encoding. How do we get from encoded |
| 11 |
strings to Python unicode? :: |
| 12 |
|
| 13 |
encoded File | (decode?) OGR (encode?) | (decode) Fiona |
| 14 |
|
| 15 |
E_f R E_i |
| 16 |
|
| 17 |
The internal encoding `E_i` is used by the ``FeatureBuilder`` class to create |
| 18 |
Fiona's record dicts. `E_f` is the encoding of the data file. `R` is ``True`` |
| 19 |
if OGR is recoding record attribute values to UTF-8 (a recent feature that |
| 20 |
isn't implemented for all format drivers, hence the question marks in the |
| 21 |
sketch above), else ``False``. |
| 22 |
|
| 23 |
The value of E_i is determined like this:: |
| 24 |
|
| 25 |
E_i = (R and 'utf-8') or E_f |
| 26 |
|
| 27 |
In the real world of sloppy data, we may not know the exact encoding of the |
| 28 |
data file. Fiona's best guess at it is this:: |
| 29 |
|
| 30 |
E_f = E_u or (R and E_o) or (S and 'iso-8859-1') or E_p |
| 31 |
|
| 32 |
`E_u`, here, is any encoding provided by the programmer (through the |
| 33 |
``Collection`` constructor). `E_o` is an encoding detected by OGR (which |
| 34 |
doesn't provide an API to get the detected encoding). `S` is ``True`` if the |
| 35 |
file is a Shapefile (because that's the format default). `E_p` is |
| 36 |
locale.getpreferredencoding(). |
| 37 |
|
| 38 |
Bottom line: if you know that your data file has an encoding other than |
| 39 |
ISO-8859-1, specify it. If you don't know what the encoding is, you can let the |
| 40 |
format driver try to figure it out (Requires GDAL 1.9.1+). |
| 41 |
|
| 42 |
Writing |
| 43 |
------- |
| 44 |
|
| 45 |
On the writing side:: |
| 46 |
|
| 47 |
Fiona (encode) | (decode?) OGR (encode?) | encoded File |
| 48 |
|
| 49 |
E_i R E_f |
| 50 |
|
| 51 |
We derive `E_i` from `R` and `E_f` again as above. `E_f` is:: |
| 52 |
|
| 53 |
E_f = E_u or (S and 'iso-8859-1') or E_p |
| 54 |
|
| 55 |
Appending |
| 56 |
--------- |
| 57 |
|
| 58 |
The diagram is the same as above, but `E_f` is as in the Reading section. |
| 59 |
|