| 1 |
===================== |
| 2 |
The Fiona User Manual |
| 3 |
===================== |
| 4 |
|
| 5 |
:Author: Sean Gillies, <[email protected]> |
| 6 |
:Version: |release| |
| 7 |
:Date: |today| |
| 8 |
:Copyright: |
| 9 |
This work, with the exception of code examples, is licensed under a `Creative |
| 10 |
Commons Attribution 3.0 United States License`__. The code examples are |
| 11 |
licensed under the BSD 3-clause license (see LICENSE.txt in the repository |
| 12 |
root). |
| 13 |
|
| 14 |
.. __: https://creativecommons.org/licenses/by/3.0/us/ |
| 15 |
|
| 16 |
:Abstract: |
| 17 |
Fiona is OGR's neat, nimble, no-nonsense API. This document explains how to |
| 18 |
use the Fiona package for reading and writing geospatial data files. Python |
| 19 |
3 is used in examples. See the `README <README.html>`__ for installation and |
| 20 |
quick start instructions. |
| 21 |
|
| 22 |
.. sectnum:: |
| 23 |
|
| 24 |
|
| 25 |
Introduction |
| 26 |
============ |
| 27 |
|
| 28 |
:dfn:`Geographic information systems` (GIS) help us plan, react to, and |
| 29 |
understand changes in our physical, political, economic, and cultural |
| 30 |
landscapes. A generation ago, GIS was something done only by major institutions |
| 31 |
like nations and cities, but it's become ubiquitous today thanks to accurate |
| 32 |
and inexpensive global positioning systems, commoditization of satellite |
| 33 |
imagery, and open source software. |
| 34 |
|
| 35 |
The kinds of data in GIS are roughly divided into :dfn:`rasters` representing |
| 36 |
continuous scalar fields (land surface temperature or elevation, for example) |
| 37 |
and :dfn:`vectors` representing discrete entities like roads and administrative |
| 38 |
boundaries. Fiona is concerned exclusively with the latter. It is a Python |
| 39 |
wrapper for vector data access functions from the `GDAL/OGR |
| 40 |
<https://gdal.org>`_ library. A very simple wrapper for minimalists. |
| 41 |
It reads data records from files as GeoJSON-like mappings and writes the same |
| 42 |
kind of mappings as records back to files. That's it. There are no layers, no |
| 43 |
cursors, no geometric operations, no transformations between coordinate |
| 44 |
systems, no remote method calls; all these concerns are left to other Python |
| 45 |
packages such as :py:mod:`Shapely <https://github.com/shapely/shapely>` and |
| 46 |
:py:mod:`pyproj <https://pypi.org/project/pyproj/>` and Python language |
| 47 |
protocols. Why? To eliminate unnecessary complication. Fiona aims to be simple |
| 48 |
to understand and use, with no gotchas. |
| 49 |
|
| 50 |
Please understand this: Fiona is designed to excel in a certain range of tasks |
| 51 |
and is less optimal in others. Fiona trades memory and speed for simplicity and |
| 52 |
reliability. Where OGR's Python bindings (for example) use C pointers, Fiona |
| 53 |
copies vector data from the data source to Python objects. These are simpler |
| 54 |
and safer to use, but more memory intensive. Fiona's performance is relatively |
| 55 |
more slow if you only need access to a single record field – and of course if |
| 56 |
you just want to reproject or filter data files, nothing beats the |
| 57 |
:command:`ogr2ogr` program – but Fiona's performance is much better than OGR's |
| 58 |
Python bindings if you want *all* fields and coordinates of a record. The |
| 59 |
copying is a constraint, but it simplifies programs. With Fiona, you don't have |
| 60 |
to track references to C objects to avoid crashes, and you can work with vector |
| 61 |
data using familiar Python mapping accessors. Less worry, less time spent |
| 62 |
reading API documentation. |
| 63 |
|
| 64 |
Rules of Thumb |
| 65 |
-------------- |
| 66 |
|
| 67 |
In what cases would you benefit from using Fiona? |
| 68 |
|
| 69 |
* If the features of interest are from or destined for a file in a non-text |
| 70 |
format like ESRI Shapefiles, Mapinfo TAB files, etc. |
| 71 |
* If you're more interested in the values of many feature properties than in |
| 72 |
a single property's value. |
| 73 |
* If you're more interested in all the coordinate values of a feature's |
| 74 |
geometry than in a single value. |
| 75 |
* If your processing system is distributed or not contained to a single |
| 76 |
process. |
| 77 |
|
| 78 |
In what cases would you not benefit from using Fiona? |
| 79 |
|
| 80 |
* If your data is in or destined for a JSON document you should use Python's |
| 81 |
:py:mod:`json` or :py:mod:`simplejson` modules. |
| 82 |
* If your data is in a RDBMS like PostGIS, use a Python DB package or ORM like |
| 83 |
:py:mod:`SQLAlchemy` or :py:mod:`GeoAlchemy`. Maybe you're using |
| 84 |
:py:mod:`GeoDjango` already. If so, carry on. |
| 85 |
* If your data is served via HTTP from CouchDB or CartoDB, etc, use an HTTP |
| 86 |
package (:py:mod:`httplib2`, :py:mod:`Requests`, etc) or the provider's |
| 87 |
Python API. |
| 88 |
* If you can use :command:`ogr2ogr`, do so. |
| 89 |
|
| 90 |
Example |
| 91 |
------- |
| 92 |
|
| 93 |
The first example of using Fiona is this: copying features (another word for |
| 94 |
record) from one file to another, adding two attributes and making sure that |
| 95 |
all polygons are facing "up". Orientation of polygons is significant in some |
| 96 |
applications, extruded polygons in Google Earth for one. No other library (like |
| 97 |
:py:mod:`Shapely`) is needed here, which keeps it uncomplicated. There's a |
| 98 |
:file:`coutwildrnp.zip` file in the Fiona repository for use in this and other |
| 99 |
examples. |
| 100 |
|
| 101 |
.. code-block:: python |
| 102 |
|
| 103 |
import datetime |
| 104 |
|
| 105 |
import fiona |
| 106 |
from fiona import Geometry, Feature, Properties |
| 107 |
|
| 108 |
|
| 109 |
def signed_area(coords): |
| 110 |
"""Return the signed area enclosed by a ring using the linear time |
| 111 |
algorithm at http://www.cgafaq.info/wiki/Polygon_Area. A value >= 0 |
| 112 |
indicates a counter-clockwise oriented ring. |
| 113 |
""" |
| 114 |
xs, ys = map(list, zip(*coords)) |
| 115 |
xs.append(xs[1]) |
| 116 |
ys.append(ys[1]) |
| 117 |
return sum(xs[i] * (ys[i + 1] - ys[i - 1]) for i in range(1, len(coords))) / 2.0 |
| 118 |
|
| 119 |
|
| 120 |
with fiona.open( |
| 121 |
"zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 122 |
) as src: |
| 123 |
|
| 124 |
# Copy the source schema and add two new properties. |
| 125 |
dst_schema = src.schema |
| 126 |
dst_schema["properties"]["signed_area"] = "float" |
| 127 |
dst_schema["properties"]["timestamp"] = "datetime" |
| 128 |
|
| 129 |
# Create a sink for processed features with the same format and |
| 130 |
# coordinate reference system as the source. |
| 131 |
with fiona.open( |
| 132 |
"example.gpkg", |
| 133 |
mode="w", |
| 134 |
layer="oriented-ccw", |
| 135 |
crs=src.crs, |
| 136 |
driver="GPKG", |
| 137 |
schema=dst_schema, |
| 138 |
) as dst: |
| 139 |
for feat in src: |
| 140 |
# If any feature's polygon is facing "down" (has rings |
| 141 |
# wound clockwise), its rings will be reordered to flip |
| 142 |
# it "up". |
| 143 |
geom = feat.geometry |
| 144 |
assert geom.type == "Polygon" |
| 145 |
rings = geom.coordinates |
| 146 |
sa = sum(signed_area(ring) for ring in rings) |
| 147 |
|
| 148 |
if sa < 0.0: |
| 149 |
rings = [r[::-1] for r in rings] |
| 150 |
geom = Geometry(type=geom.type, coordinates=rings) |
| 151 |
|
| 152 |
# Add the signed area of the polygon and a timestamp |
| 153 |
# to the feature properties map. |
| 154 |
props = Properties.from_dict( |
| 155 |
**feat.properties, |
| 156 |
signed_area=sa, |
| 157 |
timestamp=datetime.datetime.now().isoformat() |
| 158 |
) |
| 159 |
|
| 160 |
dst.write(Feature(geometry=geom, properties=props)) |
| 161 |
|
| 162 |
Data Model |
| 163 |
========== |
| 164 |
|
| 165 |
Discrete geographic features are usually represented in geographic information |
| 166 |
systems by :dfn:`records`. The characteristics of records and their semantic |
| 167 |
implications are well known [Kent1978]_. Among those most significant for |
| 168 |
geographic data: records have a single type, all records of that type have the |
| 169 |
same fields, and a record's fields concern a single geographic feature. |
| 170 |
Different systems model records in different ways, but the various models have |
| 171 |
enough in common that programmers have been able to create useful abstract data |
| 172 |
models. The `OGR model <https://gdal.org/user/vector_data_model.html>`__ is one. Its |
| 173 |
primary entities are :dfn:`Data Sources`, :dfn:`Layers`, and :dfn:`Features`. |
| 174 |
Features have not fields, but attributes and a :dfn:`Geometry`. An OGR Layer |
| 175 |
contains Features of a single type ("roads" or "wells", for example). The |
| 176 |
GeoJSON model is a bit more simple, keeping Features and substituting |
| 177 |
:dfn:`Feature Collections` for OGR Data Sources and Layers. The term "Feature" |
| 178 |
is thus overloaded in GIS modeling, denoting entities in both our conceptual |
| 179 |
and data models. |
| 180 |
|
| 181 |
Various formats for record files exist. The :dfn:`ESRI Shapefile` [ESRI1998]_ |
| 182 |
has been, at least in the United States, the most significant of these up to |
| 183 |
about 2005 and remains popular today. It is a binary format. The shape fields |
| 184 |
are stored in one .shp file and the other fields in another .dbf file. The |
| 185 |
GeoJSON [GeoJSON]_ format, from 2008, proposed a human readable text format in |
| 186 |
which geometry and other attribute fields are encoded together using |
| 187 |
:dfn:`Javascript Object Notation` [JSON]_. In GeoJSON, there's a uniformity of |
| 188 |
data access. Attributes of features are accessed in the same manner as |
| 189 |
attributes of a feature collection. Coordinates of a geometry are accessed in |
| 190 |
the same manner as features of a collection. |
| 191 |
|
| 192 |
The GeoJSON format turns out to be a good model for a Python API. JSON objects |
| 193 |
and Python dictionaries are semantically and syntactically similar. Replacing |
| 194 |
object-oriented Layer and Feature APIs with interfaces based on Python mappings |
| 195 |
provides a uniformity of access to data and reduces the amount of time spent |
| 196 |
reading documentation. A Python programmer knows how to use a mapping, so why |
| 197 |
not treat features as dictionaries? Use of existing Python idioms is one of |
| 198 |
Fiona's major design principles. |
| 199 |
|
| 200 |
.. admonition:: TL;DR |
| 201 |
|
| 202 |
Fiona subscribes to the conventional record model of data, but provides |
| 203 |
GeoJSON-like access to the data via Python file-like and mapping protocols. |
| 204 |
|
| 205 |
Reading Vector Data |
| 206 |
=================== |
| 207 |
|
| 208 |
Reading a GIS vector file begins by opening it in mode ``'r'`` using Fiona's |
| 209 |
:py:func:`~fiona.open` function. It returns an opened |
| 210 |
:py:class:`~fiona.collection.Collection` object. |
| 211 |
|
| 212 |
.. code-block:: pycon |
| 213 |
|
| 214 |
>>> import fiona |
| 215 |
>>> colxn = fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip", "r") |
| 216 |
>>> colxn |
| 217 |
<open Collection '/vsizip/vsicurl/https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip:coutwildrnp', mode 'r' at 0x7f9555af8f50> |
| 218 |
>>> collection.closed |
| 219 |
False |
| 220 |
|
| 221 |
Mode ``'r'`` is the default and will be omitted in following examples. |
| 222 |
|
| 223 |
Fiona's :py:class:`~fiona.collection.Collection` is like a Python |
| 224 |
:py:class:`file`, but is iterable for records rather than lines. |
| 225 |
|
| 226 |
.. code-block:: pycon |
| 227 |
|
| 228 |
>>> next(iter(colxn)) |
| 229 |
{'geometry': {'type': 'Polygon', 'coordinates': ... |
| 230 |
>>> len(list(colxn)) |
| 231 |
67 |
| 232 |
|
| 233 |
Note that :py:func:`list` iterates over the entire collection, effectively |
| 234 |
emptying it as with a Python :py:class:`file`. |
| 235 |
|
| 236 |
.. code-block:: pycon |
| 237 |
|
| 238 |
>>> next(iter(colxn)) |
| 239 |
Traceback (most recent call last): |
| 240 |
... |
| 241 |
StopIteration |
| 242 |
>>> len(list(colxn)) |
| 243 |
0 |
| 244 |
|
| 245 |
Seeking the beginning of the file is not supported. You must reopen the |
| 246 |
collection to get back to the beginning. |
| 247 |
|
| 248 |
.. code-block:: pycon |
| 249 |
|
| 250 |
>>> colxn = fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") |
| 251 |
>>> len(list(colxn)) |
| 252 |
67 |
| 253 |
|
| 254 |
.. admonition:: File Encoding |
| 255 |
|
| 256 |
The format drivers will attempt to detect the encoding of your data, but may |
| 257 |
fail. In this case, the proper encoding can be specified explicitly by using |
| 258 |
the ``encoding`` keyword parameter of :py:func:`fiona.open`, for example: |
| 259 |
``encoding='Windows-1252'``. |
| 260 |
|
| 261 |
New in version 0.9.1. |
| 262 |
|
| 263 |
Collection indexing |
| 264 |
------------------- |
| 265 |
|
| 266 |
Features of a collection may also be accessed by index. |
| 267 |
|
| 268 |
.. code-block:: pycon |
| 269 |
|
| 270 |
>>> with fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") as colxn: |
| 271 |
... print(colxn[1]) |
| 272 |
... |
| 273 |
<fiona.model.Feature object at 0x7f954bfc5f50> |
| 274 |
|
| 275 |
Note that these indices are controlled by GDAL, and do not always follow Python |
| 276 |
conventions. They can start from 0, 1 (e.g. geopackages), or even other values, |
| 277 |
and have no guarantee of contiguity. Negative indices will only function |
| 278 |
correctly if indices start from 0 and are contiguous. |
| 279 |
|
| 280 |
New in version 1.1.6 |
| 281 |
|
| 282 |
Closing Files |
| 283 |
------------- |
| 284 |
|
| 285 |
A :py:class:`~fiona.collection.Collection` involves external resources. There's |
| 286 |
no guarantee that these will be released unless you explicitly |
| 287 |
:py:meth:`~fiona.collection.Collection.close` the object or use |
| 288 |
a :keyword:`with` statement. When a :py:class:`~fiona.collection.Collection` |
| 289 |
is a context guard, it is closed no matter what happens within the block. |
| 290 |
|
| 291 |
.. code-block:: pycon |
| 292 |
|
| 293 |
>>> try: |
| 294 |
... with fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") as colxn: |
| 295 |
... print(len(list(colxn))) |
| 296 |
... assert True is False |
| 297 |
... except Exception: |
| 298 |
... print(colxn.closed) |
| 299 |
... raise |
| 300 |
... |
| 301 |
67 |
| 302 |
True |
| 303 |
Traceback (most recent call last): |
| 304 |
... |
| 305 |
AssertionError |
| 306 |
|
| 307 |
An exception is raised in the :keyword:`with` block above, but as you can see |
| 308 |
from the print statement in the :keyword:`except` clause :py:meth:`colxn.__exit__` |
| 309 |
(and thereby :py:meth:`colxn.close`) has been called. |
| 310 |
|
| 311 |
.. important:: Always call :py:meth:`~fiona.collection.Collection.close` or |
| 312 |
use :keyword:`with` and you'll never stumble over tied-up external resources, |
| 313 |
locked files, etc. |
| 314 |
|
| 315 |
Format Drivers, CRS, Bounds, and Schema |
| 316 |
======================================= |
| 317 |
|
| 318 |
In addition to attributes like those of :py:class:`file` |
| 319 |
(:py:attr:`~file.name`, :py:attr:`~file.mode`, :py:attr:`~file.closed`), |
| 320 |
a :py:class:`~fiona.collection.Collection` has a read-only |
| 321 |
:py:attr:`~fiona.collection.Collection.driver` attribute which names the |
| 322 |
:program:`OGR` :dfn:`format driver` used to open the vector file. |
| 323 |
|
| 324 |
.. code-block:: pycon |
| 325 |
|
| 326 |
>>> colxn = fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") |
| 327 |
>>> colxn.driver |
| 328 |
'ESRI Shapefile' |
| 329 |
|
| 330 |
The :dfn:`coordinate reference system` (CRS) of the collection's vector data is |
| 331 |
accessed via a read-only :py:attr:`~fiona.collection.Collection.crs` attribute. |
| 332 |
|
| 333 |
.. code-block:: pycon |
| 334 |
|
| 335 |
>>> colxn.crs |
| 336 |
CRS.from_epsg(4326) |
| 337 |
|
| 338 |
The :py:mod:`fiona.crs` module provides 3 functions to assist with these |
| 339 |
mappings. :py:func:`~fiona.crs.to_string` converts mappings to PROJ.4 strings: |
| 340 |
|
| 341 |
.. code-block:: pycon |
| 342 |
|
| 343 |
>>> from fiona.crs import to_string |
| 344 |
>>> to_string(colxn.crs) |
| 345 |
'EPSG:4326' |
| 346 |
|
| 347 |
:py:func:`~fiona.crs.from_string` does the inverse. |
| 348 |
|
| 349 |
.. code-block:: pycon |
| 350 |
|
| 351 |
>>> from fiona.crs import from_string |
| 352 |
>>> from_string("+datum=WGS84 +ellps=WGS84 +no_defs +proj=longlat") |
| 353 |
CRS.from_epsg(4326) |
| 354 |
|
| 355 |
:py:func:`~fiona.crs.from_epsg` is a shortcut to CRS mappings from EPSG codes. |
| 356 |
|
| 357 |
.. code-block:: pycon |
| 358 |
|
| 359 |
>>> from fiona.crs import from_epsg |
| 360 |
>>> from_epsg(3857) |
| 361 |
CRS.from_epsg(3857) |
| 362 |
|
| 363 |
.. admonition:: No Validation |
| 364 |
|
| 365 |
Both :py:func:`~fiona.crs.from_epsg` and :py:func:`~fiona.crs.from_string` |
| 366 |
simply restructure data, they do not ensure that the resulting mapping is |
| 367 |
a pre-defined or otherwise valid CRS in any way. |
| 368 |
|
| 369 |
The number of records in the collection's file can be obtained via Python's |
| 370 |
built in :py:func:`len` function. |
| 371 |
|
| 372 |
.. code-block:: pycon |
| 373 |
|
| 374 |
>>> len(colxn) |
| 375 |
67 |
| 376 |
|
| 377 |
The :dfn:`minimum bounding rectangle` (MBR) or :dfn:`bounds` of the |
| 378 |
collection's records is obtained via a read-only |
| 379 |
:py:attr:`~fiona.collection.Collection.bounds` attribute. |
| 380 |
|
| 381 |
.. code-block:: pycon |
| 382 |
|
| 383 |
>>> colxn.bounds |
| 384 |
(-113.56424713134766, 37.0689811706543, -104.97087097167969, 41.99627685546875) |
| 385 |
|
| 386 |
Finally, the schema of its record type (a vector file has a single type of |
| 387 |
record, remember) is accessed via a read-only |
| 388 |
:py:attr:`~fiona.collection.Collection.schema` attribute. It has 'geometry' |
| 389 |
and 'properties' items. The former is a string and the latter is a |
| 390 |
dict with items having the same order as the fields in the data file. |
| 391 |
|
| 392 |
.. code-block:: pycon |
| 393 |
|
| 394 |
>>> import pprint |
| 395 |
>>> pprint.pprint(colxn.schema) |
| 396 |
{'geometry': 'Polygon', |
| 397 |
'properties': {'AGBUR': 'str:80', |
| 398 |
'AREA': 'float:24.15', |
| 399 |
'FEATURE1': 'str:80', |
| 400 |
'FEATURE2': 'str:80', |
| 401 |
'NAME': 'str:80', |
| 402 |
'PERIMETER': 'float:24.15', |
| 403 |
'STATE': 'str:80', |
| 404 |
'STATE_FIPS': 'str:80', |
| 405 |
'URL': 'str:101', |
| 406 |
'WILDRNP020': 'int:10'}} |
| 407 |
|
| 408 |
Keeping Schemas Simple |
| 409 |
---------------------- |
| 410 |
|
| 411 |
Fiona takes a less is more approach to record types and schemas. Data about |
| 412 |
record types is structured as closely to data about records as can be done. |
| 413 |
Modulo a record's 'id' key, the keys of a schema mapping are the same as the |
| 414 |
keys of the collection's record mappings. |
| 415 |
|
| 416 |
.. code-block:: pycon |
| 417 |
|
| 418 |
>>> feat = next(iter(colxn)) |
| 419 |
>>> set(feat.keys()) - set(colxn.schema.keys()) |
| 420 |
{'id'} |
| 421 |
>>> set(feat['properties'].keys()) == set(colxn.schema['properties'].keys()) |
| 422 |
True |
| 423 |
|
| 424 |
The values of the schema mapping are either additional mappings or field type |
| 425 |
names like 'Polygon', 'float', and 'str'. The corresponding Python types can |
| 426 |
be found in a dictionary named :py:attr:`fiona.FIELD_TYPES_MAP`. |
| 427 |
|
| 428 |
.. code-block:: pycon |
| 429 |
|
| 430 |
>>> pprint.pprint(fiona.FIELD_TYPES_MAP) |
| 431 |
{'List[str]': typing.List[str], |
| 432 |
'bytes': <class 'bytes'>, |
| 433 |
'date': <class 'fiona.rfc3339.FionaDateType'>, |
| 434 |
'datetime': <class 'fiona.rfc3339.FionaDateTimeType'>, |
| 435 |
'float': <class 'float'>, |
| 436 |
'int': <class 'int'>, |
| 437 |
'int32': <class 'int'>, |
| 438 |
'int64': <class 'int'>, |
| 439 |
'str': <class 'str'>, |
| 440 |
'time': <class 'fiona.rfc3339.FionaTimeType'>} |
| 441 |
|
| 442 |
Field Types |
| 443 |
----------- |
| 444 |
|
| 445 |
In a nutshell, the types and their names are as near to what you'd expect in |
| 446 |
Python (or Javascript) as possible. Since Python 3, the 'str' field type |
| 447 |
may contain Unicode characters. |
| 448 |
|
| 449 |
.. code-block:: pycon |
| 450 |
|
| 451 |
>>> type(feat.properties['NAME']) |
| 452 |
<class 'str'> |
| 453 |
>>> colxn.schema['properties']['NAME'] |
| 454 |
'str' |
| 455 |
>>> fiona.FIELD_TYPES_MAP[colxn.schema['properties']['NAME']] |
| 456 |
<class 'str'> |
| 457 |
|
| 458 |
String type fields may also indicate their maximum width. A value of 'str:25' |
| 459 |
indicates that all values will be no longer than 25 characters. If this value |
| 460 |
is used in the schema of a file opened for writing, values of that property |
| 461 |
will be truncated at 25 characters. The default width is 80 chars, which means |
| 462 |
'str' and 'str:80' are more or less equivalent. |
| 463 |
|
| 464 |
Fiona provides a function to get the width of a property. |
| 465 |
|
| 466 |
.. code-block:: pycon |
| 467 |
|
| 468 |
>>> from fiona import prop_width |
| 469 |
>>> prop_width('str:25') |
| 470 |
25 |
| 471 |
>>> prop_width('str') |
| 472 |
80 |
| 473 |
|
| 474 |
Another function gets the proper Python type of a property. |
| 475 |
|
| 476 |
.. code-block:: pycon |
| 477 |
|
| 478 |
>>> from fiona import prop_type |
| 479 |
>>> prop_type('int') |
| 480 |
<type 'int'> |
| 481 |
>>> prop_type('float') |
| 482 |
<type 'float'> |
| 483 |
>>> prop_type('str:25') |
| 484 |
<class 'str'> |
| 485 |
|
| 486 |
Geometry Types |
| 487 |
-------------- |
| 488 |
|
| 489 |
Fiona supports the geometry types in GeoJSON and their 3D variants. This means |
| 490 |
that the value of a schema's geometry item will be one of the following: |
| 491 |
|
| 492 |
- Point |
| 493 |
- LineString |
| 494 |
- Polygon |
| 495 |
- MultiPoint |
| 496 |
- MultiLineString |
| 497 |
- MultiPolygon |
| 498 |
- GeometryCollection |
| 499 |
- 3D Point |
| 500 |
- 3D LineString |
| 501 |
- 3D Polygon |
| 502 |
- 3D MultiPoint |
| 503 |
- 3D MultiLineString |
| 504 |
- 3D MultiPolygon |
| 505 |
- 3D GeometryCollection |
| 506 |
|
| 507 |
The last seven of these, the 3D types, apply only to collection schema. The |
| 508 |
geometry types of features are always one of the first seven. A '3D Point' |
| 509 |
collection, for example, always has features with geometry type 'Point'. The |
| 510 |
coordinates of those geometries will be (x, y, z) tuples. |
| 511 |
|
| 512 |
Note that one of the most common vector data formats, Esri's Shapefile, has no |
| 513 |
'MultiLineString' or 'MultiPolygon' schema geometries. However, a Shapefile |
| 514 |
that indicates 'Polygon' in its schema may yield either 'Polygon' or |
| 515 |
'MultiPolygon' features. |
| 516 |
|
| 517 |
Features |
| 518 |
======== |
| 519 |
|
| 520 |
A record you get from a collection is structured like a GeoJSON Feature. Fiona |
| 521 |
records are self-describing; the names of its fields are contained within the |
| 522 |
data structure and the values in the fields are typed properly for the type of |
| 523 |
record. Numeric field values are instances of type :py:class:`int` and |
| 524 |
:py:class:`float`, for example, not strings. |
| 525 |
|
| 526 |
The record data has no references to the |
| 527 |
:py:class:`~fiona.collection.Collection` from which it originates or to any |
| 528 |
other external resource. It's entirely independent and safe to use in any way. |
| 529 |
Closing the collection does not affect the record at all. |
| 530 |
|
| 531 |
.. admonition:: Features are mappings, not dicts |
| 532 |
|
| 533 |
In Fiona versions before 1.9.0 features were Python dicts, mutable and JSON |
| 534 |
serializable. Since 1.9.0 features are mappings and not immediately JSON |
| 535 |
serializable. |
| 536 |
|
| 537 |
Instances of Feature can be converted to dicts with |
| 538 |
:py:func:`fiona.model.to_dict` or serialized using the json module and |
| 539 |
:py:class:`fiona.model.ObjectEncoder`. |
| 540 |
|
| 541 |
Feature Id |
| 542 |
---------- |
| 543 |
|
| 544 |
A feature has an ``id`` attribute. As in the GeoJSON specification, its |
| 545 |
corresponding value is a string unique within the data file. |
| 546 |
|
| 547 |
.. code-block:: pycon |
| 548 |
|
| 549 |
>>> colxn = fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") |
| 550 |
>>> feat = next(iter(colxn)) |
| 551 |
>>> feat.id |
| 552 |
'0' |
| 553 |
|
| 554 |
.. admonition:: OGR Details |
| 555 |
|
| 556 |
In the :program:`OGR` model, feature ids are long integers. Fiona record ids |
| 557 |
are therefore usually string representations of integer record indexes. |
| 558 |
|
| 559 |
Feature Properties |
| 560 |
------------------ |
| 561 |
|
| 562 |
A feature has a ``properties`` attribute. Its value is a mapping. The keys of |
| 563 |
the properties mapping are the same as the keys of the properties mapping in |
| 564 |
the schema of the collection the record comes from (see above). |
| 565 |
|
| 566 |
.. code-block:: pycon |
| 567 |
|
| 568 |
>>> for k, v in feat.properties.items(): |
| 569 |
... print(k, v) |
| 570 |
... |
| 571 |
PERIMETER 1.22107 |
| 572 |
FEATURE2 None |
| 573 |
NAME Mount Naomi Wilderness |
| 574 |
FEATURE1 Wilderness |
| 575 |
URL http://www.wilderness.net/index.cfm?fuse=NWPS&sec=wildView&wname=Mount%20Naomi |
| 576 |
AGBUR FS |
| 577 |
AREA 0.0179264 |
| 578 |
STATE_FIPS 49 |
| 579 |
WILDRNP020 332 |
| 580 |
STATE UT |
| 581 |
|
| 582 |
Feature Geometry |
| 583 |
---------------- |
| 584 |
|
| 585 |
A feature has a ``geometry`` attribute. Its value is a mapping with ``type`` |
| 586 |
and ``coordinates`` keys. |
| 587 |
|
| 588 |
.. code-block:: pycon |
| 589 |
|
| 590 |
>>> feat.geometry["type"] |
| 591 |
'Polygon' |
| 592 |
>>> feat.geometry["coordinates"] |
| 593 |
[[(-111.73527526855469, 41.995094299316406), ..., (-111.73527526855469, 41.995094299316406)]] |
| 594 |
|
| 595 |
Since the coordinates are just tuples, or lists of tuples, or lists of lists of |
| 596 |
tuples, the ``type`` tells you how to interpret them. |
| 597 |
|
| 598 |
+-------------------+---------------------------------------------------+ |
| 599 |
| Type | Coordinates | |
| 600 |
+===================+===================================================+ |
| 601 |
| Point | A single (x, y) tuple | |
| 602 |
+-------------------+---------------------------------------------------+ |
| 603 |
| LineString | A list of (x, y) tuple vertices | |
| 604 |
+-------------------+---------------------------------------------------+ |
| 605 |
| Polygon | A list of rings (each a list of (x, y) tuples) | |
| 606 |
+-------------------+---------------------------------------------------+ |
| 607 |
| MultiPoint | A list of points (each a single (x, y) tuple) | |
| 608 |
+-------------------+---------------------------------------------------+ |
| 609 |
| MultiLineString | A list of lines (each a list of (x, y) tuples) | |
| 610 |
+-------------------+---------------------------------------------------+ |
| 611 |
| MultiPolygon | A list of polygons (see above) | |
| 612 |
+-------------------+---------------------------------------------------+ |
| 613 |
|
| 614 |
Fiona, like the GeoJSON format, has both Northern Hemisphere "North is up" and |
| 615 |
Cartesian "X-Y" biases. The values within a tuple that denoted as ``(x, y)`` |
| 616 |
above are either (longitude E of the prime meridian, latitude N of the equator) |
| 617 |
or, for other projected coordinate systems, (easting, northing). |
| 618 |
|
| 619 |
.. admonition:: Long-Lat, not Lat-Long |
| 620 |
|
| 621 |
Even though most of us say "lat, long" out loud, Fiona's ``x,y`` is always |
| 622 |
easting, northing, which means ``(long, lat)``. Longitude first and latitude |
| 623 |
second, consistent with the GeoJSON format specification. |
| 624 |
|
| 625 |
Point Set Theory and Simple Features |
| 626 |
------------------------------------ |
| 627 |
|
| 628 |
In a proper, well-scrubbed vector data file the geometry mappings explained |
| 629 |
above are representations of geometric objects made up of :dfn:`point sets`. |
| 630 |
The following |
| 631 |
|
| 632 |
.. code-block:: python |
| 633 |
|
| 634 |
{"type": "LineString", "coordinates": [(0.0, 0.0), (0.0, 1.0)]} |
| 635 |
|
| 636 |
represents not just two points, but the set of infinitely many points along the |
| 637 |
line of length 1.0 from ``(0.0, 0.0)`` to ``(0.0, 1.0)``. In the application of |
| 638 |
point set theory commonly called :dfn:`Simple Features Access` [SFA]_ two |
| 639 |
geometric objects are equal if their point sets are equal whether they are |
| 640 |
equal in the Python sense or not. If you have Shapely (which implements Simple |
| 641 |
Features Access) installed, you can see this in by verifying the following. |
| 642 |
|
| 643 |
.. code-block:: pycon |
| 644 |
|
| 645 |
>>> from shapely.geometry import shape |
| 646 |
>>> l1 = shape( |
| 647 |
... {'type': 'LineString', 'coordinates': [(0, 0), (2, 2)]}) |
| 648 |
>>> l2 = shape( |
| 649 |
... {'type': 'LineString', 'coordinates': [(0, 0), (1, 1), (2, 2)]}) |
| 650 |
>>> l1 == l2 |
| 651 |
False |
| 652 |
>>> l1.equals(l2) |
| 653 |
True |
| 654 |
|
| 655 |
.. admonition:: Dirty data |
| 656 |
|
| 657 |
Some files may contain vectors that are :dfn:`invalid` from a simple |
| 658 |
features standpoint due to accident (inadequate quality control on the |
| 659 |
producer's end), intention ("dirty" vectors saved to a file for special |
| 660 |
treatment) or discrepancies of the numeric precision models (Fiona can't |
| 661 |
handle fixed precision models yet). Fiona doesn't sniff for or attempt to |
| 662 |
clean dirty data, so make sure you're getting yours from a clean source. |
| 663 |
|
| 664 |
Writing Vector Data |
| 665 |
=================== |
| 666 |
|
| 667 |
A vector file can be opened for writing in mode ``'a'`` (append) or mode |
| 668 |
``'w'`` (write). |
| 669 |
|
| 670 |
.. admonition:: Note |
| 671 |
|
| 672 |
The in situ "update" mode of :program:`OGR` is quite format dependent |
| 673 |
and is therefore not supported by Fiona. |
| 674 |
|
| 675 |
Appending Data to Existing Files |
| 676 |
-------------------------------- |
| 677 |
|
| 678 |
Let's start with the simplest if not most common use case, adding new records |
| 679 |
to an existing file. |
| 680 |
|
| 681 |
.. code-block:: console |
| 682 |
|
| 683 |
$ wget https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip |
| 684 |
$ unzip coutwildrnp.zip |
| 685 |
|
| 686 |
The coordinate reference system. format, and schema of the file are already |
| 687 |
defined, so it's opened with just two arguments as for reading, but in ``'a'`` |
| 688 |
mode. The new record is written to the end of the file using the |
| 689 |
:py:meth:`~fiona.collection.Collection.write` method. Accordingly, the length |
| 690 |
of the file grows from 67 to 68. |
| 691 |
|
| 692 |
.. code-block:: pycon |
| 693 |
|
| 694 |
>>> with fiona.open("coutwildrnp.shp", "a") as dst: |
| 695 |
... print(len(dst)) |
| 696 |
... with fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") as src: |
| 697 |
... feat = src[0] |
| 698 |
... print(feat.id, feat.properties["NAME"]) |
| 699 |
... dst.write(feat) |
| 700 |
... print(len(c)) |
| 701 |
... |
| 702 |
67 |
| 703 |
('0', 'Mount Naomi Wilderness') |
| 704 |
68 |
| 705 |
|
| 706 |
The feature you write must match the file's schema (because a file contains one |
| 707 |
type of record, remember). You'll get a :py:class:`ValueError` if it doesn't. |
| 708 |
|
| 709 |
.. code-block:: pycon |
| 710 |
|
| 711 |
>>> with fiona.open("coutwildrnp.shp", "a") as dst: |
| 712 |
... dst.write({'properties': {'foo': 'bar'}}) |
| 713 |
... |
| 714 |
Traceback (most recent call last): |
| 715 |
... |
| 716 |
ValueError: Record data not match collection schema |
| 717 |
|
| 718 |
Now, what about record ids? The id of a record written to a file is ignored and |
| 719 |
replaced by the next value appropriate for the file. If you read the file just |
| 720 |
appended to above, |
| 721 |
|
| 722 |
.. code-block:: pycon |
| 723 |
|
| 724 |
>>> with fiona.open("coutwildrnp.shp") as colxn: |
| 725 |
... feat = colxn[-1] |
| 726 |
... |
| 727 |
>>> feat.id |
| 728 |
'67' |
| 729 |
>>> feat.properties["NAME"] |
| 730 |
'Mount Naomi Wilderness' |
| 731 |
|
| 732 |
You'll see that the id of ``'0'`` which the record had when written is replaced |
| 733 |
by ``'67'``. |
| 734 |
|
| 735 |
The :py:meth:`~fiona.collection.Collection.write` method writes a single |
| 736 |
record to the collection's file. Its sibling |
| 737 |
:py:meth:`~fiona.collection.Collection.writerecords` writes a sequence (or |
| 738 |
iterator) of records. |
| 739 |
|
| 740 |
.. code-block:: pycon |
| 741 |
|
| 742 |
>>> with fiona.open("coutwildrnp.shp", "a") as colxn: |
| 743 |
... colxn.writerecords([feat, feat, feat]) |
| 744 |
... print(len(colxn)) |
| 745 |
... |
| 746 |
71 |
| 747 |
|
| 748 |
.. admonition:: Duplication |
| 749 |
|
| 750 |
Fiona's collections do not guard against duplication. The code above will |
| 751 |
write 3 duplicate records to the file, and they will be given unique |
| 752 |
sequential ids. |
| 753 |
|
| 754 |
.. admonition:: Transactions |
| 755 |
|
| 756 |
Fiona uses transactions during write operations to ensure data integrity. |
| 757 |
:py:meth:`writerecords` will start and commit one transaction. If there |
| 758 |
are lots of records, intermediate commits will be performed at reasonable |
| 759 |
intervals. |
| 760 |
|
| 761 |
Depending on the driver, a transaction can be a very costly operation. |
| 762 |
Since :py:meth:`write` is just a thin convenience wrapper that calls |
| 763 |
:py:meth:`writerecords` with a single record, you may experience significant |
| 764 |
performance issue if you write lots of features one by one using this method. |
| 765 |
Consider preparing your data first and then writing it in a single call to |
| 766 |
:py:meth:`writerecords`. |
| 767 |
|
| 768 |
.. admonition:: Buffering |
| 769 |
|
| 770 |
Fiona's output is buffered. The records passed to :py:meth:`write` and |
| 771 |
:py:meth:`writerecords` are flushed to disk when the collection is closed. |
| 772 |
You may also call :py:meth:`flush` periodically to write the buffer contents |
| 773 |
to disk. |
| 774 |
|
| 775 |
.. admonition:: Format requirements |
| 776 |
|
| 777 |
Format drivers may have specific requirements about what they store. For |
| 778 |
example, the Shapefile driver may "fix" topologically invalid features. |
| 779 |
|
| 780 |
Creating files of the same structure |
| 781 |
------------------------------------ |
| 782 |
|
| 783 |
Writing a new file is more complex than appending to an existing file because |
| 784 |
the file CRS, format, and schema have not yet been defined and must be done so |
| 785 |
by the programmer. Still, it's not very complicated. A schema is just |
| 786 |
a mapping, as described above. The possible |
| 787 |
formats are enumerated in the :py:attr:`fiona.supported_drivers` dictionary. |
| 788 |
|
| 789 |
Review the parameters of our demo file. |
| 790 |
|
| 791 |
.. code-block:: pycon |
| 792 |
|
| 793 |
>>> with fiona.open( |
| 794 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 795 |
) as src: |
| 796 |
... driver = src.driver |
| 797 |
... crs = src.crs |
| 798 |
... schema = src.schema |
| 799 |
... feat = src[1] |
| 800 |
... |
| 801 |
>>> driver |
| 802 |
'ESRI Shapefile' |
| 803 |
>>> crs |
| 804 |
CRS.from_epsg(4326) |
| 805 |
>>> pprint.pprint(schema) |
| 806 |
{'geometry': 'Polygon', |
| 807 |
'properties': {'AGBUR': 'str:80', |
| 808 |
'AREA': 'float:24.15', |
| 809 |
'FEATURE1': 'str:80', |
| 810 |
'FEATURE2': 'str:80', |
| 811 |
'NAME': 'str:80', |
| 812 |
'PERIMETER': 'float:24.15', |
| 813 |
'STATE': 'str:80', |
| 814 |
'STATE_FIPS': 'str:80', |
| 815 |
'URL': 'str:101', |
| 816 |
'WILDRNP020': 'int:10'}} |
| 817 |
|
| 818 |
We can create a new file using them. |
| 819 |
|
| 820 |
.. code-block:: pycon |
| 821 |
|
| 822 |
>>> with fiona.open("example.shp", "w", driver=driver, crs=crs, schema=schema) as dst: |
| 823 |
... print(len(dst)) |
| 824 |
... dst.write(feat) |
| 825 |
... print(len(dst)) |
| 826 |
... |
| 827 |
0 |
| 828 |
1 |
| 829 |
>>> dst.closed |
| 830 |
True |
| 831 |
>>> len(dst) |
| 832 |
1 |
| 833 |
|
| 834 |
Because the properties of the source schema are ordered and are passed in the |
| 835 |
same order to the write-mode collection, the written file's fields have the |
| 836 |
same order as those of the source file. |
| 837 |
|
| 838 |
The :py:attr:`~fiona.collection.Collection.profile` attribute makes duplication of |
| 839 |
a file's meta properties even easier. |
| 840 |
|
| 841 |
.. code-block:: pycon |
| 842 |
|
| 843 |
>>> src = fiona.open("zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip") |
| 844 |
>>> dst = fiona.open("example.shp", "w", **src.profile) |
| 845 |
|
| 846 |
Writing new files from scratch |
| 847 |
------------------------------- |
| 848 |
|
| 849 |
To write a new file from scratch we have to define our own specific driver, crs |
| 850 |
and schema. |
| 851 |
|
| 852 |
.. code-block:: pycon |
| 853 |
|
| 854 |
Consider the following record, structured in accordance to the `Python geo |
| 855 |
protocol <https://gist.github.com/sgillies/2217756>`__, representing the Eiffel |
| 856 |
Tower using a point geometry with UTM coordinates in zone 31N. |
| 857 |
|
| 858 |
.. code-block:: pycon |
| 859 |
|
| 860 |
>>> eiffel_tower = { |
| 861 |
... 'geometry': { |
| 862 |
... 'type': 'Point', |
| 863 |
... 'coordinates': (448252, 5411935) |
| 864 |
... }, |
| 865 |
... 'properties': dict([ |
| 866 |
... ('name', 'Eiffel Tower'), |
| 867 |
... ('height', 300.01), |
| 868 |
... ('view', 'scenic'), |
| 869 |
... ('year', 1889) |
| 870 |
... ]) |
| 871 |
... } |
| 872 |
|
| 873 |
A corresponding scheme could be: |
| 874 |
|
| 875 |
.. code-block:: pycon |
| 876 |
|
| 877 |
>>> landmarks_schema = { |
| 878 |
... 'geometry': 'Point', |
| 879 |
... 'properties': dict([ |
| 880 |
... ('name', 'str'), |
| 881 |
... ('height', 'float'), |
| 882 |
... ('view', 'str'), |
| 883 |
... ('year', 'int') |
| 884 |
... ]) |
| 885 |
... } |
| 886 |
|
| 887 |
The coordinate reference system of these landmark coordinates is ETRS89 / UTM |
| 888 |
zone 31N which is referenced in the EPSG database as EPSG:25831. |
| 889 |
|
| 890 |
.. code-block:: pycon |
| 891 |
|
| 892 |
>>> from fiona.crs import CRS |
| 893 |
>>> landmarks_crs = CRS.from_epsg(25831) |
| 894 |
|
| 895 |
An appropriate driver could be: |
| 896 |
|
| 897 |
.. code-block:: pycon |
| 898 |
|
| 899 |
>>> driver = "GeoJSON" |
| 900 |
|
| 901 |
Having specified schema, crs and driver, we are ready to open a file for |
| 902 |
writing our record: |
| 903 |
|
| 904 |
.. code-block:: pycon |
| 905 |
|
| 906 |
>>> with fiona.open( |
| 907 |
... "landmarks.geojson", |
| 908 |
... "w", |
| 909 |
... driver="GeoJSON", |
| 910 |
... crs=CRS.from_epsg(25831), |
| 911 |
... schema=landmarks_schema |
| 912 |
... ) as colxn: |
| 913 |
... colxn.write(eiffel_tower) |
| 914 |
... |
| 915 |
|
| 916 |
Ordering Record Fields |
| 917 |
...................... |
| 918 |
|
| 919 |
The 'properties' item of :py:func:`fiona.open`'s |
| 920 |
'schema' keyword argument is used to specify the ordering with written files. |
| 921 |
This usually a :py:func:`dict` built-in |
| 922 |
|
| 923 |
.. code-block:: python |
| 924 |
|
| 925 |
fiona.open( |
| 926 |
"/tmp/file.shp", |
| 927 |
"w", |
| 928 |
schema={"properties": {"bar": "int", "foo": "str"}}, |
| 929 |
**kwargs |
| 930 |
) |
| 931 |
|
| 932 |
The order may alternatively be supplied as a list of property items |
| 933 |
|
| 934 |
.. code-block:: python |
| 935 |
|
| 936 |
fiona.open( |
| 937 |
"/tmp/file.shp", |
| 938 |
"w", |
| 939 |
schema={"properties": [("bar", "int"), ("foo", "str")]}, |
| 940 |
**kwargs |
| 941 |
) |
| 942 |
|
| 943 |
3D Coordinates and Geometry Types |
| 944 |
--------------------------------- |
| 945 |
|
| 946 |
If you write 3D coordinates, ones having (x, y, z) tuples, to a 2D file |
| 947 |
('Point' schema geometry, for example) the z values will be lost. |
| 948 |
|
| 949 |
.. code-block:: python |
| 950 |
|
| 951 |
>>> feat = {"geometry": {"type": "Point", "coordinates": (-1, 1, 5)}} |
| 952 |
>>> with fiona.open( |
| 953 |
... "example.shp", |
| 954 |
... "w", |
| 955 |
... driver="Shapefile", |
| 956 |
... schema={"geometry": "Point", "properties": {}} |
| 957 |
... ) as dst: |
| 958 |
... dst.write(feat) |
| 959 |
... |
| 960 |
>>> with fiona.open("example.shp") as src: |
| 961 |
... print(src[0].geometry.coordinates) |
| 962 |
... |
| 963 |
(-1.0, 1.0) |
| 964 |
|
| 965 |
If you write 2D coordinates, ones having only (x, y) tuples, to a 3D file ('3D |
| 966 |
Point' schema geometry, for example) a default z value of 0 will be provided. |
| 967 |
|
| 968 |
.. code-block:: python |
| 969 |
|
| 970 |
>>> feat = {"geometry": {"type": "Point", "coordinates": (-1, 1)}} |
| 971 |
>>> with fiona.open( |
| 972 |
... "example.shp", |
| 973 |
... "w", |
| 974 |
... driver="Shapefile", |
| 975 |
... schema={"geometry": "3D Point", "properties": {}} |
| 976 |
... ) as dst: |
| 977 |
... dst.write(feat) |
| 978 |
... |
| 979 |
>>> with fiona.open("example.shp") as src: |
| 980 |
... print(src[0].geometry.coordinates) |
| 981 |
... |
| 982 |
(-1.0, 1.0, 0.0) |
| 983 |
|
| 984 |
Advanced Topics |
| 985 |
=============== |
| 986 |
|
| 987 |
OGR configuration options |
| 988 |
------------------------- |
| 989 |
|
| 990 |
GDAL/OGR has a large number of features that are controlled by global or |
| 991 |
thread-local `configuration options. <https://gdal.org/user/configoptions.html>`_ |
| 992 |
Fiona allows you to configure these options using a context manager, ``fiona.Env``. |
| 993 |
This class's constructor takes GDAL/OGR configuration options as keyword arguments. |
| 994 |
To see debugging information from GDAL/OGR, for example, you may do the following. |
| 995 |
|
| 996 |
.. code-block:: python |
| 997 |
|
| 998 |
import logging |
| 999 |
import fiona |
| 1000 |
|
| 1001 |
logging.basicConfig(level=logging.DEBUG) |
| 1002 |
|
| 1003 |
with fiona.Env(CPL_DEBUG=True): |
| 1004 |
fiona.open( |
| 1005 |
"zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 1006 |
) |
| 1007 |
|
| 1008 |
The following extra messages will appear in the Python logger's output. |
| 1009 |
|
| 1010 |
.. code-block:: |
| 1011 |
|
| 1012 |
DEBUG:fiona._env:CPLE_None in GNM: GNMRegisterAllInternal |
| 1013 |
DEBUG:fiona._env:CPLE_None in GNM: RegisterGNMFile |
| 1014 |
DEBUG:fiona._env:CPLE_None in GNM: RegisterGNMdatabase |
| 1015 |
DEBUG:fiona._env:CPLE_None in GNM: GNMRegisterAllInternal |
| 1016 |
DEBUG:fiona._env:CPLE_None in GNM: RegisterGNMFile |
| 1017 |
DEBUG:fiona._env:CPLE_None in GNM: RegisterGNMdatabase |
| 1018 |
DEBUG:fiona._env:CPLE_None in GDAL: GDALOpen(tests/data/coutwildrnp.shp, this=0x1683930) succeeds as ESRI Shapefile. |
| 1019 |
|
| 1020 |
If you call ``fiona.open()`` with no surrounding ``Env`` environment, one will |
| 1021 |
be created for you. |
| 1022 |
|
| 1023 |
When your program exits the environment's ``with`` block the configuration reverts |
| 1024 |
to its previous state. |
| 1025 |
|
| 1026 |
Driver configuration options |
| 1027 |
---------------------------- |
| 1028 |
|
| 1029 |
Drivers can have dataset open, dataset creation, respectively layer creation |
| 1030 |
options. These options can be found on the drivers page on `GDAL's homepage. |
| 1031 |
<https://gdal.org/drivers/vector/index.html>`_ or using the ``fiona.meta`` |
| 1032 |
module: |
| 1033 |
|
| 1034 |
.. code-block:: pycon |
| 1035 |
|
| 1036 |
>>> import fiona.meta |
| 1037 |
>>> fiona.meta.print_driver_options("GeoJSON") |
| 1038 |
|
| 1039 |
These options can be passed to ``fiona.open``: |
| 1040 |
|
| 1041 |
.. code-block:: python |
| 1042 |
|
| 1043 |
import fiona |
| 1044 |
fiona.open('tests/data/coutwildrnp.json', ARRAY_AS_STRING="YES") |
| 1045 |
|
| 1046 |
Cloud storage credentials |
| 1047 |
------------------------- |
| 1048 |
|
| 1049 |
One of the most important uses of ``fiona.Env`` is to set credentials for |
| 1050 |
accessing data stored in AWS S3 or another cloud storage system. |
| 1051 |
|
| 1052 |
.. code-block:: python |
| 1053 |
|
| 1054 |
import fiona |
| 1055 |
from fiona.session import AWSSession |
| 1056 |
|
| 1057 |
with fiona.Env( |
| 1058 |
session=AWSSession( |
| 1059 |
aws_access_key_id="key", |
| 1060 |
aws_secret_access_key="secret" |
| 1061 |
) |
| 1062 |
): |
| 1063 |
fiona.open("zip+s3://example-bucket/example.zip") |
| 1064 |
|
| 1065 |
The AWSSession class is currently the only credential session manager in Fiona. |
| 1066 |
The source code has an example of how classes for other cloud storage providers |
| 1067 |
may be implemented. AWSSession relies upon boto3 and botocore, which will be |
| 1068 |
installed as extra dependencies of Fiona if you run ``pip install fiona[s3]``. |
| 1069 |
|
| 1070 |
If you call ``fiona.open()`` with no surrounding ``Env`` and pass a path to an |
| 1071 |
S3 object, a session will be created for you using code equivalent to the |
| 1072 |
following code. |
| 1073 |
|
| 1074 |
.. code-block:: python |
| 1075 |
|
| 1076 |
import boto3 |
| 1077 |
|
| 1078 |
from fiona.session import AWSSession |
| 1079 |
import fiona |
| 1080 |
|
| 1081 |
with fiona.Env(session=AWSSession(boto3.Session())): |
| 1082 |
fiona.open("zip+s3://fiona-testing/coutwildrnp.zip") |
| 1083 |
|
| 1084 |
Slicing and masking iterators |
| 1085 |
----------------------------- |
| 1086 |
|
| 1087 |
With some vector data formats a spatial index accompanies the data file, |
| 1088 |
allowing efficient bounding box searches. A collection's |
| 1089 |
:py:meth:`~fiona.collection.Collection.items` method returns an iterator over |
| 1090 |
pairs of FIDs and records that intersect a given ``(minx, miny, maxx, maxy)`` |
| 1091 |
bounding box or geometry object. Spatial filtering may be inaccurate and |
| 1092 |
returning all features overlapping the envelope of the geometry. The |
| 1093 |
collection's own coordinate reference system (see below) is used to interpret |
| 1094 |
the box's values. If you want a list of the iterator's items, pass it to |
| 1095 |
Python's builtin :py:func:`list` as shown below. |
| 1096 |
|
| 1097 |
.. code-block:: pycon |
| 1098 |
|
| 1099 |
>>> colxn = fiona.open( |
| 1100 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 1101 |
... ) |
| 1102 |
>>> hits = list(colxn.items(bbox=(-110.0, 36.0, -108.0, 38.0))) |
| 1103 |
>>> len(hits) |
| 1104 |
5 |
| 1105 |
|
| 1106 |
The iterator method takes the same ``stop`` or ``start, stop[, step]`` |
| 1107 |
slicing arguments as :py:func:`itertools.islice`. |
| 1108 |
To get just the first two items from that iterator, pass a stop index. |
| 1109 |
|
| 1110 |
.. code-block:: pycon |
| 1111 |
|
| 1112 |
>>> hits = colxn.items(2, bbox=(-110.0, 36.0, -108.0, 38.0))) |
| 1113 |
>>> len(list(hits)) |
| 1114 |
2 |
| 1115 |
|
| 1116 |
To get the third through fifth items from that iterator, pass start and stop |
| 1117 |
indexes. |
| 1118 |
|
| 1119 |
.. code-block:: pycon |
| 1120 |
|
| 1121 |
>>> hits = colxn.items(2, 5, bbox=(-110.0, 36.0, -108.0, 38.0))) |
| 1122 |
>>> len(list(hits)) |
| 1123 |
3 |
| 1124 |
|
| 1125 |
To filter features by property values, use Python's builtin :py:func:`filter` and |
| 1126 |
:keyword:`lambda` or your own filter function that takes a single feature |
| 1127 |
record and returns ``True`` or ``False``. |
| 1128 |
|
| 1129 |
.. code-block:: pycon |
| 1130 |
|
| 1131 |
>>> def pass_positive_area(rec): |
| 1132 |
... return rec['properties'].get('AREA', 0.0) > 0.0 |
| 1133 |
... |
| 1134 |
>>> colxn = fiona.open( |
| 1135 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 1136 |
... ) |
| 1137 |
>>> hits = filter(pass_positive_area, colxn) |
| 1138 |
>>> len(list(hits)) |
| 1139 |
67 |
| 1140 |
|
| 1141 |
Reading Multilayer data |
| 1142 |
----------------------- |
| 1143 |
|
| 1144 |
Up to this point, only simple datasets with one thematic layer or feature type |
| 1145 |
per file have been shown and the venerable Esri Shapefile has been the primary |
| 1146 |
example. Other GIS data formats can encode multiple layers or feature types |
| 1147 |
within a single file or directory. GeoPackage is one example of such a format. |
| 1148 |
A more useful example, for the purpose of this manual, is a directory or |
| 1149 |
zipfile comprising multiple shapefiles. The GitHub-hosted zipfile we've been |
| 1150 |
using in these examples is, in fact, such a multilayer dataset. |
| 1151 |
|
| 1152 |
The layers of a dataset can be listed using :py:func:`fiona.listlayers`. In |
| 1153 |
the shapefile format case, layer names match base names of the files. |
| 1154 |
|
| 1155 |
.. code-block:: pycon |
| 1156 |
|
| 1157 |
>>> fiona.listlayers( |
| 1158 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 1159 |
... ) |
| 1160 |
['coutwildrnp'] |
| 1161 |
|
| 1162 |
Unlike OGR, Fiona has no classes representing layers or data sources. To access |
| 1163 |
the features of a layer, open a collection using the path to the data source |
| 1164 |
and specify the layer by name using the `layer` keyword. |
| 1165 |
|
| 1166 |
.. code-block:: pycon |
| 1167 |
|
| 1168 |
dataset_path = "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip" |
| 1169 |
>>> for name in fiona.listlayers(dataset_path): |
| 1170 |
... with fiona.open(dataset_path, layer=name) as colxn: |
| 1171 |
... pprint.pprint(colxn.schema) |
| 1172 |
... |
| 1173 |
{'geometry': 'Polygon', |
| 1174 |
'properties': {'AGBUR': 'str:80', |
| 1175 |
'AREA': 'float:24.15', |
| 1176 |
'FEATURE1': 'str:80', |
| 1177 |
'FEATURE2': 'str:80', |
| 1178 |
'NAME': 'str:80', |
| 1179 |
'PERIMETER': 'float:24.15', |
| 1180 |
'STATE': 'str:80', |
| 1181 |
'STATE_FIPS': 'str:80', |
| 1182 |
'URL': 'str:101', |
| 1183 |
'WILDRNP020': 'int:10'}} |
| 1184 |
|
| 1185 |
Layers may also be specified by their numerical index. |
| 1186 |
|
| 1187 |
.. code-block:: pycon |
| 1188 |
|
| 1189 |
>>> for index, name in enumerate(fiona.listlayers(dataset_path)): |
| 1190 |
... with fiona.open(dataset_path, layer=index) as colxn: |
| 1191 |
... print(len(colxn)) |
| 1192 |
... |
| 1193 |
67 |
| 1194 |
|
| 1195 |
If no layer is specified, :py:func:`fiona.open` returns an open collection |
| 1196 |
using the first layer. |
| 1197 |
|
| 1198 |
.. code-block:: pycon |
| 1199 |
|
| 1200 |
>>> with fiona.open(dataset_path) as colxn: |
| 1201 |
... colxn.name == fiona.listlayers(datasset_path)[0] |
| 1202 |
... |
| 1203 |
True |
| 1204 |
|
| 1205 |
We've been relying on this implicit behavior throughout the manual. |
| 1206 |
|
| 1207 |
The most general way to open a shapefile for reading, using all of the |
| 1208 |
parameters of :py:func:`fiona.open`, is to treat it as a data source with |
| 1209 |
a named layer. |
| 1210 |
|
| 1211 |
.. code-block:: pycon |
| 1212 |
|
| 1213 |
>>> fiona.open( |
| 1214 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip", |
| 1215 |
... mode="r", |
| 1216 |
... layer="coutwildrnp" |
| 1217 |
... ) |
| 1218 |
|
| 1219 |
In practice, it is fine to rely on the implicit first layer and default ``'r'`` |
| 1220 |
mode and open a shapefile like this: |
| 1221 |
|
| 1222 |
.. code-block:: pycon |
| 1223 |
|
| 1224 |
>>> fiona.open( |
| 1225 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip", |
| 1226 |
... ) |
| 1227 |
|
| 1228 |
Writing Multilayer data |
| 1229 |
----------------------- |
| 1230 |
|
| 1231 |
To write an entirely new layer to a multilayer data source, simply provide |
| 1232 |
a unique name to the `layer` keyword argument. |
| 1233 |
|
| 1234 |
.. code-block:: pycon |
| 1235 |
|
| 1236 |
>>> with fiona.open( |
| 1237 |
... "zip+https://github.com/Toblerity/Fiona/files/11151652/coutwildrnp.zip", |
| 1238 |
... ) as src: |
| 1239 |
... with fiona.open("example.gpkg", "w", layer="example one", **src.profile) as dst: |
| 1240 |
... dst.writerecords(src) |
| 1241 |
... |
| 1242 |
>>> fiona.listlayers("example.gpkg") |
| 1243 |
['example one'] |
| 1244 |
|
| 1245 |
In ``'w'`` mode, existing layers will be overwritten if specified, just as normal |
| 1246 |
files are overwritten by Python's :py:func:`open` function. |
| 1247 |
|
| 1248 |
Unsupported drivers |
| 1249 |
------------------- |
| 1250 |
|
| 1251 |
Fiona maintains a list of OGR drivers in :py:attr:`fiona.supported_drivers` |
| 1252 |
that are tested and known to work together with Fiona. Opening a dataset using |
| 1253 |
an unsupported driver or access mode results in an :py:attr: `DriverError` |
| 1254 |
exception. By passing `allow_unsupported_drivers=True` to :py:attr:`fiona.open` |
| 1255 |
no compatibility checks are performed and unsupported OGR drivers can be used. |
| 1256 |
However, there are no guarantees that Fiona will be able to access or write |
| 1257 |
data correctly using an unsupported driver. |
| 1258 |
|
| 1259 |
.. code-block:: python |
| 1260 |
|
| 1261 |
import fiona |
| 1262 |
|
| 1263 |
with fiona.open("file.kmz", allow_unsupported_drivers=True) as collection: |
| 1264 |
... |
| 1265 |
|
| 1266 |
Not all OGR drivers are necessarily enabled in every GDAL distribution. The |
| 1267 |
following code snippet lists the drivers included in the GDAL installation |
| 1268 |
used by Fiona: |
| 1269 |
|
| 1270 |
.. code-block:: python |
| 1271 |
|
| 1272 |
from fiona.env import Env |
| 1273 |
|
| 1274 |
with Env() as gdalenv: |
| 1275 |
print(gdalenv.drivers().keys()) |
| 1276 |
|
| 1277 |
MemoryFile and ZipMemoryFile |
| 1278 |
---------------------------- |
| 1279 |
|
| 1280 |
:py:class:`fiona.io.MemoryFile` and :py:class:`fiona.io.ZipMemoryFile` allow |
| 1281 |
formatted feature collections, even zipped feature collections, to be read or |
| 1282 |
written in memory, with no filesystem access required. For example, you may |
| 1283 |
have a zipped shapefile in a stream of bytes coming from a web upload or |
| 1284 |
download. |
| 1285 |
|
| 1286 |
.. code-block:: pycon |
| 1287 |
|
| 1288 |
>>> data = open('tests/data/coutwildrnp.zip', 'rb').read() |
| 1289 |
>>> len(data) |
| 1290 |
154006 |
| 1291 |
>>> data[:20] |
| 1292 |
b'PK\x03\x04\x14\x00\x00\x00\x00\x00\xaa~VM\xech\xae\x1e\xec\xab' |
| 1293 |
|
| 1294 |
The feature collection in this stream of bytes can be accessed by wrapping it |
| 1295 |
in an instance of ZipMemoryFile. |
| 1296 |
|
| 1297 |
.. code-block:: pycon |
| 1298 |
|
| 1299 |
>>> from fiona.io import ZipMemoryFile |
| 1300 |
>>> with ZipMemoryFile(data) as zip: |
| 1301 |
... with zip.open('coutwildrnp.shp') as collection: |
| 1302 |
... print(len(collection)) |
| 1303 |
... print(collection.schema) |
| 1304 |
... |
| 1305 |
67 |
| 1306 |
{'properties': {'PERIMETER': 'float:24.15', 'FEATURE2': 'str:80', 'NAME': 'str:80', 'FEATURE1': 'str:80', 'URL': 'str:101', 'AGBUR': 'str:80', 'AREA': 'float:24.15', 'STATE_FIPS': 'str:80', 'WILDRNP020': 'int:10', 'STATE': 'str:80'}, 'geometry': 'Polygon'} |
| 1307 |
|
| 1308 |
*New in 1.8.0* |
| 1309 |
|
| 1310 |
Fiona command line interface |
| 1311 |
============================ |
| 1312 |
|
| 1313 |
Fiona comes with a command line interface called "fio". See the |
| 1314 |
`CLI Documentation <cli.html>`__ for detailed usage instructions. |
| 1315 |
|
| 1316 |
Final Notes |
| 1317 |
=========== |
| 1318 |
|
| 1319 |
This manual is a work in progress and will grow and improve with Fiona. |
| 1320 |
Questions and suggestions are very welcome. Please feel free to use the `issue |
| 1321 |
tracker <https://github.com/Toblerity/Fiona/issues>`__ or email the author |
| 1322 |
directly. |
| 1323 |
|
| 1324 |
Do see the `README <README.html>`__ for installation instructions and |
| 1325 |
information about supported versions of Python and other software dependencies. |
| 1326 |
|
| 1327 |
Fiona would not be possible without the `contributions of other developers |
| 1328 |
<README.html#credits>`__, especially Frank Warmerdam and Even Rouault, the |
| 1329 |
developers of GDAL/OGR; and Mike Weisman, who saved Fiona from neglect and |
| 1330 |
obscurity. |
| 1331 |
|
| 1332 |
References |
| 1333 |
========== |
| 1334 |
|
| 1335 |
.. [Kent1978] William Kent, Data and Reality, North Holland, 1978. |
| 1336 |
.. [ESRI1998] ESRI Shapefile Technical Description. July 1998. https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf |
| 1337 |
.. [GeoJSON] https://geojson.org |
| 1338 |
.. [JSON] https://www.ietf.org/rfc/rfc4627 |
| 1339 |
.. [SFA] https://en.wikipedia.org/wiki/Simple_feature_access |