Command Line Interface ====================== Fiona's new command line interface is a program named "fio". .. code-block:: console Usage: fio [OPTIONS] COMMAND [ARGS]... Fiona command line interface. Options: -v, --verbose Increase verbosity. -q, --quiet Decrease verbosity. --aws-profile TEXT Select a profile from the AWS credentials file --aws-no-sign-requests Make requests anonymously --aws-requester-pays Requester pays data transfer costs --version Show the version and exit. --gdal-version Show the version and exit. --python-version Show the version and exit. --help Show this message and exit. Commands: bounds Print the extent of GeoJSON objects calc Calculate GeoJSON property by Python expression cat Concatenate and print the features of datasets collect Collect a sequence of features. distrib Distribute features from a collection. dump Dump a dataset to GeoJSON. env Print information about the fio environment. filter Evaluate pipeline expressions to filter GeoJSON features. info Print information about a dataset. insp Open a dataset and start an interpreter. load Load GeoJSON to a dataset in another format. ls List layers in a datasource. map Map a pipeline expression over GeoJSON features. reduce Reduce a stream of GeoJSON features to one value. rm Remove a datasource or an individual layer. It is developed using the ``click`` package. bounds ------ The bounds command reads LF or RS-delimited GeoJSON texts, either features or collections, from stdin and prints their bounds with or without other data to stdout. With no options, it works like this: .. code-block:: console $ fio cat docs/data/test_uk.shp | head -n 1 \ > | fio bounds [0.735, 51.357216, 0.947778, 51.444717] Using ``--with-id`` gives you .. code-block:: console $ fio cat docs/data/test_uk.shp | head -n 1 \ > | fio bounds --with-id {"id": "0", "bbox": [0.735, 51.357216, 0.947778, 51.444717]} calc ---- The calc command creates a new property on GeoJSON features using the specified expression. The expression is evaluated in a restricted namespace containing 4 functions (`sum`, `pow`, `min`, `max`), the `math` module, the shapely `shape` function, type conversions (`bool`, `int`, `str`, `len`, `float`), and an object `f` representing the feature to be evaluated. This `f` object allows access in javascript-style dot notation for convenience. The expression will be evaluated for each feature and its return value will be added to the properties as the specified property_name. Existing properties will not be overwritten by default (an `Exception` is raised). .. code-block:: console $ fio cat data.shp | fio calc sumAB "f.properties.A + f.properties.B" .. note:: ``fio calc`` requires installation of the "calc" set of extra requirements that will be installed by ``pip install fiona[calc]``. cat --- The cat command concatenates the features of one or more datasets and prints them as a `JSON text sequence `__ of features. In other words: GeoJSON feature objects, possibly pretty printed, optionally separated by ASCII RS (\x1e) chars using `--rs`. The output of ``fio cat`` can be piped to ``fio load`` to create new concatenated datasets. .. code-block:: console $ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \ > | fio load /tmp/double.shp --driver Shapefile $ fio info /tmp/double.shp --count 96 $ fio info docs/data/test_uk.shp --count 48 The cat command provides optional methods to filter data, which are different to the ``fio filter`` tool. A bounding box ``--bbox w,s,e,n`` tests for a spatial intersection with the geometries. An attribute filter ``--where TEXT`` can use an `SQL WHERE clause `__. If more than one datasets is passed to ``fio cat``, the attributes used in the WHERE clause must be valid for each dataset. collect ------- The collect command takes a JSON text sequence of GeoJSON feature objects, such as the output of ``fio cat`` and writes a GeoJSON feature collection. .. code-block:: console $ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \ > | fio collect > /tmp/collected.json $ fio info /tmp/collected.json --count 96 distrib ------- The inverse of fio-collect, fio-distrib takes a GeoJSON feature collection and writes a JSON text sequence of GeoJSON feature objects. .. code-block:: console $ fio info --count tests/data/coutwildrnp.shp 67 $ fio cat tests/data/coutwildrnp.shp | fio collect | fio distrib | wc -l 67 dump ---- The dump command reads a vector dataset and writes a GeoJSON feature collection to stdout. Its output can be piped to ``fio load`` (see below). .. code-block:: console $ fio dump docs/data/test_uk.shp --indent 2 --precision 2 | head { "features": [ { "geometry": { "coordinates": [ [ [ 0.9, 51.36 ], You can optionally dump out JSON text sequences using ``--x-json-seq``. Since version 1.4.0, ``fio cat`` is the better tool for generating sequences. .. code-block:: console $ fio dump docs/data/test_uk.shp --precision 2 --x-json-seq | head -n 2 {"geometry": {"coordinates": [[[0.9, 51.36], [0.89, 51.36], [0.79, 51.37], [0.78, 51.37], [0.77, 51.38], [0.76, 51.38], [0.75, 51.39], [0.74, 51.4], [0.73, 51.41], [0.74, 51.43], [0.75, 51.44], [0.76, 51.44], [0.79, 51.44], [0.89, 51.42], [0.9, 51.42], [0.91, 51.42], [0.93, 51.4], [0.94, 51.39], [0.94, 51.38], [0.95, 51.38], [0.95, 51.37], [0.95, 51.37], [0.94, 51.37], [0.9, 51.36], [0.9, 51.36]]], "type": "Polygon"}, "id": "0", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"} {"geometry": {"coordinates": [[[-4.66, 51.16], [-4.67, 51.16], [-4.67, 51.16], [-4.67, 51.17], [-4.67, 51.19], [-4.67, 51.19], [-4.67, 51.2], [-4.66, 51.2], [-4.66, 51.19], [-4.65, 51.16], [-4.65, 51.16], [-4.65, 51.16], [-4.66, 51.16]]], "type": "Polygon"}, "id": "1", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"} info ---- The info command prints information about a dataset as a JSON object. .. code-block:: console $ fio info docs/data/test_uk.shp --indent 2 { "count": 48, "crs": "+datum=WGS84 +no_defs +proj=longlat", "driver": "ESRI Shapefile", "bounds": [ -8.621389, 49.911659, 1.749444, 60.844444 ], "schema": { "geometry": "Polygon", "properties": { "CAT": "float:16", "FIPS_CNTRY": "str:80", "CNTRY_NAME": "str:80", "AREA": "float:15.2", "POP_CNTRY": "float:15.2" } } } You can process this JSON using, e.g., `underscore-cli `__. .. code-block:: console $ fio info docs/data/test_uk.shp | underscore extract count 48 You can also optionally get single info items as plain text (not JSON) strings .. code-block:: console $ fio info docs/data/test_uk.shp --count 48 $ fio info docs/data/test_uk.shp --bounds -8.621389 49.911659 1.749444 60.844444 load ---- The load command reads GeoJSON features from stdin and writes them to a vector dataset using another format. .. code-block:: console $ fio dump docs/data/test_uk.shp \ > | fio load /tmp/test.shp --driver Shapefile This command also supports GeoJSON text sequences. RS-separated sequences will be detected. If you want to load LF-separated sequences, you must specify ``--x-json-seq``. .. code-block:: console $ fio cat docs/data/test_uk.shp | fio load /tmp/foo.shp --driver Shapefile $ fio info /tmp/foo.shp --indent 2 { "count": 48, "crs": "+datum=WGS84 +no_defs +proj=longlat", "driver": "ESRI Shapefile", "bounds": [ -8.621389, 49.911659, 1.749444, 60.844444 ], "schema": { "geometry": "Polygon", "properties": { "AREA": "float:24.15", "CNTRY_NAME": "str:80", "POP_CNTRY": "float:24.15", "FIPS_CNTRY": "str:80", "CAT": "float:24.15" } } } The underscore-cli process command is another way of turning a GeoJSON feature collection into a feature sequence. .. code-block:: console $ fio dump docs/data/test_uk.shp \ > | underscore process \ > 'each(data.features,function(o){console.log(JSON.stringify(o))})' \ > | fio load /tmp/test-seq.shp --x-json-seq --driver Shapefile filter ------ For each feature read from stdin, filter evaluates a pipeline of one or more steps described using methods from the Shapely library in Lisp-like expressions. If the pipeline expression evaluates to True, the feature passes through the filter. Otherwise the feature does not pass. For example, this pipeline expression .. code-block:: console $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \ | fio filter '< (distance g (Point -109.0 38.5)) 100' lets through all features that are less than 100 meters from the given point and filters out all other features. *New in version 1.10*: these parenthesized list expressions. The older style Python expressions like .. code-block:: 'f.properties.area > 1000.0' are deprecated and will not be supported in version 2.0. Note this tool is different from ``fio cat --where TEXT ...``, which provides SQL WHERE clause filtering of feature attributes. .. note:: ``fio filter`` requires installation of the "calc" set of extra requirements that will be installed by ``pip install fiona[calc]``. map --- For each feature read from stdin, ``fio map`` applies a transformation pipeline and writes a copy of the feature, containing the modified geometry, to stdout. For example, polygonal features can be roughly "cleaned" by using a ``buffer g 0`` pipeline. .. code-block:: console $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \ | fio map 'buffer g 0' *New in version 1.10*. .. note:: ``fio map`` requires installation of the "calc" set of extra requirements that will be installed by ``pip install fiona[calc]``. reduce ------ Given a sequence of GeoJSON features (RS-delimited or not) on stdin this prints a single value using a provided transformation pipeline. The set of geometries of the input features in the context of these expressions is named ``c``. For example, the pipeline expression .. code-block:: console $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \ | fio reduce 'unary_union c' dissolves the geometries of input features. *New in version 1.10*. .. note:: ``fio reduce`` requires installation of the "calc" set of extra requirements that will be installed by ``pip install fiona[calc]``. rm -- The rm command deletes an entire datasource or a single layer in a multi-layer datasource. If the datasource is composed of multiple files (e.g. an ESRI Shapefile) all of the files will be removed. .. code-block:: console $ fio rm countries.shp $ fio rm --layer forests land_cover.gpkg Expressions and functions ------------------------- ``fio filter``, ``fio map``, and ``fio reduce`` expressions take the form of parenthesized lists that may contain other expressions. The first item in a list is the name of a function or method, or an expression that evaluates to a function. The second item is the function's first argument or the object to which the method is bound. The remaining list items are the positional and keyword arguments for the named function or method. The list of functions and callables available in an expression includes: * Python operators such as ``+``, ``/``, and ``<=`` * Python builtins such as ``dict``, ``list``, and ``map`` * All public functions from itertools, e.g. ``islice``, and ``repeat`` * All functions importable from Shapely 2.0, e.g. ``Point``, and ``unary_union`` * All methods of Shapely geometry classes * Functions specific to Fiona Expressions are evaluated by ``fiona.features.snuggs.eval()``. Let's look at some examples using that function. .. note:: The outer parentheses are not optional within ``snuggs.eval()``. .. note:: ``snuggs.eval()`` does not use Python's builtin ``eval()`` but isn't intended to be a secure computing environment. Expressions which access the computer's filesystem and create new processes are possible. Builtin Python functions ------------------------ ``bool()`` .. code-block:: python >>> snuggs.eval('(bool 0)') False ``range()`` .. code-block:: python >>> snuggs.eval('(range 1 4)') range(1, 4) ``list()`` .. code-block:: python >>> snuggs.eval('(list (range 1 4))') [1, 2, 3] Values can be bound to names for use in expressions. .. code-block:: python >>> snuggs.eval('(list (range start stop))', start=0, stop=5) [0, 1, 2, 3, 4] Itertools functions ------------------- Here's an example of using ``itertools.repeat()``. .. code-block:: python >>> snuggs.eval('(list (repeat "*" times))', times=6) ['*', '*', '*', '*', '*', '*'] Shapely functions ----------------- Here's an expression that evaluates to a Shapely Point instance. .. code-block:: python >>> snuggs.eval('(Point 0 0)') The expression below evaluates to a MultiPoint instance. .. code-block:: python >>> snuggs.eval('(union (Point 0 0) (Point 1 1))') Functions specific to fiona --------------------------- The fio CLI introduces four new functions not available in Python's standard library, or Shapely: ``collect()``, ``dump()``, ``identity()``, and ``vertex_count()``. The ``collect()`` function turns a list of geometries into a geometry collection and ``dump()`` does the inverse, turning a geometry collection into a sequence of geometries. .. code-block:: python >>> snuggs.eval('(collect (Point 0 0) (Point 1 1))') >>> snuggs.eval('(list (dump (collect (Point 0 0) (Point 1 1))))') [, ] The ``identity()`` function returns its single argument. .. code-block:: python >>> snuggs.eval('(identity 42)') 42 To count the number of vertices in a geometry, use ``vertex_count()``. .. code-block:: python >>> snuggs.eval('(vertex_count (Point 0 0))') 1 The ``area()``, ``buffer()``, ``distance()``, ``length()``, ``simplify()``, and ``set_precision()`` functions shadow, or override, functions from the shapely module. They automatically reproject geometry objects from their natural coordinate reference system (CRS) of ``OGC:CRS84`` to ``EPSG:6933`` so that the shapes can be measured or modified using meters as units. ``buffer()`` dilates (or erodes) a given geometry, with coordinates in decimal longitude and latitude degrees, by a given distance in meters. .. code-block:: python >>> snuggs.eval('(buffer (Point 0 0) :distance 100)') The ``area()`` and ``length()`` of this polygon have units of square meter and meter. .. code-block:: python >>> snuggs.eval('(area (buffer (Point 0 0) :distance 100))') 31214.451487413342 >>> snuggs.eval('(length (buffer (Point 0 0) :distance 100))') 627.3096977558143 The ``distance()`` between two geometries is in meters. .. code-block:: python >>> snuggs.eval('(distance (Point 0 0) (Point 0.1 0.1))') 15995.164946207413 A geometry can be simplified to a tolerance value in meters using ``simplify()``. There are more examples of this function later in this document. .. code-block:: python >>> snuggs.eval('(simplify (buffer (Point 0 0) :distance 100) :tolerance 100)') The ``set_precision()`` function snaps a geometry to a fixed precision grid with a size in meters. .. code-block:: python >>> snuggs.eval('(set_precision (Point 0.001 0.001) :grid_size 500)') Feature and geometry context for expressions -------------------------------------------- ``fio filter`` and ``fio map`` evaluate expressions in the context of a GeoJSON feature and its geometry attribute. These are named ``f`` and ``g``. For example, here is an expression that tests whether the input feature is within 62.5 kilometers of the given point. .. code-block:: lisp < (distance g (Point 4 43)) 62.5E3 ``fio reduce`` evaluates expressions in the context of the sequence of all input geometries, named ``c``. For example, this expression dissolves input geometries using Shapely's ``unary_union``. .. code-block:: lisp unary_union c Coordinate Reference System Transformations ------------------------------------------- The ``fio cat`` command can optionally transform feature geometries to a new coordinate reference system specified with ``--dst_crs``. The ``fio collect`` command can optionally transform from a coordinate reference system specified with ``--src_crs`` to the default WGS84 GeoJSON CRS. Like collect, ``fio load`` can accept non-WGS84 features, but as it can write files in formats other than GeoJSON, you can optionally specify a ``--dst_crs``. For example, the WGS84 features read from docs/data/test_uk.shp, .. code-block:: console $ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \ > | fio collect --src_crs EPSG:3857 > /tmp/foo.json make a detour through EPSG:3857 (Web Mercator) and are transformed back to WGS84 by fio cat. The following, .. code-block:: console $ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \ > | fio load --src_crs EPSG:3857 --dst_crs EPSG:4326 --driver Shapefile \ > /tmp/foo.shp does the same thing, but for ESRI Shapefile output. Sizing up and simplifying shapes -------------------------------- The following examples use the program ``jq`` and a 25-feature shapefile. You can get the data from from `rmnp.zip `__ or access it in a streaming fashion as shown in the examples below. Counting vertices in a feature collection +++++++++++++++++++++++++++++++++++++++++ The builtin ``vertex_count()`` function, in conjunction with ``fio map``'s ``--raw`` option, prints out the number of vertices in each feature. The default for fio-map is to wrap the result of every evaluated expression in a GeoJSON feature; ``--raw`` disables this. The program ``jq`` provides a nice way of summing the sequence of numbers. .. code-block:: console fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \ | fio map 'vertex_count g' --raw \ | jq -s 'add' 28915 Here's what the RMNP wilderness patrol zones features look like in QGIS. .. image:: img/zones.png Counting vertices after making a simplified buffer ++++++++++++++++++++++++++++++++++++++++++++++++++ One traditional way of simplifying an area of interest is to buffer and simplify. There's no need to use ``jq`` here because ``fio reduce`` prints out a sequence of exactly one feature. The effectiveness of this method depends a bit on the nature of the data, especially the distance between vertices. The total length of the perimeters of all zones is 889 kilometers. .. code-block:: console fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \ | fio map 'length g' --raw \ | jq -s 'add' 889332.0900809917 The mean distance between vertices on the edges of zones is 889332 / 28915, or 30.7 meters. You need to buffer and simplify by this value or more to get a significant reduction in the number of vertices. Choosing 40 as a buffer distance and simplification tolerance results in a shape with 469 vertices. It's a suitable area of interest for applications that require this number to be less than 500. .. code-block:: console fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \ | fio reduce 'unary_union c' \ | fio map 'simplify (buffer g 40) 40' \ | fio map 'vertex_count g' --raw 469 .. image:: img/simplified-buffer.png Counting vertices after dissolving convex hulls of features +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Convex hulls are an easy means of simplification as there are no distance parameters to tweak. The ``--dump-parts`` option of ``fio map`` turns the parts of multi-part features into separate single-part features. This is one of the ways in which fio-map can multiply its inputs, printing out more features than it receives. .. code-block:: console fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \ | fio map 'convex_hull g' --dump-parts \ | fio reduce 'unary_union c' \ | fio map 'vertex_count g' --raw 157 .. image:: img/convex.png Counting vertices after dissolving concave hulls of features ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Convex hulls simplify, but also dilate concave areas of interest. They fill the "bays", so to speak, and this can be undesirable. Concave hulls do a better job at preserving the concave nature of a shape and result in a smaller increase of area. .. code-block:: console fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \ | fio map 'concave_hull g :ratio 0.4' --dump-parts \ | fio reduce 'unary_union c' \ | fio map 'vertex_count g' --raw 301 .. image:: img/concave.png