Debian Codegraph @

docs/cli.rst

raw
1 Command Line Interface
2 ======================
3
4 Fiona's new command line interface is a program named "fio".
5
6 .. code-block:: console
7
8 Usage: fio [OPTIONS] COMMAND [ARGS]...
9
10 Fiona command line interface.
11
12 Options:
13 -v, --verbose Increase verbosity.
14 -q, --quiet Decrease verbosity.
15 --aws-profile TEXT Select a profile from the AWS credentials file
16 --aws-no-sign-requests Make requests anonymously
17 --aws-requester-pays Requester pays data transfer costs
18 --version Show the version and exit.
19 --gdal-version Show the version and exit.
20 --python-version Show the version and exit.
21 --help Show this message and exit.
22
23 Commands:
24 bounds Print the extent of GeoJSON objects
25 calc Calculate GeoJSON property by Python expression
26 cat Concatenate and print the features of datasets
27 collect Collect a sequence of features.
28 distrib Distribute features from a collection.
29 dump Dump a dataset to GeoJSON.
30 env Print information about the fio environment.
31 filter Evaluate pipeline expressions to filter GeoJSON features.
32 info Print information about a dataset.
33 insp Open a dataset and start an interpreter.
34 load Load GeoJSON to a dataset in another format.
35 ls List layers in a datasource.
36 map Map a pipeline expression over GeoJSON features.
37 reduce Reduce a stream of GeoJSON features to one value.
38 rm Remove a datasource or an individual layer.
39
40 It is developed using the ``click`` package.
41
42 bounds
43 ------
44
45 The bounds command reads LF or RS-delimited GeoJSON texts, either features or
46 collections, from stdin and prints their bounds with or without other data to
47 stdout.
48
49 With no options, it works like this:
50
51 .. code-block:: console
52
53 $ fio cat docs/data/test_uk.shp | head -n 1 \
54 > | fio bounds
55 [0.735, 51.357216, 0.947778, 51.444717]
56
57 Using ``--with-id`` gives you
58
59 .. code-block:: console
60
61 $ fio cat docs/data/test_uk.shp | head -n 1 \
62 > | fio bounds --with-id
63 {"id": "0", "bbox": [0.735, 51.357216, 0.947778, 51.444717]}
64
65 calc
66 ----
67
68 The calc command creates a new property on GeoJSON features using the
69 specified expression.
70
71 The expression is evaluated in a restricted namespace containing 4 functions
72 (`sum`, `pow`, `min`, `max`), the `math` module, the shapely `shape` function,
73 type conversions (`bool`, `int`, `str`, `len`, `float`), and an object `f`
74 representing the feature to be evaluated. This `f` object allows access in
75 javascript-style dot notation for convenience.
76
77 The expression will be evaluated for each feature and its return value will be
78 added to the properties as the specified property_name. Existing properties
79 will not be overwritten by default (an `Exception` is raised).
80
81 .. code-block:: console
82
83 $ fio cat data.shp | fio calc sumAB "f.properties.A + f.properties.B"
84
85 .. note::
86
87 ``fio calc`` requires installation of the "calc" set of extra requirements
88 that will be installed by ``pip install fiona[calc]``.
89
90 cat
91 ---
92
93 The cat command concatenates the features of one or more datasets and prints
94 them as a `JSON text sequence
95 <https://datatracker.ietf.org/doc/html/rfc7464>`__ of features.
96 In other words: GeoJSON feature objects, possibly pretty printed, optionally
97 separated by ASCII RS (\x1e) chars using `--rs`.
98
99 The output of ``fio cat`` can be piped to ``fio load`` to create new
100 concatenated datasets.
101
102 .. code-block:: console
103
104 $ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \
105 > | fio load /tmp/double.shp --driver Shapefile
106 $ fio info /tmp/double.shp --count
107 96
108 $ fio info docs/data/test_uk.shp --count
109 48
110
111 The cat command provides optional methods to filter data, which are
112 different to the ``fio filter`` tool.
113 A bounding box ``--bbox w,s,e,n`` tests for a spatial intersection with
114 the geometries. An attribute filter ``--where TEXT`` can use
115 an `SQL WHERE clause <https://gdal.org/user/ogr_sql_dialect.html#where>`__.
116 If more than one datasets is passed to ``fio cat``, the attributes used
117 in the WHERE clause must be valid for each dataset.
118
119 collect
120 -------
121
122 The collect command takes a JSON text sequence of GeoJSON feature objects, such
123 as the output of ``fio cat`` and writes a GeoJSON feature collection.
124
125 .. code-block:: console
126
127 $ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \
128 > | fio collect > /tmp/collected.json
129 $ fio info /tmp/collected.json --count
130 96
131
132 distrib
133 -------
134
135 The inverse of fio-collect, fio-distrib takes a GeoJSON feature collection
136 and writes a JSON text sequence of GeoJSON feature objects.
137
138 .. code-block:: console
139
140 $ fio info --count tests/data/coutwildrnp.shp
141 67
142 $ fio cat tests/data/coutwildrnp.shp | fio collect | fio distrib | wc -l
143 67
144
145 dump
146 ----
147
148 The dump command reads a vector dataset and writes a GeoJSON feature collection
149 to stdout. Its output can be piped to ``fio load`` (see below).
150
151 .. code-block:: console
152
153 $ fio dump docs/data/test_uk.shp --indent 2 --precision 2 | head
154 {
155 "features": [
156 {
157 "geometry": {
158 "coordinates": [
159 [
160 [
161 0.9,
162 51.36
163 ],
164
165 You can optionally dump out JSON text sequences using ``--x-json-seq``. Since
166 version 1.4.0, ``fio cat`` is the better tool for generating sequences.
167
168 .. code-block:: console
169
170 $ fio dump docs/data/test_uk.shp --precision 2 --x-json-seq | head -n 2
171 {"geometry": {"coordinates": [[[0.9, 51.36], [0.89, 51.36], [0.79, 51.37], [0.78, 51.37], [0.77, 51.38], [0.76, 51.38], [0.75, 51.39], [0.74, 51.4], [0.73, 51.41], [0.74, 51.43], [0.75, 51.44], [0.76, 51.44], [0.79, 51.44], [0.89, 51.42], [0.9, 51.42], [0.91, 51.42], [0.93, 51.4], [0.94, 51.39], [0.94, 51.38], [0.95, 51.38], [0.95, 51.37], [0.95, 51.37], [0.94, 51.37], [0.9, 51.36], [0.9, 51.36]]], "type": "Polygon"}, "id": "0", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}
172 {"geometry": {"coordinates": [[[-4.66, 51.16], [-4.67, 51.16], [-4.67, 51.16], [-4.67, 51.17], [-4.67, 51.19], [-4.67, 51.19], [-4.67, 51.2], [-4.66, 51.2], [-4.66, 51.19], [-4.65, 51.16], [-4.65, 51.16], [-4.65, 51.16], [-4.66, 51.16]]], "type": "Polygon"}, "id": "1", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}
173
174
175 info
176 ----
177
178 The info command prints information about a dataset as a JSON object.
179
180 .. code-block:: console
181
182 $ fio info docs/data/test_uk.shp --indent 2
183 {
184 "count": 48,
185 "crs": "+datum=WGS84 +no_defs +proj=longlat",
186 "driver": "ESRI Shapefile",
187 "bounds": [
188 -8.621389,
189 49.911659,
190 1.749444,
191 60.844444
192 ],
193 "schema": {
194 "geometry": "Polygon",
195 "properties": {
196 "CAT": "float:16",
197 "FIPS_CNTRY": "str:80",
198 "CNTRY_NAME": "str:80",
199 "AREA": "float:15.2",
200 "POP_CNTRY": "float:15.2"
201 }
202 }
203 }
204
205 You can process this JSON using, e.g.,
206 `underscore-cli <https://github.com/ddopson/underscore-cli>`__.
207
208 .. code-block:: console
209
210 $ fio info docs/data/test_uk.shp | underscore extract count
211 48
212
213 You can also optionally get single info items as plain text (not JSON)
214 strings
215
216 .. code-block:: console
217
218 $ fio info docs/data/test_uk.shp --count
219 48
220 $ fio info docs/data/test_uk.shp --bounds
221 -8.621389 49.911659 1.749444 60.844444
222
223 load
224 ----
225
226 The load command reads GeoJSON features from stdin and writes them to a vector
227 dataset using another format.
228
229 .. code-block:: console
230
231 $ fio dump docs/data/test_uk.shp \
232 > | fio load /tmp/test.shp --driver Shapefile
233
234 This command also supports GeoJSON text sequences. RS-separated sequences will
235 be detected. If you want to load LF-separated sequences, you must specify
236 ``--x-json-seq``.
237
238 .. code-block:: console
239
240 $ fio cat docs/data/test_uk.shp | fio load /tmp/foo.shp --driver Shapefile
241 $ fio info /tmp/foo.shp --indent 2
242 {
243 "count": 48,
244 "crs": "+datum=WGS84 +no_defs +proj=longlat",
245 "driver": "ESRI Shapefile",
246 "bounds": [
247 -8.621389,
248 49.911659,
249 1.749444,
250 60.844444
251 ],
252 "schema": {
253 "geometry": "Polygon",
254 "properties": {
255 "AREA": "float:24.15",
256 "CNTRY_NAME": "str:80",
257 "POP_CNTRY": "float:24.15",
258 "FIPS_CNTRY": "str:80",
259 "CAT": "float:24.15"
260 }
261 }
262 }
263
264 The underscore-cli process command is another way of turning a GeoJSON feature
265 collection into a feature sequence.
266
267 .. code-block:: console
268
269 $ fio dump docs/data/test_uk.shp \
270 > | underscore process \
271 > 'each(data.features,function(o){console.log(JSON.stringify(o))})' \
272 > | fio load /tmp/test-seq.shp --x-json-seq --driver Shapefile
273
274
275 filter
276 ------
277
278 For each feature read from stdin, filter evaluates a pipeline of one or
279 more steps described using methods from the Shapely library in Lisp-like
280 expressions. If the pipeline expression evaluates to True, the feature passes
281 through the filter. Otherwise the feature does not pass.
282
283 For example, this pipeline expression
284
285 .. code-block:: console
286
287 $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
288 | fio filter '< (distance g (Point -109.0 38.5)) 100'
289
290 lets through all features that are less than 100 meters from the given point
291 and filters out all other features.
292
293 *New in version 1.10*: these parenthesized list expressions.
294
295 The older style Python expressions like
296
297 .. code-block::
298
299 'f.properties.area > 1000.0'
300
301 are deprecated and will not be supported in version 2.0.
302
303 Note this tool is different from ``fio cat --where TEXT ...``, which provides
304 SQL WHERE clause filtering of feature attributes.
305
306 .. note::
307
308 ``fio filter`` requires installation of the "calc" set of extra requirements
309 that will be installed by ``pip install fiona[calc]``.
310
311 map
312 ---
313
314 For each feature read from stdin, ``fio map`` applies a transformation pipeline and
315 writes a copy of the feature, containing the modified geometry, to stdout. For
316 example, polygonal features can be roughly "cleaned" by using a ``buffer g 0``
317 pipeline.
318
319 .. code-block:: console
320
321 $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
322 | fio map 'buffer g 0'
323
324 *New in version 1.10*.
325
326 .. note::
327
328 ``fio map`` requires installation of the "calc" set of extra requirements
329 that will be installed by ``pip install fiona[calc]``.
330
331
332 reduce
333 ------
334
335 Given a sequence of GeoJSON features (RS-delimited or not) on stdin this prints
336 a single value using a provided transformation pipeline. The set of geometries
337 of the input features in the context of these expressions is named ``c``.
338
339 For example, the pipeline expression
340
341 .. code-block:: console
342
343 $ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
344 | fio reduce 'unary_union c'
345
346 dissolves the geometries of input features.
347
348 *New in version 1.10*.
349
350 .. note::
351
352 ``fio reduce`` requires installation of the "calc" set of extra requirements
353 that will be installed by ``pip install fiona[calc]``.
354
355 rm
356 --
357
358 The rm command deletes an entire datasource or a single layer in a multi-layer
359 datasource. If the datasource is composed of multiple files (e.g. an ESRI
360 Shapefile) all of the files will be removed.
361
362 .. code-block:: console
363
364 $ fio rm countries.shp
365 $ fio rm --layer forests land_cover.gpkg
366
367 Expressions and functions
368 -------------------------
369
370 ``fio filter``, ``fio map``, and ``fio reduce`` expressions take the form of
371 parenthesized lists that may contain other expressions. The first item in a
372 list is the name of a function or method, or an expression that evaluates to a
373 function. The second item is the function's first argument or the object to
374 which the method is bound. The remaining list items are the positional and
375 keyword arguments for the named function or method. The list of functions and
376 callables available in an expression includes:
377
378 * Python operators such as ``+``, ``/``, and ``<=``
379 * Python builtins such as ``dict``, ``list``, and ``map``
380 * All public functions from itertools, e.g. ``islice``, and ``repeat``
381 * All functions importable from Shapely 2.0, e.g. ``Point``, and ``unary_union``
382 * All methods of Shapely geometry classes
383 * Functions specific to Fiona
384
385 Expressions are evaluated by ``fiona.features.snuggs.eval()``. Let's look at
386 some examples using that function.
387
388 .. note::
389
390 The outer parentheses are not optional within ``snuggs.eval()``.
391
392 .. note::
393
394 ``snuggs.eval()`` does not use Python's builtin ``eval()`` but isn't intended
395 to be a secure computing environment. Expressions which access the
396 computer's filesystem and create new processes are possible.
397
398 Builtin Python functions
399 ------------------------
400
401 ``bool()``
402
403 .. code-block:: python
404
405 >>> snuggs.eval('(bool 0)')
406 False
407
408 ``range()``
409
410 .. code-block:: python
411
412 >>> snuggs.eval('(range 1 4)')
413 range(1, 4)
414
415 ``list()``
416
417 .. code-block:: python
418
419 >>> snuggs.eval('(list (range 1 4))')
420 [1, 2, 3]
421
422 Values can be bound to names for use in expressions.
423
424 .. code-block:: python
425
426 >>> snuggs.eval('(list (range start stop))', start=0, stop=5)
427 [0, 1, 2, 3, 4]
428
429 Itertools functions
430 -------------------
431
432 Here's an example of using ``itertools.repeat()``.
433
434 .. code-block:: python
435
436 >>> snuggs.eval('(list (repeat "*" times))', times=6)
437 ['*', '*', '*', '*', '*', '*']
438
439 Shapely functions
440 -----------------
441
442 Here's an expression that evaluates to a Shapely Point instance.
443
444 .. code-block:: python
445
446 >>> snuggs.eval('(Point 0 0)')
447 <POINT (0 0)>
448
449 The expression below evaluates to a MultiPoint instance.
450
451 .. code-block:: python
452
453 >>> snuggs.eval('(union (Point 0 0) (Point 1 1))')
454 <MULTIPOINT (0 0, 1 1)>
455
456 Functions specific to fiona
457 ---------------------------
458
459 The fio CLI introduces four new functions not available in Python's standard
460 library, or Shapely: ``collect()``, ``dump()``, ``identity()``, and
461 ``vertex_count()``.
462
463 The ``collect()`` function turns a list of geometries into a geometry
464 collection and ``dump()`` does the inverse, turning a geometry collection into
465 a sequence of geometries.
466
467 .. code-block:: python
468
469 >>> snuggs.eval('(collect (Point 0 0) (Point 1 1))')
470 <GEOMETRYCOLLECTION (POINT (0 0), POINT (1 1))>
471 >>> snuggs.eval('(list (dump (collect (Point 0 0) (Point 1 1))))')
472 [<POINT (0 0)>, <POINT (1 1)>]
473
474 The ``identity()`` function returns its single argument.
475
476 .. code-block:: python
477
478 >>> snuggs.eval('(identity 42)')
479 42
480
481 To count the number of vertices in a geometry, use ``vertex_count()``.
482
483 .. code-block:: python
484
485 >>> snuggs.eval('(vertex_count (Point 0 0))')
486 1
487
488 The ``area()``, ``buffer()``, ``distance()``, ``length()``, ``simplify()``, and
489 ``set_precision()`` functions shadow, or override, functions from the shapely
490 module. They automatically reproject geometry objects from their natural
491 coordinate reference system (CRS) of ``OGC:CRS84`` to ``EPSG:6933`` so that the
492 shapes can be measured or modified using meters as units.
493
494 ``buffer()`` dilates (or erodes) a given geometry, with coordinates in decimal
495 longitude and latitude degrees, by a given distance in meters.
496
497 .. code-block:: python
498
499 >>> snuggs.eval('(buffer (Point 0 0) :distance 100)')
500 <POLYGON ((0.001 0, 0.001 0, 0.001 0, 0.001 0, 0.001 -0.001, 0.001 -0.001, 0...>
501
502 The ``area()`` and ``length()`` of this polygon have units of square meter and
503 meter.
504
505 .. code-block:: python
506
507 >>> snuggs.eval('(area (buffer (Point 0 0) :distance 100))')
508 31214.451487413342
509 >>> snuggs.eval('(length (buffer (Point 0 0) :distance 100))')
510 627.3096977558143
511
512 The ``distance()`` between two geometries is in meters.
513
514 .. code-block:: python
515
516 >>> snuggs.eval('(distance (Point 0 0) (Point 0.1 0.1))')
517 15995.164946207413
518
519 A geometry can be simplified to a tolerance value in meters using
520 ``simplify()``. There are more examples of this function later in this
521 document.
522
523 .. code-block:: python
524
525 >>> snuggs.eval('(simplify (buffer (Point 0 0) :distance 100) :tolerance 100)')
526 <POLYGON ((0.001 0, 0 -0.001, -0.001 0, 0 0.001, 0.001 0))>
527
528 The ``set_precision()`` function snaps a geometry to a fixed precision grid
529 with a size in meters.
530
531 .. code-block:: python
532
533 >>> snuggs.eval('(set_precision (Point 0.001 0.001) :grid_size 500)')
534 <POINT (0 0)>
535
536 Feature and geometry context for expressions
537 --------------------------------------------
538
539 ``fio filter`` and ``fio map`` evaluate expressions in the context of a GeoJSON
540 feature and its geometry attribute. These are named ``f`` and ``g``. For example,
541 here is an expression that tests whether the input feature is within 62.5
542 kilometers of the given point.
543
544 .. code-block:: lisp
545
546 < (distance g (Point 4 43)) 62.5E3
547
548 ``fio reduce`` evaluates expressions in the context of the sequence of all input
549 geometries, named ``c``. For example, this expression dissolves input
550 geometries using Shapely's ``unary_union``.
551
552 .. code-block:: lisp
553
554 unary_union c
555
556 Coordinate Reference System Transformations
557 -------------------------------------------
558
559 The ``fio cat`` command can optionally transform feature geometries to a new
560 coordinate reference system specified with ``--dst_crs``. The ``fio collect``
561 command can optionally transform from a coordinate reference system specified
562 with ``--src_crs`` to the default WGS84 GeoJSON CRS. Like collect, ``fio load``
563 can accept non-WGS84 features, but as it can write files in formats other than
564 GeoJSON, you can optionally specify a ``--dst_crs``. For example, the WGS84
565 features read from docs/data/test_uk.shp,
566
567 .. code-block:: console
568
569 $ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \
570 > | fio collect --src_crs EPSG:3857 > /tmp/foo.json
571
572 make a detour through EPSG:3857 (Web Mercator) and are transformed back to WGS84
573 by fio cat. The following,
574
575 .. code-block:: console
576
577 $ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \
578 > | fio load --src_crs EPSG:3857 --dst_crs EPSG:4326 --driver Shapefile \
579 > /tmp/foo.shp
580
581 does the same thing, but for ESRI Shapefile output.
582
583 Sizing up and simplifying shapes
584 --------------------------------
585
586 The following examples use the program ``jq`` and a 25-feature shapefile. You
587 can get the data from from `rmnp.zip
588 <https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip>`__ or access it in
589 a streaming fashion as shown in the examples below.
590
591 Counting vertices in a feature collection
592 +++++++++++++++++++++++++++++++++++++++++
593
594 The builtin ``vertex_count()`` function, in conjunction with ``fio map``'s
595 ``--raw`` option, prints out the number of vertices in each feature. The
596 default for fio-map is to wrap the result of every evaluated expression in
597 a GeoJSON feature; ``--raw`` disables this. The program ``jq`` provides a nice
598 way of summing the sequence of numbers.
599
600 .. code-block:: console
601
602 fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
603 | fio map 'vertex_count g' --raw \
604 | jq -s 'add'
605 28915
606
607 Here's what the RMNP wilderness patrol zones features look like in QGIS.
608
609 .. image:: img/zones.png
610
611 Counting vertices after making a simplified buffer
612 ++++++++++++++++++++++++++++++++++++++++++++++++++
613
614 One traditional way of simplifying an area of interest is to buffer and
615 simplify. There's no need to use ``jq`` here because ``fio reduce`` prints out
616 a sequence of exactly one feature. The effectiveness of this method depends
617 a bit on the nature of the data, especially the distance between vertices. The
618 total length of the perimeters of all zones is 889 kilometers.
619
620 .. code-block:: console
621
622 fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
623 | fio map 'length g' --raw \
624 | jq -s 'add'
625 889332.0900809917
626
627 The mean distance between vertices on the edges of zones is 889332 / 28915, or
628 30.7 meters. You need to buffer and simplify by this value or more to get
629 a significant reduction in the number of vertices. Choosing 40 as a buffer
630 distance and simplification tolerance results in a shape with 469 vertices.
631 It's a suitable area of interest for applications that require this number to
632 be less than 500.
633
634 .. code-block:: console
635
636 fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
637 | fio reduce 'unary_union c' \
638 | fio map 'simplify (buffer g 40) 40' \
639 | fio map 'vertex_count g' --raw
640 469
641
642 .. image:: img/simplified-buffer.png
643
644 Counting vertices after dissolving convex hulls of features
645 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
646
647 Convex hulls are an easy means of simplification as there are no distance
648 parameters to tweak. The ``--dump-parts`` option of ``fio map`` turns the parts of
649 multi-part features into separate single-part features. This is one of the ways
650 in which fio-map can multiply its inputs, printing out more features than it
651 receives.
652
653 .. code-block:: console
654
655 fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
656 | fio map 'convex_hull g' --dump-parts \
657 | fio reduce 'unary_union c' \
658 | fio map 'vertex_count g' --raw
659 157
660
661 .. image:: img/convex.png
662
663 Counting vertices after dissolving concave hulls of features
664 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
665
666 Convex hulls simplify, but also dilate concave areas of interest. They fill the
667 "bays", so to speak, and this can be undesirable. Concave hulls do a better job
668 at preserving the concave nature of a shape and result in a smaller increase of
669 area.
670
671 .. code-block:: console
672
673 fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
674 | fio map 'concave_hull g :ratio 0.4' --dump-parts \
675 | fio reduce 'unary_union c' \
676 | fio map 'vertex_count g' --raw
677 301
678
679 .. image:: img/concave.png
680