| 1 |
==================================== |
| 2 |
``LayerMapping`` data import utility |
| 3 |
==================================== |
| 4 |
|
| 5 |
.. module:: django.contrib.gis.utils.layermapping |
| 6 |
:synopsis: Spatial data import utility for GeoDjango models. |
| 7 |
|
| 8 |
.. currentmodule:: django.contrib.gis.utils |
| 9 |
|
| 10 |
The :class:`LayerMapping` class provides a way to map the contents of |
| 11 |
vector spatial data files (e.g. shapefiles) into GeoDjango models. |
| 12 |
|
| 13 |
This utility grew out of the author's personal needs to eliminate |
| 14 |
the code repetition that went into pulling geometries and fields out of |
| 15 |
a vector layer, converting to another coordinate system (e.g. WGS84), and |
| 16 |
then inserting into a GeoDjango model. |
| 17 |
|
| 18 |
.. note:: |
| 19 |
|
| 20 |
Use of :class:`LayerMapping` requires GDAL. |
| 21 |
|
| 22 |
.. warning :: |
| 23 |
|
| 24 |
GIS data sources, like shapefiles, may be very large. If you find |
| 25 |
that :class:`LayerMapping` is using too much memory, set |
| 26 |
:setting:`DEBUG` to ``False`` in your settings. When :setting:`DEBUG` |
| 27 |
is set to ``True``, Django :ref:`automatically logs <faq-see-raw-sql-queries>` |
| 28 |
*every* SQL query -- and when SQL statements contain geometries, this may |
| 29 |
consume more memory than is typical. |
| 30 |
|
| 31 |
Example |
| 32 |
======= |
| 33 |
|
| 34 |
#. You need a GDAL-supported data source, like a shapefile (here we're using |
| 35 |
a simple polygon shapefile, ``test_poly.shp``, with three features): |
| 36 |
|
| 37 |
.. code-block:: pycon |
| 38 |
|
| 39 |
>>> from django.contrib.gis.gdal import DataSource |
| 40 |
>>> ds = DataSource("test_poly.shp") |
| 41 |
>>> layer = ds[0] |
| 42 |
>>> print(layer.fields) # Exploring the fields in the layer, we only want the 'str' field. |
| 43 |
['float', 'int', 'str'] |
| 44 |
>>> print(len(layer)) # getting the number of features in the layer (should be 3) |
| 45 |
3 |
| 46 |
>>> print(layer.geom_type) # Should be 'Polygon' |
| 47 |
Polygon |
| 48 |
>>> print(layer.srs) # WGS84 in WKT |
| 49 |
GEOGCS["GCS_WGS_1984", |
| 50 |
DATUM["WGS_1984", |
| 51 |
SPHEROID["WGS_1984",6378137,298.257223563]], |
| 52 |
PRIMEM["Greenwich",0], |
| 53 |
UNIT["Degree",0.017453292519943295]] |
| 54 |
|
| 55 |
#. Now we define our corresponding Django model (make sure to use :djadmin:`migrate`):: |
| 56 |
|
| 57 |
from django.contrib.gis.db import models |
| 58 |
|
| 59 |
|
| 60 |
class TestGeo(models.Model): |
| 61 |
name = models.CharField(max_length=25) # corresponds to the 'str' field |
| 62 |
poly = models.PolygonField(srid=4269) # we want our model in a different SRID |
| 63 |
|
| 64 |
def __str__(self): |
| 65 |
return "Name: %s" % self.name |
| 66 |
|
| 67 |
#. Use :class:`LayerMapping` to extract all the features and place them in the |
| 68 |
database: |
| 69 |
|
| 70 |
.. code-block:: pycon |
| 71 |
|
| 72 |
>>> from django.contrib.gis.utils import LayerMapping |
| 73 |
>>> from geoapp.models import TestGeo |
| 74 |
>>> mapping = { |
| 75 |
... "name": "str", # The 'name' model field maps to the 'str' layer field. |
| 76 |
... "poly": "POLYGON", # For geometry fields use OGC name. |
| 77 |
... } # The mapping is a dictionary |
| 78 |
>>> lm = LayerMapping(TestGeo, "test_poly.shp", mapping) |
| 79 |
>>> lm.save(verbose=True) # Save the layermap, imports the data. |
| 80 |
Saved: Name: 1 |
| 81 |
Saved: Name: 2 |
| 82 |
Saved: Name: 3 |
| 83 |
|
| 84 |
Here, :class:`LayerMapping` transformed the three geometries from the shapefile |
| 85 |
in their original spatial reference system (WGS84) to the spatial reference |
| 86 |
system of the GeoDjango model (NAD83). If no spatial reference system is |
| 87 |
defined for the layer, use the ``source_srs`` keyword with a |
| 88 |
:class:`~django.contrib.gis.gdal.SpatialReference` object to specify one. |
| 89 |
|
| 90 |
``LayerMapping`` API |
| 91 |
==================== |
| 92 |
|
| 93 |
.. class:: LayerMapping(model, data_source, mapping, layer=0, source_srs=None, encoding=None, transaction_mode='commit_on_success', transform=True, unique=True, using='default') |
| 94 |
|
| 95 |
The following are the arguments and keywords that may be used during |
| 96 |
instantiation of ``LayerMapping`` objects. |
| 97 |
|
| 98 |
================= ========================================================= |
| 99 |
Argument Description |
| 100 |
================= ========================================================= |
| 101 |
``model`` The geographic model, *not* an instance. |
| 102 |
|
| 103 |
``data_source`` The path to the OGR-supported data source file |
| 104 |
(e.g., a shapefile). Also accepts |
| 105 |
:class:`django.contrib.gis.gdal.DataSource` instances. |
| 106 |
|
| 107 |
``mapping`` A dictionary: keys are strings corresponding to |
| 108 |
the model field, and values correspond to |
| 109 |
string field names for the OGR feature, or if the |
| 110 |
model field is a geographic then it should |
| 111 |
correspond to the OGR geometry type, |
| 112 |
e.g., ``'POINT'``, ``'LINESTRING'``, ``'POLYGON'``. |
| 113 |
================= ========================================================= |
| 114 |
|
| 115 |
===================== ===================================================== |
| 116 |
Keyword Arguments |
| 117 |
===================== ===================================================== |
| 118 |
``layer`` The index of the layer to use from the Data Source |
| 119 |
(defaults to 0) |
| 120 |
|
| 121 |
``source_srs`` Use this to specify the source SRS manually (for |
| 122 |
example, some shapefiles don't come with a ``'.prj'`` |
| 123 |
file). An integer SRID, WKT or PROJ strings, and |
| 124 |
:class:`django.contrib.gis.gdal.SpatialReference` |
| 125 |
objects are accepted. |
| 126 |
|
| 127 |
``encoding`` Specifies the character set encoding of the strings |
| 128 |
in the OGR data source. For example, ``'latin-1'``, |
| 129 |
``'utf-8'``, and ``'cp437'`` are all valid encoding |
| 130 |
parameters. |
| 131 |
|
| 132 |
``transaction_mode`` May be ``'commit_on_success'`` (default) or |
| 133 |
``'autocommit'``. |
| 134 |
|
| 135 |
``transform`` Setting this to False will disable coordinate |
| 136 |
transformations. In other words, geometries will |
| 137 |
be inserted into the database unmodified from their |
| 138 |
original state in the data source. |
| 139 |
|
| 140 |
``unique`` Setting this to the name, or a tuple of names, |
| 141 |
from the given model will create models unique |
| 142 |
only to the given name(s). Geometries from |
| 143 |
each feature will be added into the collection |
| 144 |
associated with the unique model. Forces |
| 145 |
the transaction mode to be ``'autocommit'``. |
| 146 |
|
| 147 |
``using`` Sets the database to use when importing spatial data. |
| 148 |
Default is ``'default'``. |
| 149 |
===================== ===================================================== |
| 150 |
|
| 151 |
``save()`` Keyword Arguments |
| 152 |
---------------------------- |
| 153 |
|
| 154 |
.. method:: LayerMapping.save(verbose=False, fid_range=False, step=False, progress=False, silent=False, stream=sys.stdout, strict=False) |
| 155 |
|
| 156 |
The ``save()`` method also accepts keywords. These keywords are |
| 157 |
used for controlling output logging, error handling, and for importing |
| 158 |
specific feature ranges. |
| 159 |
|
| 160 |
=========================== ================================================= |
| 161 |
Save Keyword Arguments Description |
| 162 |
=========================== ================================================= |
| 163 |
``fid_range`` May be set with a slice or tuple of |
| 164 |
(begin, end) feature ID's to map from |
| 165 |
the data source. In other words, this |
| 166 |
keyword enables the user to selectively |
| 167 |
import a subset range of features in the |
| 168 |
geographic data source. |
| 169 |
|
| 170 |
``progress`` When this keyword is set, status information |
| 171 |
will be printed giving the number of features |
| 172 |
processed and successfully saved. By default, |
| 173 |
progress information will be printed every 1000 |
| 174 |
features processed, however, this default may |
| 175 |
be overridden by setting this keyword with an |
| 176 |
integer for the desired interval. |
| 177 |
|
| 178 |
``silent`` By default, non-fatal error notifications are |
| 179 |
printed to ``sys.stdout``, but this keyword may |
| 180 |
be set to disable these notifications. |
| 181 |
|
| 182 |
``step`` If set with an integer, transactions will |
| 183 |
occur at every step interval. For example, if |
| 184 |
``step=1000``, a commit would occur after the |
| 185 |
1,000th feature, the 2,000th feature etc. |
| 186 |
|
| 187 |
|
| 188 |
``stream`` Status information will be written to this file |
| 189 |
handle. Defaults to using ``sys.stdout``, but |
| 190 |
any object with a ``write`` method is supported. |
| 191 |
|
| 192 |
``strict`` Execution of the model mapping will cease upon |
| 193 |
the first error encountered. The default value |
| 194 |
(``False``) |
| 195 |
behavior is to attempt to continue. |
| 196 |
|
| 197 |
``verbose`` If set, information will be printed |
| 198 |
subsequent to each model save |
| 199 |
executed on the database. |
| 200 |
=========================== ================================================= |
| 201 |
|
| 202 |
Troubleshooting |
| 203 |
=============== |
| 204 |
|
| 205 |
Running out of memory |
| 206 |
--------------------- |
| 207 |
|
| 208 |
As noted in the warning at the top of this section, Django stores all SQL |
| 209 |
queries when ``DEBUG=True``. Set ``DEBUG=False`` in your settings, and this |
| 210 |
should stop excessive memory use when running ``LayerMapping`` scripts. |
| 211 |
|
| 212 |
MySQL: ``max_allowed_packet`` error |
| 213 |
----------------------------------- |
| 214 |
|
| 215 |
If you encounter the following error when using ``LayerMapping`` and MySQL: |
| 216 |
|
| 217 |
.. code-block:: pytb |
| 218 |
|
| 219 |
OperationalError: (1153, "Got a packet bigger than 'max_allowed_packet' bytes") |
| 220 |
|
| 221 |
Then the solution is to increase the value of the ``max_allowed_packet`` |
| 222 |
setting in your MySQL configuration. For example, the default value may |
| 223 |
be something low like one megabyte -- the setting may be modified in MySQL's |
| 224 |
configuration file (``my.cnf``) in the ``[mysqld]`` section: |
| 225 |
|
| 226 |
.. code-block:: ini |
| 227 |
|
| 228 |
max_allowed_packet = 10M |