| 1 |
GRASS 6 vector TODO |
| 2 |
--------------------- |
| 3 |
(Radim Blazek, May 2006) |
| 4 |
|
| 5 |
This document is summary of my ideas on how vector part of GRASS GIS |
| 6 |
could be improved. |
| 7 |
|
| 8 |
It can be that you come to conclusion that vectors in GRASS are bad |
| 9 |
and it is necessary to start from scratch. In that case I would |
| 10 |
recommend to leave current library and modules intact and start the |
| 11 |
new work in parallel (the new modules could start with w.* or v2.*). I |
| 12 |
was thinking for example about completely new vector library based on |
| 13 |
OGC standard, using either OGR directly or an abstraction layer and |
| 14 |
OGR as an option (driver). That does not mean that I prefer simple |
| 15 |
feature specification over current GRASS implementation, I am not sure |
| 16 |
which one is better. In any case it would be pity to drop current |
| 17 |
topological format with all its flexibility. Each approach has |
| 18 |
advantages and disadvantages. I think that it the best to have in OS |
| 19 |
GIS all alternatives file/database and topology/simple feature. |
| 20 |
|
| 21 |
|
| 22 |
|
| 23 |
Historical notes |
| 24 |
---------------- |
| 25 |
The current implementation of vectors is based on previous work which |
| 26 |
was present in GRASS5 (the vector library and modules and DBMI |
| 27 |
library). We started this work together with David D. Gray in autumn |
| 28 |
2000 (IIRC) but David had to leave GRASS project soon so that I almost |
| 29 |
all responsibility for vector development in GRASS6 and its results is |
| 30 |
mine. |
| 31 |
|
| 32 |
The current design of GRASS vectors is result of these factor: |
| 33 |
+ very limited resources for development (necessity to use existing |
| 34 |
free code/libraries/applications whenever possible) |
| 35 |
+ relatively little experience with development of GIS application |
| 36 |
+ respect for certain features of GRASS5 vector model and for existing |
| 37 |
community which is using it |
| 38 |
+ bad experience with quality of data produced in simple feature based |
| 39 |
applications (ArcView) |
| 40 |
|
| 41 |
|
| 42 |
1. Library |
| 43 |
---------- |
| 44 |
|
| 45 |
1.1 Geometry |
| 46 |
------------ |
| 47 |
|
| 48 |
Keep topology and spatial index in file instead of in memory |
| 49 |
------------------------------------------------------------ |
| 50 |
Scalability seems to be currently the biggest problem of GRASS |
| 51 |
vectors. The geometry of GRASS vectors (coor file) is never loaded |
| 52 |
whole to the memory. OTOH the support structures (topology and spatial |
| 53 |
index) are loaded to memory on runtime. It should be possible to use |
| 54 |
files for topology and spatial index also on runtime and that way |
| 55 |
decrease the memory occupied by running module (practically to |
| 56 |
zero). The speed will decrease a bit but not significantly because |
| 57 |
files are usually cached by system. |
| 58 |
|
| 59 |
* Update: implemented in r38385 (2009/07) by Markus Metz |
| 60 |
|
| 61 |
Temporary vector |
| 62 |
---------------- |
| 63 |
Analytical modules process data in the output vector (for example |
| 64 |
v.overlay and v.buffer). Because many lines can be deleted (broken |
| 65 |
lines for example) and new lines are written at the end of coor file |
| 66 |
the output file can contain many 'dead' lines (not used space). It |
| 67 |
would be better to do processing in a temporary vector and copy only |
| 68 |
alive lines to the output when processing finished. That means |
| 69 |
implement Vect_open_temporary() which will work like Vect_open_new() |
| 70 |
but the files will be opened in temporary directory (it should not |
| 71 |
write to $MAPSET/vector). |
| 72 |
|
| 73 |
|
| 74 |
Recycle deleted lines |
| 75 |
--------------------- |
| 76 |
The space which was occupied by a line in coor file is lost after call |
| 77 |
to Vect_delete_line(). A list of the free positions be kept and |
| 78 |
Vect_write_line() should write in that free space if possible instead |
| 79 |
of to append a new line to the end of the file. There is already empty |
| 80 |
structure 'recycle' in 'dig_head' where the list could be implemented |
| 81 |
(without changing 'dig_head' size, to keep binary compatibility). |
| 82 |
|
| 83 |
* Note: currently wxGUI vector digitizer 'undo' depends on this 'feature' |
| 84 |
|
| 85 |
Vect_rewrite_line |
| 86 |
----------------- |
| 87 |
Implement properly Vect_rewrite_line(). Currently it simply calls |
| 88 |
Vect_delete_line() and Vect_write_line(). It should be implemented so |
| 89 |
that if the new size of the line is the same as the old size it will |
| 90 |
be written in the same place in the coor file where the original line |
| 91 |
existed. |
| 92 |
|
| 93 |
* Note: see above |
| 94 |
|
| 95 |
Remove bounding box from support structures (?) |
| 96 |
----------------------------------------------- |
| 97 |
The vector structures (P_line, P_area, P_isle) store bounding box in |
| 98 |
N,S,E,W,T,B (doubles). Especially in case of element type GV_POINT the |
| 99 |
bounding box occupies a lot of space (2-3 times more than the point |
| 100 |
itself). I am not sure if this is really good idea, it is necessary to |
| 101 |
valutate also how often Vect_line_box() is called and the impact of |
| 102 |
the necessity to calculate always the box on the fly (when it is not |
| 103 |
stored in the structure) which can be time consuming for example for |
| 104 |
areas or long lines. |
| 105 |
|
| 106 |
* See also https://trac.osgeo.org/grass/ticket/542 |
| 107 |
* Update: implemented in r46898 (2011/07) by Markus Metz |
| 108 |
|
| 109 |
Switching to update mode |
| 110 |
------------------------ |
| 111 |
It would be useful to have a possibility to switch to 'update' mode a |
| 112 |
map which was opened by Vect_open_old/new() and similarly to switch |
| 113 |
back to 'normal' mode. Currently it is necessary to call Vect_close() |
| 114 |
and Vect_open_update(). |
| 115 |
|
| 116 |
Layer names |
| 117 |
----------- |
| 118 |
The layers are currently identified only by numbers but it is possible |
| 119 |
to assign to each layer number a name. The library can read these |
| 120 |
names but it is not possible to use the name as parameter for |
| 121 |
modules. It is necessary to write int Vect_get_layer_by_name ( struct |
| 122 |
Map_info *map, char *name) which will accept both names and numbers |
| 123 |
and use this function in vector modules. This is also important for |
| 124 |
OGR interface improvements (see below). |
| 125 |
|
| 126 |
* Update: implemented in r38548 (2009/07) by Martin Landa |
| 127 |
|
| 128 |
OGR interface |
| 129 |
------------- |
| 130 |
It is important to enable direct access to OGR data sources without |
| 131 |
v.external and without necessity to store anything in files. The |
| 132 |
problem of v.external is that topology is stored in file that means it |
| 133 |
can be wrong when the source is opened next time. It should be |
| 134 |
relatively easy to call Vect_build_ogr() whenever an OGR vector is |
| 135 |
opened with level2 (topology) requested and topology will be built on |
| 136 |
the fly. OGR vectors would be specified by virtual mapset name |
| 137 |
'OGR'. Each OGR datasource will be equivalent to GRASS vector and each |
| 138 |
OGR layer will be equivalent to GRASS layer (it is necessary to |
| 139 |
implement layer names, see above). It would be for example possible to |
| 140 |
display a shapefile or PostGIS layers directly: |
| 141 |
|
| 142 |
d.vect map=./shapefiles/@OGR layer=roads # display shapefile ./shapefiles/roads.shp |
| 143 |
d.vect map=PG:dbname=test@OGR layer=roads # display table roads from database test |
| 144 |
|
| 145 |
* Update: in progress, |
| 146 |
see https://trac.osgeo.org/grass/wiki/Grass7/VectorLib/OGRInterface#DirectOGRreadaccess |
| 147 |
|
| 148 |
Simple feature API and sequential reading |
| 149 |
----------------------------------------- |
| 150 |
Most GRASS modules are currently using random access to the data which |
| 151 |
reflects GRASS format. This works well with GRASS data but it can |
| 152 |
become very slow or even impossible with OGR data sources because some |
| 153 |
OGR drivers don't support random access or random access is very |
| 154 |
slow. Because conversion from topological format to simple feature is |
| 155 |
very simple and sequential reading of GRASS vectors is not problem it |
| 156 |
would be desirable to implement in GRASS vector library 'simple |
| 157 |
feature' API to GRASS vectors and map it directly to OGR API in case |
| 158 |
of OGR data sources. Then many GRASS modules can be modified to use |
| 159 |
sequentil reading and simple feature API and that will make more |
| 160 |
efficient processing of data directly read from OGR data sources. |
| 161 |
|
| 162 |
|
| 163 |
1.2 Attributes |
| 164 |
-------------- |
| 165 |
|
| 166 |
In general I found the use of true RDBMS for attributes as a |
| 167 |
problem. The data are stored in two distinct places (vector files + |
| 168 |
database) and it makes it difficult to keep them consistent and manage |
| 169 |
(move, backup). Another problem is random access to the data in RDBMS |
| 170 |
from an application which is terribly slow (due to communication with |
| 171 |
server). RDBMS is not bad, bad is the combination of files and |
| 172 |
RDBMS. I think that either everything must be stored in RDBMS |
| 173 |
(PostGIS) or nothing. Eric G. Miller (IIRC) was right when he said |
| 174 |
that data are 'too distant' when RDBMS is used with geometry in file. |
| 175 |
|
| 176 |
I think that more work should be done on the drivers which are using |
| 177 |
embedded databases stored in files (SQLite,MySQL,DBF) with scope to |
| 178 |
reach similar functionality (functions, queries) which are in true |
| 179 |
RDBMS without penalty of communication with server. It should be also |
| 180 |
considered the possibility to change the default location of database |
| 181 |
files to vector directory ($MAPSET/vector/test). That means to keep |
| 182 |
all the data of one vector in a single directory. It is already |
| 183 |
possible but it is not the default settings, for example: |
| 184 |
|
| 185 |
db.connect driver=dbf database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/' |
| 186 |
db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/db.sqlite' |
| 187 |
db.connect driver=mesql database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/' |
| 188 |
|
| 189 |
Implement insert/update cursors |
| 190 |
------------------------------- |
| 191 |
GRASS modules are currently sending all data to database drivers as |
| 192 |
individual SQL insert/update statements. This makes the update process |
| 193 |
slow (cunstructing and parsing queries) and number precision can be |
| 194 |
lost. The solution is to implement db_open_insert/update_cursor() and |
| 195 |
db_insert/update() in database drivers and use these functions in |
| 196 |
modules. The drivers should then use precompiled statements |
| 197 |
(e.g. SQLite) or they could update the database directly (DBF). |
| 198 |
|
| 199 |
Note that it is not necessary to implement these functions in all |
| 200 |
drivers at the same time. You can implement lib/db/stubs functions |
| 201 |
which will create SQL statement and send it to db_execute() which is |
| 202 |
implemented in all drivers until the functions are properly |
| 203 |
implemented in all drivers. |
| 204 |
|
| 205 |
SQLite driver |
| 206 |
------------- |
| 207 |
Current implementation is very slow with large updates/inserts. I |
| 208 |
think that it is because all statements are parsed and it should be |
| 209 |
possible to improve by insert/update cursors (see above). |
| 210 |
|
| 211 |
DBF driver |
| 212 |
---------- |
| 213 |
Add on the fly index for select/update. |
| 214 |
|
| 215 |
Implement db_copy_table() in drivers |
| 216 |
---------------------------------- |
| 217 |
db_copy_table() is implement in client library and it always reads and |
| 218 |
writes all the data which is slow. It would be better to send this |
| 219 |
request to the driver (if possible, i.e. input and output driver are |
| 220 |
the same) which can copy tables much faster. For example true RDBMS |
| 221 |
can use 'create table new as select * from old' and DBF driver can |
| 222 |
simply copy files. |
| 223 |
|
| 224 |
Load drivers as dynamic libraries |
| 225 |
--------------------------------- |
| 226 |
Database drivers are implemented as executables which communicate with |
| 227 |
modules via pipes. This implementation creates some problems with |
| 228 |
portability (especially on Windows) and it makes communication slow (I |
| 229 |
am not sure how much). It would be probably desirable to implement |
| 230 |
drivers as loadable modules (dlopen() and equivalents). |
| 231 |
|
| 232 |
|
| 233 |
2. Modules |
| 234 |
---------- |
| 235 |
|
| 236 |
v.overlay |
| 237 |
--------- |
| 238 |
Select only relevant features which will be written to the output if |
| 239 |
'and,not,nor' operators are used. An inspiration is available in |
| 240 |
v.select. |
| 241 |
|
| 242 |
v.pack/v.unpack |
| 243 |
--------------- |
| 244 |
Write it. New modules to pack/unpack a vector to/from single file |
| 245 |
(probably tar). I am not sure about format. Originally I was thinking |
| 246 |
about ASCII+DBF as it can be read also without GRASS but ASCII and DBF |
| 247 |
can lose precision and DBF has other limitations. It would be |
| 248 |
probably better to use copy of 'coor' file and attributes written to |
| 249 |
SQLite database. |
| 250 |
|
| 251 |
Update: see |
| 252 |
https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.pack |
| 253 |
and |
| 254 |
https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.unpack |
| 255 |
by Luca Delucchi |
| 256 |
|
| 257 |
1/2009: Other suggestions moved to |
| 258 |
https://trac.osgeo.org/grass/ |