Debian Codegraph @

doc/examples/vector/TODO

raw
1 GRASS 6 vector TODO
2 ---------------------
3 (Radim Blazek, May 2006)
4
5 This document is summary of my ideas on how vector part of GRASS GIS
6 could be improved.
7
8 It can be that you come to conclusion that vectors in GRASS are bad
9 and it is necessary to start from scratch. In that case I would
10 recommend to leave current library and modules intact and start the
11 new work in parallel (the new modules could start with w.* or v2.*). I
12 was thinking for example about completely new vector library based on
13 OGC standard, using either OGR directly or an abstraction layer and
14 OGR as an option (driver). That does not mean that I prefer simple
15 feature specification over current GRASS implementation, I am not sure
16 which one is better. In any case it would be pity to drop current
17 topological format with all its flexibility. Each approach has
18 advantages and disadvantages. I think that it the best to have in OS
19 GIS all alternatives file/database and topology/simple feature.
20
21
22
23 Historical notes
24 ----------------
25 The current implementation of vectors is based on previous work which
26 was present in GRASS5 (the vector library and modules and DBMI
27 library). We started this work together with David D. Gray in autumn
28 2000 (IIRC) but David had to leave GRASS project soon so that I almost
29 all responsibility for vector development in GRASS6 and its results is
30 mine.
31
32 The current design of GRASS vectors is result of these factor:
33 + very limited resources for development (necessity to use existing
34 free code/libraries/applications whenever possible)
35 + relatively little experience with development of GIS application
36 + respect for certain features of GRASS5 vector model and for existing
37 community which is using it
38 + bad experience with quality of data produced in simple feature based
39 applications (ArcView)
40
41
42 1. Library
43 ----------
44
45 1.1 Geometry
46 ------------
47
48 Keep topology and spatial index in file instead of in memory
49 ------------------------------------------------------------
50 Scalability seems to be currently the biggest problem of GRASS
51 vectors. The geometry of GRASS vectors (coor file) is never loaded
52 whole to the memory. OTOH the support structures (topology and spatial
53 index) are loaded to memory on runtime. It should be possible to use
54 files for topology and spatial index also on runtime and that way
55 decrease the memory occupied by running module (practically to
56 zero). The speed will decrease a bit but not significantly because
57 files are usually cached by system.
58
59 * Update: implemented in r38385 (2009/07) by Markus Metz
60
61 Temporary vector
62 ----------------
63 Analytical modules process data in the output vector (for example
64 v.overlay and v.buffer). Because many lines can be deleted (broken
65 lines for example) and new lines are written at the end of coor file
66 the output file can contain many 'dead' lines (not used space). It
67 would be better to do processing in a temporary vector and copy only
68 alive lines to the output when processing finished. That means
69 implement Vect_open_temporary() which will work like Vect_open_new()
70 but the files will be opened in temporary directory (it should not
71 write to $MAPSET/vector).
72
73
74 Recycle deleted lines
75 ---------------------
76 The space which was occupied by a line in coor file is lost after call
77 to Vect_delete_line(). A list of the free positions be kept and
78 Vect_write_line() should write in that free space if possible instead
79 of to append a new line to the end of the file. There is already empty
80 structure 'recycle' in 'dig_head' where the list could be implemented
81 (without changing 'dig_head' size, to keep binary compatibility).
82
83 * Note: currently wxGUI vector digitizer 'undo' depends on this 'feature'
84
85 Vect_rewrite_line
86 -----------------
87 Implement properly Vect_rewrite_line(). Currently it simply calls
88 Vect_delete_line() and Vect_write_line(). It should be implemented so
89 that if the new size of the line is the same as the old size it will
90 be written in the same place in the coor file where the original line
91 existed.
92
93 * Note: see above
94
95 Remove bounding box from support structures (?)
96 -----------------------------------------------
97 The vector structures (P_line, P_area, P_isle) store bounding box in
98 N,S,E,W,T,B (doubles). Especially in case of element type GV_POINT the
99 bounding box occupies a lot of space (2-3 times more than the point
100 itself). I am not sure if this is really good idea, it is necessary to
101 valutate also how often Vect_line_box() is called and the impact of
102 the necessity to calculate always the box on the fly (when it is not
103 stored in the structure) which can be time consuming for example for
104 areas or long lines.
105
106 * See also https://trac.osgeo.org/grass/ticket/542
107 * Update: implemented in r46898 (2011/07) by Markus Metz
108
109 Switching to update mode
110 ------------------------
111 It would be useful to have a possibility to switch to 'update' mode a
112 map which was opened by Vect_open_old/new() and similarly to switch
113 back to 'normal' mode. Currently it is necessary to call Vect_close()
114 and Vect_open_update().
115
116 Layer names
117 -----------
118 The layers are currently identified only by numbers but it is possible
119 to assign to each layer number a name. The library can read these
120 names but it is not possible to use the name as parameter for
121 modules. It is necessary to write int Vect_get_layer_by_name ( struct
122 Map_info *map, char *name) which will accept both names and numbers
123 and use this function in vector modules. This is also important for
124 OGR interface improvements (see below).
125
126 * Update: implemented in r38548 (2009/07) by Martin Landa
127
128 OGR interface
129 -------------
130 It is important to enable direct access to OGR data sources without
131 v.external and without necessity to store anything in files. The
132 problem of v.external is that topology is stored in file that means it
133 can be wrong when the source is opened next time. It should be
134 relatively easy to call Vect_build_ogr() whenever an OGR vector is
135 opened with level2 (topology) requested and topology will be built on
136 the fly. OGR vectors would be specified by virtual mapset name
137 'OGR'. Each OGR datasource will be equivalent to GRASS vector and each
138 OGR layer will be equivalent to GRASS layer (it is necessary to
139 implement layer names, see above). It would be for example possible to
140 display a shapefile or PostGIS layers directly:
141
142 d.vect map=./shapefiles/@OGR layer=roads # display shapefile ./shapefiles/roads.shp
143 d.vect map=PG:dbname=test@OGR layer=roads # display table roads from database test
144
145 * Update: in progress,
146 see https://trac.osgeo.org/grass/wiki/Grass7/VectorLib/OGRInterface#DirectOGRreadaccess
147
148 Simple feature API and sequential reading
149 -----------------------------------------
150 Most GRASS modules are currently using random access to the data which
151 reflects GRASS format. This works well with GRASS data but it can
152 become very slow or even impossible with OGR data sources because some
153 OGR drivers don't support random access or random access is very
154 slow. Because conversion from topological format to simple feature is
155 very simple and sequential reading of GRASS vectors is not problem it
156 would be desirable to implement in GRASS vector library 'simple
157 feature' API to GRASS vectors and map it directly to OGR API in case
158 of OGR data sources. Then many GRASS modules can be modified to use
159 sequentil reading and simple feature API and that will make more
160 efficient processing of data directly read from OGR data sources.
161
162
163 1.2 Attributes
164 --------------
165
166 In general I found the use of true RDBMS for attributes as a
167 problem. The data are stored in two distinct places (vector files +
168 database) and it makes it difficult to keep them consistent and manage
169 (move, backup). Another problem is random access to the data in RDBMS
170 from an application which is terribly slow (due to communication with
171 server). RDBMS is not bad, bad is the combination of files and
172 RDBMS. I think that either everything must be stored in RDBMS
173 (PostGIS) or nothing. Eric G. Miller (IIRC) was right when he said
174 that data are 'too distant' when RDBMS is used with geometry in file.
175
176 I think that more work should be done on the drivers which are using
177 embedded databases stored in files (SQLite,MySQL,DBF) with scope to
178 reach similar functionality (functions, queries) which are in true
179 RDBMS without penalty of communication with server. It should be also
180 considered the possibility to change the default location of database
181 files to vector directory ($MAPSET/vector/test). That means to keep
182 all the data of one vector in a single directory. It is already
183 possible but it is not the default settings, for example:
184
185 db.connect driver=dbf database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/'
186 db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/db.sqlite'
187 db.connect driver=mesql database='$GISDBASE/$LOCATION_NAME/$MAPSET/vector/$MAP/'
188
189 Implement insert/update cursors
190 -------------------------------
191 GRASS modules are currently sending all data to database drivers as
192 individual SQL insert/update statements. This makes the update process
193 slow (cunstructing and parsing queries) and number precision can be
194 lost. The solution is to implement db_open_insert/update_cursor() and
195 db_insert/update() in database drivers and use these functions in
196 modules. The drivers should then use precompiled statements
197 (e.g. SQLite) or they could update the database directly (DBF).
198
199 Note that it is not necessary to implement these functions in all
200 drivers at the same time. You can implement lib/db/stubs functions
201 which will create SQL statement and send it to db_execute() which is
202 implemented in all drivers until the functions are properly
203 implemented in all drivers.
204
205 SQLite driver
206 -------------
207 Current implementation is very slow with large updates/inserts. I
208 think that it is because all statements are parsed and it should be
209 possible to improve by insert/update cursors (see above).
210
211 DBF driver
212 ----------
213 Add on the fly index for select/update.
214
215 Implement db_copy_table() in drivers
216 ----------------------------------
217 db_copy_table() is implement in client library and it always reads and
218 writes all the data which is slow. It would be better to send this
219 request to the driver (if possible, i.e. input and output driver are
220 the same) which can copy tables much faster. For example true RDBMS
221 can use 'create table new as select * from old' and DBF driver can
222 simply copy files.
223
224 Load drivers as dynamic libraries
225 ---------------------------------
226 Database drivers are implemented as executables which communicate with
227 modules via pipes. This implementation creates some problems with
228 portability (especially on Windows) and it makes communication slow (I
229 am not sure how much). It would be probably desirable to implement
230 drivers as loadable modules (dlopen() and equivalents).
231
232
233 2. Modules
234 ----------
235
236 v.overlay
237 ---------
238 Select only relevant features which will be written to the output if
239 'and,not,nor' operators are used. An inspiration is available in
240 v.select.
241
242 v.pack/v.unpack
243 ---------------
244 Write it. New modules to pack/unpack a vector to/from single file
245 (probably tar). I am not sure about format. Originally I was thinking
246 about ASCII+DBF as it can be read also without GRASS but ASCII and DBF
247 can lose precision and DBF has other limitations. It would be
248 probably better to use copy of 'coor' file and attributes written to
249 SQLite database.
250
251 Update: see
252 https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.pack
253 and
254 https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.unpack
255 by Luca Delucchi
256
257 1/2009: Other suggestions moved to
258 https://trac.osgeo.org/grass/