Debian Codegraph @

docs/design.rst

raw
1 Thoughts about redesign, well actually "design" since originally there
2 were none, of datalad crawl.
3
4 Global portion of the config
5 ============================
6
7 ::
8
9 [dataset]
10 path =
11 description =
12 exec =
13
14 Data providers
15 ==============
16
17 `crawl` command collects data present possibly across
18 different remote data providers (regular HTTP websites, AWS S3
19 buckets, etc) and then consolidates access to them within a single
20 git-annex'ed repository. `crawl` should also keep track of
21 status/versions of the files, so in case of the updates (changes,
22 removals, etc) on remote sites, git-annex repository could be
23 correspondingly updated.
24
25 Common config specs::
26
27 [provider:XXX]
28 type = (web|s3http|merge|git-annex) # default to web
29 branch = master # default to master
30 commit_to_git = # regexps of file names to commit directly to git
31 ignore = # files to ignore entirely
32 drop = False # either to drop the load upon 'completion'
33 # some sanity checks
34 check_entries_limit = -1 # no limit by default
35
36
37 (To be) Supported data providers
38 --------------------------------
39
40 Web
41 ~~~
42
43 In many usecases data are hosted on a public portal, lab website,
44 personal page, etc. Such data are often provided in tarballs, which
45 need to be downloaded and extracted later on. Extraction will not be
46 a part of this provider -- only download from the web::
47
48 [provider:incoming_http]
49 type = web
50 mode = (download|fast|relaxed) # fast/relaxed/download
51 filename = (url|request) # of cause also could be _e'valuated given the bs4 link get_video_filename(link, filename)
52 recurse_(a|href) = # regexes to recurse
53 # mimicking scrapy
54 start_urls = http://... #
55 certificates = # if there are https -- we need to allow specifying those
56 allowed_domains = example.com/buga/duga # to limit recursion
57 sample.com
58 excluded_hrefs = # do not even search for "download" URLs on given pages. Should also allow to be a function/callback to decide based on request?
59 include_(a|href) = # what to download
60 exclude_(a|href) = # and not (even if matches)
61 ???generators = generate_readme # Define some additional actions to be performed....
62
63 We need to separate options for crawling (recursion etc) and deciding
64 what to download/annex.
65
66 Q: should we just specify xpath's for information to get extracted
67 from a response corresponding to a matching url? just any crawled page?
68
69 Q: allow to use xpath syntax for better control of what to recurse/include?
70
71 Q: authentication -- we should here relate to the Hostings
72 !: scrapy's Spider provides start_requests() which could be used to
73 initiate the connection, e.g. to authenticate and then use that connection.
74 Authentication detail must not be a part of the configuration, BUT
75 it must know HOW authentication should be achieved. In many cases could
76 be a regular netrc-style support (so username/password).
77
78 Those authenticators should later be reused by "download clients"
79
80 Q: we might need to worry/test virtually about every possible associated
81 to http downloads scenario, e.g. support proxy (with authentication).
82 May be we could just switch to aria2 and allow to specify access options?
83
84 Q: may be (a new provider?) allow to use a scrapy spider's output to
85 harvest the table of links which need to be fetched
86
87
88
89 Use cases to keep in mind
90 +++++++++++++++++++++++++
91
92 - versioning present in the file names
93 ftp://ftp.ncbi.nlm.nih.gov/1000genomes/ftp/sequence_indices/
94
95 - ha -- idea, all those should be referred in some other branch, like
96 with archives, and then 'public' one would just take care about
97 pointing to the "correct one" and serve a "compressed" view.
98 Hence: monitor original, point "compressed" to a branch giving it
99 a set of rules on how to determine version, i.e. on which files
100 This way we could have both referenced in the same repository.
101
102
103 Amazon S3
104 ~~~~~~~~~
105
106 Initial accent will be made on S3 buckets which have versioning
107 enabled, and which expose their content via regular http/https.
108
109 tricky points:
110 - versioning (must be enabled. If uploaded before enabled, version is Null)
111
112 - etags are md5s BUT only if upload was not multi-chunked, so
113 it becomes difficult to identify files by md5sums (must be
114 downloaded first then, or some meta-info of file should be modified so
115 etag gets regenerated -- should result in file md5sum appearing as etag)
116
117 - S3 most probably would be just an additional -- not the primary provider
118
119
120 Generated
121 ~~~~~~~~~
122
123 We should allow for files to be generated based on the content of the
124 repository and/or original information from the data providers,
125 e.g. content of the webpages containing the files to be
126 downloaded/referenced. Originally envisioned as a separate branch,
127 where only archived content would be downloaded and later extracted
128 into corresponding locations of the "public" branch (e.g. master).
129
130 But may be it should be more similar to the stated above "versioning"
131 idea where it would simply be an alternative "view" of another branch,
132 where some content is simply extracted. I.e. all those modifications
133 could be assembled as a set of "filters"::
134
135 [generator:generate_readme]
136 filename = README.txt
137 content_e = generate_readme(link, filename) # those should be obtained/provided while crawling
138
139 or
140
141 [generator:fetch_license]
142 filename = LICENSE.txt
143 content_e = fetch_license(link, filename) # those should be obtained/provided while crawling
144
145
146 Merge
147 ~~~~~
148
149 Originally fetched Files might reside in e.g. 'incoming' branch while
150 'master' branch then could be 'assembled' from few other branches with
151 help of filtering::
152
153 [provider:master]
154 type = merge
155 branch = master # here matches the name but see below if we need to repeat
156 merge = incoming_data_http
157 incoming_meta_website
158 filters = extract_models
159 extract_data
160 generate_some_more_files_if_you_like
161
162
163 Q: should we may be 'git merge --no-commit' and then apply the
164 filters???
165
166 probably not since there could be conflicts if similarly named file
167 is present in target branch (e.g. generated) and was present
168 (moved/renamed via filters) in the original branch.
169
170 Q: but merging of branches is way too cool and better establishes the
171 'timeline' and dependencies...
172 So merge should be done "manually" by doing (there must be cleaner way)::
173
174 git merge -s ours --no-commit
175 git rm -r *
176 # - collect and copy files for all the File's from branches to .
177 # - stage all the files
178 # - pipe those "File"s from all the branches through the filters
179 # (those should where necessary use git rm, mv, etc)
180 # - add those File's to git/git-annex
181 git commit
182
183 but what if a filter (e.g. cmd) requires current state of files from
184 different branches?... all possible conflict problems could be
185 mitigated by storing content in branches under some directories,
186 then manipulating upon "merge" and renaming before actually 'git merging'
187
188
189 Q: what about filters per incoming branch??? we could options for
190 filters specification
191 (e.g. extract_models[branches=incoming_data_http]) or allow
192 only regular 2-edge merge at a time but multiple times...
193
194
195 XNAT, COINS, ...
196 ~~~~~~~~~~~~~~~~
197
198 Later ... but the idea should be the same I guess: they should expose
199 collections of File's with a set of URIs so they could be addurl'ed to
200 the files. It is not clear yet either they would need to be crawled
201 or would provide some API similar to S3 to request all the necessary
202 information?
203
204
205 git/git-annex
206 ~~~~~~~~~~~~~
207
208 If provider is already a Git(-annex) repository. Usecase:
209 forrest_gump. So it is pretty much a regular remote **but** it might
210 benefit from our filters etc.
211
212
213 torrent
214 ~~~~~~~
215
216 I guess similar/identical to archives if torrent points to a single
217 file -- so just 'addurl'. If torrent provides multiple files, would
218 need mapping of UUIDs I guess back to torrents/corresponding files.
219 So again -- similar to archives...?
220
221 aria2 seems to provide a single unified HTTP/HTTPS/FTP/BitTorrent
222 support, with fancy simultaneous fetching from multiple
223 remotes/feeding back to the torrent swarm (caution for non-free data).
224 It also has RPC support, which seems to be quite cool and might come
225 handy (e.g. to monitor progress etc)
226
227
228 Wild: Git repository for being rewritten
229 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
230
231 theoretically we could collect all the information to rewrite some
232 other Git repo but now injecting some files into git-annex (while
233 possibly even pointing for the load to e.g. original SVN repo).
234
235 Tricky:
236 - branches and merges -- would be really tricky and so far not
237 envisioned how
238 - "updates" should correspond to commits in original repository
239 - all the commit information should be extracted/provided for the
240 commit here
241
242
243 Filters
244 =======
245
246 Considering idea that all the modifications (archives extraction,
247 versioning etc) could be made through monitoring of another branch(es)
248 and applying a set of filters.
249
250 - files which aren't modified, should also propagate into target
251 branch, along with all their urls
252
253 file by file wouldn't work since filter might need to analyze the
254 entire list of files...::
255
256 def apply_filters(self):
257 files_out = files_in
258 for filter in self.filters:
259 files_out = filter.apply(files_out)
260 return files_out
261
262 then each filter would decide on how to treat the list of files.
263 May be some filters' subtyping would be desired
264 (PerfileFilter/AllfilesFilter)
265
266 - filters should provide API to 'rerun' their action to obtain the
267 same result.
268
269
270 Cross-branch
271 ------------
272
273 Some filters to be applied on files from one branch to have results
274 placed into another:
275
276
277 Extract
278 ~~~~~~~
279
280 Special kind of a beast: while keeping the original archive under
281 git-annex obtained from any other provider (e.g. 'Web'), we extract
282 the load (possibly with some filtering/selection):
283
284 Q: how to deal with extract from archives -- extraction should
285 better be queued to extract multiple files from the archive at
286 once. But ATM it would not happen since all those URIs will
287 simply be requested by simple wget/curl calls by git-annex file
288 at a time.
289 A: upon such a first call, check if there is .../extracted_key/key/, if
290 there is -- use. If not -- extract and then use. use = hardlink
291 into the target file.
292 Upon completion of `datalad get` (or some other command) verify
293 that all `/extracted/` are removed (and/or provide setting -- may
294 be we could/should just keep those around)
295
296
297 Config Examples:
298 ++++++++++++++++
299
300 ::
301
302 [filter:extract_models]
303 filter = extract # by default would be taken as the element after "filter:"
304 input = *(\S+)_models\.tgz$ # and those files are not provided into output
305 output_prefix = models/$1/ # somehow we should allow to reuse input regex's groups
306 exclude = # regex for files to be excluded from extraction or straight for tar?
307 strip_path = 1
308
309 Probably will just use patoolib (do not remember if has
310 strip_path... seems not:
311 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=757483)
312
313 URI: dl:extract:UID
314
315 and we keep information for what 'key' it came into what file (which
316 might later get renamed, so extraction from the archive shouldn't
317 later happen in-place, but rather outside and then moved accordingly)
318
319 Tricky point(s):
320
321 - may be by default should still extract all known archives types and
322 just rely on the filename logic?
323 - the same file might be available from multiple archives.
324 So we would need to keep track from previous updates, from which
325 archive files could be fetched.
326 - how to remove if archive is no longer avail?
327 probably some fsck should take care about checking if archives
328 are still avail, and if not -- remove the url
329
330 - keep track which files came from the archive, so we could later
331 remove them happen if archive misses the file now.
332
333 Q: allow for 'relaxed' handling?
334 If tarballs are not versioned at all, but we would like to create
335 overall (? or just per files) 'relaxed' git-annex?
336
337 Probably no complication if URIs will be based (natively) on the
338 fast or relaxed keys. Sure thing things would fail if archive was
339 changed and lacks the file.
340
341 Q: hm -- what about MD5SUM checking? e.g. if archive was posted with
342 the MD5SUMs file
343
344 I guess some kind of additional filter which could be attached
345 somehow?
346
347
348 Move/Rename/Delete
349 ~~~~~~~~~~~~~~~~~~
350
351 Just move/rename/delete some files around e.g. for a custom view of
352 the dataset (e.g. to conform to OpenfMRI layout). Key would simply be
353 reused ;)
354
355 Q: should it be 'Within-branch' filter?
356
357
358 Command
359 ~~~~~~~
360
361 A universal filter which would operate on some files and output
362 possibly in place or modified ones...
363
364 Then it would need to harvest and encode into file's URI the
365 provenance -- i.e. so it could later be recreated automagically.
366
367 For simple usecases (e.g. creation of lateralized atlas in HOX, some
368 data curation, etc)
369
370 URI: dl:cmd:UID
371
372 while we keep a file providing the corresponding command for each UID,
373 where ARGUMENTS will would point to the original files keys in the git
374 annex. Should it be kept in PROV format may be???
375
376 Config Examples::
377
378 [filter:command_gunzip]
379 in1 = *\.gz
380 in2_e = in1.replace('.gz', '')
381 #eval_order=in1 in2
382 command = zcat {in1} > {in2}
383 output_files = {in2}
384
385 Problems:
386
387 - might be tricky to provide generic enough interface?
388 - we need plentiful of use-cases to get it right, so this one is just
389 to keep in mind for future -- might be quite cool after all.
390
391
392 Within-branch
393 -------------
394
395 Other "Filters" should operate within the branch, primarily simply for
396 checking the content
397
398
399 Checksum
400 ~~~~~~~~
401
402 e.g. point to MD5SUMS file stored in the branch, provide how file
403 names must be augmented, run verification -- no files output, just the
404 status
405
406 Addurl
407 ~~~~~~
408
409 If the repository is going/was published also online under some URL.
410 We might like to populate files with corresponding urls.
411
412 [filter:addurl]
413 prefix = http://psydata.ovgu.de/forrest_gump/.git/annex/
414 check = (False|True) # to verify presence or not ???
415
416 Usecase -- Michael's forrest_gump repository. Now files are not
417 associated explicitly with that URL -- only via a regular git remote.
418 This cumbersomes work with clones which then all must have original
419 repository added as a remote.
420
421 `check = False` could be the one needed for a 'publish' operation
422 where this data present locally is not yet published anywhere.
423
424 Tagging
425 ~~~~~~~
426
427 We might like to tag files... TODO: think what to provide/use to
428 develop nice tags.
429
430 Ideas:
431
432 - a tag given a set of filename regexps
433
434 [tag:anatomicals]
435 files = .*\_anat\.nii\.gz
436 tag = modality=anatomy
437
438 or just
439
440 [tag:anatomicals]
441 files = .*\_anat\.nii\.gz
442 tag = anatomy
443
444 if it is just a tag (anatomy) without a field
445
446 - (full)filename regexp with groups defining possibly multiple
447 tag/value pairs
448
449 [tag:modality]
450 files = .*\_(?P<modality>\S*)\.nii\.gz
451 translate = anat: T1 # might need some translation dictionary?
452 dwi: DTI
453
454
455 Notes:
456 - metadata cane be added only to files under git-annex control so those
457 directly committed
458
459 Design thoughts
460 ===============
461
462 Data providers should provide a unified interface
463
464 DataProvider
465 ~~~~~~~~~~~~
466
467 Common Parameters
468 - add_to_git - what files to commit to git directly (should we leverage
469 git-annex largefiles option somehow?)
470 - ignore - what files to ignore
471
472 - get_items(version=None) - return a list of Files
473 - get_item_by_name
474 - get_item_by_md5
475 - should those be additional interfaces?
476 - what if multiple items fulfill (content is the same, e.g. empty, names differ,
477 we better get the most appropriate in the name or don't give a damn?)
478 - what if a collision????
479 - get_item_by_sha256
480 - e.g. natively provided by 'Branch' provider for annexed files
481 (what to do about git committed ones -- compute/keep info?)
482 - get_versions(min_version=None)
483 provider-wide version (i.e. not per file). E.g. S3
484 provider can have multiple versions of files.
485 Might be that it needs to return a DAG of versions i.e. a
486 (version, [prev_version1, prev_version2, ...]) to represent e.g.
487 history of a Git repo. In most of the cases would be degenerate to just
488 one prev version, in which case could just be (version, ).
489 We would need to store that meta-information for future updates at least
490 for the last version so we could 'grow' next ones on top.
491 - ? get_release_versions() -- by default identical to above... but might
492 differ (update was, but no new official release (yet), so no release
493 tag)
494 - get_version_metainformation() -- primarily conceived when thinking
495 about monitoring other VCS repos... so should be information to be
496 used for a new Git commit into this new repository
497
498 .. note:
499
500 Keep in mind
501 - Web DataProvider must have an option to request the content filename
502 (addressing use case with redirects etc)
503 - Some providers might have multiple URIs (mirrors) so right away
504 assign them per each file... As such they might be from
505 different Hostings!
506
507
508 File
509 ~~~~
510
511 what would be saved as a file. Should know about itself... and origins!
512
513 - filename
514 - URIs - list containing origins (e.g. URLs) on where to fetch it from.
515 First provided by the
516 original DataProvider, but then might be expanded using
517 other DataProviders
518 Q: Those might need to be not just URIs but some classes associated
519 with original Hosting's, e.g. for the cases of authentication etc?
520 or we would associate with a Hosting based on the URI?
521 # combination of known fields should be stored/used to detect changes
522 # Different data providers might rely on a different subset of below
523 # to see if there was a change. We should probably assume some
524 # "correspondence"
525 - key # was thinking about Branch as DataProvider -- those must be reused
526 - md5
527 - sha256
528 - mtime
529 - size
530
531 It will be the job of a DataProvider to initiate File with the
532 appropriate filename.
533
534 URI
535 ~~~
536
537 -> URL(URI): will be our first and main "target" but it could
538 also be direct S3, etc.
539
540 a URI should be associated with an "Hosting" (many-to-one), so we could
541 e.g. provide authentication information per actual "Hosting" as the
542 entity. But now we are getting back to DataProvider, which is the
543 Hosting, or actually also a part of it (since Hosting could serve
544 multiple Providers, e.g. OpenfMRI -> providers per each dataset?)
545 But also Provider might use/point to multiple Hostings (e.g. mirrors
546 listed on nitp-2013).
547
548 Hosting
549 ~~~~~~~
550
551 Each DataProvider would be a factory of File's.
552
553
554 Ideas to not forget
555 ~~~~~~~~~~~~~~~~~~~
556
557 - Before carrying out some operation, remember the state of all
558 (involved) branches, so it would be very easy later on to "cancel"
559 the entire transaction through a set of 'git reset --hard' or
560 'update-ref's.
561
562 Keep log of the above!
563
564 - multiple data providers could be specified but there should be
565 'primary' and 'complimentary' ones:
566
567 - primary provider(s) define the layout/content
568 - complimentary providers just provide references to additional
569 locations where that data (uniquely identified via checksums etc)
570 could be obtained, so we could add more data providing urls
571 - Q: should all DataProvider's be able to serve as primary and complimentary?
572 - most probably we should allow for an option to 'fail' or issue a
573 warning in some cases
574 - secondary provider doesn't carry a requested load/file
575 - secondary provider provides some files not provided by the primary
576 data provider
577
578 - at the end of the crawl operation, verify that all the files have all
579 and only urls from the provided data providers
580
581 - allow to add/specify conventional git/annex clones as additional,
582 conventional (non special) remotes to be added.
583
584 - allow to prepopulate URLs given e.g. perspective hosting on HTTP.
585 This way whenever content gets published there -- all files would
586 have appropriate URLs associated and would 'transcend' through the
587 clones without requiring adding original remote.
588
589 Updates
590 =======
591
592 - must track updates and removals of the files
593 - must verify presence (add, remove) of the urls associated with the
594 files given a list of data providers
595
596
597 Meta information
598 ================
599
600 Since a while `git annex` provides a neat feature allowing to assign
601 tags to the files and later use e.g. `git annex view` to quickly
602 generate customized views of the repository.
603
604
605 Some cool related tools
606 =======================
607
608 https://github.com/scrapy/scrapely
609 Pure Python (no DOM, lxml, etc) scraping of pages, "training" the
610 scraper given a sample. May be could be handy???
611 https://github.com/scrapy/slybot
612 Brings together scrapy + scrapely to provide json-specs for
613 spiders/items/etc
614 Might be worth at least adopting spiders specs...?
615 Has a neat slybot/validation/schemas.json which validates the schematic