| 1 |
Thoughts about redesign, well actually "design" since originally there |
| 2 |
were none, of datalad crawl. |
| 3 |
|
| 4 |
Global portion of the config |
| 5 |
============================ |
| 6 |
|
| 7 |
:: |
| 8 |
|
| 9 |
[dataset] |
| 10 |
path = |
| 11 |
description = |
| 12 |
exec = |
| 13 |
|
| 14 |
Data providers |
| 15 |
============== |
| 16 |
|
| 17 |
`crawl` command collects data present possibly across |
| 18 |
different remote data providers (regular HTTP websites, AWS S3 |
| 19 |
buckets, etc) and then consolidates access to them within a single |
| 20 |
git-annex'ed repository. `crawl` should also keep track of |
| 21 |
status/versions of the files, so in case of the updates (changes, |
| 22 |
removals, etc) on remote sites, git-annex repository could be |
| 23 |
correspondingly updated. |
| 24 |
|
| 25 |
Common config specs:: |
| 26 |
|
| 27 |
[provider:XXX] |
| 28 |
type = (web|s3http|merge|git-annex) # default to web |
| 29 |
branch = master # default to master |
| 30 |
commit_to_git = # regexps of file names to commit directly to git |
| 31 |
ignore = # files to ignore entirely |
| 32 |
drop = False # either to drop the load upon 'completion' |
| 33 |
# some sanity checks |
| 34 |
check_entries_limit = -1 # no limit by default |
| 35 |
|
| 36 |
|
| 37 |
(To be) Supported data providers |
| 38 |
-------------------------------- |
| 39 |
|
| 40 |
Web |
| 41 |
~~~ |
| 42 |
|
| 43 |
In many usecases data are hosted on a public portal, lab website, |
| 44 |
personal page, etc. Such data are often provided in tarballs, which |
| 45 |
need to be downloaded and extracted later on. Extraction will not be |
| 46 |
a part of this provider -- only download from the web:: |
| 47 |
|
| 48 |
[provider:incoming_http] |
| 49 |
type = web |
| 50 |
mode = (download|fast|relaxed) # fast/relaxed/download |
| 51 |
filename = (url|request) # of cause also could be _e'valuated given the bs4 link get_video_filename(link, filename) |
| 52 |
recurse_(a|href) = # regexes to recurse |
| 53 |
# mimicking scrapy |
| 54 |
start_urls = http://... # |
| 55 |
certificates = # if there are https -- we need to allow specifying those |
| 56 |
allowed_domains = example.com/buga/duga # to limit recursion |
| 57 |
sample.com |
| 58 |
excluded_hrefs = # do not even search for "download" URLs on given pages. Should also allow to be a function/callback to decide based on request? |
| 59 |
include_(a|href) = # what to download |
| 60 |
exclude_(a|href) = # and not (even if matches) |
| 61 |
???generators = generate_readme # Define some additional actions to be performed.... |
| 62 |
|
| 63 |
We need to separate options for crawling (recursion etc) and deciding |
| 64 |
what to download/annex. |
| 65 |
|
| 66 |
Q: should we just specify xpath's for information to get extracted |
| 67 |
from a response corresponding to a matching url? just any crawled page? |
| 68 |
|
| 69 |
Q: allow to use xpath syntax for better control of what to recurse/include? |
| 70 |
|
| 71 |
Q: authentication -- we should here relate to the Hostings |
| 72 |
!: scrapy's Spider provides start_requests() which could be used to |
| 73 |
initiate the connection, e.g. to authenticate and then use that connection. |
| 74 |
Authentication detail must not be a part of the configuration, BUT |
| 75 |
it must know HOW authentication should be achieved. In many cases could |
| 76 |
be a regular netrc-style support (so username/password). |
| 77 |
|
| 78 |
Those authenticators should later be reused by "download clients" |
| 79 |
|
| 80 |
Q: we might need to worry/test virtually about every possible associated |
| 81 |
to http downloads scenario, e.g. support proxy (with authentication). |
| 82 |
May be we could just switch to aria2 and allow to specify access options? |
| 83 |
|
| 84 |
Q: may be (a new provider?) allow to use a scrapy spider's output to |
| 85 |
harvest the table of links which need to be fetched |
| 86 |
|
| 87 |
|
| 88 |
|
| 89 |
Use cases to keep in mind |
| 90 |
+++++++++++++++++++++++++ |
| 91 |
|
| 92 |
- versioning present in the file names |
| 93 |
ftp://ftp.ncbi.nlm.nih.gov/1000genomes/ftp/sequence_indices/ |
| 94 |
|
| 95 |
- ha -- idea, all those should be referred in some other branch, like |
| 96 |
with archives, and then 'public' one would just take care about |
| 97 |
pointing to the "correct one" and serve a "compressed" view. |
| 98 |
Hence: monitor original, point "compressed" to a branch giving it |
| 99 |
a set of rules on how to determine version, i.e. on which files |
| 100 |
This way we could have both referenced in the same repository. |
| 101 |
|
| 102 |
|
| 103 |
Amazon S3 |
| 104 |
~~~~~~~~~ |
| 105 |
|
| 106 |
Initial accent will be made on S3 buckets which have versioning |
| 107 |
enabled, and which expose their content via regular http/https. |
| 108 |
|
| 109 |
tricky points: |
| 110 |
- versioning (must be enabled. If uploaded before enabled, version is Null) |
| 111 |
|
| 112 |
- etags are md5s BUT only if upload was not multi-chunked, so |
| 113 |
it becomes difficult to identify files by md5sums (must be |
| 114 |
downloaded first then, or some meta-info of file should be modified so |
| 115 |
etag gets regenerated -- should result in file md5sum appearing as etag) |
| 116 |
|
| 117 |
- S3 most probably would be just an additional -- not the primary provider |
| 118 |
|
| 119 |
|
| 120 |
Generated |
| 121 |
~~~~~~~~~ |
| 122 |
|
| 123 |
We should allow for files to be generated based on the content of the |
| 124 |
repository and/or original information from the data providers, |
| 125 |
e.g. content of the webpages containing the files to be |
| 126 |
downloaded/referenced. Originally envisioned as a separate branch, |
| 127 |
where only archived content would be downloaded and later extracted |
| 128 |
into corresponding locations of the "public" branch (e.g. master). |
| 129 |
|
| 130 |
But may be it should be more similar to the stated above "versioning" |
| 131 |
idea where it would simply be an alternative "view" of another branch, |
| 132 |
where some content is simply extracted. I.e. all those modifications |
| 133 |
could be assembled as a set of "filters":: |
| 134 |
|
| 135 |
[generator:generate_readme] |
| 136 |
filename = README.txt |
| 137 |
content_e = generate_readme(link, filename) # those should be obtained/provided while crawling |
| 138 |
|
| 139 |
or |
| 140 |
|
| 141 |
[generator:fetch_license] |
| 142 |
filename = LICENSE.txt |
| 143 |
content_e = fetch_license(link, filename) # those should be obtained/provided while crawling |
| 144 |
|
| 145 |
|
| 146 |
Merge |
| 147 |
~~~~~ |
| 148 |
|
| 149 |
Originally fetched Files might reside in e.g. 'incoming' branch while |
| 150 |
'master' branch then could be 'assembled' from few other branches with |
| 151 |
help of filtering:: |
| 152 |
|
| 153 |
[provider:master] |
| 154 |
type = merge |
| 155 |
branch = master # here matches the name but see below if we need to repeat |
| 156 |
merge = incoming_data_http |
| 157 |
incoming_meta_website |
| 158 |
filters = extract_models |
| 159 |
extract_data |
| 160 |
generate_some_more_files_if_you_like |
| 161 |
|
| 162 |
|
| 163 |
Q: should we may be 'git merge --no-commit' and then apply the |
| 164 |
filters??? |
| 165 |
|
| 166 |
probably not since there could be conflicts if similarly named file |
| 167 |
is present in target branch (e.g. generated) and was present |
| 168 |
(moved/renamed via filters) in the original branch. |
| 169 |
|
| 170 |
Q: but merging of branches is way too cool and better establishes the |
| 171 |
'timeline' and dependencies... |
| 172 |
So merge should be done "manually" by doing (there must be cleaner way):: |
| 173 |
|
| 174 |
git merge -s ours --no-commit |
| 175 |
git rm -r * |
| 176 |
# - collect and copy files for all the File's from branches to . |
| 177 |
# - stage all the files |
| 178 |
# - pipe those "File"s from all the branches through the filters |
| 179 |
# (those should where necessary use git rm, mv, etc) |
| 180 |
# - add those File's to git/git-annex |
| 181 |
git commit |
| 182 |
|
| 183 |
but what if a filter (e.g. cmd) requires current state of files from |
| 184 |
different branches?... all possible conflict problems could be |
| 185 |
mitigated by storing content in branches under some directories, |
| 186 |
then manipulating upon "merge" and renaming before actually 'git merging' |
| 187 |
|
| 188 |
|
| 189 |
Q: what about filters per incoming branch??? we could options for |
| 190 |
filters specification |
| 191 |
(e.g. extract_models[branches=incoming_data_http]) or allow |
| 192 |
only regular 2-edge merge at a time but multiple times... |
| 193 |
|
| 194 |
|
| 195 |
XNAT, COINS, ... |
| 196 |
~~~~~~~~~~~~~~~~ |
| 197 |
|
| 198 |
Later ... but the idea should be the same I guess: they should expose |
| 199 |
collections of File's with a set of URIs so they could be addurl'ed to |
| 200 |
the files. It is not clear yet either they would need to be crawled |
| 201 |
or would provide some API similar to S3 to request all the necessary |
| 202 |
information? |
| 203 |
|
| 204 |
|
| 205 |
git/git-annex |
| 206 |
~~~~~~~~~~~~~ |
| 207 |
|
| 208 |
If provider is already a Git(-annex) repository. Usecase: |
| 209 |
forrest_gump. So it is pretty much a regular remote **but** it might |
| 210 |
benefit from our filters etc. |
| 211 |
|
| 212 |
|
| 213 |
torrent |
| 214 |
~~~~~~~ |
| 215 |
|
| 216 |
I guess similar/identical to archives if torrent points to a single |
| 217 |
file -- so just 'addurl'. If torrent provides multiple files, would |
| 218 |
need mapping of UUIDs I guess back to torrents/corresponding files. |
| 219 |
So again -- similar to archives...? |
| 220 |
|
| 221 |
aria2 seems to provide a single unified HTTP/HTTPS/FTP/BitTorrent |
| 222 |
support, with fancy simultaneous fetching from multiple |
| 223 |
remotes/feeding back to the torrent swarm (caution for non-free data). |
| 224 |
It also has RPC support, which seems to be quite cool and might come |
| 225 |
handy (e.g. to monitor progress etc) |
| 226 |
|
| 227 |
|
| 228 |
Wild: Git repository for being rewritten |
| 229 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 230 |
|
| 231 |
theoretically we could collect all the information to rewrite some |
| 232 |
other Git repo but now injecting some files into git-annex (while |
| 233 |
possibly even pointing for the load to e.g. original SVN repo). |
| 234 |
|
| 235 |
Tricky: |
| 236 |
- branches and merges -- would be really tricky and so far not |
| 237 |
envisioned how |
| 238 |
- "updates" should correspond to commits in original repository |
| 239 |
- all the commit information should be extracted/provided for the |
| 240 |
commit here |
| 241 |
|
| 242 |
|
| 243 |
Filters |
| 244 |
======= |
| 245 |
|
| 246 |
Considering idea that all the modifications (archives extraction, |
| 247 |
versioning etc) could be made through monitoring of another branch(es) |
| 248 |
and applying a set of filters. |
| 249 |
|
| 250 |
- files which aren't modified, should also propagate into target |
| 251 |
branch, along with all their urls |
| 252 |
|
| 253 |
file by file wouldn't work since filter might need to analyze the |
| 254 |
entire list of files...:: |
| 255 |
|
| 256 |
def apply_filters(self): |
| 257 |
files_out = files_in |
| 258 |
for filter in self.filters: |
| 259 |
files_out = filter.apply(files_out) |
| 260 |
return files_out |
| 261 |
|
| 262 |
then each filter would decide on how to treat the list of files. |
| 263 |
May be some filters' subtyping would be desired |
| 264 |
(PerfileFilter/AllfilesFilter) |
| 265 |
|
| 266 |
- filters should provide API to 'rerun' their action to obtain the |
| 267 |
same result. |
| 268 |
|
| 269 |
|
| 270 |
Cross-branch |
| 271 |
------------ |
| 272 |
|
| 273 |
Some filters to be applied on files from one branch to have results |
| 274 |
placed into another: |
| 275 |
|
| 276 |
|
| 277 |
Extract |
| 278 |
~~~~~~~ |
| 279 |
|
| 280 |
Special kind of a beast: while keeping the original archive under |
| 281 |
git-annex obtained from any other provider (e.g. 'Web'), we extract |
| 282 |
the load (possibly with some filtering/selection): |
| 283 |
|
| 284 |
Q: how to deal with extract from archives -- extraction should |
| 285 |
better be queued to extract multiple files from the archive at |
| 286 |
once. But ATM it would not happen since all those URIs will |
| 287 |
simply be requested by simple wget/curl calls by git-annex file |
| 288 |
at a time. |
| 289 |
A: upon such a first call, check if there is .../extracted_key/key/, if |
| 290 |
there is -- use. If not -- extract and then use. use = hardlink |
| 291 |
into the target file. |
| 292 |
Upon completion of `datalad get` (or some other command) verify |
| 293 |
that all `/extracted/` are removed (and/or provide setting -- may |
| 294 |
be we could/should just keep those around) |
| 295 |
|
| 296 |
|
| 297 |
Config Examples: |
| 298 |
++++++++++++++++ |
| 299 |
|
| 300 |
:: |
| 301 |
|
| 302 |
[filter:extract_models] |
| 303 |
filter = extract # by default would be taken as the element after "filter:" |
| 304 |
input = *(\S+)_models\.tgz$ # and those files are not provided into output |
| 305 |
output_prefix = models/$1/ # somehow we should allow to reuse input regex's groups |
| 306 |
exclude = # regex for files to be excluded from extraction or straight for tar? |
| 307 |
strip_path = 1 |
| 308 |
|
| 309 |
Probably will just use patoolib (do not remember if has |
| 310 |
strip_path... seems not: |
| 311 |
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=757483) |
| 312 |
|
| 313 |
URI: dl:extract:UID |
| 314 |
|
| 315 |
and we keep information for what 'key' it came into what file (which |
| 316 |
might later get renamed, so extraction from the archive shouldn't |
| 317 |
later happen in-place, but rather outside and then moved accordingly) |
| 318 |
|
| 319 |
Tricky point(s): |
| 320 |
|
| 321 |
- may be by default should still extract all known archives types and |
| 322 |
just rely on the filename logic? |
| 323 |
- the same file might be available from multiple archives. |
| 324 |
So we would need to keep track from previous updates, from which |
| 325 |
archive files could be fetched. |
| 326 |
- how to remove if archive is no longer avail? |
| 327 |
probably some fsck should take care about checking if archives |
| 328 |
are still avail, and if not -- remove the url |
| 329 |
|
| 330 |
- keep track which files came from the archive, so we could later |
| 331 |
remove them happen if archive misses the file now. |
| 332 |
|
| 333 |
Q: allow for 'relaxed' handling? |
| 334 |
If tarballs are not versioned at all, but we would like to create |
| 335 |
overall (? or just per files) 'relaxed' git-annex? |
| 336 |
|
| 337 |
Probably no complication if URIs will be based (natively) on the |
| 338 |
fast or relaxed keys. Sure thing things would fail if archive was |
| 339 |
changed and lacks the file. |
| 340 |
|
| 341 |
Q: hm -- what about MD5SUM checking? e.g. if archive was posted with |
| 342 |
the MD5SUMs file |
| 343 |
|
| 344 |
I guess some kind of additional filter which could be attached |
| 345 |
somehow? |
| 346 |
|
| 347 |
|
| 348 |
Move/Rename/Delete |
| 349 |
~~~~~~~~~~~~~~~~~~ |
| 350 |
|
| 351 |
Just move/rename/delete some files around e.g. for a custom view of |
| 352 |
the dataset (e.g. to conform to OpenfMRI layout). Key would simply be |
| 353 |
reused ;) |
| 354 |
|
| 355 |
Q: should it be 'Within-branch' filter? |
| 356 |
|
| 357 |
|
| 358 |
Command |
| 359 |
~~~~~~~ |
| 360 |
|
| 361 |
A universal filter which would operate on some files and output |
| 362 |
possibly in place or modified ones... |
| 363 |
|
| 364 |
Then it would need to harvest and encode into file's URI the |
| 365 |
provenance -- i.e. so it could later be recreated automagically. |
| 366 |
|
| 367 |
For simple usecases (e.g. creation of lateralized atlas in HOX, some |
| 368 |
data curation, etc) |
| 369 |
|
| 370 |
URI: dl:cmd:UID |
| 371 |
|
| 372 |
while we keep a file providing the corresponding command for each UID, |
| 373 |
where ARGUMENTS will would point to the original files keys in the git |
| 374 |
annex. Should it be kept in PROV format may be??? |
| 375 |
|
| 376 |
Config Examples:: |
| 377 |
|
| 378 |
[filter:command_gunzip] |
| 379 |
in1 = *\.gz |
| 380 |
in2_e = in1.replace('.gz', '') |
| 381 |
#eval_order=in1 in2 |
| 382 |
command = zcat {in1} > {in2} |
| 383 |
output_files = {in2} |
| 384 |
|
| 385 |
Problems: |
| 386 |
|
| 387 |
- might be tricky to provide generic enough interface? |
| 388 |
- we need plentiful of use-cases to get it right, so this one is just |
| 389 |
to keep in mind for future -- might be quite cool after all. |
| 390 |
|
| 391 |
|
| 392 |
Within-branch |
| 393 |
------------- |
| 394 |
|
| 395 |
Other "Filters" should operate within the branch, primarily simply for |
| 396 |
checking the content |
| 397 |
|
| 398 |
|
| 399 |
Checksum |
| 400 |
~~~~~~~~ |
| 401 |
|
| 402 |
e.g. point to MD5SUMS file stored in the branch, provide how file |
| 403 |
names must be augmented, run verification -- no files output, just the |
| 404 |
status |
| 405 |
|
| 406 |
Addurl |
| 407 |
~~~~~~ |
| 408 |
|
| 409 |
If the repository is going/was published also online under some URL. |
| 410 |
We might like to populate files with corresponding urls. |
| 411 |
|
| 412 |
[filter:addurl] |
| 413 |
prefix = http://psydata.ovgu.de/forrest_gump/.git/annex/ |
| 414 |
check = (False|True) # to verify presence or not ??? |
| 415 |
|
| 416 |
Usecase -- Michael's forrest_gump repository. Now files are not |
| 417 |
associated explicitly with that URL -- only via a regular git remote. |
| 418 |
This cumbersomes work with clones which then all must have original |
| 419 |
repository added as a remote. |
| 420 |
|
| 421 |
`check = False` could be the one needed for a 'publish' operation |
| 422 |
where this data present locally is not yet published anywhere. |
| 423 |
|
| 424 |
Tagging |
| 425 |
~~~~~~~ |
| 426 |
|
| 427 |
We might like to tag files... TODO: think what to provide/use to |
| 428 |
develop nice tags. |
| 429 |
|
| 430 |
Ideas: |
| 431 |
|
| 432 |
- a tag given a set of filename regexps |
| 433 |
|
| 434 |
[tag:anatomicals] |
| 435 |
files = .*\_anat\.nii\.gz |
| 436 |
tag = modality=anatomy |
| 437 |
|
| 438 |
or just |
| 439 |
|
| 440 |
[tag:anatomicals] |
| 441 |
files = .*\_anat\.nii\.gz |
| 442 |
tag = anatomy |
| 443 |
|
| 444 |
if it is just a tag (anatomy) without a field |
| 445 |
|
| 446 |
- (full)filename regexp with groups defining possibly multiple |
| 447 |
tag/value pairs |
| 448 |
|
| 449 |
[tag:modality] |
| 450 |
files = .*\_(?P<modality>\S*)\.nii\.gz |
| 451 |
translate = anat: T1 # might need some translation dictionary? |
| 452 |
dwi: DTI |
| 453 |
|
| 454 |
|
| 455 |
Notes: |
| 456 |
- metadata cane be added only to files under git-annex control so those |
| 457 |
directly committed |
| 458 |
|
| 459 |
Design thoughts |
| 460 |
=============== |
| 461 |
|
| 462 |
Data providers should provide a unified interface |
| 463 |
|
| 464 |
DataProvider |
| 465 |
~~~~~~~~~~~~ |
| 466 |
|
| 467 |
Common Parameters |
| 468 |
- add_to_git - what files to commit to git directly (should we leverage |
| 469 |
git-annex largefiles option somehow?) |
| 470 |
- ignore - what files to ignore |
| 471 |
|
| 472 |
- get_items(version=None) - return a list of Files |
| 473 |
- get_item_by_name |
| 474 |
- get_item_by_md5 |
| 475 |
- should those be additional interfaces? |
| 476 |
- what if multiple items fulfill (content is the same, e.g. empty, names differ, |
| 477 |
we better get the most appropriate in the name or don't give a damn?) |
| 478 |
- what if a collision???? |
| 479 |
- get_item_by_sha256 |
| 480 |
- e.g. natively provided by 'Branch' provider for annexed files |
| 481 |
(what to do about git committed ones -- compute/keep info?) |
| 482 |
- get_versions(min_version=None) |
| 483 |
provider-wide version (i.e. not per file). E.g. S3 |
| 484 |
provider can have multiple versions of files. |
| 485 |
Might be that it needs to return a DAG of versions i.e. a |
| 486 |
(version, [prev_version1, prev_version2, ...]) to represent e.g. |
| 487 |
history of a Git repo. In most of the cases would be degenerate to just |
| 488 |
one prev version, in which case could just be (version, ). |
| 489 |
We would need to store that meta-information for future updates at least |
| 490 |
for the last version so we could 'grow' next ones on top. |
| 491 |
- ? get_release_versions() -- by default identical to above... but might |
| 492 |
differ (update was, but no new official release (yet), so no release |
| 493 |
tag) |
| 494 |
- get_version_metainformation() -- primarily conceived when thinking |
| 495 |
about monitoring other VCS repos... so should be information to be |
| 496 |
used for a new Git commit into this new repository |
| 497 |
|
| 498 |
.. note: |
| 499 |
|
| 500 |
Keep in mind |
| 501 |
- Web DataProvider must have an option to request the content filename |
| 502 |
(addressing use case with redirects etc) |
| 503 |
- Some providers might have multiple URIs (mirrors) so right away |
| 504 |
assign them per each file... As such they might be from |
| 505 |
different Hostings! |
| 506 |
|
| 507 |
|
| 508 |
File |
| 509 |
~~~~ |
| 510 |
|
| 511 |
what would be saved as a file. Should know about itself... and origins! |
| 512 |
|
| 513 |
- filename |
| 514 |
- URIs - list containing origins (e.g. URLs) on where to fetch it from. |
| 515 |
First provided by the |
| 516 |
original DataProvider, but then might be expanded using |
| 517 |
other DataProviders |
| 518 |
Q: Those might need to be not just URIs but some classes associated |
| 519 |
with original Hosting's, e.g. for the cases of authentication etc? |
| 520 |
or we would associate with a Hosting based on the URI? |
| 521 |
# combination of known fields should be stored/used to detect changes |
| 522 |
# Different data providers might rely on a different subset of below |
| 523 |
# to see if there was a change. We should probably assume some |
| 524 |
# "correspondence" |
| 525 |
- key # was thinking about Branch as DataProvider -- those must be reused |
| 526 |
- md5 |
| 527 |
- sha256 |
| 528 |
- mtime |
| 529 |
- size |
| 530 |
|
| 531 |
It will be the job of a DataProvider to initiate File with the |
| 532 |
appropriate filename. |
| 533 |
|
| 534 |
URI |
| 535 |
~~~ |
| 536 |
|
| 537 |
-> URL(URI): will be our first and main "target" but it could |
| 538 |
also be direct S3, etc. |
| 539 |
|
| 540 |
a URI should be associated with an "Hosting" (many-to-one), so we could |
| 541 |
e.g. provide authentication information per actual "Hosting" as the |
| 542 |
entity. But now we are getting back to DataProvider, which is the |
| 543 |
Hosting, or actually also a part of it (since Hosting could serve |
| 544 |
multiple Providers, e.g. OpenfMRI -> providers per each dataset?) |
| 545 |
But also Provider might use/point to multiple Hostings (e.g. mirrors |
| 546 |
listed on nitp-2013). |
| 547 |
|
| 548 |
Hosting |
| 549 |
~~~~~~~ |
| 550 |
|
| 551 |
Each DataProvider would be a factory of File's. |
| 552 |
|
| 553 |
|
| 554 |
Ideas to not forget |
| 555 |
~~~~~~~~~~~~~~~~~~~ |
| 556 |
|
| 557 |
- Before carrying out some operation, remember the state of all |
| 558 |
(involved) branches, so it would be very easy later on to "cancel" |
| 559 |
the entire transaction through a set of 'git reset --hard' or |
| 560 |
'update-ref's. |
| 561 |
|
| 562 |
Keep log of the above! |
| 563 |
|
| 564 |
- multiple data providers could be specified but there should be |
| 565 |
'primary' and 'complimentary' ones: |
| 566 |
|
| 567 |
- primary provider(s) define the layout/content |
| 568 |
- complimentary providers just provide references to additional |
| 569 |
locations where that data (uniquely identified via checksums etc) |
| 570 |
could be obtained, so we could add more data providing urls |
| 571 |
- Q: should all DataProvider's be able to serve as primary and complimentary? |
| 572 |
- most probably we should allow for an option to 'fail' or issue a |
| 573 |
warning in some cases |
| 574 |
- secondary provider doesn't carry a requested load/file |
| 575 |
- secondary provider provides some files not provided by the primary |
| 576 |
data provider |
| 577 |
|
| 578 |
- at the end of the crawl operation, verify that all the files have all |
| 579 |
and only urls from the provided data providers |
| 580 |
|
| 581 |
- allow to add/specify conventional git/annex clones as additional, |
| 582 |
conventional (non special) remotes to be added. |
| 583 |
|
| 584 |
- allow to prepopulate URLs given e.g. perspective hosting on HTTP. |
| 585 |
This way whenever content gets published there -- all files would |
| 586 |
have appropriate URLs associated and would 'transcend' through the |
| 587 |
clones without requiring adding original remote. |
| 588 |
|
| 589 |
Updates |
| 590 |
======= |
| 591 |
|
| 592 |
- must track updates and removals of the files |
| 593 |
- must verify presence (add, remove) of the urls associated with the |
| 594 |
files given a list of data providers |
| 595 |
|
| 596 |
|
| 597 |
Meta information |
| 598 |
================ |
| 599 |
|
| 600 |
Since a while `git annex` provides a neat feature allowing to assign |
| 601 |
tags to the files and later use e.g. `git annex view` to quickly |
| 602 |
generate customized views of the repository. |
| 603 |
|
| 604 |
|
| 605 |
Some cool related tools |
| 606 |
======================= |
| 607 |
|
| 608 |
https://github.com/scrapy/scrapely |
| 609 |
Pure Python (no DOM, lxml, etc) scraping of pages, "training" the |
| 610 |
scraper given a sample. May be could be handy??? |
| 611 |
https://github.com/scrapy/slybot |
| 612 |
Brings together scrapy + scrapely to provide json-specs for |
| 613 |
spiders/items/etc |
| 614 |
Might be worth at least adopting spiders specs...? |
| 615 |
Has a neat slybot/validation/schemas.json which validates the schematic |