| 1 |
say "With a few lines DataLad is set up to track data posted on a website, and obtain changes made in the future..." |
| 2 |
|
| 3 |
say "The website http://www.fmri-data-analysis.org/code provides code and data file for examples in a text book." |
| 4 |
say "We will set up a dataset that DataLad uses to track the content linked from this webpage" |
| 5 |
say "Let's create the dataset, and configure it to track any text file directly in Git. This will make it very convenient to see how source code changed over time." |
| 6 |
run "datalad create --text-no-annex demo" |
| 7 |
run "cd demo" |
| 8 |
|
| 9 |
say "DataLad's crawler functionality is used to monitor the webpage. It's configuration is stored in the dataset itself." |
| 10 |
say "The crawler comes with a bunch of configuration templates. Here we are using one that extract all URLs that match a particular pattern, and obtains the linked data. In case of this webpage, all URLs of interest on that page seems to have 'd=1' suffix" |
| 11 |
run "datalad crawl-init --save --template=simple_with_archives url=http://www.fmri-data-analysis.org/code 'a_href_match_=.*d=1$'" |
| 12 |
run "datalad diff --revision @~1" |
| 13 |
run "cat .datalad/crawl/crawl.cfg" |
| 14 |
|
| 15 |
say "With this configuration in place, we can ask DataLad to crawl the webpage." |
| 16 |
run "datalad crawl" |
| 17 |
|
| 18 |
say "All files have been obtained and are ready to use. Here is what DataLad recorded for this update" |
| 19 |
run "git show @ -s" |
| 20 |
|
| 21 |
say "Any file from the webpage is available locally." |
| 22 |
run "ls" |
| 23 |
|
| 24 |
say "The webpage can be queried for potential updates at any time by re-running the 'crawl' command." |
| 25 |
run "datalad crawl" |
| 26 |
|
| 27 |
say "Files can be added, or removed from this dataset without impairing the ability to get updates from the webpage. DataLad keeps the necessary information in dedicated Git branches." |
| 28 |
run "git branch" |