| 1 |
# this cast requires the ability to create 'datalad/publish-demo' on Github (must not exist yet) |
| 2 |
# it also require SSH access to the URL demo.datalad.org (can be localhost) |
| 3 |
|
| 4 |
say "Sharing is best done on a platform that many people visit. One of them is GitHub." |
| 5 |
say "However, Git is not designed to handle large data files directly, and GitHub will refuse large files" |
| 6 |
say "One can either use GitHub's own LFS tool ... or use DataLad to flexibly combine many possible data hosting solutions with a repository hosted on GitHub" |
| 7 |
say "Here is how this looks..." |
| 8 |
|
| 9 |
say "It starts with a dataset. We give it a description to more easily keep track of where data are." |
| 10 |
run "datalad create demo --description \"original dataset location\"" |
| 11 |
run "cd demo" |
| 12 |
|
| 13 |
say "For this demo, we are generating a large (600MB file) comprised of random data. A file of this size cannot be hosted on GitHub directly." |
| 14 |
run "datalad run dd if=/dev/urandom of=big.dat bs=1M count=600" |
| 15 |
|
| 16 |
say "Publishing this dataset in a way that allows anyone to simply install it from GitHub, AND get the big data file, requires two steps" |
| 17 |
say "1. Host the data file at some publicly accessible location" |
| 18 |
say "2. Configure DataLad to make sure that getting data from GitHub transparently requests from this other location instead" |
| 19 |
|
| 20 |
say "Here we use a personal webserver with SSH access, but, in principle, any hosting solution supported by git-annex is equally suitable" |
| 21 |
say "We create a remote sibling of our dataset under the name 'myserver' via SSH, and tell datalad to track it as a common data source that is available for any future installation of this dataset. Access to this location will happen via the given http:// URL, and --ui true tells to install DataLad web UI as on https://datasets.datalad.org. Note that /.git in the URL most likely to be necessary in your case." |
| 22 |
run "datalad create-sibling -s myserver demo.datalad.org:public_html/publish-demo --ui true --as-common-datasrc demo-server --target-url http://demo.datalad.org/publish-demo/.git" |
| 23 |
|
| 24 |
say "With this configuration in place, we can now create a repository on GitHub, and configure the remote sibling on the SSH server as a publication dependency" |
| 25 |
run "datalad create-sibling-github --github-organization datalad --publish-depends myserver --access-protocol ssh publish-demo" |
| 26 |
|
| 27 |
say "Let's quickly recap that the data file is just in our local dataset" |
| 28 |
run "git annex whereis" |
| 29 |
|
| 30 |
say "From now on, we can simply 'publish to GitHub' and DataLad will take care of the rest" |
| 31 |
run "datalad publish --to github --transfer-data all" |
| 32 |
|
| 33 |
say "We can confirm that our data file ended up on our server" |
| 34 |
run "git annex whereis big.dat" |
| 35 |
|
| 36 |
say "Now we simulate how it would look for a random person to obtain the data in this dataset from GitHub -- by installing into a new location, straight from GitHub (this does not require a GitHub account)" |
| 37 |
run "cd ../" |
| 38 |
run "datalad install -s [email protected]:datalad/publish-demo.git fromgh" |
| 39 |
run "cd fromgh" |
| 40 |
|
| 41 |
say "This dataset does not know how to access the original dataset location, only GitHub and our server" |
| 42 |
run "git remote -v" |
| 43 |
|
| 44 |
say "There is no further setup necessary, anyone can get the data -- if they have permission to access the URL of our own server" |
| 45 |
run "datalad get big.dat" |
| 46 |
run "ls -sLh big.dat" |
| 47 |
|
| 48 |
say "Go publish!" |