| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_design_file_url_handling: |
| 5 |
|
| 6 |
***************** |
| 7 |
File URL handling |
| 8 |
***************** |
| 9 |
|
| 10 |
.. topic:: Specification scope and status |
| 11 |
|
| 12 |
This specification describes the current implementation. |
| 13 |
|
| 14 |
DataLad datasets can record URLs for file content access as metadata. This is a |
| 15 |
feature provided by git-annex and is available for any annexed file. DataLad |
| 16 |
improves upon the git-annex functionality in two ways: |
| 17 |
|
| 18 |
1. Support for a variety of (additional) protocols and authentication methods. |
| 19 |
|
| 20 |
2. Support for special URLs pointing to individual files located in registered |
| 21 |
(annexed) archives, such as tarballs and ZIP files. |
| 22 |
|
| 23 |
These additional features are available to all functionality that is processing |
| 24 |
URLs, such as ``get``, ``addurls``, or ``download-url``. |
| 25 |
|
| 26 |
|
| 27 |
Extensible protocol and authentication support |
| 28 |
============================================== |
| 29 |
|
| 30 |
DataLad ships with a dedicated implementation of an external `git-annex special |
| 31 |
remote`_ named ``git-annex-remote-datalad``. This is a somewhat atypical special |
| 32 |
remote, because it cannot receive files and store them, but only supports |
| 33 |
read operations. |
| 34 |
|
| 35 |
Specifically, it uses the ``CLAIMURL`` feature of the `external special remote |
| 36 |
protocol`_ to take over processing of URLs with supported protocols in all |
| 37 |
datasets that have this special remote configured and enabled. |
| 38 |
|
| 39 |
This special remote is automatically configured and enabled in DataLad dataset |
| 40 |
as a ``datalad`` remote, by commands that utilize its features, such as |
| 41 |
``download-url``. Once enabled, DataLad (but also git-annex) is able to act on |
| 42 |
additional protocols, such as ``s3://``, and the respective URLs can be given |
| 43 |
directly to commands like ``git annex addurl``, or ``datalad download-url``. |
| 44 |
|
| 45 |
Beyond additional protocol support, the ``datalad`` special remote also |
| 46 |
interfaces with DataLad's :ref:`chap_design_credentials`. It can identify a |
| 47 |
particular credential required for a given URL (based on something called a |
| 48 |
"provider" configuration), ask for the credential or retrieve it from a |
| 49 |
credential store, and supply it to the respective service in an appropriate |
| 50 |
form. Importantly, this feature neither requires the necessary credential or |
| 51 |
provider configuration to be encoded in a URL (where it would become part of |
| 52 |
the git-annex metadata), nor to be committed to a dataset. Hence all |
| 53 |
information that may depend on which entity is performing a URL request |
| 54 |
and in what environment is completely separated from the location information |
| 55 |
on a particular file content. This minimizes the required dataset maintenance |
| 56 |
effort (when credentials change), and offers a clean separation of identity |
| 57 |
and availability tracking vs. authentication management. |
| 58 |
|
| 59 |
|
| 60 |
Indexing and access of archive content |
| 61 |
====================================== |
| 62 |
|
| 63 |
Another `git-annex special remote`_, named |
| 64 |
``git-annex-remote-datalad-archives``, is used to enable file content retrieval |
| 65 |
from annexed archive files, such as tarballs and ZIP files. Its implementation |
| 66 |
concept is closely related to the ``git-annex-remote-datalad``, described |
| 67 |
above. Its main difference is that it claims responsibility for a particular |
| 68 |
type of "URL" (starting with ``dl+archive:``). These URLs encode the identity |
| 69 |
of an archive file, in terms of its git-annex key name, and a relative path |
| 70 |
inside this archive pointing to a particular file. |
| 71 |
|
| 72 |
Like ``git-annex-remote-datalad``, only read operations are supported. When |
| 73 |
a request to a ``dl+archive:`` "URL" is made, the special remote identifies |
| 74 |
the archive file, if necessary obtains it at the precise version needed, and |
| 75 |
extracts the respected file content from the archive at the correct location. |
| 76 |
|
| 77 |
This special remote is automatically configured and enabled as |
| 78 |
``datalad-archives`` by the ``add-archive-content`` command. This command |
| 79 |
indexes annexed archives, extracts, and registers their content to a |
| 80 |
dataset. File content availability information is recorded in terms of the |
| 81 |
``dl+archive:`` "URLs", which are put into the git-annex metadata on a file's |
| 82 |
content. |
| 83 |
|
| 84 |
|
| 85 |
.. _git-annex special remote: https://git-annex.branchable.com/special_remotes/ |
| 86 |
.. _external special remote protocol: https://git-annex.branchable.com/design/external_special_remote_protocol |