Debian Codegraph @

docs/source/design/file_url_handling.rst

raw
1 .. -*- mode: rst -*-
2 .. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
3
4 .. _chap_design_file_url_handling:
5
6 *****************
7 File URL handling
8 *****************
9
10 .. topic:: Specification scope and status
11
12 This specification describes the current implementation.
13
14 DataLad datasets can record URLs for file content access as metadata. This is a
15 feature provided by git-annex and is available for any annexed file. DataLad
16 improves upon the git-annex functionality in two ways:
17
18 1. Support for a variety of (additional) protocols and authentication methods.
19
20 2. Support for special URLs pointing to individual files located in registered
21 (annexed) archives, such as tarballs and ZIP files.
22
23 These additional features are available to all functionality that is processing
24 URLs, such as ``get``, ``addurls``, or ``download-url``.
25
26
27 Extensible protocol and authentication support
28 ==============================================
29
30 DataLad ships with a dedicated implementation of an external `git-annex special
31 remote`_ named ``git-annex-remote-datalad``. This is a somewhat atypical special
32 remote, because it cannot receive files and store them, but only supports
33 read operations.
34
35 Specifically, it uses the ``CLAIMURL`` feature of the `external special remote
36 protocol`_ to take over processing of URLs with supported protocols in all
37 datasets that have this special remote configured and enabled.
38
39 This special remote is automatically configured and enabled in DataLad dataset
40 as a ``datalad`` remote, by commands that utilize its features, such as
41 ``download-url``. Once enabled, DataLad (but also git-annex) is able to act on
42 additional protocols, such as ``s3://``, and the respective URLs can be given
43 directly to commands like ``git annex addurl``, or ``datalad download-url``.
44
45 Beyond additional protocol support, the ``datalad`` special remote also
46 interfaces with DataLad's :ref:`chap_design_credentials`. It can identify a
47 particular credential required for a given URL (based on something called a
48 "provider" configuration), ask for the credential or retrieve it from a
49 credential store, and supply it to the respective service in an appropriate
50 form. Importantly, this feature neither requires the necessary credential or
51 provider configuration to be encoded in a URL (where it would become part of
52 the git-annex metadata), nor to be committed to a dataset. Hence all
53 information that may depend on which entity is performing a URL request
54 and in what environment is completely separated from the location information
55 on a particular file content. This minimizes the required dataset maintenance
56 effort (when credentials change), and offers a clean separation of identity
57 and availability tracking vs. authentication management.
58
59
60 Indexing and access of archive content
61 ======================================
62
63 Another `git-annex special remote`_, named
64 ``git-annex-remote-datalad-archives``, is used to enable file content retrieval
65 from annexed archive files, such as tarballs and ZIP files. Its implementation
66 concept is closely related to the ``git-annex-remote-datalad``, described
67 above. Its main difference is that it claims responsibility for a particular
68 type of "URL" (starting with ``dl+archive:``). These URLs encode the identity
69 of an archive file, in terms of its git-annex key name, and a relative path
70 inside this archive pointing to a particular file.
71
72 Like ``git-annex-remote-datalad``, only read operations are supported. When
73 a request to a ``dl+archive:`` "URL" is made, the special remote identifies
74 the archive file, if necessary obtains it at the precise version needed, and
75 extracts the respected file content from the archive at the correct location.
76
77 This special remote is automatically configured and enabled as
78 ``datalad-archives`` by the ``add-archive-content`` command. This command
79 indexes annexed archives, extracts, and registers their content to a
80 dataset. File content availability information is recorded in terms of the
81 ``dl+archive:`` "URLs", which are put into the git-annex metadata on a file's
82 content.
83
84
85 .. _git-annex special remote: https://git-annex.branchable.com/special_remotes/
86 .. _external special remote protocol: https://git-annex.branchable.com/design/external_special_remote_protocol