Debian Codegraph @

docs/source/design/url_substitution.rst

raw
1 .. -*- mode: rst -*-
2 .. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
3
4 .. _chap_design_url_substitution:
5
6 ****************
7 URL substitution
8 ****************
9
10 .. topic:: Specification scope and status
11
12 This specification describes the current implementation. This implementation
13 is covering URL substitution in ``clone`` only. A further extension to
14 URL processing elsewhere is possible.
15
16 URL substitution is a transformation of a given URL using a set of
17 specifications. Such specification can be provided as configuration settings
18 (via all supported configuration sources). These configuration items must
19 follow the naming scheme ``datalad.clone.url-substitute.<label>``, where
20 ``<label>`` is an arbitrary identifier.
21
22 A substitution specification is a string with a match and substitution
23 expression, each following Python's regular expression syntax. Both
24 expressions are concatenated into a single string with an arbitrary delimiter
25 character. The delimiter is defined by prefixing the string with the delimiter.
26 Prefix and delimiter are stripped from the expressions before processing.
27 Example::
28
29 ,^http://(.*)$,https://\\1
30
31 A particular configuration item can be defined multiple times (see examples
32 below) to form a substitution series. Substitutions in the same series will be
33 applied incrementally, in order of their definition. If the first substitution
34 expression does not match, the entire series will be ignored. However,
35 following a first positive match all further substitutions in a series are
36 processed, regardless whether intermediate expressions match or not.
37
38 Any number of substitution series can be configured. They will be considered in
39 no particular order. Consequently, it advisable to implement the first match
40 specification of any series as specific as possible, in order to prevent
41 undesired transformations.
42
43
44 Examples
45 ========
46
47 Change the protocol component of a given URL in order to hand over further
48 processing to a dedicated Git remote helper. Specifically, the following
49 example converts Open Science Framework project URLs like
50 ``https://osf.io/f5j3e/`` into ``osf://f5j3e``, a URL that can be handle by
51 ``git-remote-osf``, the Git remote helper provided by the `datalad-osf
52 extension package <https://github.com/datalad/datalad-osf>`__::
53
54 datalad.clone.url-substitute.osf = ,^https://osf.io/([^/]+)[/]*$,osf://\1
55
56 Here is a more complex examples with a series of substitutions. The first
57 expression ensures that only GitHub URLs are being processed. The associated
58 substitution disassembles the URL into its two only relevant components,
59 the organisation/user name, and the project name::
60
61 datalad.clone.url-substitute.github = ,https?://github.com/([^/]+)/(.*)$,\1###\2
62
63 All other expressions in this series that are described below will only be considered
64 if the above expression matched.
65
66 The next two expressions in the series normalize URL components that maybe be
67 auto-generated by some DataLad functionality, e.g. subdataset location
68 candidate generation from directory names::
69
70 # replace (back)slashes with a single dash
71 datalad.clone.url-substitute.github = ,[/\\]+,-
72
73 # replace with whitespace (URL-quoted or not) with a single underscore
74 datalad.clone.url-substitute.github = ,\s+|(%2520)+|(%20)+,_
75
76 The final expression in the series is recombining the organization/user name
77 and project name components back into a complete URL::
78
79 datalad.clone.url-substitute.github = ,([^#]+)###(.*),https://github.com/\1/\2