| 1 |
.. -*- mode: rst -*- |
| 2 |
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79: |
| 3 |
|
| 4 |
.. _chap_design_url_substitution: |
| 5 |
|
| 6 |
**************** |
| 7 |
URL substitution |
| 8 |
**************** |
| 9 |
|
| 10 |
.. topic:: Specification scope and status |
| 11 |
|
| 12 |
This specification describes the current implementation. This implementation |
| 13 |
is covering URL substitution in ``clone`` only. A further extension to |
| 14 |
URL processing elsewhere is possible. |
| 15 |
|
| 16 |
URL substitution is a transformation of a given URL using a set of |
| 17 |
specifications. Such specification can be provided as configuration settings |
| 18 |
(via all supported configuration sources). These configuration items must |
| 19 |
follow the naming scheme ``datalad.clone.url-substitute.<label>``, where |
| 20 |
``<label>`` is an arbitrary identifier. |
| 21 |
|
| 22 |
A substitution specification is a string with a match and substitution |
| 23 |
expression, each following Python's regular expression syntax. Both |
| 24 |
expressions are concatenated into a single string with an arbitrary delimiter |
| 25 |
character. The delimiter is defined by prefixing the string with the delimiter. |
| 26 |
Prefix and delimiter are stripped from the expressions before processing. |
| 27 |
Example:: |
| 28 |
|
| 29 |
,^http://(.*)$,https://\\1 |
| 30 |
|
| 31 |
A particular configuration item can be defined multiple times (see examples |
| 32 |
below) to form a substitution series. Substitutions in the same series will be |
| 33 |
applied incrementally, in order of their definition. If the first substitution |
| 34 |
expression does not match, the entire series will be ignored. However, |
| 35 |
following a first positive match all further substitutions in a series are |
| 36 |
processed, regardless whether intermediate expressions match or not. |
| 37 |
|
| 38 |
Any number of substitution series can be configured. They will be considered in |
| 39 |
no particular order. Consequently, it advisable to implement the first match |
| 40 |
specification of any series as specific as possible, in order to prevent |
| 41 |
undesired transformations. |
| 42 |
|
| 43 |
|
| 44 |
Examples |
| 45 |
======== |
| 46 |
|
| 47 |
Change the protocol component of a given URL in order to hand over further |
| 48 |
processing to a dedicated Git remote helper. Specifically, the following |
| 49 |
example converts Open Science Framework project URLs like |
| 50 |
``https://osf.io/f5j3e/`` into ``osf://f5j3e``, a URL that can be handle by |
| 51 |
``git-remote-osf``, the Git remote helper provided by the `datalad-osf |
| 52 |
extension package <https://github.com/datalad/datalad-osf>`__:: |
| 53 |
|
| 54 |
datalad.clone.url-substitute.osf = ,^https://osf.io/([^/]+)[/]*$,osf://\1 |
| 55 |
|
| 56 |
Here is a more complex examples with a series of substitutions. The first |
| 57 |
expression ensures that only GitHub URLs are being processed. The associated |
| 58 |
substitution disassembles the URL into its two only relevant components, |
| 59 |
the organisation/user name, and the project name:: |
| 60 |
|
| 61 |
datalad.clone.url-substitute.github = ,https?://github.com/([^/]+)/(.*)$,\1###\2 |
| 62 |
|
| 63 |
All other expressions in this series that are described below will only be considered |
| 64 |
if the above expression matched. |
| 65 |
|
| 66 |
The next two expressions in the series normalize URL components that maybe be |
| 67 |
auto-generated by some DataLad functionality, e.g. subdataset location |
| 68 |
candidate generation from directory names:: |
| 69 |
|
| 70 |
# replace (back)slashes with a single dash |
| 71 |
datalad.clone.url-substitute.github = ,[/\\]+,- |
| 72 |
|
| 73 |
# replace with whitespace (URL-quoted or not) with a single underscore |
| 74 |
datalad.clone.url-substitute.github = ,\s+|(%2520)+|(%20)+,_ |
| 75 |
|
| 76 |
The final expression in the series is recombining the organization/user name |
| 77 |
and project name components back into a complete URL:: |
| 78 |
|
| 79 |
datalad.clone.url-substitute.github = ,([^#]+)###(.*),https://github.com/\1/\2 |