Debian Codegraph @

docs/intro.rst

raw
1 .. _`intro`:
2
3 ============
4 Introduction
5 ============
6
7
8 Let's start with an example. Imagine that you have several proxy servers and
9 you want to route requests to the right one. You could define the following rules:
10
11 * ``site1.com`` →︎ ``us_proxy``
12 * ``site2.com/uk`` →︎ ``uk_proxy``
13 * ``site2.com/ie`` →︎ ``ie_proxy``
14
15 All URLs from ``site1.com`` should use the US proxy. The situation for ``site2.com`` URLs are
16 different: if the path starts with ``/uk``, then the UK proxy should be used
17 whereas if the path starts with ``/ie`` then the IE proxy should be used instead.
18 This library allows to create a matcher that can be used to match URLs with the right proxy
19 using these rules.
20
21 Let see how the library can handle this situation:
22
23 .. code-block:: python
24
25 from url_matcher import URLMatcher, Patterns
26
27 matcher = URLMatcher()
28 matcher.add_or_update("us_proxy", Patterns(["site1.com"]))
29 matcher.add_or_update("uk_proxy", Patterns(["site2.com/uk"]))
30 matcher.add_or_update("ie_proxy", Patterns(["site2.com/ie"]))
31
32 proxy = matcher.match("http://site1.com/articles/article1")
33 # proxy is "us_proxy" here
34
35 proxy = matcher.match("http://site2.com/uk/a_page")
36 # proxy is "uk_proxy" here
37
38 proxy = matcher.match("https://www.site2.com/ie/a_page")
39 # proxy is "ie_proxy" here
40
41 proxy = matcher.match("http://example.com/a_differnt_page")
42 # proxy is None here
43
44 As can be seen the the class :class:`url_matcher.URLMatcher` is handy to handle
45 this use case.
46
47 .. note::
48 Relative URLs are not supported in the ``match`` method.
49
50 Patterns, include and exclude
51 =============================
52
53 A pattern is a URL that describes a set of URLs. For example, the
54 pattern ``example.com`` describes any URL whose domain is ``example.com``
55 or any of its subdomains.
56
57 A single pattern is sometimes not enough to describe which URLs to match.
58 This is why we can define instead a set of patterns that are matched against.
59 There is then a list of positive patterns (``include``) and a list of
60 negative ones (``exclude``).
61
62 **A URL is a match** if it matches **at least one** of the patterns in ``include`` and
63 **none** of the patterns in ``exclude``.
64
65 This is an example of a rule using such a set of patterns:
66
67 .. code-block:: python
68
69 patterns = Patterns(include=["example.com", "example.org"],
70 exclude=["*.jpg|", "*.jpeg|"])
71 matcher.add_or_update("proxy_1", patterns))
72
73 Patterns
74 ========
75
76 A pattern is a URL that describes a set of URLs. It itself is just a
77 URL. The following diagram summarizes its different parts and
78 what do they mean.
79
80 .. image:: patterns/patterns-cheatsheet.png
81 :alt: Patterns Cheatsheet
82
83 .. note::
84 Matching is always **case-insensitive**.
85
86 The best way to understand how the patterns work is to look at some examples:
87
88 Basic patterns
89 --------------
90
91 .. csv-table::
92 :file: patterns/basic_patterns.csv
93 :widths: 30, 70
94 :header-rows: 1
95
96 Domain patterns
97 ---------------
98
99 .. csv-table::
100 :file: patterns/domain_patterns.csv
101 :widths: 30, 70
102 :header-rows: 1
103
104 .. note::
105
106 Rules above only differ by the ``/`` character and this
107 is enough to change the matching behaviour. The
108 general rule is that the pattern matches the domain or any
109 of the subdomains only if the pattern does not contain a
110 path, a query or a fragment. Otherwise, only URLs with the exact
111 same domain after removing ``www.`` will match the pattern.
112
113 Path patterns
114 -------------
115
116 A URL matches if the pattern path is a prefix of it.
117
118 Besides, the following modifier characters can be used:
119
120 * The ``*`` character matches any number of characters.
121 * Use the ``|`` character at the end of the pattern path if
122 a exact path matching is required.
123
124 .. csv-table::
125 :file: patterns/path_patterns.csv
126 :widths: 30, 70
127 :header-rows: 1
128
129 Query patterns
130 --------------
131
132 It serves to match URLs that have some specific parameters in the URL.
133 The order of parameters in the query string is irrelevant.
134 The wildcard char ``*`` can be used for values.
135
136 If a parameter is repeated in the pattern it will match if any
137 of the values provided is matched
138
139 .. csv-table::
140 :file: patterns/query_patterns.csv
141 :widths: 30, 70
142 :header-rows: 1
143
144 Fragment patterns
145 -----------------
146
147 It works exactly like the path.
148
149
150 Rules conflict resolution
151 =========================
152
153 Sometimes several rules can match the same URL. We have then a conflict.
154 By default the library
155 will prioritize the most specific rule. For example, if
156 a URL is matching both a rule with a pattern ``example.com`` and another
157 with the pattern ``example.com/articles`` then the later one will
158 be final match because it is more specific.
159
160 Alternatively, it is possible to control manually the order of rules
161 by using the ``priority`` parameter of the :class:`url_matcher.Patterns`.
162 In case of conflict, the rule with the highest priority will be chosen.
163
164 The full criteria applied to resolve a conflict between rules are:
165
166 1. universality (rules with non universal include patterns are prioritized over rules with universal ones)
167 2. priority (the highest wins)
168 3. specificity (the most specific include patterns for the concerning domain wins)
169 4. the rule id (the rule with the highest id wins)
170
171 Efficiency
172 ==========
173
174 Internally, the library clusters the rules by the top level domain
175 of their include patterns. This is done to speed up the matching because
176 it reduces the space of possible rules that can match a URL.
177
178 The drawback is that the rules with ``include`` patterns that do not
179 belong to any top level domain are not supported. In fact, an
180 error is raised.
181
182 An exception were done for the universal matching pattern. It is
183 the only cross-top-level-domain ``include`` pattern that is allowed.
184 The rationale is that is can be convenient to define defaults (e.g.
185 to define the default proxy to use if no other rule matches).