Debian Codegraph @

manual/xpath.rst

raw
1 .. _xpath-tutorial:
2
3 XPath tutorial
4 ==============
5
6 In this tutorial, you will be given a gentle introduction to
7 `XPath <https://en.wikipedia.org/wiki/XPath>`_, a query language that can be
8 used to select arbitrary parts of `HTML <https://en.wikipedia.org/wiki/HTML>`_
9 documents in calibre. XPath is a widely
10 used standard, and googling it will yield a ton of information. This tutorial,
11 however, focuses on using XPath for e-book related tasks like finding chapter
12 headings in an unstructured HTML document.
13
14 .. contents:: Contents
15 :depth: 1
16 :local:
17
18 Selecting by tag name
19 ----------------------------------------
20
21 The simplest form of selection is to select tags by name. For example,
22 suppose you want to select all the ``<h2>`` tags in a document. The XPath
23 query for this is simply::
24
25 //h:h2 (Selects all <h2> tags)
26
27 The prefix `//` means *search at any level of the document*. Now suppose you
28 want to search for ``<span>`` tags that are inside ``<a>`` tags. That can be
29 achieved with::
30
31 //h:a/h:span (Selects <span> tags inside <a> tags)
32
33 If you want to search for tags at a particular level in the document, change
34 the prefix::
35
36 /h:body/h:div/h:p (Selects <p> tags that are children of <div> tags that are
37 children of the <body> tag)
38
39 This will match only ``<p>A very short e-book to demonstrate the use of XPath.</p>``
40 in the :ref:`sample_ebook` but not any of the other ``<p>`` tags. The ``h:`` prefix
41 in the above examples is needed to match XHTML tags. This is because internally,
42 calibre represents all content as XHTML. In XHTML tags have a *namespace*, and
43 ``h:`` is the namespace prefix for HTML tags.
44
45 Now suppose you want to select both ``<h1>`` and ``<h2>`` tags. To do that,
46 we need an XPath construct called *predicate*. A :dfn:`predicate` is simply
47 a test that is used to select tags. Tests can be arbitrarily powerful and as
48 this tutorial progresses, you will see more powerful examples. A predicate
49 is created by enclosing the test expression in square brackets::
50
51 //*[name()='h1' or name()='h2']
52
53 There are several new features in this XPath expression. The first is the use
54 of the wildcard ``*``. It means *match any tag*. Now look at the test expression
55 ``name()='h1' or name()='h2'``. :term:`name()` is an example of a *built-in function*.
56 It simply evaluates to the name of the tag. So by using it, we can select tags
57 whose names are either `h1` or `h2`. Note that the :term:`name()` function
58 ignores namespaces so that there is no need for the ``h:`` prefix.
59 XPath has several useful built-in functions. A few more will be introduced in this tutorial.
60
61 Selecting by attributes
62 -----------------------
63
64 To select tags based on their attributes, the use of predicates is required::
65
66 //*[@style] (Select all tags that have a style attribute)
67 //*[@class="chapter"] (Select all tags that have class="chapter")
68 //h:h1[@class="bookTitle"] (Select all h1 tags that have class="bookTitle")
69
70 Here, the ``@`` operator refers to the attributes of the tag. You can use some
71 of the `XPath built-in functions`_ to perform more sophisticated
72 matching on attribute values.
73
74 .. note::
75 When XPath expressions are used for conversion options such as chapter
76 detection or the start reading at position, they are evaluated against the
77 *input* document, before calibre flattens the CSS. CSS flattening rewrites
78 class names, so the class names you see in the converted output (things like
79 ``calibre1``) do not exist yet when your expression runs. Copy
80 class names from the input document, not from the converted book. If in
81 doubt, match on tag names and text content instead, as these are not changed
82 by the conversion.
83
84
85 Selecting by tag content
86 ------------------------
87
88 Using XPath, you can even select tags based on the text they contain. The best way to do this is
89 to use the power of *regular expressions* via the built-in function :term:`re:test()`::
90
91 //h:h2[re:test(., 'chapter|section', 'i')] (Selects <h2> tags that contain the words chapter or
92 section)
93
94 Here the ``.`` operator refers to the contents of the tag, just as the ``@`` operator referred
95 to its attributes.
96
97 .. _sample_ebook :
98
99 Sample e-book
100 -------------
101
102 .. literalinclude:: xpath.xhtml
103 :language: html
104
105 XPath built-in functions
106 ------------------------
107
108 .. glossary::
109
110 name()
111 The name of the current tag.
112
113 contains()
114 ``contains(s1, s2)`` returns `true` if s1 contains s2.
115
116 re:test()
117 ``re:test(src, pattern, flags)`` returns `true` if the string `src` matches the
118 regular expression `pattern`. A particularly useful flag is ``i``, it makes matching
119 case insensitive. A good primer on the syntax for regular expressions can be found
120 at `regexp syntax <https://docs.python.org/library/re.html>`_