| 1 |
.. _xpath-tutorial: |
| 2 |
|
| 3 |
XPath tutorial |
| 4 |
============== |
| 5 |
|
| 6 |
In this tutorial, you will be given a gentle introduction to |
| 7 |
`XPath <https://en.wikipedia.org/wiki/XPath>`_, a query language that can be |
| 8 |
used to select arbitrary parts of `HTML <https://en.wikipedia.org/wiki/HTML>`_ |
| 9 |
documents in calibre. XPath is a widely |
| 10 |
used standard, and googling it will yield a ton of information. This tutorial, |
| 11 |
however, focuses on using XPath for e-book related tasks like finding chapter |
| 12 |
headings in an unstructured HTML document. |
| 13 |
|
| 14 |
.. contents:: Contents |
| 15 |
:depth: 1 |
| 16 |
:local: |
| 17 |
|
| 18 |
Selecting by tag name |
| 19 |
---------------------------------------- |
| 20 |
|
| 21 |
The simplest form of selection is to select tags by name. For example, |
| 22 |
suppose you want to select all the ``<h2>`` tags in a document. The XPath |
| 23 |
query for this is simply:: |
| 24 |
|
| 25 |
//h:h2 (Selects all <h2> tags) |
| 26 |
|
| 27 |
The prefix `//` means *search at any level of the document*. Now suppose you |
| 28 |
want to search for ``<span>`` tags that are inside ``<a>`` tags. That can be |
| 29 |
achieved with:: |
| 30 |
|
| 31 |
//h:a/h:span (Selects <span> tags inside <a> tags) |
| 32 |
|
| 33 |
If you want to search for tags at a particular level in the document, change |
| 34 |
the prefix:: |
| 35 |
|
| 36 |
/h:body/h:div/h:p (Selects <p> tags that are children of <div> tags that are |
| 37 |
children of the <body> tag) |
| 38 |
|
| 39 |
This will match only ``<p>A very short e-book to demonstrate the use of XPath.</p>`` |
| 40 |
in the :ref:`sample_ebook` but not any of the other ``<p>`` tags. The ``h:`` prefix |
| 41 |
in the above examples is needed to match XHTML tags. This is because internally, |
| 42 |
calibre represents all content as XHTML. In XHTML tags have a *namespace*, and |
| 43 |
``h:`` is the namespace prefix for HTML tags. |
| 44 |
|
| 45 |
Now suppose you want to select both ``<h1>`` and ``<h2>`` tags. To do that, |
| 46 |
we need an XPath construct called *predicate*. A :dfn:`predicate` is simply |
| 47 |
a test that is used to select tags. Tests can be arbitrarily powerful and as |
| 48 |
this tutorial progresses, you will see more powerful examples. A predicate |
| 49 |
is created by enclosing the test expression in square brackets:: |
| 50 |
|
| 51 |
//*[name()='h1' or name()='h2'] |
| 52 |
|
| 53 |
There are several new features in this XPath expression. The first is the use |
| 54 |
of the wildcard ``*``. It means *match any tag*. Now look at the test expression |
| 55 |
``name()='h1' or name()='h2'``. :term:`name()` is an example of a *built-in function*. |
| 56 |
It simply evaluates to the name of the tag. So by using it, we can select tags |
| 57 |
whose names are either `h1` or `h2`. Note that the :term:`name()` function |
| 58 |
ignores namespaces so that there is no need for the ``h:`` prefix. |
| 59 |
XPath has several useful built-in functions. A few more will be introduced in this tutorial. |
| 60 |
|
| 61 |
Selecting by attributes |
| 62 |
----------------------- |
| 63 |
|
| 64 |
To select tags based on their attributes, the use of predicates is required:: |
| 65 |
|
| 66 |
//*[@style] (Select all tags that have a style attribute) |
| 67 |
//*[@class="chapter"] (Select all tags that have class="chapter") |
| 68 |
//h:h1[@class="bookTitle"] (Select all h1 tags that have class="bookTitle") |
| 69 |
|
| 70 |
Here, the ``@`` operator refers to the attributes of the tag. You can use some |
| 71 |
of the `XPath built-in functions`_ to perform more sophisticated |
| 72 |
matching on attribute values. |
| 73 |
|
| 74 |
.. note:: |
| 75 |
When XPath expressions are used for conversion options such as chapter |
| 76 |
detection or the start reading at position, they are evaluated against the |
| 77 |
*input* document, before calibre flattens the CSS. CSS flattening rewrites |
| 78 |
class names, so the class names you see in the converted output (things like |
| 79 |
``calibre1``) do not exist yet when your expression runs. Copy |
| 80 |
class names from the input document, not from the converted book. If in |
| 81 |
doubt, match on tag names and text content instead, as these are not changed |
| 82 |
by the conversion. |
| 83 |
|
| 84 |
|
| 85 |
Selecting by tag content |
| 86 |
------------------------ |
| 87 |
|
| 88 |
Using XPath, you can even select tags based on the text they contain. The best way to do this is |
| 89 |
to use the power of *regular expressions* via the built-in function :term:`re:test()`:: |
| 90 |
|
| 91 |
//h:h2[re:test(., 'chapter|section', 'i')] (Selects <h2> tags that contain the words chapter or |
| 92 |
section) |
| 93 |
|
| 94 |
Here the ``.`` operator refers to the contents of the tag, just as the ``@`` operator referred |
| 95 |
to its attributes. |
| 96 |
|
| 97 |
.. _sample_ebook : |
| 98 |
|
| 99 |
Sample e-book |
| 100 |
------------- |
| 101 |
|
| 102 |
.. literalinclude:: xpath.xhtml |
| 103 |
:language: html |
| 104 |
|
| 105 |
XPath built-in functions |
| 106 |
------------------------ |
| 107 |
|
| 108 |
.. glossary:: |
| 109 |
|
| 110 |
name() |
| 111 |
The name of the current tag. |
| 112 |
|
| 113 |
contains() |
| 114 |
``contains(s1, s2)`` returns `true` if s1 contains s2. |
| 115 |
|
| 116 |
re:test() |
| 117 |
``re:test(src, pattern, flags)`` returns `true` if the string `src` matches the |
| 118 |
regular expression `pattern`. A particularly useful flag is ``i``, it makes matching |
| 119 |
case insensitive. A good primer on the syntax for regular expressions can be found |
| 120 |
at `regexp syntax <https://docs.python.org/library/re.html>`_ |