| 1 |
Function mode for Search & replace in the Editor |
| 2 |
================================================ |
| 3 |
|
| 4 |
The :guilabel:`Search & replace` tool in the editor supports a *function mode*. |
| 5 |
In this mode, you can combine regular expressions (see :doc:`regexp`) with |
| 6 |
arbitrarily powerful Python functions to do all sorts of advanced text |
| 7 |
processing. |
| 8 |
|
| 9 |
In the standard *regexp* mode for search and replace, you specify both a |
| 10 |
regular expression to search for as well as a template that is used to replace |
| 11 |
all found matches. In function mode, instead of using a fixed template, you |
| 12 |
specify an arbitrary function, in the |
| 13 |
`Python programming language <https://docs.python.org>`_. This allows |
| 14 |
you to do lots of things that are not possible with simple templates. |
| 15 |
|
| 16 |
Techniques for using function mode and the syntax will be described by means of |
| 17 |
examples, showing you how to create functions to perform progressively more |
| 18 |
complex tasks. |
| 19 |
|
| 20 |
|
| 21 |
.. image:: images/function_replace.png |
| 22 |
:alt: The Function mode |
| 23 |
:align: center |
| 24 |
|
| 25 |
Automatically fixing the case of headings in the document |
| 26 |
--------------------------------------------------------- |
| 27 |
|
| 28 |
Here, we will leverage one of the builtin functions in the editor to |
| 29 |
automatically change the case of all text inside heading tags to title case:: |
| 30 |
|
| 31 |
Find expression: <([Hh][1-6])[^>]*>.+?</\1> |
| 32 |
|
| 33 |
For the function, simply choose the :guilabel:`Title-case text (ignore tags)` builtin |
| 34 |
function. This will change titles that look like: ``<h1>some titLE</h1>`` to |
| 35 |
``<h1>Some Title</h1>``. It will work even if there are other HTML tags inside |
| 36 |
the heading tags. |
| 37 |
|
| 38 |
|
| 39 |
Your first custom function - smartening hyphens |
| 40 |
----------------------------------------------- |
| 41 |
|
| 42 |
The real power of function mode comes from being able to create your own |
| 43 |
functions to process text in arbitrary ways. The Smarten Punctuation tool in |
| 44 |
the editor leaves individual hyphens alone, so you can use this function to |
| 45 |
replace them with em-dashes. |
| 46 |
|
| 47 |
To create a new function, simply click the :guilabel:`Create/edit` button to create a new |
| 48 |
function and copy the Python code from below. |
| 49 |
|
| 50 |
.. code-block:: python |
| 51 |
|
| 52 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 53 |
return match.group().replace('--', '—').replace('-', '—') |
| 54 |
|
| 55 |
Every :guilabel:`Search & replace` custom function must have a unique name and consist of a |
| 56 |
Python function named replace, that accepts all the arguments shown above. |
| 57 |
For the moment, we won't worry about all the different arguments to |
| 58 |
``replace()`` function. Just focus on the ``match`` argument. It represents a |
| 59 |
match when running a search and replace. Its full documentation in available |
| 60 |
`here <https://docs.python.org/library/re.html#match-objects>`_. |
| 61 |
``match.group()`` simply returns all the matched text and all we do is replace |
| 62 |
hyphens in that text with em-dashes, first replacing double hyphens and |
| 63 |
then single hyphens. |
| 64 |
|
| 65 |
Use this function with the find regular expression:: |
| 66 |
|
| 67 |
>[^<>]+< |
| 68 |
|
| 69 |
And it will replace all hyphens with em-dashes, but only in actual text and not |
| 70 |
inside HTML tag definitions. |
| 71 |
|
| 72 |
|
| 73 |
The power of function mode - using a spelling dictionary to fix mis-hyphenated words |
| 74 |
------------------------------------------------------------------------------------ |
| 75 |
|
| 76 |
Often, e-books created from scans of printed books contain mis-hyphenated words |
| 77 |
-- words that were split at the end of the line on the printed page. We will |
| 78 |
write a simple function to automatically find and fix such words. |
| 79 |
|
| 80 |
.. code-block:: python |
| 81 |
|
| 82 |
import regex |
| 83 |
from calibre import replace_entities |
| 84 |
from calibre import prepare_string_for_xml |
| 85 |
|
| 86 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 87 |
|
| 88 |
def replace_word(wmatch): |
| 89 |
# Try to remove the hyphen and replace the words if the resulting |
| 90 |
# hyphen free word is recognized by the dictionary |
| 91 |
without_hyphen = wmatch.group(1) + wmatch.group(2) |
| 92 |
if dictionaries.recognized(without_hyphen): |
| 93 |
return without_hyphen |
| 94 |
return wmatch.group() |
| 95 |
|
| 96 |
# Search for words split by a hyphen |
| 97 |
text = replace_entities(match.group()[1:-1]) # Handle HTML entities like & |
| 98 |
corrected = regex.sub(r'(\w+)\s*-\s*(\w+)', replace_word, text, flags=regex.VERSION1 | regex.UNICODE) |
| 99 |
return '>%s<' % prepare_string_for_xml(corrected) # Put back required entities |
| 100 |
|
| 101 |
Use this function with the same find expression as before, namely:: |
| 102 |
|
| 103 |
>[^<>]+< |
| 104 |
|
| 105 |
And it will magically fix all mis-hyphenated words in the text of the book. The |
| 106 |
main trick is to use one of the useful extra arguments to the replace function, |
| 107 |
``dictionaries``. This refers to the dictionaries the editor itself uses to |
| 108 |
spell check text in the book. What this function does is look for words |
| 109 |
separated by a hyphen, remove the hyphen and check if the dictionary recognizes |
| 110 |
the composite word, if it does, the original words are replaced by the hyphen |
| 111 |
free composite word. |
| 112 |
|
| 113 |
Note that one limitation of this technique is it will only work for |
| 114 |
mono-lingual books, because, by default, ``dictionaries.recognized()`` uses the |
| 115 |
main language of the book. |
| 116 |
|
| 117 |
|
| 118 |
Auto numbering sections |
| 119 |
----------------------- |
| 120 |
|
| 121 |
Now we will see something a little different. Suppose your HTML file has many |
| 122 |
sections, each with a heading in an :code:`<h2>` tag that looks like |
| 123 |
:code:`<h2>Some text</h2>`. You can create a custom function that will |
| 124 |
automatically number these headings with consecutive section numbers, so that |
| 125 |
they look like :code:`<h2>1. Some text</h2>`. |
| 126 |
|
| 127 |
.. code-block:: python |
| 128 |
|
| 129 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 130 |
section_number = '%d. ' % number |
| 131 |
return match.group(1) + section_number + match.group(2) |
| 132 |
|
| 133 |
# Ensure that when running over multiple files, the files are processed |
| 134 |
# in the order in which they appear in the book |
| 135 |
replace.file_order = 'spine' |
| 136 |
|
| 137 |
Use it with the find expression:: |
| 138 |
|
| 139 |
(?s)(<h2[^<>]*>)(.+?</h2>) |
| 140 |
|
| 141 |
Place the cursor at the top of the file and click :guilabel:`Replace all`. |
| 142 |
|
| 143 |
This function uses another of the useful extra arguments to ``replace()``: the |
| 144 |
``number`` argument. When doing a :guilabel:`Replace All` number is |
| 145 |
automatically incremented for every successive match. |
| 146 |
|
| 147 |
Another new feature is the use of ``replace.file_order`` -- setting that to |
| 148 |
``'spine'`` means that if this search is run on multiple HTML files, the files |
| 149 |
are processed in the order in which they appear in the book. See |
| 150 |
:ref:`file_order_replace_all` for details. |
| 151 |
|
| 152 |
|
| 153 |
Auto create a Table of Contents |
| 154 |
------------------------------- |
| 155 |
|
| 156 |
Finally, let's try something a little more ambitious. Suppose your book has |
| 157 |
headings in ``h1`` and ``h2`` tags that look like |
| 158 |
``<h1 id="someid">Some Text</h1>``. We will auto-generate an HTML Table of |
| 159 |
Contents based on these headings. Create the custom function below: |
| 160 |
|
| 161 |
.. code-block:: python |
| 162 |
|
| 163 |
from calibre import replace_entities |
| 164 |
from calibre.ebooks.oeb.polish.toc import TOC, toc_to_html |
| 165 |
from calibre.gui2.tweak_book import current_container |
| 166 |
from calibre.ebooks.oeb.base import xml2str |
| 167 |
|
| 168 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 169 |
if match is None: |
| 170 |
# All matches found, output the resulting Table of Contents. |
| 171 |
# The argument metadata is the metadata of the book being edited |
| 172 |
if 'toc' in data: |
| 173 |
toc = data['toc'] |
| 174 |
root = TOC() |
| 175 |
for (file_name, tag_name, anchor, text) in toc: |
| 176 |
parent = root.children[-1] if tag_name == 'h2' and root.children else root |
| 177 |
parent.add(text, file_name, anchor) |
| 178 |
toc = toc_to_html(root, current_container(), 'toc.html', 'Table of Contents for ' + metadata.title, metadata.language) |
| 179 |
print(xml2str(toc)) |
| 180 |
else: |
| 181 |
print('No headings to build ToC from found') |
| 182 |
else: |
| 183 |
# Add an entry corresponding to this match to the Table of Contents |
| 184 |
if 'toc' not in data: |
| 185 |
# The entries are stored in the data object, which will persist |
| 186 |
# for all invocations of this function during a 'Replace All' operation |
| 187 |
data['toc'] = [] |
| 188 |
tag_name, anchor, text = match.group(1), replace_entities(match.group(2)), replace_entities(match.group(3)) |
| 189 |
data['toc'].append((file_name, tag_name, anchor, text)) |
| 190 |
return match.group() # We don't want to make any actual changes, so return the original matched text |
| 191 |
|
| 192 |
# Ensure that we are called once after the last match is found so we can |
| 193 |
# output the ToC |
| 194 |
replace.call_after_last_match = True |
| 195 |
# Ensure that when running over multiple files, this function is called, |
| 196 |
# the files are processed in the order in which they appear in the book |
| 197 |
replace.file_order = 'spine' |
| 198 |
|
| 199 |
And use it with the find expression:: |
| 200 |
|
| 201 |
<(h[12]) [^<>]* id=['"]([^'"]+)['"][^<>]*>([^<>]+) |
| 202 |
|
| 203 |
Run the search on :guilabel:`All text files` and at the end of the search, a |
| 204 |
window will popup with "Debug output from your function" which will have the |
| 205 |
HTML Table of Contents, ready to be pasted into :file:`toc.html`. |
| 206 |
|
| 207 |
The function above is heavily commented, so it should be easy to follow. The |
| 208 |
key new feature is the use of another useful extra argument to the |
| 209 |
``replace()`` function, the ``data`` object. The ``data`` object is a Python |
| 210 |
*dictionary* that persists between all successive invocations of ``replace()`` during |
| 211 |
a single :guilabel:`Replace All` operation. |
| 212 |
|
| 213 |
Another new feature is the use of ``call_after_last_match`` -- setting that to |
| 214 |
``True`` on the ``replace()`` function means that the editor will call |
| 215 |
``replace()`` one extra time after all matches have been found. For this extra |
| 216 |
call, the match object will be ``None``. |
| 217 |
|
| 218 |
This was just a demonstration to show you the power of function mode, |
| 219 |
if you really needed to generate a Table of Contents from headings in your book, |
| 220 |
you would be better off using the dedicated Table of Contents tool in |
| 221 |
:guilabel:`Tools->Table of Contents`. |
| 222 |
|
| 223 |
The API for the function mode |
| 224 |
----------------------------- |
| 225 |
|
| 226 |
All function mode functions must be Python functions named replace, with the |
| 227 |
following signature:: |
| 228 |
|
| 229 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 230 |
return a_string |
| 231 |
|
| 232 |
When a find/replace is run, for every match that is found, the ``replace()`` |
| 233 |
function will be called, it must return the replacement string for that match. |
| 234 |
If no replacements are to be done, it should return ``match.group()`` which is |
| 235 |
the original string. The various arguments to the ``replace()`` function are |
| 236 |
documented below. |
| 237 |
|
| 238 |
The ``match`` argument |
| 239 |
^^^^^^^^^^^^^^^^^^^^^^ |
| 240 |
|
| 241 |
The ``match`` argument represents the currently found match. It is a |
| 242 |
`Python Match object <https://docs.python.org/library/re.html#match-objects>`_. |
| 243 |
Its most useful method is ``group()`` which can be used to get the matched |
| 244 |
text corresponding to individual capture groups in the search regular |
| 245 |
expression. |
| 246 |
|
| 247 |
The ``number`` argument |
| 248 |
^^^^^^^^^^^^^^^^^^^^^^^ |
| 249 |
|
| 250 |
The ``number`` argument is the number of the current match. When you run |
| 251 |
:guilabel:`Replace All`, every successive match will cause ``replace()`` to be |
| 252 |
called with an increasing number. The first match has number 1. |
| 253 |
|
| 254 |
The ``file_name`` argument |
| 255 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 256 |
|
| 257 |
This is the filename of the file in which the current match was found. When |
| 258 |
searching inside marked text, the ``file_name`` is empty. The ``file_name`` is |
| 259 |
in canonical form, a path relative to the root of the book, using ``/`` as the |
| 260 |
path separator. |
| 261 |
|
| 262 |
The ``metadata`` argument |
| 263 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 264 |
|
| 265 |
This represents the metadata of the current book, such as title, authors, |
| 266 |
language, etc. It is an object of class :class:`calibre.ebooks.metadata.book.base.Metadata`. |
| 267 |
Useful attributes include, ``title``, ``authors`` (a list of authors) and |
| 268 |
``language`` (the language code). |
| 269 |
|
| 270 |
The ``dictionaries`` argument |
| 271 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 272 |
|
| 273 |
This represents the collection of dictionaries used for spell checking the |
| 274 |
current book. Its most useful method is ``dictionaries.recognized(word)`` |
| 275 |
which will return ``True`` if the passed in word is recognized by the dictionary |
| 276 |
for the current book's language. |
| 277 |
|
| 278 |
The ``data`` argument |
| 279 |
^^^^^^^^^^^^^^^^^^^^^ |
| 280 |
|
| 281 |
This a simple Python ``dictionary``. When you run |
| 282 |
:guilabel:`Replace all`, every successive match will cause ``replace()`` to be |
| 283 |
called with the same ``dictionary`` as data. You can thus use it to store arbitrary |
| 284 |
data between invocations of ``replace()`` during a :guilabel:`Replace all` |
| 285 |
operation. |
| 286 |
|
| 287 |
The ``functions`` argument |
| 288 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 289 |
|
| 290 |
The ``functions`` argument gives you access to all other user defined |
| 291 |
functions. This is useful for code re-use. You can define utility functions in |
| 292 |
one place and re-use them in all your other functions. For example, suppose you |
| 293 |
create a function named ``My Function`` like this: |
| 294 |
|
| 295 |
.. code-block:: python |
| 296 |
|
| 297 |
def utility(): |
| 298 |
# do something |
| 299 |
|
| 300 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 301 |
... |
| 302 |
|
| 303 |
Then, in another function, you can access the ``utility()`` function like this: |
| 304 |
|
| 305 |
.. code-block:: python |
| 306 |
|
| 307 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 308 |
utility = functions['My Function']['utility'] |
| 309 |
... |
| 310 |
|
| 311 |
You can also use the functions object to store persistent data, that can be |
| 312 |
re-used by other functions. For example, you could have one function that when |
| 313 |
run with :guilabel:`Replace All` collects some data and another function that |
| 314 |
uses it when it is run afterwards. Consider the following two functions: |
| 315 |
|
| 316 |
.. code-block:: python |
| 317 |
|
| 318 |
# Function One |
| 319 |
persistent_data = {} |
| 320 |
|
| 321 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 322 |
... |
| 323 |
persistent_data['something'] = 'some data' |
| 324 |
|
| 325 |
# Function Two |
| 326 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 327 |
persistent_data = functions['Function One']['persistent_data'] |
| 328 |
... |
| 329 |
|
| 330 |
Debugging your functions |
| 331 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
| 332 |
|
| 333 |
You can debug the functions you create by using the standard ``print()`` |
| 334 |
function from Python. The output of print will be displayed in a popup window |
| 335 |
after the Find/replace has completed. You saw an example of using ``print()`` |
| 336 |
to output an entire table of contents above. |
| 337 |
|
| 338 |
.. _file_order_replace_all: |
| 339 |
|
| 340 |
Choose file order when running on multiple HTML files |
| 341 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 342 |
|
| 343 |
When you run a :guilabel:`Replace all` on multiple HTML files, the order in |
| 344 |
which the files are processed depends on what files you have open for editing. |
| 345 |
You can force the search to process files in the order in which they appear by |
| 346 |
setting the ``file_order`` attribute on your function, like this: |
| 347 |
|
| 348 |
.. code-block:: python |
| 349 |
|
| 350 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 351 |
... |
| 352 |
|
| 353 |
replace.file_order = 'spine' |
| 354 |
|
| 355 |
``file_order`` accepts two values, ``spine`` and ``spine-reverse`` which cause |
| 356 |
the search to process multiple files in the order they appear in the book, |
| 357 |
either forwards or backwards, respectively. |
| 358 |
|
| 359 |
Having your function called an extra time after the last match is found |
| 360 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 361 |
|
| 362 |
Sometimes, as in the auto generate table of contents example above, it is |
| 363 |
useful to have your function called an extra time after the last match is |
| 364 |
found. You can do this by setting the ``call_after_last_match`` attribute on your |
| 365 |
function, like this: |
| 366 |
|
| 367 |
.. code-block:: python |
| 368 |
|
| 369 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 370 |
... |
| 371 |
|
| 372 |
replace.call_after_last_match = True |
| 373 |
|
| 374 |
|
| 375 |
Appending the output from the function to marked text |
| 376 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 377 |
|
| 378 |
When running search and replace on marked text, it is sometimes useful to |
| 379 |
append some text to the end of the marked text. You can do that by setting |
| 380 |
the ``append_final_output_to_marked`` attribute on your function (note that you |
| 381 |
also need to set ``call_after_last_match``), like this: |
| 382 |
|
| 383 |
.. code-block:: python |
| 384 |
|
| 385 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 386 |
... |
| 387 |
return 'some text to append' |
| 388 |
|
| 389 |
replace.call_after_last_match = True |
| 390 |
replace.append_final_output_to_marked = True |
| 391 |
|
| 392 |
Suppressing the result dialog when performing searches on marked text |
| 393 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 394 |
|
| 395 |
You can also suppress the result dialog (which can slow down the repeated |
| 396 |
application of a search/replace on many blocks of text) by setting |
| 397 |
the ``suppress_result_dialog`` attribute on your function, like this: |
| 398 |
|
| 399 |
.. code-block:: python |
| 400 |
|
| 401 |
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs): |
| 402 |
... |
| 403 |
|
| 404 |
replace.suppress_result_dialog = True |
| 405 |
|
| 406 |
|
| 407 |
More examples |
| 408 |
---------------- |
| 409 |
|
| 410 |
More useful examples, contributed by calibre users, can be found in the |
| 411 |
`calibre E-book editor forum <https://www.mobileread.com/forums/showthread.php?t=237181>`_. |