Debian Codegraph @

manual/develop.rst

raw
1 .. _develop:
2
3 Setting up a calibre development environment
4 ===========================================================
5
6 calibre is completely open source, licensed under the `GNU GPL v3 <https://www.gnu.org/licenses/gpl.html>`_.
7 This means that you are free to download and modify the program to your heart's content. In this section,
8 you will learn how to get a calibre development environment set up on the operating system of your choice.
9 calibre is written primarily in `Python <https://www.python.org>`_ with some C/C++ code for speed and system interfacing.
10 Note that calibre requires at least Python 3.8.
11
12 .. contents:: Contents
13 :depth: 2
14 :local:
15
16 Design philosophy
17 -------------------
18
19 calibre has its roots in the Unix world, which means that its design is highly modular.
20 The modules interact with each other via well defined interfaces. This makes adding new features and fixing
21 bugs in calibre very easy, resulting in a frenetic pace of development. Because of its roots, calibre has a
22 comprehensive command line interface for all its functions, documented in :doc:`generated/en/cli-index`.
23
24 The modular design of calibre is expressed via ``Plugins``. There is a :ref:`tutorial <customize>` on writing calibre plugins.
25 For example, adding support for a new device to calibre typically involves writing fewer than 100 lines of code in the form of
26 a device driver plugin. You can browse the
27 `built-in drivers <https://github.com/kovidgoyal/calibre/tree/master/src/calibre/devices>`_. Similarly, adding support
28 for new conversion formats involves writing input/output format plugins. Another example of the modular design is the :ref:`recipe system <news>` for
29 fetching news. For more examples of plugins designed to add features to calibre, see the `Index of plugins <https://www.mobileread.com/forums/showthread.php?p=1362767#post1362767>`_.
30
31 .. _code_layout:
32
33 Code layout
34 ^^^^^^^^^^^^^^
35
36 All the calibre Python code is in the ``calibre`` package. This package contains the following main sub-packages
37
38 * devices - All the device drivers. Just look through some of the built-in drivers to get an idea for how they work.
39
40 * For details, see: ``devices.interface`` which defines the interface supported by device drivers and ``devices.usbms`` which
41 defines a generic driver that connects to a USBMS device. All USBMS based drivers in calibre inherit from it.
42
43 * e-books - All the e-book conversion/metadata code. A good starting point is ``calibre.ebooks.conversion.cli`` which is the
44 module powering the :command:`ebook-convert` command. The conversion process is controlled via ``conversion.plumber``.
45 The format independent code is all in ``ebooks.oeb`` and the format dependent code is in ``ebooks.format_name``.
46
47 * Metadata reading, writing, and downloading is all in ``ebooks.metadata``
48 * Conversion happens in a pipeline, for the structure of the pipeline,
49 see :ref:`conversion-introduction`. The pipeline consists of an input
50 plugin, various transforms and an output plugin. The code that constructs
51 and drives the pipeline is in :file:`plumber.py`. The pipeline works on a
52 representation of an e-book that is like an unzipped epub, with
53 manifest, spine, toc, guide, html content, etc. The
54 class that manages this representation is OEBBook in ``ebooks.oeb.base``. The
55 various transformations that are applied to the book during
56 conversions live in :file:`oeb/transforms/*.py`. And the input and output
57 plugins live in :file:`conversion/plugins/*.py`.
58 * E-book editing happens using a different container object. It is
59 documented in :ref:`polish_api`.
60
61 * db - The database back-end. See :ref:`db_api` for the interface to the calibre library.
62
63 * Content server: ``srv`` is the calibre Content server.
64
65 * gui2 - The Graphical User Interface. GUI initialization happens in ``gui2.main`` and ``gui2.ui``. The e-book-viewer is in ``gui2.viewer``. The e-book editor is in ``gui2.tweak_book``.
66
67 If you want to locate the entry points for all the various calibre executables,
68 look at the ``entry_points`` structure in `linux.py
69 <https://github.com/kovidgoyal/calibre/blob/master/src/calibre/linux.py>`_.
70
71 If you need help understanding the code, post in the `development forum <https://www.mobileread.com/forums/forumdisplay.php?f=240>`_
72 and you will most likely get help from one of calibre's many developers.
73
74 Getting the code
75 ------------------
76
77 You can get the calibre source code in two ways, using a version control system or
78 directly downloading a :website_base:`tarball <dist/src>`.
79
80 calibre uses `Git <https://www.git-scm.com/>`_, a distributed version control
81 system. Git is available on all the platforms calibre supports. After
82 installing Git, you can get the calibre source code with the command::
83
84 git clone https://github.com/kovidgoyal/calibre.git
85
86 On Windows you will need the complete path name, that will be something like :file:`C:\\Program Files\\Git\\git.exe`.
87
88 calibre is a very large project with a very long source control history, so the
89 above can take a while (10 mins to an hour depending on your internet speed).
90
91 If you want to get the code faster, the source code for the latest release is
92 always available as an :website_base:`archive <dist/src>`.
93
94 To update a branch to the latest code, use the command::
95
96 git pull --no-edit
97
98 You can also browse the code at `GitHub <https://github.com/kovidgoyal/calibre>`_.
99
100 Submitting your changes to be included
101 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
102
103 If you only plan to make a few small changes, you can make your changes and
104 create a "merge directive" which you can then attach to a ticket in the calibre
105 `bug tracker <https://bugs.launchpad.net/calibre>`_. To do this, make your
106 changes, then run::
107
108 git commit -am "Comment describing your changes"
109 git format-patch origin/master --stdout > my-changes
110
111 This will create a :file:`my-changes` file in the current folder,
112 simply attach that to a ticket on the calibre `bug tracker <https://bugs.launchpad.net/calibre>`_.
113 Note that this will include *all* the commits you have made. If you only want
114 to send some commits, you have to change ``origin/master`` above. To send only
115 the last commit, use::
116
117 git format-patch HEAD~1 --stdout > my-changes
118
119 To send the last *n* commits, replace *1* with *n*, for example, for the last 3
120 commits::
121
122 git format-patch HEAD~3 --stdout > my-changes
123
124 Be careful to not include merges when using ``HEAD~n``.
125
126 If you plan to do a lot of development on calibre, then the best method is to create a
127 `GitHub <https://github.com>`__ account. Below is a basic guide to setting up
128 your own fork of calibre in a way that will allow you to submit pull requests
129 for inclusion into the main calibre repository:
130
131 * Setup git on your machine as described in this article: `Setup Git <https://help.github.com/articles/set-up-git>`_
132 * Setup ssh keys for authentication to GitHub, as described here: `Generating SSH keys <https://help.github.com/articles/generating-ssh-keys>`_
133 * Go to https://github.com/kovidgoyal/calibre and click the :guilabel:`Fork` button.
134 * In a Terminal do::
135
136 git clone [email protected]:<username>/calibre.git
137 git remote add upstream https://github.com/kovidgoyal/calibre.git
138
139 Replace <username> above with your GitHub username. That will get your fork checked out locally.
140 * You can make changes and commit them whenever you like. When you are ready to have your work merged, do a::
141
142 git push
143
144 and go to ``https://github.com/<username>/calibre`` and click the :guilabel:`Pull Request` button to generate a pull request that can be merged.
145 * You can update your local copy with code from the main repo at any time by doing::
146
147 git pull upstream
148
149
150 You should also keep an eye on the calibre `development forum
151 <https://www.mobileread.com/forums/forumdisplay.php?f=240>`_. Before making
152 major changes, you should discuss them in the forum or contact Kovid directly
153 (his email address is all over the source code).
154
155 Windows development environment
156 ---------------------------------
157
158 .. note:: You must also get the calibre source code separately as described above.
159
160 Install calibre normally, using the `Windows installer <https://calibre-ebook.com/download_windows>`__.
161 Then open a Command Prompt and change to the previously checked out calibre code folder. For example::
162
163 cd C:\Users\kovid\work\calibre
164
165 calibre is the folder that contains the src and resources sub-folders.
166
167 The next step is to set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src folder.
168 So, following the example above, it would be ``C:\Users\kovid\work\calibre\src``. `Here is a short
169 guide <https://docs.python.org/using/windows.html#excursus-setting-environment-variables>`_ to setting environment
170 variables on Windows.
171
172 Once you have set the environment variable, open a new command prompt and check that it was correctly set by using
173 the command::
174
175 echo %CALIBRE_DEVELOP_FROM%
176
177 Setting this environment variable means that calibre will now load all its Python code from the specified location.
178
179 That's it! You are now ready to start hacking on the calibre code. For example, open the file :file:`src\\calibre\\__init__.py`
180 in your favorite editor and add the line::
181
182 print("Hello, world!")
183
184 near the top of the file. Now run the command :command:`calibredb`. The very first line of output should be ``Hello, world!``.
185
186 You can also setup a calibre development environment inside the free Microsoft
187 Visual Studio, if you like, following the instructions `here <https://www.mobileread.com/forums/showthread.php?t=251201>`_.
188
189 macOS development environment
190 ------------------------------
191
192 .. note:: You must also get the calibre source code separately as described above.
193
194 Install calibre normally using the `provided .dmg <https://calibre-ebook.com/download_osx>`__.
195 Then open a Terminal and change to the previously checked out calibre code folder, for example::
196
197 cd /Users/kovid/work/calibre
198
199 calibre is the folder that contains the src and resources sub-folders.
200 The calibre command line tools are found inside the calibre app bundle, in
201 :file:`/Applications/calibre.app/Contents/MacOS`
202 you should add this folder to your PATH environment variable, if you want to
203 run the command line tools easily.
204
205 The next step is to create a bash script that will set the environment variable ``CALIBRE_DEVELOP_FROM``
206 to the absolute path of the src folder when running calibre in debug mode.
207
208 Create a plain text file::
209
210 #!/bin/sh
211 export CALIBRE_DEVELOP_FROM="/Users/kovid/work/calibre/src"
212 calibre-debug -g
213
214 Save this file as :file:`/usr/local/bin/calibre-develop`, then set its permissions so that it can be executed::
215
216 chmod +x /usr/local/bin/calibre-develop
217
218 Once you have done this, run::
219
220 calibre-develop
221
222 You should see some diagnostic information in the Terminal window as calibre
223 starts up, and you should see an asterisk after the version number in the GUI
224 window, indicating that you are running from source.
225
226 Linux development environment
227 ------------------------------
228
229 .. note:: You must also get the calibre source code separately as described above.
230
231 calibre is primarily developed on Linux. You have two choices in setting up the development environment. You can install the
232 calibre binary as normal and use that as a runtime environment to do your development. This approach is similar to that
233 used in Windows and macOS. Alternatively, you can install calibre from source. Instructions for setting up a development
234 environment from source are in the INSTALL file in the source tree. Here we will address using the binary as a runtime, which is the
235 recommended method.
236
237 Install calibre using the `binary installer <https://calibre-ebook.com/download_linux>`_. Then open a terminal and change to the previously checked out calibre code folder, for example::
238
239 cd /home/kovid/work/calibre
240
241 calibre is the folder that contains the src and resources sub-folders.
242
243 The next step is to set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src folder.
244 So, following the example above, it would be ``/home/kovid/work/calibre/src``. How to set environment variables depends on
245 your Linux distribution and what shell you are using.
246
247 .. note::
248 It is recommended to use the binary installer provided from upstream.
249 Should you insist on using a package provided by your distribution, use
250 the ``CALIBRE_PYTHON_PATH`` and ``CALIBRE_RESOURCES_PATH`` variables instead.
251 You can get these by running ``calibre-debug --paths``. But note that
252 distribution calibre packages are often terminally broken and are completely
253 unsupported.
254
255 Once you have set the environment variable, open a new terminal and check that it was correctly set by using
256 the command::
257
258 echo $CALIBRE_DEVELOP_FROM
259
260 Setting this environment variable means that calibre will now load all its Python code from the specified location.
261
262 That's it! You are now ready to start hacking on the calibre code. For example, open the file :file:`src/calibre/__init__.py`
263 in your favorite editor and add the line::
264
265 print("Hello, world!")
266
267 near the top of the file. Now run the command :command:`calibredb`. The very first line of output should be ``Hello, world!``.
268
269 Having separate "normal" and "development" calibre installs on the same computer
270 -----------------------------------------------------------------------------------------------------------------
271
272 The calibre source tree is very stable and rarely breaks, but if you feel the need to run from source on a separate
273 test library and run the released calibre version with your everyday library, you can achieve this easily using
274 .bat files or shell scripts to launch calibre. The example below shows how to do this on Windows using .bat files (the
275 instructions for other platforms are the same, just use a shell script instead of a .bat file)
276
277 To launch the release version of calibre with your everyday library:
278
279 calibre-normal.bat::
280
281 calibre.exe "--with-library=C:\path\to\everyday\library folder"
282
283 calibre-dev.bat::
284
285 set CALIBRE_DEVELOP_FROM=C:\path\to\calibre\checkout\src
286 calibre.exe "--with-library=C:\path\to\test\library folder"
287
288
289 Debugging tips
290 ----------------
291
292 Python is a
293 dynamically typed language with excellent facilities for introspection. Kovid wrote the core calibre code without once
294 using a debugger. There are many strategies to debug calibre code:
295
296 Using print statements
297 ^^^^^^^^^^^^^^^^^^^^^^^
298
299 This is Kovid's favorite way to debug. Simply insert print statements at points of interest and run your program in the
300 terminal. For example, you can start the GUI from the terminal as::
301
302 calibre-debug -g
303
304 Similarly, you can start the E-book viewer as::
305
306 calibre-debug -w /path/to/file/to/be/viewed
307
308 The e-book editor can be started as::
309
310 calibre-debug --edit-book /path/to/be/edited
311
312 Using an interactive Python interpreter
313 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
314
315 You can insert the following two lines of code to start an interactive Python session at that point::
316
317 from calibre import ipython
318 ipython(locals())
319
320 When running from the command line, this will start an interactive Python interpreter with access to all
321 locally defined variables (variables in the local scope). The interactive prompt even has :kbd:`Tab` completion
322 for object properties and you can use the various Python facilities for introspection, such as
323 :func:`dir`, :func:`type`, :func:`repr`, etc.
324
325 Using the Python debugger as a remote debugger
326 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
327
328 You can use the builtin Python debugger (pdb) as a remote debugger from the
329 command line. First, start the remote debugger at the point in the calibre code
330 you are interested in, like this::
331
332 from calibre.rpdb import set_trace
333 set_trace()
334
335 Then run calibre, either as normal, or using one of the calibre-debug commands
336 described in the previous section. Once the above point in the code is reached,
337 calibre will freeze, waiting for the debugger to connect.
338
339 Now open a terminal or command prompt and use the following command to start
340 the debugging session::
341
342 calibre-debug -c "from calibre.rpdb import cli; cli()"
343
344 You can read about how to use the Python debugger in the `Python stdlib docs
345 for the pdb module <https://docs.python.org/library/pdb.html#debugger-commands>`_.
346
347 .. note::
348 By default, the remote debugger will try to connect on port 4444. You can
349 change it, by passing the port parameter to both the set_trace() and the
350 cli() functions above, like this: ``set_trace(port=1234)`` and
351 ``cli(port=1234)``.
352
353 .. note::
354 The Python debugger cannot handle multiple threads, so you have to
355 call set_trace once per thread, each time with a different port number.
356
357 Using the debugger in your favorite Python IDE
358 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
359
360 It is possible to use the builtin debugger in your favorite Python IDE, if it
361 supports remote debugging. The first step is to add the calibre src checkout to
362 the ``PYTHONPATH`` in your IDE. In other words, the folder you set as
363 ``CALIBRE_DEVELOP_FROM`` above, must also be in the ``PYTHONPATH`` of your IDE.
364
365 Then place the IDE's remote debugger module into the :file:`src` sub-folder
366 of the calibre source code checkout. Add whatever code is needed to launch the
367 remote debugger to calibre at the point of interest, for example in the main
368 function. Then run calibre as normal. Your IDE should now be able to connect to
369 the remote debugger running inside calibre.
370
371 Executing arbitrary scripts in the calibre Python environment
372 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
373
374 The :command:`calibre-debug` command provides a couple of handy switches to execute your own
375 code, with access to the calibre modules::
376
377 calibre-debug -c "some Python code"
378
379 is great for testing a little snippet of code on the command line. It works in the same way as the -c switch to the Python interpreter::
380
381 calibre-debug myscript.py
382
383 can be used to execute your own Python script. It works in the same way as passing the script to the Python interpreter, except
384 that the calibre environment is fully initialized, so you can use all the calibre code in your script. To use command line arguments with your script, use the form::
385
386 calibre-debug myscript.py -- --option1 arg1
387
388 The ``--`` causes all subsequent arguments to be passed to your script.
389
390
391 Running the calibre test suite
392 ------------------------------------
393
394 The calibre test suite can be run by::
395
396 calibre-debug -t all
397
398
399 Using calibre in your projects
400 ----------------------------------------
401
402 It is possible to directly use calibre functions/code in your Python project. Two ways exist to do this:
403
404 Binary install of calibre
405 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
406
407 If you have a binary install of calibre, you can use the Python interpreter bundled with calibre, like this::
408
409 calibre-debug /path/to/your/python/script.py -- arguments to your script
410
411 Source install on Linux
412 ^^^^^^^^^^^^^^^^^^^^^^^^^^
413
414 In addition to using the above technique, if you do a source install on Linux,
415 you can also directly import calibre, as follows::
416
417 import init_calibre
418 import calibre
419
420 print(calibre.__version__)
421
422 It is essential that you import the init_calibre module before any other calibre modules/packages as
423 it sets up the interpreter to run calibre code.
424
425 API documentation for various parts of calibre
426 ------------------------------------------------
427
428 .. toctree::
429 :maxdepth: 1
430
431 news_recipe
432 plugins
433 db_api
434 polish