| 1 |
.. _develop: |
| 2 |
|
| 3 |
Setting up a calibre development environment |
| 4 |
=========================================================== |
| 5 |
|
| 6 |
calibre is completely open source, licensed under the `GNU GPL v3 <https://www.gnu.org/licenses/gpl.html>`_. |
| 7 |
This means that you are free to download and modify the program to your heart's content. In this section, |
| 8 |
you will learn how to get a calibre development environment set up on the operating system of your choice. |
| 9 |
calibre is written primarily in `Python <https://www.python.org>`_ with some C/C++ code for speed and system interfacing. |
| 10 |
Note that calibre requires at least Python 3.8. |
| 11 |
|
| 12 |
.. contents:: Contents |
| 13 |
:depth: 2 |
| 14 |
:local: |
| 15 |
|
| 16 |
Design philosophy |
| 17 |
------------------- |
| 18 |
|
| 19 |
calibre has its roots in the Unix world, which means that its design is highly modular. |
| 20 |
The modules interact with each other via well defined interfaces. This makes adding new features and fixing |
| 21 |
bugs in calibre very easy, resulting in a frenetic pace of development. Because of its roots, calibre has a |
| 22 |
comprehensive command line interface for all its functions, documented in :doc:`generated/en/cli-index`. |
| 23 |
|
| 24 |
The modular design of calibre is expressed via ``Plugins``. There is a :ref:`tutorial <customize>` on writing calibre plugins. |
| 25 |
For example, adding support for a new device to calibre typically involves writing fewer than 100 lines of code in the form of |
| 26 |
a device driver plugin. You can browse the |
| 27 |
`built-in drivers <https://github.com/kovidgoyal/calibre/tree/master/src/calibre/devices>`_. Similarly, adding support |
| 28 |
for new conversion formats involves writing input/output format plugins. Another example of the modular design is the :ref:`recipe system <news>` for |
| 29 |
fetching news. For more examples of plugins designed to add features to calibre, see the `Index of plugins <https://www.mobileread.com/forums/showthread.php?p=1362767#post1362767>`_. |
| 30 |
|
| 31 |
.. _code_layout: |
| 32 |
|
| 33 |
Code layout |
| 34 |
^^^^^^^^^^^^^^ |
| 35 |
|
| 36 |
All the calibre Python code is in the ``calibre`` package. This package contains the following main sub-packages |
| 37 |
|
| 38 |
* devices - All the device drivers. Just look through some of the built-in drivers to get an idea for how they work. |
| 39 |
|
| 40 |
* For details, see: ``devices.interface`` which defines the interface supported by device drivers and ``devices.usbms`` which |
| 41 |
defines a generic driver that connects to a USBMS device. All USBMS based drivers in calibre inherit from it. |
| 42 |
|
| 43 |
* e-books - All the e-book conversion/metadata code. A good starting point is ``calibre.ebooks.conversion.cli`` which is the |
| 44 |
module powering the :command:`ebook-convert` command. The conversion process is controlled via ``conversion.plumber``. |
| 45 |
The format independent code is all in ``ebooks.oeb`` and the format dependent code is in ``ebooks.format_name``. |
| 46 |
|
| 47 |
* Metadata reading, writing, and downloading is all in ``ebooks.metadata`` |
| 48 |
* Conversion happens in a pipeline, for the structure of the pipeline, |
| 49 |
see :ref:`conversion-introduction`. The pipeline consists of an input |
| 50 |
plugin, various transforms and an output plugin. The code that constructs |
| 51 |
and drives the pipeline is in :file:`plumber.py`. The pipeline works on a |
| 52 |
representation of an e-book that is like an unzipped epub, with |
| 53 |
manifest, spine, toc, guide, html content, etc. The |
| 54 |
class that manages this representation is OEBBook in ``ebooks.oeb.base``. The |
| 55 |
various transformations that are applied to the book during |
| 56 |
conversions live in :file:`oeb/transforms/*.py`. And the input and output |
| 57 |
plugins live in :file:`conversion/plugins/*.py`. |
| 58 |
* E-book editing happens using a different container object. It is |
| 59 |
documented in :ref:`polish_api`. |
| 60 |
|
| 61 |
* db - The database back-end. See :ref:`db_api` for the interface to the calibre library. |
| 62 |
|
| 63 |
* Content server: ``srv`` is the calibre Content server. |
| 64 |
|
| 65 |
* gui2 - The Graphical User Interface. GUI initialization happens in ``gui2.main`` and ``gui2.ui``. The e-book-viewer is in ``gui2.viewer``. The e-book editor is in ``gui2.tweak_book``. |
| 66 |
|
| 67 |
If you want to locate the entry points for all the various calibre executables, |
| 68 |
look at the ``entry_points`` structure in `linux.py |
| 69 |
<https://github.com/kovidgoyal/calibre/blob/master/src/calibre/linux.py>`_. |
| 70 |
|
| 71 |
If you need help understanding the code, post in the `development forum <https://www.mobileread.com/forums/forumdisplay.php?f=240>`_ |
| 72 |
and you will most likely get help from one of calibre's many developers. |
| 73 |
|
| 74 |
Getting the code |
| 75 |
------------------ |
| 76 |
|
| 77 |
You can get the calibre source code in two ways, using a version control system or |
| 78 |
directly downloading a :website_base:`tarball <dist/src>`. |
| 79 |
|
| 80 |
calibre uses `Git <https://www.git-scm.com/>`_, a distributed version control |
| 81 |
system. Git is available on all the platforms calibre supports. After |
| 82 |
installing Git, you can get the calibre source code with the command:: |
| 83 |
|
| 84 |
git clone https://github.com/kovidgoyal/calibre.git |
| 85 |
|
| 86 |
On Windows you will need the complete path name, that will be something like :file:`C:\\Program Files\\Git\\git.exe`. |
| 87 |
|
| 88 |
calibre is a very large project with a very long source control history, so the |
| 89 |
above can take a while (10 mins to an hour depending on your internet speed). |
| 90 |
|
| 91 |
If you want to get the code faster, the source code for the latest release is |
| 92 |
always available as an :website_base:`archive <dist/src>`. |
| 93 |
|
| 94 |
To update a branch to the latest code, use the command:: |
| 95 |
|
| 96 |
git pull --no-edit |
| 97 |
|
| 98 |
You can also browse the code at `GitHub <https://github.com/kovidgoyal/calibre>`_. |
| 99 |
|
| 100 |
Submitting your changes to be included |
| 101 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 102 |
|
| 103 |
If you only plan to make a few small changes, you can make your changes and |
| 104 |
create a "merge directive" which you can then attach to a ticket in the calibre |
| 105 |
`bug tracker <https://bugs.launchpad.net/calibre>`_. To do this, make your |
| 106 |
changes, then run:: |
| 107 |
|
| 108 |
git commit -am "Comment describing your changes" |
| 109 |
git format-patch origin/master --stdout > my-changes |
| 110 |
|
| 111 |
This will create a :file:`my-changes` file in the current folder, |
| 112 |
simply attach that to a ticket on the calibre `bug tracker <https://bugs.launchpad.net/calibre>`_. |
| 113 |
Note that this will include *all* the commits you have made. If you only want |
| 114 |
to send some commits, you have to change ``origin/master`` above. To send only |
| 115 |
the last commit, use:: |
| 116 |
|
| 117 |
git format-patch HEAD~1 --stdout > my-changes |
| 118 |
|
| 119 |
To send the last *n* commits, replace *1* with *n*, for example, for the last 3 |
| 120 |
commits:: |
| 121 |
|
| 122 |
git format-patch HEAD~3 --stdout > my-changes |
| 123 |
|
| 124 |
Be careful to not include merges when using ``HEAD~n``. |
| 125 |
|
| 126 |
If you plan to do a lot of development on calibre, then the best method is to create a |
| 127 |
`GitHub <https://github.com>`__ account. Below is a basic guide to setting up |
| 128 |
your own fork of calibre in a way that will allow you to submit pull requests |
| 129 |
for inclusion into the main calibre repository: |
| 130 |
|
| 131 |
* Setup git on your machine as described in this article: `Setup Git <https://help.github.com/articles/set-up-git>`_ |
| 132 |
* Setup ssh keys for authentication to GitHub, as described here: `Generating SSH keys <https://help.github.com/articles/generating-ssh-keys>`_ |
| 133 |
* Go to https://github.com/kovidgoyal/calibre and click the :guilabel:`Fork` button. |
| 134 |
* In a Terminal do:: |
| 135 |
|
| 136 |
git clone [email protected]:<username>/calibre.git |
| 137 |
git remote add upstream https://github.com/kovidgoyal/calibre.git |
| 138 |
|
| 139 |
Replace <username> above with your GitHub username. That will get your fork checked out locally. |
| 140 |
* You can make changes and commit them whenever you like. When you are ready to have your work merged, do a:: |
| 141 |
|
| 142 |
git push |
| 143 |
|
| 144 |
and go to ``https://github.com/<username>/calibre`` and click the :guilabel:`Pull Request` button to generate a pull request that can be merged. |
| 145 |
* You can update your local copy with code from the main repo at any time by doing:: |
| 146 |
|
| 147 |
git pull upstream |
| 148 |
|
| 149 |
|
| 150 |
You should also keep an eye on the calibre `development forum |
| 151 |
<https://www.mobileread.com/forums/forumdisplay.php?f=240>`_. Before making |
| 152 |
major changes, you should discuss them in the forum or contact Kovid directly |
| 153 |
(his email address is all over the source code). |
| 154 |
|
| 155 |
Windows development environment |
| 156 |
--------------------------------- |
| 157 |
|
| 158 |
.. note:: You must also get the calibre source code separately as described above. |
| 159 |
|
| 160 |
Install calibre normally, using the `Windows installer <https://calibre-ebook.com/download_windows>`__. |
| 161 |
Then open a Command Prompt and change to the previously checked out calibre code folder. For example:: |
| 162 |
|
| 163 |
cd C:\Users\kovid\work\calibre |
| 164 |
|
| 165 |
calibre is the folder that contains the src and resources sub-folders. |
| 166 |
|
| 167 |
The next step is to set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src folder. |
| 168 |
So, following the example above, it would be ``C:\Users\kovid\work\calibre\src``. `Here is a short |
| 169 |
guide <https://docs.python.org/using/windows.html#excursus-setting-environment-variables>`_ to setting environment |
| 170 |
variables on Windows. |
| 171 |
|
| 172 |
Once you have set the environment variable, open a new command prompt and check that it was correctly set by using |
| 173 |
the command:: |
| 174 |
|
| 175 |
echo %CALIBRE_DEVELOP_FROM% |
| 176 |
|
| 177 |
Setting this environment variable means that calibre will now load all its Python code from the specified location. |
| 178 |
|
| 179 |
That's it! You are now ready to start hacking on the calibre code. For example, open the file :file:`src\\calibre\\__init__.py` |
| 180 |
in your favorite editor and add the line:: |
| 181 |
|
| 182 |
print("Hello, world!") |
| 183 |
|
| 184 |
near the top of the file. Now run the command :command:`calibredb`. The very first line of output should be ``Hello, world!``. |
| 185 |
|
| 186 |
You can also setup a calibre development environment inside the free Microsoft |
| 187 |
Visual Studio, if you like, following the instructions `here <https://www.mobileread.com/forums/showthread.php?t=251201>`_. |
| 188 |
|
| 189 |
macOS development environment |
| 190 |
------------------------------ |
| 191 |
|
| 192 |
.. note:: You must also get the calibre source code separately as described above. |
| 193 |
|
| 194 |
Install calibre normally using the `provided .dmg <https://calibre-ebook.com/download_osx>`__. |
| 195 |
Then open a Terminal and change to the previously checked out calibre code folder, for example:: |
| 196 |
|
| 197 |
cd /Users/kovid/work/calibre |
| 198 |
|
| 199 |
calibre is the folder that contains the src and resources sub-folders. |
| 200 |
The calibre command line tools are found inside the calibre app bundle, in |
| 201 |
:file:`/Applications/calibre.app/Contents/MacOS` |
| 202 |
you should add this folder to your PATH environment variable, if you want to |
| 203 |
run the command line tools easily. |
| 204 |
|
| 205 |
The next step is to create a bash script that will set the environment variable ``CALIBRE_DEVELOP_FROM`` |
| 206 |
to the absolute path of the src folder when running calibre in debug mode. |
| 207 |
|
| 208 |
Create a plain text file:: |
| 209 |
|
| 210 |
#!/bin/sh |
| 211 |
export CALIBRE_DEVELOP_FROM="/Users/kovid/work/calibre/src" |
| 212 |
calibre-debug -g |
| 213 |
|
| 214 |
Save this file as :file:`/usr/local/bin/calibre-develop`, then set its permissions so that it can be executed:: |
| 215 |
|
| 216 |
chmod +x /usr/local/bin/calibre-develop |
| 217 |
|
| 218 |
Once you have done this, run:: |
| 219 |
|
| 220 |
calibre-develop |
| 221 |
|
| 222 |
You should see some diagnostic information in the Terminal window as calibre |
| 223 |
starts up, and you should see an asterisk after the version number in the GUI |
| 224 |
window, indicating that you are running from source. |
| 225 |
|
| 226 |
Linux development environment |
| 227 |
------------------------------ |
| 228 |
|
| 229 |
.. note:: You must also get the calibre source code separately as described above. |
| 230 |
|
| 231 |
calibre is primarily developed on Linux. You have two choices in setting up the development environment. You can install the |
| 232 |
calibre binary as normal and use that as a runtime environment to do your development. This approach is similar to that |
| 233 |
used in Windows and macOS. Alternatively, you can install calibre from source. Instructions for setting up a development |
| 234 |
environment from source are in the INSTALL file in the source tree. Here we will address using the binary as a runtime, which is the |
| 235 |
recommended method. |
| 236 |
|
| 237 |
Install calibre using the `binary installer <https://calibre-ebook.com/download_linux>`_. Then open a terminal and change to the previously checked out calibre code folder, for example:: |
| 238 |
|
| 239 |
cd /home/kovid/work/calibre |
| 240 |
|
| 241 |
calibre is the folder that contains the src and resources sub-folders. |
| 242 |
|
| 243 |
The next step is to set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src folder. |
| 244 |
So, following the example above, it would be ``/home/kovid/work/calibre/src``. How to set environment variables depends on |
| 245 |
your Linux distribution and what shell you are using. |
| 246 |
|
| 247 |
.. note:: |
| 248 |
It is recommended to use the binary installer provided from upstream. |
| 249 |
Should you insist on using a package provided by your distribution, use |
| 250 |
the ``CALIBRE_PYTHON_PATH`` and ``CALIBRE_RESOURCES_PATH`` variables instead. |
| 251 |
You can get these by running ``calibre-debug --paths``. But note that |
| 252 |
distribution calibre packages are often terminally broken and are completely |
| 253 |
unsupported. |
| 254 |
|
| 255 |
Once you have set the environment variable, open a new terminal and check that it was correctly set by using |
| 256 |
the command:: |
| 257 |
|
| 258 |
echo $CALIBRE_DEVELOP_FROM |
| 259 |
|
| 260 |
Setting this environment variable means that calibre will now load all its Python code from the specified location. |
| 261 |
|
| 262 |
That's it! You are now ready to start hacking on the calibre code. For example, open the file :file:`src/calibre/__init__.py` |
| 263 |
in your favorite editor and add the line:: |
| 264 |
|
| 265 |
print("Hello, world!") |
| 266 |
|
| 267 |
near the top of the file. Now run the command :command:`calibredb`. The very first line of output should be ``Hello, world!``. |
| 268 |
|
| 269 |
Having separate "normal" and "development" calibre installs on the same computer |
| 270 |
----------------------------------------------------------------------------------------------------------------- |
| 271 |
|
| 272 |
The calibre source tree is very stable and rarely breaks, but if you feel the need to run from source on a separate |
| 273 |
test library and run the released calibre version with your everyday library, you can achieve this easily using |
| 274 |
.bat files or shell scripts to launch calibre. The example below shows how to do this on Windows using .bat files (the |
| 275 |
instructions for other platforms are the same, just use a shell script instead of a .bat file) |
| 276 |
|
| 277 |
To launch the release version of calibre with your everyday library: |
| 278 |
|
| 279 |
calibre-normal.bat:: |
| 280 |
|
| 281 |
calibre.exe "--with-library=C:\path\to\everyday\library folder" |
| 282 |
|
| 283 |
calibre-dev.bat:: |
| 284 |
|
| 285 |
set CALIBRE_DEVELOP_FROM=C:\path\to\calibre\checkout\src |
| 286 |
calibre.exe "--with-library=C:\path\to\test\library folder" |
| 287 |
|
| 288 |
|
| 289 |
Debugging tips |
| 290 |
---------------- |
| 291 |
|
| 292 |
Python is a |
| 293 |
dynamically typed language with excellent facilities for introspection. Kovid wrote the core calibre code without once |
| 294 |
using a debugger. There are many strategies to debug calibre code: |
| 295 |
|
| 296 |
Using print statements |
| 297 |
^^^^^^^^^^^^^^^^^^^^^^^ |
| 298 |
|
| 299 |
This is Kovid's favorite way to debug. Simply insert print statements at points of interest and run your program in the |
| 300 |
terminal. For example, you can start the GUI from the terminal as:: |
| 301 |
|
| 302 |
calibre-debug -g |
| 303 |
|
| 304 |
Similarly, you can start the E-book viewer as:: |
| 305 |
|
| 306 |
calibre-debug -w /path/to/file/to/be/viewed |
| 307 |
|
| 308 |
The e-book editor can be started as:: |
| 309 |
|
| 310 |
calibre-debug --edit-book /path/to/be/edited |
| 311 |
|
| 312 |
Using an interactive Python interpreter |
| 313 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 314 |
|
| 315 |
You can insert the following two lines of code to start an interactive Python session at that point:: |
| 316 |
|
| 317 |
from calibre import ipython |
| 318 |
ipython(locals()) |
| 319 |
|
| 320 |
When running from the command line, this will start an interactive Python interpreter with access to all |
| 321 |
locally defined variables (variables in the local scope). The interactive prompt even has :kbd:`Tab` completion |
| 322 |
for object properties and you can use the various Python facilities for introspection, such as |
| 323 |
:func:`dir`, :func:`type`, :func:`repr`, etc. |
| 324 |
|
| 325 |
Using the Python debugger as a remote debugger |
| 326 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 327 |
|
| 328 |
You can use the builtin Python debugger (pdb) as a remote debugger from the |
| 329 |
command line. First, start the remote debugger at the point in the calibre code |
| 330 |
you are interested in, like this:: |
| 331 |
|
| 332 |
from calibre.rpdb import set_trace |
| 333 |
set_trace() |
| 334 |
|
| 335 |
Then run calibre, either as normal, or using one of the calibre-debug commands |
| 336 |
described in the previous section. Once the above point in the code is reached, |
| 337 |
calibre will freeze, waiting for the debugger to connect. |
| 338 |
|
| 339 |
Now open a terminal or command prompt and use the following command to start |
| 340 |
the debugging session:: |
| 341 |
|
| 342 |
calibre-debug -c "from calibre.rpdb import cli; cli()" |
| 343 |
|
| 344 |
You can read about how to use the Python debugger in the `Python stdlib docs |
| 345 |
for the pdb module <https://docs.python.org/library/pdb.html#debugger-commands>`_. |
| 346 |
|
| 347 |
.. note:: |
| 348 |
By default, the remote debugger will try to connect on port 4444. You can |
| 349 |
change it, by passing the port parameter to both the set_trace() and the |
| 350 |
cli() functions above, like this: ``set_trace(port=1234)`` and |
| 351 |
``cli(port=1234)``. |
| 352 |
|
| 353 |
.. note:: |
| 354 |
The Python debugger cannot handle multiple threads, so you have to |
| 355 |
call set_trace once per thread, each time with a different port number. |
| 356 |
|
| 357 |
Using the debugger in your favorite Python IDE |
| 358 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 359 |
|
| 360 |
It is possible to use the builtin debugger in your favorite Python IDE, if it |
| 361 |
supports remote debugging. The first step is to add the calibre src checkout to |
| 362 |
the ``PYTHONPATH`` in your IDE. In other words, the folder you set as |
| 363 |
``CALIBRE_DEVELOP_FROM`` above, must also be in the ``PYTHONPATH`` of your IDE. |
| 364 |
|
| 365 |
Then place the IDE's remote debugger module into the :file:`src` sub-folder |
| 366 |
of the calibre source code checkout. Add whatever code is needed to launch the |
| 367 |
remote debugger to calibre at the point of interest, for example in the main |
| 368 |
function. Then run calibre as normal. Your IDE should now be able to connect to |
| 369 |
the remote debugger running inside calibre. |
| 370 |
|
| 371 |
Executing arbitrary scripts in the calibre Python environment |
| 372 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 373 |
|
| 374 |
The :command:`calibre-debug` command provides a couple of handy switches to execute your own |
| 375 |
code, with access to the calibre modules:: |
| 376 |
|
| 377 |
calibre-debug -c "some Python code" |
| 378 |
|
| 379 |
is great for testing a little snippet of code on the command line. It works in the same way as the -c switch to the Python interpreter:: |
| 380 |
|
| 381 |
calibre-debug myscript.py |
| 382 |
|
| 383 |
can be used to execute your own Python script. It works in the same way as passing the script to the Python interpreter, except |
| 384 |
that the calibre environment is fully initialized, so you can use all the calibre code in your script. To use command line arguments with your script, use the form:: |
| 385 |
|
| 386 |
calibre-debug myscript.py -- --option1 arg1 |
| 387 |
|
| 388 |
The ``--`` causes all subsequent arguments to be passed to your script. |
| 389 |
|
| 390 |
|
| 391 |
Running the calibre test suite |
| 392 |
------------------------------------ |
| 393 |
|
| 394 |
The calibre test suite can be run by:: |
| 395 |
|
| 396 |
calibre-debug -t all |
| 397 |
|
| 398 |
|
| 399 |
Using calibre in your projects |
| 400 |
---------------------------------------- |
| 401 |
|
| 402 |
It is possible to directly use calibre functions/code in your Python project. Two ways exist to do this: |
| 403 |
|
| 404 |
Binary install of calibre |
| 405 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 406 |
|
| 407 |
If you have a binary install of calibre, you can use the Python interpreter bundled with calibre, like this:: |
| 408 |
|
| 409 |
calibre-debug /path/to/your/python/script.py -- arguments to your script |
| 410 |
|
| 411 |
Source install on Linux |
| 412 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 413 |
|
| 414 |
In addition to using the above technique, if you do a source install on Linux, |
| 415 |
you can also directly import calibre, as follows:: |
| 416 |
|
| 417 |
import init_calibre |
| 418 |
import calibre |
| 419 |
|
| 420 |
print(calibre.__version__) |
| 421 |
|
| 422 |
It is essential that you import the init_calibre module before any other calibre modules/packages as |
| 423 |
it sets up the interpreter to run calibre code. |
| 424 |
|
| 425 |
API documentation for various parts of calibre |
| 426 |
------------------------------------------------ |
| 427 |
|
| 428 |
.. toctree:: |
| 429 |
:maxdepth: 1 |
| 430 |
|
| 431 |
news_recipe |
| 432 |
plugins |
| 433 |
db_api |
| 434 |
polish |