Debian Codegraph @

manual/news.rst

raw
1 .. _news:
2
3 Adding your favorite news website
4 ==================================
5
6 calibre has a powerful, flexible and easy-to-use framework for downloading news from the Internet and converting it into an e-book. The following will show you, by means of examples, how to get news from various websites.
7
8 To gain an understanding of how to use the framework, follow the examples in the order listed below:
9
10 .. contents::
11 :depth: 2
12 :local:
13
14 Completely automatic fetching
15 -------------------------------
16
17 If your news source is simple enough, calibre may well be able to fetch it
18 completely automatically, all you need to do is provide the URL. calibre
19 gathers all the information needed to download a news source into a
20 :term:`recipe`. In order to tell calibre about a news source, you have to
21 create a :term:`recipe` for it. Let's see some examples:
22
23 .. _calibre_blog:
24
25 The calibre blog
26 ~~~~~~~~~~~~~~~~~~~
27
28 The calibre blog is a blog of posts that describe many useful calibre features
29 in a simple and accessible way for new calibre users. In order to download this
30 blog into an e-book, we rely on the :term:`RSS` feed of the blog::
31
32 http://blog.calibre-ebook.com/feeds/posts/default
33
34 I got the RSS URL by looking under "Subscribe to" at the bottom of the blog
35 page and choosing :guilabel:`Posts->Atom`. To make calibre download the feeds and convert
36 them into an e-book, you should right click the :guilabel:`Fetch news` button
37 and then the :guilabel:`Add a custom news source` menu item and then the
38 :guilabel:`New Recipe` button. A dialog similar to that shown below should open up.
39
40 .. image:: images/custom_news.png
41 :align: center
42
43 First enter ``Calibre Blog`` into the :guilabel:`Recipe title` field. This will be the title of the e-book that will be created from the articles in the above feeds.
44
45 The next two fields (:guilabel:`Oldest article` and :guilabel:`Max. number of articles`) allow you some control over how many articles should be downloaded from each feed, and they are pretty self explanatory.
46
47 To add the feeds to the recipe, enter the feed title and the feed URL and click
48 the :guilabel:`Add feed` button. Once you have added the feed, simply click the
49 :guilabel:`Save` button and you're done! Close the dialog.
50
51 To test your new :term:`recipe`, click the :guilabel:`Fetch news` button and in the :guilabel:`Custom news sources` sub-menu click :guilabel:`calibre Blog`. After a couple of minutes, the newly downloaded e-book of blog posts will appear in the main library view (if you have your reader connected, it will be put onto the reader instead of into the library). Select it and hit the :guilabel:`View` button to read!
52
53 The reason this worked so well, with so little effort is that the blog provides *full-content* :term:`RSS` feeds, i.e., the article content is embedded in the feed itself. For most news sources that provide news in this fashion, with *full-content* feeds, you don't need any more effort to convert them to e-books. Now we will look at a news source that does not provide full content feeds. In such feeds, the full article is a webpage and the feed only contains a link to the webpage with a short summary of the article.
54
55 .. _bbc:
56
57 bbc.co.uk
58 ~~~~~~~~~~~~~~
59
60 Let's try the following two feeds from *The BBC*:
61
62 #. News Front Page: https://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml
63 #. Science/Nature: https://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml
64
65 Follow the procedure outlined in :ref:`calibre_blog` above to create a recipe for *The BBC* (using the feeds above). Looking at the downloaded e-book, we see that calibre has done a creditable job of extracting only the content you care about from each article's webpage. However, the extraction process is not perfect. Sometimes it leaves in undesirable content like menus and navigation aids or it removes content that should have been left alone, like article headings. In order, to have perfect content extraction, we will need to customize the fetch process, as described in the next section.
66
67 Customizing the fetch process
68 --------------------------------
69
70 When you want to perfect the download process, or download content from a particularly complex website, you can avail yourself of all the power and flexibility of the :term:`recipe` framework. In order to do that, in the :guilabel:`Add custom news sources` dialog, simply click the :guilabel:`Switch to Advanced mode` button.
71
72 The easiest and often most productive customization is to use the print version of the online articles. The print version typically has much less cruft and translates much more smoothly to an e-book. Let's try to use the print version of the articles from *The BBC*.
73
74 Using the print version of bbc.co.uk
75 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
76
77 The first step is to look at the e-book we downloaded previously from :ref:`bbc`. At the end of each article, in the e-book is a little blurb telling you where the article was downloaded from. Copy and paste that URL into a browser. Now on the article webpage look for a link that points to the "Printable version". Click it to see the print version of the article. It looks much neater! Now compare the two URLs. For me they were:
78
79 Article URL
80 https://news.bbc.co.uk/2/hi/science/nature/7312016.stm
81
82 Print version URL
83 https://newsvote.bbc.co.uk/mpapps/pagetools/print/news.bbc.co.uk/2/hi/science/nature/7312016.stm
84
85 So it looks like to get the print version, we need to prefix every article URL with:
86
87 newsvote.bbc.co.uk/mpapps/pagetools/print/
88
89 Now in the :guilabel:`Advanced mode` of the Custom news sources dialog, you should see something like (remember to select *The BBC* recipe before switching to advanced mode):
90
91 .. image:: images/bbc_advanced.png
92 :align: center
93
94 You can see that the fields from the :guilabel:`Basic mode` have been translated to Python code in a straightforward manner. We need to add instructions to this recipe to use the print version of the articles. All that's needed is to add the following two lines:
95
96 .. code-block:: python
97
98 def print_version(self, url):
99 return url.replace('https://', 'https://newsvote.bbc.co.uk/mpapps/pagetools/print/')
100
101 This is Python, so indentation is important. After you've added the lines, it should look like:
102
103 .. image:: images/bbc_altered.png
104 :align: center
105
106 In the above, ``def print_version(self, url)`` defines a *method* that is called by calibre for every article. ``url`` is the URL of the original article. What ``print_version`` does is take that url and replace it with the new URL that points to the print version of the article. To learn about `Python <https://www.python.org>`_ see the `tutorial <https://docs.python.org/tutorial/>`_.
107
108 Now, click the :guilabel:`Add/update recipe` button and your changes will be saved. Re-download the e-book. You should have a much improved e-book. One of the problems with the new version is that the fonts on the print version webpage are too small. This is automatically fixed when converting to an e-book, but even after the fixing process, the font size of the menus and navigation bar become too large relative to the article text. To fix this, we will do some more customization, in the next section.
109
110 Replacing article styles
111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
112
113 In the previous section, we saw that the font size for articles from the print version of *The BBC* was too small. In most websites, *The BBC* included, this font size is set by means of :term:`CSS` stylesheets. We can disable the fetching of such stylesheets by adding the line::
114
115 no_stylesheets = True
116
117 .. _bbc1:
118
119 The recipe now looks like:
120
121 .. image:: images/bbc_altered1.png
122 :align: center
123
124 The new version looks pretty good. If you're a perfectionist, you'll want to read the next section, which deals with actually modifying the downloaded content.
125
126 Slicing and dicing
127 ~~~~~~~~~~~~~~~~~~~~~~~
128
129 calibre contains very powerful and flexible abilities when it comes to manipulating downloaded content. To show off a couple of these, let's look at our old friend the :ref:`The BBC <bbc1>` recipe again. Looking at the source code (:term:`HTML`) of a couple of articles (print version), we see that they have a footer that contains no useful information, contained in
130
131 .. code-block:: html
132
133 <div class="footer">
134 ...
135 </div>
136
137 This can be removed by adding::
138
139 remove_tags = [dict(name='div', attrs={'class':'footer'})]
140
141 to the recipe. Finally, let's replace some of the :term:`CSS` that we disabled earlier, with our own :term:`CSS` that is suitable for conversion to an e-book::
142
143 extra_css = '.headline {font-size: x-large;} \n .fact { padding-top: 10pt }'
144
145 With these additions, our recipe has become "production quality".
146
147 This :term:`recipe` explores only the tip of the iceberg when it comes to the power of calibre. To explore more of the abilities of calibre we'll examine a more complex real life example in the next section.
148
149 Real life example
150 ~~~~~~~~~~~~~~~~~~~~~
151
152 A reasonably complex real life example that exposes more of the :term:`API` of ``BasicNewsRecipe`` is the :term:`recipe` for *The New York Times*
153
154 .. code-block:: python
155
156 import string, re
157 from calibre import strftime
158 from calibre.web.feeds.recipes import BasicNewsRecipe
159 from calibre.ebooks.BeautifulSoup import BeautifulSoup
160
161 class NYTimes(BasicNewsRecipe):
162
163 title = 'The New York Times'
164 __author__ = 'Kovid Goyal'
165 description = 'Daily news from the New York Times'
166 timefmt = ' [%a, %d %b, %Y]'
167 needs_subscription = True
168 remove_tags_before = dict(id='article')
169 remove_tags_after = dict(id='article')
170 remove_tags = [dict(attrs={'class':['articleTools', 'post-tools', 'side_tool', 'nextArticleLink clearfix']}),
171 dict(id=['footer', 'toolsRight', 'articleInline', 'navigation', 'archive', 'side_search', 'blog_sidebar', 'side_tool', 'side_index']),
172 dict(name=['script', 'noscript', 'style'])]
173 encoding = 'cp1252'
174 no_stylesheets = True
175 extra_css = 'h1 {font: sans-serif large;}\n.byline {font:monospace;}'
176
177 def get_browser(self):
178 br = BasicNewsRecipe.get_browser(self)
179 if self.username is not None and self.password is not None:
180 br.open('https://www.nytimes.com/auth/login')
181 br.select_form(name='login')
182 br['USERID'] = self.username
183 br['PASSWORD'] = self.password
184 br.submit()
185 return br
186
187 def parse_index(self):
188 soup = self.index_to_soup('https://www.nytimes.com/pages/todayspaper/index.html')
189
190 def feed_title(div):
191 return ''.join(div.findAll(text=True, recursive=False)).strip()
192
193 articles = {}
194 key = None
195 ans = []
196 for div in soup.findAll(True,
197 attrs={'class':['section-headline', 'story', 'story headline']}):
198
199 if ''.join(div['class']) == 'section-headline':
200 key = string.capwords(feed_title(div))
201 articles[key] = []
202 ans.append(key)
203
204 elif ''.join(div['class']) in ['story', 'story headline']:
205 a = div.find('a', href=True)
206 if not a:
207 continue
208 url = re.sub(r'\?.*', '', a['href'])
209 url += '?pagewanted=all'
210 title = self.tag_to_string(a, use_alt=True).strip()
211 description = ''
212 pubdate = strftime('%a, %d %b')
213 summary = div.find(True, attrs={'class':'summary'})
214 if summary:
215 description = self.tag_to_string(summary, use_alt=False)
216
217 feed = key if key is not None else 'Uncategorized'
218 if feed not in articles:
219 articles[feed] = []
220 if not 'podcasts' in url:
221 articles[feed].append(
222 dict(title=title, url=url, date=pubdate,
223 description=description,
224 content=''))
225 ans = self.sort_index_by(ans, {'The Front Page':-1, 'Dining In, Dining Out':1, 'Obituaries':2})
226 ans = [(key, articles[key]) for key in ans if key in articles]
227 return ans
228
229 def preprocess_html(self, soup):
230 refresh = soup.find('meta', {'http-equiv':'refresh'})
231 if refresh is None:
232 return soup
233 content = refresh.get('content').partition('=')[2]
234 raw = self.browser.open('https://www.nytimes.com'+content).read()
235 return BeautifulSoup(raw.decode('cp1252', 'replace'))
236
237
238 We see several new features in this :term:`recipe`. First, we have::
239
240 timefmt = ' [%a, %d %b, %Y]'
241
242 This sets the displayed time on the front page of the created e-book to be in the format,
243 ``Day, Day_Number Month, Year``. See :attr:`timefmt <calibre.web.feeds.news.BasicNewsRecipe.timefmt>`.
244
245 Then we see a group of directives to cleanup the downloaded :term:`HTML`::
246
247 remove_tags_before = dict(name='h1')
248 remove_tags_after = dict(id='footer')
249 remove_tags = ...
250
251 These remove everything before the first ``<h1>`` tag and everything after the first tag whose id is ``footer``. See :attr:`remove_tags <calibre.web.feeds.news.BasicNewsRecipe.remove_tags>`, :attr:`remove_tags_before <calibre.web.feeds.news.BasicNewsRecipe.remove_tags_before>`, :attr:`remove_tags_after <calibre.web.feeds.news.BasicNewsRecipe.remove_tags_after>`.
252
253 The next interesting feature is::
254
255 needs_subscription = True
256 ...
257 def get_browser(self):
258 ...
259
260 ``needs_subscription = True`` tells calibre that this recipe needs a username and password in order to access the content. This causes, calibre to ask for a username and password whenever you try to use this recipe. The code in :meth:`calibre.web.feeds.news.BasicNewsRecipe.get_browser` actually does the login into the NYT website. Once logged in, calibre will use the same, logged in, browser instance to fetch all content. See `mechanize <https://mechanize.readthedocs.io/en/latest/>`_ to understand the code in ``get_browser``.
261
262 The next new feature is the
263 :meth:`calibre.web.feeds.news.BasicNewsRecipe.parse_index` method. Its job is
264 to go to https://www.nytimes.com/pages/todayspaper/index.html and fetch the list
265 of articles that appear in *today's* paper. While more complex than simply using
266 :term:`RSS`, the recipe creates an e-book that corresponds very closely to the
267 day's paper. ``parse_index`` makes heavy use of `BeautifulSoup
268 <https://www.crummy.com/software/BeautifulSoup/bs4/doc/>`_ to parse
269 the daily paper webpage. You can also use other, more modern parsers if you
270 dislike BeautifulSoup. calibre comes with `lxml <https://lxml.de/>`_ and
271 `html5lib <https://github.com/html5lib/html5lib-python>`_, which are the
272 recommended parsers. To use them, replace the call to ``index_to_soup()`` with
273 the following::
274
275 raw = self.index_to_soup(url, raw=True)
276 # For html5lib
277 import html5lib
278 root = html5lib.parse(raw, namespaceHTMLElements=False, treebuilder='lxml')
279 # For the lxml html 4 parser
280 from lxml import html
281 root = html.fromstring(raw)
282
283 The final new feature is the :meth:`calibre.web.feeds.news.BasicNewsRecipe.preprocess_html` method. It can be used to perform arbitrary transformations on every downloaded HTML page. Here it is used to bypass the ads that the nytimes shows you before each article.
284
285 Tips for developing new recipes
286 ---------------------------------
287
288 The best way to develop new recipes is to use the command line interface. Create the recipe using your favorite Python editor and save it to a file say :file:`myrecipe.recipe`. The `.recipe` extension is required. You can download content using this recipe with the command::
289
290 ebook-convert myrecipe.recipe .epub --test -vv --debug-pipeline debug
291
292 The command :command:`ebook-convert` will download all the webpages and save
293 them to the EPUB file :file:`myrecipe.epub`. The ``-vv`` option makes
294 ebook-convert spit out a lot of information about what it is doing. The
295 :option:`ebook-convert-recipe-input --test` option makes it download only a couple of articles from at most two
296 feeds. In addition, ebook-convert will put the downloaded HTML into the
297 ``debug/input`` folder, where ``debug`` is the folder you specified in
298 the :option:`ebook-convert --debug-pipeline` option.
299
300 Once the download is complete, you can look at the downloaded :term:`HTML` by opening the file :file:`debug/input/index.html` in a browser. Once you're satisfied that the download and preprocessing is happening correctly, you can generate e-books in different formats as shown below::
301
302 ebook-convert myrecipe.recipe myrecipe.epub
303 ebook-convert myrecipe.recipe myrecipe.mobi
304 ...
305
306
307 If you're satisfied with your recipe, and you feel there is enough demand to justify its inclusion into the set of built-in recipes, post your recipe in the `calibre recipes forum <https://www.mobileread.com/forums/forumdisplay.php?f=228>`_ to share it with other calibre users.
308
309 .. note::
310 On macOS, the command line tools are inside the calibre bundle, for example,
311 if you installed calibre in :file:`/Applications` the command line tools
312 are in :file:`/Applications/calibre.app/Contents/MacOS/`.
313
314 .. seealso::
315
316 :doc:`generated/en/ebook-convert`
317 The command line interface for all e-book conversion.
318
319
320 Further reading
321 --------------------
322
323 To learn more about writing advanced recipes using some of the facilities, available in ``BasicNewsRecipe`` you should consult the following sources:
324
325 :ref:`API documentation <news_recipe>`
326 Documentation of the ``BasicNewsRecipe`` class and all its important methods and fields.
327
328 `BasicNewsRecipe <https://github.com/kovidgoyal/calibre/blob/master/src/calibre/web/feeds/news.py>`_
329 The source code of ``BasicNewsRecipe``
330
331 `Built-in recipes <https://github.com/kovidgoyal/calibre/tree/master/recipes>`_
332 The source code for the built-in recipes that come with calibre
333
334 `The calibre recipes forum <https://www.mobileread.com/forums/forumdisplay.php?f=228>`_
335 Lots of knowledgeable calibre recipe writers hang out here.
336
337
338 API documentation
339 --------------------
340
341 .. toctree::
342
343 news_recipe