Debian Codegraph @

Doc/library/html.parser.rst

raw
1 :mod:`!html.parser` --- Simple HTML and XHTML parser
2 ====================================================
3
4 .. module:: html.parser
5 :synopsis: A simple parser that can handle HTML and XHTML.
6
7 **Source code:** :source:`Lib/html/parser.py`
8
9 .. index::
10 single: HTML
11 single: XHTML
12
13 --------------
14
15 This module defines a class :class:`HTMLParser` which serves as the basis for
16 parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.
17
18 .. class:: HTMLParser(*, convert_charrefs=True, scripting=False)
19
20 Create a parser instance able to parse invalid markup.
21
22 If *convert_charrefs* is true (the default), all character
23 references (except the ones in elements like ``script`` and ``style``) are
24 automatically converted to the corresponding Unicode characters.
25
26 If *scripting* is false (the default), the content of the ``noscript``
27 element is parsed normally; if it's true, it's returned as is without
28 being parsed.
29
30 An :class:`.HTMLParser` instance is fed HTML data and calls handler methods
31 when start tags, end tags, text, comments, and other markup elements are
32 encountered. The user should subclass :class:`.HTMLParser` and override its
33 methods to implement the desired behavior.
34
35 This parser does not check that end tags match start tags or call the end-tag
36 handler for elements which are closed implicitly by closing an outer element.
37
38 .. versionchanged:: 3.4
39 *convert_charrefs* keyword argument added.
40
41 .. versionchanged:: 3.5
42 The default value for argument *convert_charrefs* is now ``True``.
43
44 .. versionchanged:: 3.14.1
45 Added the *scripting* parameter.
46
47
48 Example HTML Parser Application
49 -------------------------------
50
51 As a basic example, below is a simple HTML parser that uses the
52 :class:`HTMLParser` class to print out start tags, end tags, and data
53 as they are encountered:
54
55 .. testcode::
56
57 from html.parser import HTMLParser
58
59 class MyHTMLParser(HTMLParser):
60 def handle_starttag(self, tag, attrs):
61 print("Encountered a start tag:", tag)
62
63 def handle_endtag(self, tag):
64 print("Encountered an end tag :", tag)
65
66 def handle_data(self, data):
67 print("Encountered some data :", data)
68
69 parser = MyHTMLParser()
70 parser.feed('<html><head><title>Test</title></head>'
71 '<body><h1>Parse me!</h1></body></html>')
72
73 The output will then be:
74
75 .. testoutput::
76
77 Encountered a start tag: html
78 Encountered a start tag: head
79 Encountered a start tag: title
80 Encountered some data : Test
81 Encountered an end tag : title
82 Encountered an end tag : head
83 Encountered a start tag: body
84 Encountered a start tag: h1
85 Encountered some data : Parse me!
86 Encountered an end tag : h1
87 Encountered an end tag : body
88 Encountered an end tag : html
89
90
91 :class:`.HTMLParser` Methods
92 ----------------------------
93
94 :class:`HTMLParser` instances have the following methods:
95
96
97 .. method:: HTMLParser.feed(data)
98
99 Feed some text to the parser. It is processed insofar as it consists of
100 complete elements; incomplete data is buffered until more data is fed or
101 :meth:`close` is called. *data* must be :class:`str`.
102
103
104 .. method:: HTMLParser.close()
105
106 Force processing of all buffered data as if it were followed by an end-of-file
107 mark. This method may be redefined by a derived class to define additional
108 processing at the end of the input, but the redefined version should always call
109 the :class:`HTMLParser` base class method :meth:`close`.
110
111
112 .. method:: HTMLParser.reset()
113
114 Reset the instance. Loses all unprocessed data. This is called implicitly at
115 instantiation time.
116
117
118 .. method:: HTMLParser.getpos()
119
120 Return current line number and offset.
121
122
123 .. method:: HTMLParser.get_starttag_text()
124
125 Return the text of the most recently opened start tag. This should not normally
126 be needed for structured processing, but may be useful in dealing with HTML "as
127 deployed" or for re-generating input with minimal changes (whitespace between
128 attributes can be preserved, etc.).
129
130
131 The following methods are called when data or markup elements are encountered
132 and they are meant to be overridden in a subclass. The base class
133 implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`):
134
135
136 .. method:: HTMLParser.handle_starttag(tag, attrs)
137
138 This method is called to handle the start tag of an element (e.g. ``<div id="main">``).
139
140 The *tag* argument is the name of the tag converted to lower case. The *attrs*
141 argument is a list of ``(name, value)`` pairs containing the attributes found
142 inside the tag's ``<>`` brackets. The *name* will be translated to lower case,
143 and quotes in the *value* have been removed, and character and entity references
144 have been replaced. For empty attributes, *value* is ``None``.
145
146 For instance, for the tag ``<A HREF="https://www.cwi.nl/">``, this method
147 would be called as ``handle_starttag('a', [('href', 'https://www.cwi.nl/')])``.
148
149 All entity references from :mod:`html.entities` are replaced in the attribute
150 values.
151
152
153 .. method:: HTMLParser.handle_endtag(tag)
154
155 This method is called to handle the end tag of an element (e.g. ``</div>``).
156
157 The *tag* argument is the name of the tag converted to lower case.
158
159
160 .. method:: HTMLParser.handle_startendtag(tag, attrs)
161
162 Similar to :meth:`handle_starttag`, but called when the parser encounters an
163 XHTML-style empty tag (``<img ... />``). This method may be overridden by
164 subclasses which require this particular lexical information; the default
165 implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`.
166
167
168 .. method:: HTMLParser.handle_data(data)
169
170 This method is called to process arbitrary data (e.g. text nodes and the
171 content of elements like ``script`` and ``style``).
172
173
174 .. method:: HTMLParser.handle_entityref(name)
175
176 This method is called to process a named character reference of the form
177 ``&name;`` (e.g. ``&gt;``), where *name* is a general entity reference
178 (e.g. ``'gt'``).
179 This method is only called if *convert_charrefs* is false.
180
181
182 .. method:: HTMLParser.handle_charref(name)
183
184 This method is called to process decimal and hexadecimal numeric character
185 references of the form :samp:`&#{NNN};` and :samp:`&#x{NNN};`. For example, the decimal
186 equivalent for ``&gt;`` is ``&#62;``, whereas the hexadecimal is ``&#x3E;``;
187 in this case the method will receive ``'62'`` or ``'x3E'``.
188 This method is only called if *convert_charrefs* is false.
189
190
191 .. method:: HTMLParser.handle_comment(data)
192
193 This method is called when a comment is encountered (e.g. ``<!--comment-->``).
194
195 For example, the comment ``<!-- comment -->`` will cause this method to be
196 called with the argument ``' comment '``.
197
198 The content of Internet Explorer conditional comments (condcoms) will also be
199 sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``,
200 this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``.
201
202
203 .. method:: HTMLParser.handle_decl(decl)
204
205 This method is called to handle an HTML doctype declaration (e.g.
206 ``<!DOCTYPE html>``).
207
208 The *decl* parameter will be the entire contents of the declaration inside
209 the ``<!...>`` markup (e.g. ``'DOCTYPE html'``).
210
211
212 .. method:: HTMLParser.handle_pi(data)
213
214 Method called when a processing instruction is encountered. The *data*
215 parameter will contain the entire processing instruction. For example, for the
216 processing instruction ``<?proc color='red'>``, this method would be called as
217 ``handle_pi("proc color='red'")``. It is intended to be overridden by a derived
218 class; the base class implementation does nothing.
219
220 .. note::
221
222 The :class:`HTMLParser` class uses the SGML syntactic rules for processing
223 instructions. An XHTML processing instruction using the trailing ``'?'`` will
224 cause the ``'?'`` to be included in *data*.
225
226
227 .. method:: HTMLParser.unknown_decl(data)
228
229 This method is called when an unrecognized declaration is read by the parser.
230
231 The *data* parameter will be the entire contents of the declaration inside
232 the ``<![...]>`` markup. It is sometimes useful to be overridden by a
233 derived class. The base class implementation does nothing.
234
235
236 .. _htmlparser-examples:
237
238 Examples
239 --------
240
241 The following class implements a parser that will be used to illustrate more
242 examples:
243
244 .. testcode::
245
246 from html.parser import HTMLParser
247 from html.entities import name2codepoint
248
249 class MyHTMLParser(HTMLParser):
250 def handle_starttag(self, tag, attrs):
251 print("Start tag:", tag)
252 for attr in attrs:
253 print(" attr:", attr)
254
255 def handle_endtag(self, tag):
256 print("End tag :", tag)
257
258 def handle_data(self, data):
259 print("Data :", data)
260
261 def handle_comment(self, data):
262 print("Comment :", data)
263
264 def handle_entityref(self, name):
265 c = chr(name2codepoint[name])
266 print("Named ent:", c)
267
268 def handle_charref(self, name):
269 if name.startswith('x'):
270 c = chr(int(name[1:], 16))
271 else:
272 c = chr(int(name))
273 print("Num ent :", c)
274
275 def handle_decl(self, data):
276 print("Decl :", data)
277
278 parser = MyHTMLParser()
279
280 Parsing a doctype:
281
282 .. doctest::
283
284 >>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
285 ... '"http://www.w3.org/TR/html4/strict.dtd">')
286 Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"
287
288 Parsing an element with a few attributes and a title:
289
290 .. doctest::
291
292 >>> parser.feed('<img src="python-logo.png" alt="The Python logo">')
293 Start tag: img
294 attr: ('src', 'python-logo.png')
295 attr: ('alt', 'The Python logo')
296 >>>
297 >>> parser.feed('<h1>Python</h1>')
298 Start tag: h1
299 Data : Python
300 End tag : h1
301
302 The content of elements like ``script`` and ``style`` is returned as is,
303 without further parsing:
304
305 .. doctest::
306
307 >>> parser.feed('<style type="text/css">#python { color: green }</style>')
308 Start tag: style
309 attr: ('type', 'text/css')
310 Data : #python { color: green }
311 End tag : style
312
313 >>> parser.feed('<script type="text/javascript">'
314 ... 'alert("<strong>hello! &#9786;</strong>");</script>')
315 Start tag: script
316 attr: ('type', 'text/javascript')
317 Data : alert("<strong>hello! &#9786;</strong>");
318 End tag : script
319
320 Attribute names are converted to lowercase, quotes from attribute values removed,
321 and ``None`` is returned as *value* for empty attributes (such as ``checked``):
322
323 .. doctest::
324
325 >>> parser.feed("<input TYPE='checkbox' checked required='' disabled=disabled>")
326 Start tag: input
327 attr: ('type', 'checkbox')
328 attr: ('checked', None)
329 attr: ('required', '')
330 attr: ('disabled', 'disabled')
331
332 Parsing comments:
333
334 .. doctest::
335
336 >>> parser.feed('<!--a comment-->'
337 ... '<!--[if IE 9]>IE-specific content<![endif]-->')
338 Comment : a comment
339 Comment : [if IE 9]>IE-specific content<![endif]
340
341 Parsing named and numeric character references and converting them to the
342 correct char (note: these 3 references are all equivalent to ``'>'``):
343
344 .. doctest::
345
346 >>> parser = MyHTMLParser()
347 >>> parser.feed('&gt;&#62;&#x3E;')
348 Data : >>>
349
350 >>> parser = MyHTMLParser(convert_charrefs=False)
351 >>> parser.feed('&gt;&#62;&#x3E;')
352 Named ent: >
353 Num ent : >
354 Num ent : >
355
356 Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but
357 :meth:`~HTMLParser.handle_data` might be called more than once
358 if *convert_charrefs* is false:
359
360 .. doctest::
361
362 >>> for chunk in ['<sp', 'an>buff', 'ered', ' text</s', 'pan>']:
363 ... parser.feed(chunk)
364 ...
365 Start tag: span
366 Data : buff
367 Data : ered
368 Data : text
369 End tag : span
370
371 Parsing invalid HTML (e.g. unquoted attributes) also works:
372
373 .. doctest::
374
375 >>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>')
376 Start tag: p
377 Start tag: a
378 attr: ('class', 'link')
379 attr: ('href', '#main')
380 Data : tag soup
381 End tag : p
382 End tag : a