Debian Codegraph @

speed_test/README

raw
1 ABOUT
2
3 This is a benchmark system for XML parsers against various language editions of
4 the Wikipedia. The benchmark is to print all the article titles and text of a
5 dump file specified on the command line to standard output. There are
6 implementations for many perl parsing modules both high and low level. There
7 are even implementations written in C that perform very fast.
8
9 The benchmark.pl program is used to run a series of benchmarks. It takes two
10 required arguments and one optional. The first required argument is a path to
11 a directory full of tests to execute. The second required argument is a path
12 to a directory full of dump files to execute the tests against. Both of these
13 directories will be executed according to sort() on their file names. The third
14 argument is a number of iterations to perform, the default being 1.
15
16 Output goes to two files: results.log and results.data - they both are the
17 output from YAML of an internal data structure that represents the
18 test report. The results.log file is written to each time all the tests
19 have been run against a specific file and lets you keep an eye on how long
20 running jobs are performing. The results.data file is the cumulative data
21 for all iterations and is written at the end of the entire run.
22
23 The benchmark.pl utility and all of the tests are only guaranteed to work
24 if executed from the root directory of this software package. The C based
25 parsers are in the bin/ directory and can be compiled by executing make in
26 that directory. The Iksemel parser is not currently functional for unknown
27 reasons.
28
29 THE CHALLENGE
30
31 First and foremost the most important thing to keep in mind is that the English
32 Wikipedia is currently 22 gigabytes of XML in a single file. You will not be
33 able to use any XML processing system that requires the entire document to
34 fit into RAM.
35
36 Each benchmark must gather up the title and text for each Wikipedia article
37 for an arbitrary XML dump file. In the spirit of making this test approximate
38 a real world scenario you must collect all character data together and make it
39 available at one time. For instance in the perl benchmarks they actually invoke
40 a common method that prints the article title and text for them. In the C based
41 tests they simply collect all the data and print it out at once.
42
43 EXAMPLES
44
45 Doing a test run:
46
47 foodmotron:XML_Speed_Test tyler$ ./benchmark.pl test_cases data
48 Iterations remaining: 1
49 Benchmarking 20-simplewiki-20091021-pages-articles.xml
50 Generating md5sum: 8fa1e9de18b8da7523ebfe2dac53482a
51 running test_cases/MediaWiki-DumpFile-SimplePages.t data/20-simplewiki-20091021-pages-articles.xml: 12 seconds
52 running test_cases/Parse-MediaWikiDump.t data/20-simplewiki-20091021-pages-articles.xml: 66 seconds
53 running test_cases/XML-Bare.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
54 running test_cases/XML-LibXML-Reader.t data/20-simplewiki-20091021-pages-articles.xml: 12 seconds
55 running test_cases/XML-LibXML-SAX.t data/20-simplewiki-20091021-pages-articles.xml: 68 seconds
56 running test_cases/XML-Parser-ExpatNB.t data/20-simplewiki-20091021-pages-articles.xml: 44 seconds
57 running test_cases/XML-Parser.t data/20-simplewiki-20091021-pages-articles.xml: 42 seconds
58 running test_cases/XML-SAX-Expat.t data/20-simplewiki-20091021-pages-articles.xml: 183 seconds
59 running test_cases/XML-SAX-ExpatXS.t data/20-simplewiki-20091021-pages-articles.xml: 33 seconds
60 running test_cases/XML-SAX-ExpatXS_nocharjoin.t data/20-simplewiki-20091021-pages-articles.xml: 62 seconds
61 running test_cases/XML-SAX-PurePerl.t data/20-simplewiki-20091021-pages-articles.xml: 585 seconds
62 running test_cases/XML-Twig.t data/20-simplewiki-20091021-pages-articles.xml: 204 seconds
63 running test_cases/expat.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
64 running test_cases/libxml.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
65 foodmotron:XML_Speed_Test tyler$
66
67 The report:
68
69 $VAR1 = [
70 {
71 'filename' => '20-simplewiki-20091021-pages-articles.xml',
72 'tests' => [
73 {
74 'runtimes' => {
75 'system' => '0.4',
76 'user' => '5.78',
77 'total' => '6.18'
78 },
79 'name' => 'libxml.t',
80 'percentage' => 100,
81 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
82 'MiB/sec' => '35.1349971055213'
83 },
84 {
85 'runtimes' => {
86 'system' => '0.37',
87 'user' => '6.32',
88 'total' => '6.69'
89 },
90 'name' => 'XML-Bare.t',
91 'percentage' => 108,
92 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
93 'MiB/sec' => '32.4565444113784'
94 },
95 {
96 'runtimes' => {
97 'system' => '0.4',
98 'user' => '6.55',
99 'total' => '6.95'
100 },
101 'name' => 'expat.t',
102 'percentage' => 112,
103 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
104 'MiB/sec' => '31.2423427499455'
105 },
106 {
107 'runtimes' => {
108 'system' => '0.83',
109 'user' => '10.62',
110 'total' => '11.45'
111 },
112 'name' => 'XML-LibXML-Reader.t',
113 'percentage' => 185,
114 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
115 'MiB/sec' => '18.963692760884'
116 },
117 {
118 'runtimes' => {
119 'system' => '0.42',
120 'user' => '11.33',
121 'total' => '11.75'
122 },
123 'name' => 'MediaWiki-DumpFile-SimplePages.t',
124 'percentage' => 190,
125 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
126 'MiB/sec' => '18.4795133712444'
127 },
128 {
129 'runtimes' => {
130 'system' => '0.55',
131 'user' => '32',
132 'total' => '32.55'
133 },
134 'name' => 'XML-SAX-ExpatXS.t',
135 'percentage' => 526,
136 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
137 'MiB/sec' => '6.67079207717731'
138 },
139 {
140 'runtimes' => {
141 'system' => '0.26',
142 'user' => '41.55',
143 'total' => '41.81'
144 },
145 'name' => 'XML-Parser.t',
146 'percentage' => 676,
147 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
148 'MiB/sec' => '5.19335762047648'
149 },
150 {
151 'runtimes' => {
152 'system' => '0.46',
153 'user' => '42.1',
154 'total' => '42.56'
155 },
156 'name' => 'XML-Parser-ExpatNB.t',
157 'percentage' => 688,
158 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
159 'MiB/sec' => '5.1018393353412'
160 },
161 {
162 'runtimes' => {
163 'system' => '0.53',
164 'user' => '60.13',
165 'total' => '60.66'
166 },
167 'name' => 'XML-SAX-ExpatXS_nocharjoin.t',
168 'percentage' => 981,
169 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
170 'MiB/sec' => '3.5795298732628'
171 },
172 {
173 'runtimes' => {
174 'system' => '0.49',
175 'user' => '65.33',
176 'total' => '65.82'
177 },
178 'name' => 'Parse-MediaWikiDump.t',
179 'percentage' => 1065,
180 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
181 'MiB/sec' => '3.29891039368158'
182 },
183 {
184 'runtimes' => {
185 'system' => '0.87',
186 'user' => '66.01',
187 'total' => '66.88'
188 },
189 'name' => 'XML-LibXML-SAX.t',
190 'percentage' => 1082,
191 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
192 'MiB/sec' => '3.24662503158076'
193 },
194 {
195 'runtimes' => {
196 'system' => '1.32',
197 'user' => '179.77',
198 'total' => '181.09'
199 },
200 'name' => 'XML-SAX-Expat.t',
201 'percentage' => 2930,
202 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
203 'MiB/sec' => '1.19904070965885'
204 },
205 {
206 'runtimes' => {
207 'system' => '1.95',
208 'user' => '201.49',
209 'total' => '203.44'
210 },
211 'name' => 'XML-Twig.t',
212 'percentage' => 3291,
213 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
214 'MiB/sec' => '1.06731361635923'
215 },
216 {
217 'runtimes' => {
218 'system' => '3.45',
219 'user' => '577.07',
220 'total' => '580.52'
221 },
222 'name' => 'XML-SAX-PurePerl.t',
223 'percentage' => 9393,
224 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
225 'MiB/sec' => '0.374034110990356'
226 }
227 ],
228 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
229 'size' => 227681797
230 }
231 ];
232
233 One of the fastest benchmarks:
234
235 #!/usr/bin/env perl
236
237 use strict;
238 use warnings;
239 use Data::Dumper;
240
241 use XML::LibXML;
242 use XML::LibXML::Reader;
243
244 binmode(STDOUT, ':utf8');
245 binmode(STDERR, ':utf8');
246
247 $| = 1;
248 print '';
249
250 use Bench;
251
252 my $reader = XML::LibXML::Reader->new(location => shift(@ARGV));
253 my $title;
254
255 while(1) {
256 my $type = $reader->nodeType;
257
258 if ($type == XML_READER_TYPE_ELEMENT) {
259 if ($reader->name eq 'title') {
260 $title = get_text($reader);
261 } elsif ($reader->name eq 'text') {
262 my $text = get_text($reader);
263 Bench::Article($title, $text);
264 }
265
266 $reader->nextElement;
267 next;
268 }
269
270 last unless $reader->read;
271 }
272
273 sub get_text {
274 my ($r) = @_;
275 my @buffer;
276 my $type;
277
278 while($r->nodeType != XML_READER_TYPE_TEXT && $r->nodeType != XML_READER_TYPE_END_ELEMENT) {
279 $r->read or die "could not read";
280 }
281
282 while($r->nodeType != XML_READER_TYPE_END_ELEMENT) {
283 if ($r->nodeType == XML_READER_TYPE_TEXT) {
284 push(@buffer, $r->value);
285 }
286
287 $r->read or die "could not read";
288 }
289
290 return join('', @buffer);
291 }
292
293 __END__
294
295
296 TEST DATA
297 You can find various MediaWiki dump files via http://download.wikimedia.org/
298 I use the following various language Wikipedia dump files for my testing:
299
300 http://download.wikimedia.org/cvwiki/20091208/cvwiki-20091208-pages-articles.xml.bz2
301 http://download.wikimedia.org/simplewiki/20091203/simplewiki-20091203-pages-articles.xml.bz2
302 http://download.wikimedia.org/enwiki/20091103/enwiki-20091103-pages-articles.xml.bz2
303
304 TODO
305
306 * It would be nice if the C based parsers were glued to perl with XS so they invoke the
307 Bench::Article method just like the perl based parsers do.
308
309 * One common string buffering library between all C based parsers would be nice
310 but I could not get this functional. There is a lot of other code duplication
311 as well.
312
313 * A C implementation of libxml's reader interface would be fun to compare
314 against the perl one.
315
316 AUTHOR
317
318 Test suite and initial tests created by Tyler Riddle <[email protected]>
319 Please send any patches to me and feel free to add yourself to the
320 contributors list.
321
322 CONTRIBUTORS
323
324 * "Sebastian Bober <[email protected]>" - Concept behind the XML::Bare implementation