| 1 |
ABOUT |
| 2 |
|
| 3 |
This is a benchmark system for XML parsers against various language editions of |
| 4 |
the Wikipedia. The benchmark is to print all the article titles and text of a |
| 5 |
dump file specified on the command line to standard output. There are |
| 6 |
implementations for many perl parsing modules both high and low level. There |
| 7 |
are even implementations written in C that perform very fast. |
| 8 |
|
| 9 |
The benchmark.pl program is used to run a series of benchmarks. It takes two |
| 10 |
required arguments and one optional. The first required argument is a path to |
| 11 |
a directory full of tests to execute. The second required argument is a path |
| 12 |
to a directory full of dump files to execute the tests against. Both of these |
| 13 |
directories will be executed according to sort() on their file names. The third |
| 14 |
argument is a number of iterations to perform, the default being 1. |
| 15 |
|
| 16 |
Output goes to two files: results.log and results.data - they both are the |
| 17 |
output from YAML of an internal data structure that represents the |
| 18 |
test report. The results.log file is written to each time all the tests |
| 19 |
have been run against a specific file and lets you keep an eye on how long |
| 20 |
running jobs are performing. The results.data file is the cumulative data |
| 21 |
for all iterations and is written at the end of the entire run. |
| 22 |
|
| 23 |
The benchmark.pl utility and all of the tests are only guaranteed to work |
| 24 |
if executed from the root directory of this software package. The C based |
| 25 |
parsers are in the bin/ directory and can be compiled by executing make in |
| 26 |
that directory. The Iksemel parser is not currently functional for unknown |
| 27 |
reasons. |
| 28 |
|
| 29 |
THE CHALLENGE |
| 30 |
|
| 31 |
First and foremost the most important thing to keep in mind is that the English |
| 32 |
Wikipedia is currently 22 gigabytes of XML in a single file. You will not be |
| 33 |
able to use any XML processing system that requires the entire document to |
| 34 |
fit into RAM. |
| 35 |
|
| 36 |
Each benchmark must gather up the title and text for each Wikipedia article |
| 37 |
for an arbitrary XML dump file. In the spirit of making this test approximate |
| 38 |
a real world scenario you must collect all character data together and make it |
| 39 |
available at one time. For instance in the perl benchmarks they actually invoke |
| 40 |
a common method that prints the article title and text for them. In the C based |
| 41 |
tests they simply collect all the data and print it out at once. |
| 42 |
|
| 43 |
EXAMPLES |
| 44 |
|
| 45 |
Doing a test run: |
| 46 |
|
| 47 |
foodmotron:XML_Speed_Test tyler$ ./benchmark.pl test_cases data |
| 48 |
Iterations remaining: 1 |
| 49 |
Benchmarking 20-simplewiki-20091021-pages-articles.xml |
| 50 |
Generating md5sum: 8fa1e9de18b8da7523ebfe2dac53482a |
| 51 |
running test_cases/MediaWiki-DumpFile-SimplePages.t data/20-simplewiki-20091021-pages-articles.xml: 12 seconds |
| 52 |
running test_cases/Parse-MediaWikiDump.t data/20-simplewiki-20091021-pages-articles.xml: 66 seconds |
| 53 |
running test_cases/XML-Bare.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds |
| 54 |
running test_cases/XML-LibXML-Reader.t data/20-simplewiki-20091021-pages-articles.xml: 12 seconds |
| 55 |
running test_cases/XML-LibXML-SAX.t data/20-simplewiki-20091021-pages-articles.xml: 68 seconds |
| 56 |
running test_cases/XML-Parser-ExpatNB.t data/20-simplewiki-20091021-pages-articles.xml: 44 seconds |
| 57 |
running test_cases/XML-Parser.t data/20-simplewiki-20091021-pages-articles.xml: 42 seconds |
| 58 |
running test_cases/XML-SAX-Expat.t data/20-simplewiki-20091021-pages-articles.xml: 183 seconds |
| 59 |
running test_cases/XML-SAX-ExpatXS.t data/20-simplewiki-20091021-pages-articles.xml: 33 seconds |
| 60 |
running test_cases/XML-SAX-ExpatXS_nocharjoin.t data/20-simplewiki-20091021-pages-articles.xml: 62 seconds |
| 61 |
running test_cases/XML-SAX-PurePerl.t data/20-simplewiki-20091021-pages-articles.xml: 585 seconds |
| 62 |
running test_cases/XML-Twig.t data/20-simplewiki-20091021-pages-articles.xml: 204 seconds |
| 63 |
running test_cases/expat.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds |
| 64 |
running test_cases/libxml.t data/20-simplewiki-20091021-pages-articles.xml: 7 seconds |
| 65 |
foodmotron:XML_Speed_Test tyler$ |
| 66 |
|
| 67 |
The report: |
| 68 |
|
| 69 |
$VAR1 = [ |
| 70 |
{ |
| 71 |
'filename' => '20-simplewiki-20091021-pages-articles.xml', |
| 72 |
'tests' => [ |
| 73 |
{ |
| 74 |
'runtimes' => { |
| 75 |
'system' => '0.4', |
| 76 |
'user' => '5.78', |
| 77 |
'total' => '6.18' |
| 78 |
}, |
| 79 |
'name' => 'libxml.t', |
| 80 |
'percentage' => 100, |
| 81 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 82 |
'MiB/sec' => '35.1349971055213' |
| 83 |
}, |
| 84 |
{ |
| 85 |
'runtimes' => { |
| 86 |
'system' => '0.37', |
| 87 |
'user' => '6.32', |
| 88 |
'total' => '6.69' |
| 89 |
}, |
| 90 |
'name' => 'XML-Bare.t', |
| 91 |
'percentage' => 108, |
| 92 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 93 |
'MiB/sec' => '32.4565444113784' |
| 94 |
}, |
| 95 |
{ |
| 96 |
'runtimes' => { |
| 97 |
'system' => '0.4', |
| 98 |
'user' => '6.55', |
| 99 |
'total' => '6.95' |
| 100 |
}, |
| 101 |
'name' => 'expat.t', |
| 102 |
'percentage' => 112, |
| 103 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 104 |
'MiB/sec' => '31.2423427499455' |
| 105 |
}, |
| 106 |
{ |
| 107 |
'runtimes' => { |
| 108 |
'system' => '0.83', |
| 109 |
'user' => '10.62', |
| 110 |
'total' => '11.45' |
| 111 |
}, |
| 112 |
'name' => 'XML-LibXML-Reader.t', |
| 113 |
'percentage' => 185, |
| 114 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 115 |
'MiB/sec' => '18.963692760884' |
| 116 |
}, |
| 117 |
{ |
| 118 |
'runtimes' => { |
| 119 |
'system' => '0.42', |
| 120 |
'user' => '11.33', |
| 121 |
'total' => '11.75' |
| 122 |
}, |
| 123 |
'name' => 'MediaWiki-DumpFile-SimplePages.t', |
| 124 |
'percentage' => 190, |
| 125 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 126 |
'MiB/sec' => '18.4795133712444' |
| 127 |
}, |
| 128 |
{ |
| 129 |
'runtimes' => { |
| 130 |
'system' => '0.55', |
| 131 |
'user' => '32', |
| 132 |
'total' => '32.55' |
| 133 |
}, |
| 134 |
'name' => 'XML-SAX-ExpatXS.t', |
| 135 |
'percentage' => 526, |
| 136 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 137 |
'MiB/sec' => '6.67079207717731' |
| 138 |
}, |
| 139 |
{ |
| 140 |
'runtimes' => { |
| 141 |
'system' => '0.26', |
| 142 |
'user' => '41.55', |
| 143 |
'total' => '41.81' |
| 144 |
}, |
| 145 |
'name' => 'XML-Parser.t', |
| 146 |
'percentage' => 676, |
| 147 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 148 |
'MiB/sec' => '5.19335762047648' |
| 149 |
}, |
| 150 |
{ |
| 151 |
'runtimes' => { |
| 152 |
'system' => '0.46', |
| 153 |
'user' => '42.1', |
| 154 |
'total' => '42.56' |
| 155 |
}, |
| 156 |
'name' => 'XML-Parser-ExpatNB.t', |
| 157 |
'percentage' => 688, |
| 158 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 159 |
'MiB/sec' => '5.1018393353412' |
| 160 |
}, |
| 161 |
{ |
| 162 |
'runtimes' => { |
| 163 |
'system' => '0.53', |
| 164 |
'user' => '60.13', |
| 165 |
'total' => '60.66' |
| 166 |
}, |
| 167 |
'name' => 'XML-SAX-ExpatXS_nocharjoin.t', |
| 168 |
'percentage' => 981, |
| 169 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 170 |
'MiB/sec' => '3.5795298732628' |
| 171 |
}, |
| 172 |
{ |
| 173 |
'runtimes' => { |
| 174 |
'system' => '0.49', |
| 175 |
'user' => '65.33', |
| 176 |
'total' => '65.82' |
| 177 |
}, |
| 178 |
'name' => 'Parse-MediaWikiDump.t', |
| 179 |
'percentage' => 1065, |
| 180 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 181 |
'MiB/sec' => '3.29891039368158' |
| 182 |
}, |
| 183 |
{ |
| 184 |
'runtimes' => { |
| 185 |
'system' => '0.87', |
| 186 |
'user' => '66.01', |
| 187 |
'total' => '66.88' |
| 188 |
}, |
| 189 |
'name' => 'XML-LibXML-SAX.t', |
| 190 |
'percentage' => 1082, |
| 191 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 192 |
'MiB/sec' => '3.24662503158076' |
| 193 |
}, |
| 194 |
{ |
| 195 |
'runtimes' => { |
| 196 |
'system' => '1.32', |
| 197 |
'user' => '179.77', |
| 198 |
'total' => '181.09' |
| 199 |
}, |
| 200 |
'name' => 'XML-SAX-Expat.t', |
| 201 |
'percentage' => 2930, |
| 202 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 203 |
'MiB/sec' => '1.19904070965885' |
| 204 |
}, |
| 205 |
{ |
| 206 |
'runtimes' => { |
| 207 |
'system' => '1.95', |
| 208 |
'user' => '201.49', |
| 209 |
'total' => '203.44' |
| 210 |
}, |
| 211 |
'name' => 'XML-Twig.t', |
| 212 |
'percentage' => 3291, |
| 213 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 214 |
'MiB/sec' => '1.06731361635923' |
| 215 |
}, |
| 216 |
{ |
| 217 |
'runtimes' => { |
| 218 |
'system' => '3.45', |
| 219 |
'user' => '577.07', |
| 220 |
'total' => '580.52' |
| 221 |
}, |
| 222 |
'name' => 'XML-SAX-PurePerl.t', |
| 223 |
'percentage' => 9393, |
| 224 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 225 |
'MiB/sec' => '0.374034110990356' |
| 226 |
} |
| 227 |
], |
| 228 |
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a', |
| 229 |
'size' => 227681797 |
| 230 |
} |
| 231 |
]; |
| 232 |
|
| 233 |
One of the fastest benchmarks: |
| 234 |
|
| 235 |
#!/usr/bin/env perl |
| 236 |
|
| 237 |
use strict; |
| 238 |
use warnings; |
| 239 |
use Data::Dumper; |
| 240 |
|
| 241 |
use XML::LibXML; |
| 242 |
use XML::LibXML::Reader; |
| 243 |
|
| 244 |
binmode(STDOUT, ':utf8'); |
| 245 |
binmode(STDERR, ':utf8'); |
| 246 |
|
| 247 |
$| = 1; |
| 248 |
print ''; |
| 249 |
|
| 250 |
use Bench; |
| 251 |
|
| 252 |
my $reader = XML::LibXML::Reader->new(location => shift(@ARGV)); |
| 253 |
my $title; |
| 254 |
|
| 255 |
while(1) { |
| 256 |
my $type = $reader->nodeType; |
| 257 |
|
| 258 |
if ($type == XML_READER_TYPE_ELEMENT) { |
| 259 |
if ($reader->name eq 'title') { |
| 260 |
$title = get_text($reader); |
| 261 |
} elsif ($reader->name eq 'text') { |
| 262 |
my $text = get_text($reader); |
| 263 |
Bench::Article($title, $text); |
| 264 |
} |
| 265 |
|
| 266 |
$reader->nextElement; |
| 267 |
next; |
| 268 |
} |
| 269 |
|
| 270 |
last unless $reader->read; |
| 271 |
} |
| 272 |
|
| 273 |
sub get_text { |
| 274 |
my ($r) = @_; |
| 275 |
my @buffer; |
| 276 |
my $type; |
| 277 |
|
| 278 |
while($r->nodeType != XML_READER_TYPE_TEXT && $r->nodeType != XML_READER_TYPE_END_ELEMENT) { |
| 279 |
$r->read or die "could not read"; |
| 280 |
} |
| 281 |
|
| 282 |
while($r->nodeType != XML_READER_TYPE_END_ELEMENT) { |
| 283 |
if ($r->nodeType == XML_READER_TYPE_TEXT) { |
| 284 |
push(@buffer, $r->value); |
| 285 |
} |
| 286 |
|
| 287 |
$r->read or die "could not read"; |
| 288 |
} |
| 289 |
|
| 290 |
return join('', @buffer); |
| 291 |
} |
| 292 |
|
| 293 |
__END__ |
| 294 |
|
| 295 |
|
| 296 |
TEST DATA |
| 297 |
You can find various MediaWiki dump files via http://download.wikimedia.org/ |
| 298 |
I use the following various language Wikipedia dump files for my testing: |
| 299 |
|
| 300 |
http://download.wikimedia.org/cvwiki/20091208/cvwiki-20091208-pages-articles.xml.bz2 |
| 301 |
http://download.wikimedia.org/simplewiki/20091203/simplewiki-20091203-pages-articles.xml.bz2 |
| 302 |
http://download.wikimedia.org/enwiki/20091103/enwiki-20091103-pages-articles.xml.bz2 |
| 303 |
|
| 304 |
TODO |
| 305 |
|
| 306 |
* It would be nice if the C based parsers were glued to perl with XS so they invoke the |
| 307 |
Bench::Article method just like the perl based parsers do. |
| 308 |
|
| 309 |
* One common string buffering library between all C based parsers would be nice |
| 310 |
but I could not get this functional. There is a lot of other code duplication |
| 311 |
as well. |
| 312 |
|
| 313 |
* A C implementation of libxml's reader interface would be fun to compare |
| 314 |
against the perl one. |
| 315 |
|
| 316 |
AUTHOR |
| 317 |
|
| 318 |
Test suite and initial tests created by Tyler Riddle <[email protected]> |
| 319 |
Please send any patches to me and feel free to add yourself to the |
| 320 |
contributors list. |
| 321 |
|
| 322 |
CONTRIBUTORS |
| 323 |
|
| 324 |
* "Sebastian Bober <[email protected]>" - Concept behind the XML::Bare implementation |