Debian Codegraph @

upstream/debian-unstable/man1/perlperf.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "PERLPERF 1"
58 .TH PERLPERF 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 perlperf \- Perl Performance and Optimization Techniques
65 .SH DESCRIPTION
66 .IX Header "DESCRIPTION"
67 This is an introduction to the use of performance and optimization techniques
68 which can be used with particular reference to perl programs. While many perl
69 developers have come from other languages, and can use their prior knowledge
70 where appropriate, there are many other people who might benefit from a few
71 perl specific pointers. If you want the condensed version, perhaps the best
72 advice comes from the renowned Japanese Samurai, Miyamoto Musashi, who said:
73 .PP
74 .Vb 1
75 \& "Do Not Engage in Useless Activity"
76 .Ve
77 .PP
78 in 1645.
79 .SH OVERVIEW
80 .IX Header "OVERVIEW"
81 Perhaps the most common mistake programmers make is to attempt to optimize
82 their code before a program actually does anything useful \- this is a bad idea.
83 There's no point in having an extremely fast program that doesn't work. The
84 first job is to get a program to \fIcorrectly\fR do something \fBuseful\fR, (not to
85 mention ensuring the test suite is fully functional), and only then to consider
86 optimizing it. Having decided to optimize existing working code, there are
87 several simple but essential steps to consider which are intrinsic to any
88 optimization process.
89 .SS "ONE STEP SIDEWAYS"
90 .IX Subsection "ONE STEP SIDEWAYS"
91 Firstly, you need to establish a baseline time for the existing code, which
92 timing needs to be reliable and repeatable. You'll probably want to use the
93 \&\f(CW\*(C`Benchmark\*(C'\fR or \f(CW\*(C`Devel::NYTProf\*(C'\fR modules, or something similar, for this step,
94 or perhaps the Unix system \f(CW\*(C`time\*(C'\fR utility, whichever is appropriate. See the
95 base of this document for a longer list of benchmarking and profiling modules,
96 and recommended further reading.
97 .SS "ONE STEP FORWARD"
98 .IX Subsection "ONE STEP FORWARD"
99 Next, having examined the program for \fIhot spots\fR, (places where the code
100 seems to run slowly), change the code with the intention of making it run
101 faster. Using version control software, like \f(CW\*(C`subversion\*(C'\fR, will ensure no
102 changes are irreversible. It's too easy to fiddle here and fiddle there \-
103 don't change too much at any one time or you might not discover which piece of
104 code \fBreally\fR was the slow bit.
105 .SS "ANOTHER STEP SIDEWAYS"
106 .IX Subsection "ANOTHER STEP SIDEWAYS"
107 It's not enough to say: "that will make it run faster", you have to check it.
108 Rerun the code under control of the benchmarking or profiling modules, from the
109 first step above, and check that the new code executed the \fBsame task\fR in
110 \&\fIless time\fR. Save your work and repeat...
111 .SH "GENERAL GUIDELINES"
112 .IX Header "GENERAL GUIDELINES"
113 The critical thing when considering performance is to remember there is no such
114 thing as a \f(CW\*(C`Golden Bullet\*(C'\fR, which is why there are no rules, only guidelines.
115 .PP
116 It is clear that inline code is going to be faster than subroutine or method
117 calls, because there is less overhead, but this approach has the disadvantage
118 of being less maintainable and comes at the cost of greater memory usage \-
119 there is no such thing as a free lunch. If you are searching for an element in
120 a list, it can be more efficient to store the data in a hash structure, and
121 then simply look to see whether the key is defined, rather than to loop through
122 the entire array using \fBgrep()\fR for instance. \fBsubstr()\fR may be (a lot) faster
123 than \fBgrep()\fR but not as flexible, so you have another trade-off to access. Your
124 code may contain a line which takes 0.01 of a second to execute which if you
125 call it 1,000 times, quite likely in a program parsing even medium sized files
126 for instance, you already have a 10 second delay, in just one single code
127 location, and if you call that line 100,000 times, your entire program will
128 slow down to an unbearable crawl.
129 .PP
130 Using a subroutine as part of your sort is a powerful way to get exactly what
131 you want, but will usually be slower than the built-in \fIalphabetic\fR \f(CW\*(C`cmp\*(C'\fR and
132 \&\fInumeric\fR \f(CW\*(C`<=>\*(C'\fR sort operators. It is possible to make multiple
133 passes over your data, building indices to make the upcoming sort more
134 efficient, and to use what is known as the \f(CW\*(C`OM\*(C'\fR (Orcish Maneuver) to cache the
135 sort keys in advance. The cache lookup, while a good idea, can itself be a
136 source of slowdown by enforcing a double pass over the data \- once to setup the
137 cache, and once to sort the data. Using \f(CWpack()\fR to extract the required sort
138 key into a consistent string can be an efficient way to build a single string
139 to compare, instead of using multiple sort keys, which makes it possible to use
140 the standard, written in \f(CW\*(C`c\*(C'\fR and fast, perl \f(CWsort()\fR function on the output,
141 and is the basis of the \f(CW\*(C`GRT\*(C'\fR (Guttman Rossler Transform). Some string
142 combinations can slow the \f(CW\*(C`GRT\*(C'\fR down, by just being too plain complex for its
143 own good.
144 .PP
145 For applications using database backends, the standard \f(CW\*(C`DBIx\*(C'\fR namespace has
146 tried to help with keeping things nippy, not least because it tries to \fInot\fR
147 query the database until the latest possible moment, but always read the docs
148 which come with your choice of libraries. Among the many issues facing
149 developers dealing with databases should remain aware of is to always use
150 \&\f(CW\*(C`SQL\*(C'\fR placeholders and to consider pre-fetching data sets when this might
151 prove advantageous. Splitting up a large file by assigning multiple processes
152 to parsing a single file, using say \f(CW\*(C`POE\*(C'\fR, \f(CW\*(C`threads\*(C'\fR or \f(CW\*(C`fork\*(C'\fR can also be a
153 useful way of optimizing your usage of the available \f(CW\*(C`CPU\*(C'\fR resources, though
154 this technique is fraught with concurrency issues and demands high attention to
155 detail.
156 .PP
157 Every case has a specific application and one or more exceptions, and there is
158 no replacement for running a few tests and finding out which method works best
159 for your particular environment, this is why writing optimal code is not an
160 exact science, and why we love using Perl so much \- TMTOWTDI.
161 .SH BENCHMARKS
162 .IX Header "BENCHMARKS"
163 Here are a few examples to demonstrate usage of Perl's benchmarking tools.
164 .SS "Assigning and Dereferencing Variables"
165 .IX Subsection "Assigning and Dereferencing Variables"
166 I'm sure most of us have seen code which looks like, (or worse than), this:
167 .PP
168 .Vb 2
169 \& if ( $obj\->{_ref}\->{_myscore} >= $obj\->{_ref}\->{_yourscore} ) {
170 \& ...
171 .Ve
172 .PP
173 This sort of code can be a real eyesore to read, as well as being very
174 sensitive to typos, and it's much clearer to dereference the variable
175 explicitly. We're side-stepping the issue of working with object-oriented
176 programming techniques to encapsulate variable access via methods, only
177 accessible through an object. Here we're just discussing the technical
178 implementation of choice, and whether this has an effect on performance. We
179 can see whether this dereferencing operation, has any overhead by putting
180 comparative code in a file and running a \f(CW\*(C`Benchmark\*(C'\fR test.
181 .PP
182 # dereference
183 .PP
184 .Vb 1
185 \& #!/usr/bin/perl
186 \&
187 \& use v5.36;
188 \&
189 \& use Benchmark;
190 \&
191 \& my $ref = {
192 \& \*(Aqref\*(Aq => {
193 \& _myscore => \*(Aq100 + 1\*(Aq,
194 \& _yourscore => \*(Aq102 \- 1\*(Aq,
195 \& },
196 \& };
197 \&
198 \& timethese(1000000, {
199 \& \*(Aqdirect\*(Aq => sub {
200 \& my $x = $ref\->{ref}\->{_myscore} . $ref\->{ref}\->{_yourscore} ;
201 \& },
202 \& \*(Aqdereference\*(Aq => sub {
203 \& my $ref = $ref\->{ref};
204 \& my $myscore = $ref\->{_myscore};
205 \& my $yourscore = $ref\->{_yourscore};
206 \& my $x = $myscore . $yourscore;
207 \& },
208 \& });
209 .Ve
210 .PP
211 It's essential to run any timing measurements a sufficient number of times so
212 the numbers settle on a numerical average, otherwise each run will naturally
213 fluctuate due to variations in the environment, to reduce the effect of
214 contention for \f(CW\*(C`CPU\*(C'\fR resources and network bandwidth for instance. Running
215 the above code for one million iterations, we can take a look at the report
216 output by the \f(CW\*(C`Benchmark\*(C'\fR module, to see which approach is the most effective.
217 .PP
218 .Vb 1
219 \& $> perl dereference
220 \&
221 \& Benchmark: timing 1000000 iterations of dereference, direct...
222 \& dereference: 2 wallclock secs ( 1.59 usr + 0.00 sys = 1.59 CPU) @ 628930.82/s (n=1000000)
223 \& direct: 1 wallclock secs ( 1.20 usr + 0.00 sys = 1.20 CPU) @ 833333.33/s (n=1000000)
224 .Ve
225 .PP
226 The difference is clear to see and the dereferencing approach is slower. While
227 it managed to execute an average of 628,930 times a second during our test, the
228 direct approach managed to run an additional 204,403 times, unfortunately.
229 Unfortunately, because there are many examples of code written using the
230 multiple layer direct variable access, and it's usually horrible. It is,
231 however, minusculy faster. The question remains whether the minute gain is
232 actually worth the eyestrain, or the loss of maintainability.
233 .SS "Search and replace or tr"
234 .IX Subsection "Search and replace or tr"
235 If we have a string which needs to be modified, while a regex will almost
236 always be much more flexible, \f(CW\*(C`tr\*(C'\fR, an oft underused tool, can still be a
237 useful. One scenario might be replace all vowels with another character. The
238 regex solution might look like this:
239 .PP
240 .Vb 1
241 \& $str =~ s/[aeiou]/x/g
242 .Ve
243 .PP
244 The \f(CW\*(C`tr\*(C'\fR alternative might look like this:
245 .PP
246 .Vb 1
247 \& $str =~ tr/aeiou/xxxxx/
248 .Ve
249 .PP
250 We can put that into a test file which we can run to check which approach is
251 the fastest, using a global \f(CW$STR\fR variable to assign to the \f(CW\*(C`my $str\*(C'\fR
252 variable so as to avoid perl trying to optimize any of the work away by
253 noticing it's assigned only the once.
254 .PP
255 # regex-transliterate
256 .PP
257 .Vb 1
258 \& #!/usr/bin/perl
259 \&
260 \& use v5.36;
261 \&
262 \& use Benchmark;
263 \&
264 \& my $STR = "$$\-this and that";
265 \&
266 \& timethese( 1000000, {
267 \& \*(Aqsr\*(Aq => sub { my $str = $STR; $str =~ s/[aeiou]/x/g; return $str; },
268 \& \*(Aqtr\*(Aq => sub { my $str = $STR; $str =~ tr/aeiou/xxxxx/; return $str; },
269 \& });
270 .Ve
271 .PP
272 Running the code gives us our results:
273 .PP
274 .Vb 1
275 \& $> perl regex\-transliterate
276 \&
277 \& Benchmark: timing 1000000 iterations of sr, tr...
278 \& sr: 2 wallclock secs ( 1.19 usr + 0.00 sys = 1.19 CPU) @ 840336.13/s (n=1000000)
279 \& tr: 0 wallclock secs ( 0.49 usr + 0.00 sys = 0.49 CPU) @ 2040816.33/s (n=1000000)
280 .Ve
281 .PP
282 The \f(CW\*(C`tr\*(C'\fR version is a clear winner. One solution is flexible, the other is
283 fast \- and it's appropriately the programmer's choice which to use.
284 .PP
285 Check the \f(CW\*(C`Benchmark\*(C'\fR docs for further useful techniques.
286 .SH "PROFILING TOOLS"
287 .IX Header "PROFILING TOOLS"
288 A slightly larger piece of code will provide something on which a profiler can
289 produce more extensive reporting statistics. This example uses the simplistic
290 \&\f(CW\*(C`wordmatch\*(C'\fR program which parses a given input file and spews out a short
291 report on the contents.
292 .PP
293 # wordmatch
294 .PP
295 .Vb 1
296 \& #!/usr/bin/perl
297 \&
298 \& use v5.36;
299 \&
300 \& =head1 NAME
301 \&
302 \& filewords \- word analysis of input file
303 \&
304 \& =head1 SYNOPSIS
305 \&
306 \& filewords \-f inputfilename [\-d]
307 \&
308 \& =head1 DESCRIPTION
309 \&
310 \& This program parses the given filename, specified with C<\-f>, and
311 \& displays a simple analysis of the words found therein. Use the C<\-d>
312 \& switch to enable debugging messages.
313 \&
314 \& =cut
315 \&
316 \& use FileHandle;
317 \& use Getopt::Long;
318 \&
319 \& my $debug = 0;
320 \& my $file = \*(Aq\*(Aq;
321 \&
322 \& my $result = GetOptions (
323 \& \*(Aqdebug\*(Aq => \e$debug,
324 \& \*(Aqfile=s\*(Aq => \e$file,
325 \& );
326 \& die("invalid args") unless $result;
327 \&
328 \& unless ( \-f $file ) {
329 \& die("Usage: $0 \-f filename [\-d]");
330 \& }
331 \& my $FH = FileHandle\->new("< $file")
332 \& or die("unable to open file($file): $!");
333 \&
334 \& my $i_LINES = 0;
335 \& my $i_WORDS = 0;
336 \& my %count = ();
337 \&
338 \& my @lines = <$FH>;
339 \& foreach my $line ( @lines ) {
340 \& $i_LINES++;
341 \& $line =~ s/\en//;
342 \& my @words = split(/ +/, $line);
343 \& my $i_words = scalar(@words);
344 \& $i_WORDS = $i_WORDS + $i_words;
345 \& debug("line: $i_LINES supplying $i_words words: @words");
346 \& my $i_word = 0;
347 \& foreach my $word ( @words ) {
348 \& $i_word++;
349 \& $count{$i_LINES}{spec} += matches($i_word, $word,
350 \& \*(Aq[^a\-zA\-Z0\-9]\*(Aq);
351 \& $count{$i_LINES}{only} += matches($i_word, $word,
352 \& \*(Aq^[^a\-zA\-Z0\-9]+$\*(Aq);
353 \& $count{$i_LINES}{cons} += matches($i_word, $word,
354 \& \*(Aq^[(?i:bcdfghjklmnpqrstvwxyz)]+$\*(Aq);
355 \& $count{$i_LINES}{vows} += matches($i_word, $word,
356 \& \*(Aq^[(?i:aeiou)]+$\*(Aq);
357 \& $count{$i_LINES}{caps} += matches($i_word, $word,
358 \& \*(Aq^[(A\-Z)]+$\*(Aq);
359 \& }
360 \& }
361 \&
362 \& print report( %count );
363 \&
364 \& sub matches {
365 \& my $i_wd = shift;
366 \& my $word = shift;
367 \& my $regex = shift;
368 \& my $has = 0;
369 \&
370 \& if ( $word =~ /($regex)/ ) {
371 \& $has++ if $1;
372 \& }
373 \&
374 \& debug( "word: $i_wd "
375 \& . ($has ? \*(Aqmatches\*(Aq : \*(Aqdoes not match\*(Aq)
376 \& . " chars: /$regex/");
377 \&
378 \& return $has;
379 \& }
380 \&
381 \& sub report {
382 \& my %report = @_;
383 \& my %rep;
384 \&
385 \& foreach my $line ( keys %report ) {
386 \& foreach my $key ( keys $report{$line}\->%* ) {
387 \& $rep{$key} += $report{$line}{$key};
388 \& }
389 \& }
390 \&
391 \& my $report = qq|
392 \& $0 report for $file:
393 \& lines in file: $i_LINES
394 \& words in file: $i_WORDS
395 \& words with special (non\-word) characters: $i_spec
396 \& words with only special (non\-word) characters: $i_only
397 \& words with only consonants: $i_cons
398 \& words with only capital letters: $i_caps
399 \& words with only vowels: $i_vows
400 \& |;
401 \&
402 \& return $report;
403 \& }
404 \&
405 \& sub debug {
406 \& my $message = shift;
407 \&
408 \& if ( $debug ) {
409 \& print STDERR "DBG: $message\en";
410 \& }
411 \& }
412 \&
413 \& exit 0;
414 .Ve
415 .SS Devel::DProf
416 .IX Subsection "Devel::DProf"
417 This venerable module has been the de-facto standard for Perl code profiling
418 for more than a decade, but has been replaced by a number of other modules
419 which have brought us back to the 21st century. Although you're recommended to
420 evaluate your tool from the several mentioned here and from the CPAN list at
421 the base of this document, (and currently Devel::NYTProf seems to be the
422 weapon of choice \- see below), we'll take a quick look at the output from
423 Devel::DProf first, to set a baseline for Perl profiling tools. Run the
424 above program under the control of \f(CW\*(C`Devel::DProf\*(C'\fR by using the \f(CW\*(C`\-d\*(C'\fR switch on
425 the command-line.
426 .PP
427 .Vb 1
428 \& $> perl \-d:DProf wordmatch \-f perl5db.pl
429 \&
430 \& <...multiple lines snipped...>
431 \&
432 \& wordmatch report for perl5db.pl:
433 \& lines in file: 9428
434 \& words in file: 50243
435 \& words with special (non\-word) characters: 20480
436 \& words with only special (non\-word) characters: 7790
437 \& words with only consonants: 4801
438 \& words with only capital letters: 1316
439 \& words with only vowels: 1701
440 .Ve
441 .PP
442 \&\f(CW\*(C`Devel::DProf\*(C'\fR produces a special file, called \fItmon.out\fR by default, and
443 this file is read by the \f(CW\*(C`dprofpp\*(C'\fR program, which is already installed as part
444 of the \f(CW\*(C`Devel::DProf\*(C'\fR distribution. If you call \f(CW\*(C`dprofpp\*(C'\fR with no options,
445 it will read the \fItmon.out\fR file in the current directory and produce a human
446 readable statistics report of the run of your program. Note that this may take
447 a little time.
448 .PP
449 .Vb 1
450 \& $> dprofpp
451 \&
452 \& Total Elapsed Time = 2.951677 Seconds
453 \& User+System Time = 2.871677 Seconds
454 \& Exclusive Times
455 \& %Time ExclSec CumulS #Calls sec/call Csec/c Name
456 \& 102. 2.945 3.003 251215 0.0000 0.0000 main::matches
457 \& 2.40 0.069 0.069 260643 0.0000 0.0000 main::debug
458 \& 1.74 0.050 0.050 1 0.0500 0.0500 main::report
459 \& 1.04 0.030 0.049 4 0.0075 0.0123 main::BEGIN
460 \& 0.35 0.010 0.010 3 0.0033 0.0033 Exporter::as_heavy
461 \& 0.35 0.010 0.010 7 0.0014 0.0014 IO::File::BEGIN
462 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::FindOption
463 \& 0.00 \- \-0.000 1 \- \- Symbol::BEGIN
464 \& 0.00 \- \-0.000 1 \- \- Fcntl::BEGIN
465 \& 0.00 \- \-0.000 1 \- \- Fcntl::bootstrap
466 \& 0.00 \- \-0.000 1 \- \- warnings::BEGIN
467 \& 0.00 \- \-0.000 1 \- \- IO::bootstrap
468 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::ConfigDefaults
469 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::Configure
470 \& 0.00 \- \-0.000 1 \- \- Symbol::gensym
471 .Ve
472 .PP
473 \&\f(CW\*(C`dprofpp\*(C'\fR will produce some quite detailed reporting on the activity of the
474 \&\f(CW\*(C`wordmatch\*(C'\fR program. The wallclock, user and system, times are at the top of
475 the analysis, and after this are the main columns defining which define the
476 report. Check the \f(CW\*(C`dprofpp\*(C'\fR docs for details of the many options it supports.
477 .PP
478 See also \f(CW\*(C`Apache::DProf\*(C'\fR which hooks \f(CW\*(C`Devel::DProf\*(C'\fR into \f(CW\*(C`mod_perl\*(C'\fR.
479 .SS Devel::Profiler
480 .IX Subsection "Devel::Profiler"
481 Let's take a look at the same program using a different profiler:
482 \&\f(CW\*(C`Devel::Profiler\*(C'\fR, a drop-in Perl-only replacement for \f(CW\*(C`Devel::DProf\*(C'\fR. The
483 usage is very slightly different in that instead of using the special \f(CW\*(C`\-d:\*(C'\fR
484 flag, you pull \f(CW\*(C`Devel::Profiler\*(C'\fR in directly as a module using \f(CW\*(C`\-M\*(C'\fR.
485 .PP
486 .Vb 1
487 \& $> perl \-MDevel::Profiler wordmatch \-f perl5db.pl
488 \&
489 \& <...multiple lines snipped...>
490 \&
491 \& wordmatch report for perl5db.pl:
492 \& lines in file: 9428
493 \& words in file: 50243
494 \& words with special (non\-word) characters: 20480
495 \& words with only special (non\-word) characters: 7790
496 \& words with only consonants: 4801
497 \& words with only capital letters: 1316
498 \& words with only vowels: 1701
499 .Ve
500 .PP
501 \&\f(CW\*(C`Devel::Profiler\*(C'\fR generates a tmon.out file which is compatible with the
502 \&\f(CW\*(C`dprofpp\*(C'\fR program, thus saving the construction of a dedicated statistics
503 reader program. \f(CW\*(C`dprofpp\*(C'\fR usage is therefore identical to the above example.
504 .PP
505 .Vb 1
506 \& $> dprofpp
507 \&
508 \& Total Elapsed Time = 20.984 Seconds
509 \& User+System Time = 19.981 Seconds
510 \& Exclusive Times
511 \& %Time ExclSec CumulS #Calls sec/call Csec/c Name
512 \& 49.0 9.792 14.509 251215 0.0000 0.0001 main::matches
513 \& 24.4 4.887 4.887 260643 0.0000 0.0000 main::debug
514 \& 0.25 0.049 0.049 1 0.0490 0.0490 main::report
515 \& 0.00 0.000 0.000 1 0.0000 0.0000 Getopt::Long::GetOptions
516 \& 0.00 0.000 0.000 2 0.0000 0.0000 Getopt::Long::ParseOptionSpec
517 \& 0.00 0.000 0.000 1 0.0000 0.0000 Getopt::Long::FindOption
518 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::File::new
519 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::Handle::new
520 \& 0.00 0.000 0.000 1 0.0000 0.0000 Symbol::gensym
521 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::File::open
522 .Ve
523 .PP
524 Interestingly we get slightly different results, which is mostly because the
525 algorithm which generates the report is different, even though the output file
526 format was allegedly identical. The elapsed, user and system times are clearly
527 showing the time it took for \f(CW\*(C`Devel::Profiler\*(C'\fR to execute its own run, but
528 the column listings feel more accurate somehow than the ones we had earlier
529 from \f(CW\*(C`Devel::DProf\*(C'\fR. The 102% figure has disappeared, for example. This is
530 where we have to use the tools at our disposal, and recognise their pros and
531 cons, before using them. Interestingly, the numbers of calls for each
532 subroutine are identical in the two reports, it's the percentages which differ.
533 As the author of \f(CW\*(C`Devel::Profiler\*(C'\fR writes:
534 .PP
535 .Vb 6
536 \& ...running HTML::Template\*(Aqs test suite under Devel::DProf shows
537 \& output() taking NO time but Devel::Profiler shows around 10% of the
538 \& time is in output(). I don\*(Aqt know which to trust but my gut tells me
539 \& something is wrong with Devel::DProf. HTML::Template::output() is a
540 \& big routine that\*(Aqs called for every test. Either way, something needs
541 \& fixing.
542 .Ve
543 .PP
544 YMMV.
545 .PP
546 See also \f(CW\*(C`Devel::Apache::Profiler\*(C'\fR which hooks \f(CW\*(C`Devel::Profiler\*(C'\fR
547 into \f(CW\*(C`mod_perl\*(C'\fR.
548 .SS Devel::SmallProf
549 .IX Subsection "Devel::SmallProf"
550 The \f(CW\*(C`Devel::SmallProf\*(C'\fR profiler examines the runtime of your Perl program and
551 produces a line-by-line listing to show how many times each line was called,
552 and how long each line took to execute. It is called by supplying the familiar
553 \&\f(CW\*(C`\-d\*(C'\fR flag to Perl at runtime.
554 .PP
555 .Vb 1
556 \& $> perl \-d:SmallProf wordmatch \-f perl5db.pl
557 \&
558 \& <...multiple lines snipped...>
559 \&
560 \& wordmatch report for perl5db.pl:
561 \& lines in file: 9428
562 \& words in file: 50243
563 \& words with special (non\-word) characters: 20480
564 \& words with only special (non\-word) characters: 7790
565 \& words with only consonants: 4801
566 \& words with only capital letters: 1316
567 \& words with only vowels: 1701
568 .Ve
569 .PP
570 \&\f(CW\*(C`Devel::SmallProf\*(C'\fR writes its output into a file called \fIsmallprof.out\fR, by
571 default. The format of the file looks like this:
572 .PP
573 .Vb 1
574 \& <num> <time> <ctime> <line>:<text>
575 .Ve
576 .PP
577 When the program has terminated, the output may be examined and sorted using
578 any standard text filtering utilities. Something like the following may be
579 sufficient:
580 .PP
581 .Vb 1
582 \& $> cat smallprof.out | grep \ed*: | sort \-k3 | tac | head \-n20
583 \&
584 \& 251215 1.65674 7.68000 75: if ( $word =~ /($regex)/ ) {
585 \& 251215 0.03264 4.40000 79: debug("word: $i_wd ".($has ? \*(Aqmatches\*(Aq :
586 \& 251215 0.02693 4.10000 81: return $has;
587 \& 260643 0.02841 4.07000 128: if ( $debug ) {
588 \& 260643 0.02601 4.04000 126: my $message = shift;
589 \& 251215 0.02641 3.91000 73: my $has = 0;
590 \& 251215 0.03311 3.71000 70: my $i_wd = shift;
591 \& 251215 0.02699 3.69000 72: my $regex = shift;
592 \& 251215 0.02766 3.68000 71: my $word = shift;
593 \& 50243 0.59726 1.00000 59: $count{$i_LINES}{cons} =
594 \& 50243 0.48175 0.92000 61: $count{$i_LINES}{spec} =
595 \& 50243 0.00644 0.89000 56: my $i_cons = matches($i_word, $word,
596 \& 50243 0.48837 0.88000 63: $count{$i_LINES}{caps} =
597 \& 50243 0.00516 0.88000 58: my $i_caps = matches($i_word, $word, \*(Aq^[(A\-
598 \& 50243 0.00631 0.81000 54: my $i_spec = matches($i_word, $word, \*(Aq[^a\-
599 \& 50243 0.00496 0.80000 57: my $i_vows = matches($i_word, $word,
600 \& 50243 0.00688 0.80000 53: $i_word++;
601 \& 50243 0.48469 0.79000 62: $count{$i_LINES}{only} =
602 \& 50243 0.48928 0.77000 60: $count{$i_LINES}{vows} =
603 \& 50243 0.00683 0.75000 55: my $i_only = matches($i_word, $word, \*(Aq^[^a\-
604 .Ve
605 .PP
606 You can immediately see a slightly different focus to the subroutine profiling
607 modules, and we start to see exactly which line of code is taking the most
608 time. That regex line is looking a bit suspicious, for example. Remember that
609 these tools are supposed to be used together, there is no single best way to
610 profile your code, you need to use the best tools for the job.
611 .PP
612 See also \f(CW\*(C`Apache::SmallProf\*(C'\fR which hooks \f(CW\*(C`Devel::SmallProf\*(C'\fR into
613 \&\f(CW\*(C`mod_perl\*(C'\fR.
614 .SS Devel::FastProf
615 .IX Subsection "Devel::FastProf"
616 \&\f(CW\*(C`Devel::FastProf\*(C'\fR is another Perl line profiler. This was written with a view
617 to getting a faster line profiler, than is possible with for example
618 \&\f(CW\*(C`Devel::SmallProf\*(C'\fR, because it's written in \f(CW\*(C`C\*(C'\fR. To use \f(CW\*(C`Devel::FastProf\*(C'\fR,
619 supply the \f(CW\*(C`\-d\*(C'\fR argument to Perl:
620 .PP
621 .Vb 1
622 \& $> perl \-d:FastProf wordmatch \-f perl5db.pl
623 \&
624 \& <...multiple lines snipped...>
625 \&
626 \& wordmatch report for perl5db.pl:
627 \& lines in file: 9428
628 \& words in file: 50243
629 \& words with special (non\-word) characters: 20480
630 \& words with only special (non\-word) characters: 7790
631 \& words with only consonants: 4801
632 \& words with only capital letters: 1316
633 \& words with only vowels: 1701
634 .Ve
635 .PP
636 \&\f(CW\*(C`Devel::FastProf\*(C'\fR writes statistics to the file \fIfastprof.out\fR in the current
637 directory. The output file, which can be specified, can be interpreted by using
638 the \f(CW\*(C`fprofpp\*(C'\fR command-line program.
639 .PP
640 .Vb 1
641 \& $> fprofpp | head \-n20
642 \&
643 \& # fprofpp output format is:
644 \& # filename:line time count: source
645 \& wordmatch:75 3.93338 251215: if ( $word =~ /($regex)/ ) {
646 \& wordmatch:79 1.77774 251215: debug("word: $i_wd ".($has ? \*(Aqmatches\*(Aq : \*(Aqdoes not match\*(Aq)." chars: /$regex/");
647 \& wordmatch:81 1.47604 251215: return $has;
648 \& wordmatch:126 1.43441 260643: my $message = shift;
649 \& wordmatch:128 1.42156 260643: if ( $debug ) {
650 \& wordmatch:70 1.36824 251215: my $i_wd = shift;
651 \& wordmatch:71 1.36739 251215: my $word = shift;
652 \& wordmatch:72 1.35939 251215: my $regex = shift;
653 .Ve
654 .PP
655 Straightaway we can see that the number of times each line has been called is
656 identical to the \f(CW\*(C`Devel::SmallProf\*(C'\fR output, and the sequence is only very
657 slightly different based on the ordering of the amount of time each line took
658 to execute, \f(CW\*(C`if ( $debug ) { \*(C'\fR and \f(CW\*(C`my $message = shift;\*(C'\fR, for example. The
659 differences in the actual times recorded might be in the algorithm used
660 internally, or it could be due to system resource limitations or contention.
661 .PP
662 See also the DBIx::Profile which will profile database queries running
663 under the \f(CW\*(C`DBIx::*\*(C'\fR namespace.
664 .SS Devel::NYTProf
665 .IX Subsection "Devel::NYTProf"
666 \&\f(CW\*(C`Devel::NYTProf\*(C'\fR is the \fBnext generation\fR of Perl code profiler, fixing many
667 shortcomings in other tools and implementing many cool features. First of all it
668 can be used as either a \fIline\fR profiler, a \fIblock\fR or a \fIsubroutine\fR
669 profiler, all at once. It can also use sub-microsecond (100ns) resolution on
670 systems which provide \f(CWclock_gettime()\fR. It can be started and stopped even
671 by the program being profiled. It's a one-line entry to profile \f(CW\*(C`mod_perl\*(C'\fR
672 applications. It's written in \f(CW\*(C`c\*(C'\fR and is probably the fastest profiler
673 available for Perl. The list of coolness just goes on. Enough of that, let's
674 see how to it works \- just use the familiar \f(CW\*(C`\-d\*(C'\fR switch to plug it in and run
675 the code.
676 .PP
677 .Vb 1
678 \& $> perl \-d:NYTProf wordmatch \-f perl5db.pl
679 \&
680 \& wordmatch report for perl5db.pl:
681 \& lines in file: 9427
682 \& words in file: 50243
683 \& words with special (non\-word) characters: 20480
684 \& words with only special (non\-word) characters: 7790
685 \& words with only consonants: 4801
686 \& words with only capital letters: 1316
687 \& words with only vowels: 1701
688 .Ve
689 .PP
690 \&\f(CW\*(C`NYTProf\*(C'\fR will generate a report database into the file \fInytprof.out\fR by
691 default. Human readable reports can be generated from here by using the
692 supplied \f(CW\*(C`nytprofhtml\*(C'\fR (HTML output) and \f(CW\*(C`nytprofcsv\*(C'\fR (CSV output) programs.
693 We've used the Unix system \f(CW\*(C`html2text\*(C'\fR utility to convert the
694 \&\fInytprof/index.html\fR file for convenience here.
695 .PP
696 .Vb 1
697 \& $> html2text nytprof/index.html
698 \&
699 \& Performance Profile Index
700 \& For wordmatch
701 \& Run on Fri Sep 26 13:46:39 2008
702 \& Reported on Fri Sep 26 13:47:23 2008
703 \&
704 \& Top 15 Subroutines \-\- ordered by exclusive time
705 \& |Calls |P |F |Inclusive|Exclusive|Subroutine |
706 \& | | | |Time |Time | |
707 \& |251215|5 |1 |13.09263 |10.47692 |main:: |matches |
708 \& |260642|2 |1 |2.71199 |2.71199 |main:: |debug |
709 \& |1 |1 |1 |0.21404 |0.21404 |main:: |report |
710 \& |2 |2 |2 |0.00511 |0.00511 |XSLoader:: |load (xsub) |
711 \& |14 |14|7 |0.00304 |0.00298 |Exporter:: |import |
712 \& |3 |1 |1 |0.00265 |0.00254 |Exporter:: |as_heavy |
713 \& |10 |10|4 |0.00140 |0.00140 |vars:: |import |
714 \& |13 |13|1 |0.00129 |0.00109 |constant:: |import |
715 \& |1 |1 |1 |0.00360 |0.00096 |FileHandle:: |import |
716 \& |3 |3 |3 |0.00086 |0.00074 |warnings::register::|import |
717 \& |9 |3 |1 |0.00036 |0.00036 |strict:: |bits |
718 \& |13 |13|13|0.00032 |0.00029 |strict:: |import |
719 \& |2 |2 |2 |0.00020 |0.00020 |warnings:: |import |
720 \& |2 |1 |1 |0.00020 |0.00020 |Getopt::Long:: |ParseOptionSpec|
721 \& |7 |7 |6 |0.00043 |0.00020 |strict:: |unimport |
722 \&
723 \& For more information see the full list of 189 subroutines.
724 .Ve
725 .PP
726 The first part of the report already shows the critical information regarding
727 which subroutines are using the most time. The next gives some statistics
728 about the source files profiled.
729 .PP
730 .Vb 10
731 \& Source Code Files \-\- ordered by exclusive time then name
732 \& |Stmts |Exclusive|Avg. |Reports |Source File |
733 \& | |Time | | | |
734 \& |2699761|15.66654 |6e\-06 |line . block . sub|wordmatch |
735 \& |35 |0.02187 |0.00062|line . block . sub|IO/Handle.pm |
736 \& |274 |0.01525 |0.00006|line . block . sub|Getopt/Long.pm |
737 \& |20 |0.00585 |0.00029|line . block . sub|Fcntl.pm |
738 \& |128 |0.00340 |0.00003|line . block . sub|Exporter/Heavy.pm |
739 \& |42 |0.00332 |0.00008|line . block . sub|IO/File.pm |
740 \& |261 |0.00308 |0.00001|line . block . sub|Exporter.pm |
741 \& |323 |0.00248 |8e\-06 |line . block . sub|constant.pm |
742 \& |12 |0.00246 |0.00021|line . block . sub|File/Spec/Unix.pm |
743 \& |191 |0.00240 |0.00001|line . block . sub|vars.pm |
744 \& |77 |0.00201 |0.00003|line . block . sub|FileHandle.pm |
745 \& |12 |0.00198 |0.00016|line . block . sub|Carp.pm |
746 \& |14 |0.00175 |0.00013|line . block . sub|Symbol.pm |
747 \& |15 |0.00130 |0.00009|line . block . sub|IO.pm |
748 \& |22 |0.00120 |0.00005|line . block . sub|IO/Seekable.pm |
749 \& |198 |0.00085 |4e\-06 |line . block . sub|warnings/register.pm|
750 \& |114 |0.00080 |7e\-06 |line . block . sub|strict.pm |
751 \& |47 |0.00068 |0.00001|line . block . sub|warnings.pm |
752 \& |27 |0.00054 |0.00002|line . block . sub|overload.pm |
753 \& |9 |0.00047 |0.00005|line . block . sub|SelectSaver.pm |
754 \& |13 |0.00045 |0.00003|line . block . sub|File/Spec.pm |
755 \& |2701595|15.73869 | |Total |
756 \& |128647 |0.74946 | |Average |
757 \& | |0.00201 |0.00003|Median |
758 \& | |0.00121 |0.00003|Deviation |
759 \&
760 \& Report produced by the NYTProf 2.03 Perl profiler, developed by Tim Bunce and
761 \& Adam Kaplan.
762 .Ve
763 .PP
764 At this point, if you're using the \fIhtml\fR report, you can click through the
765 various links to bore down into each subroutine and each line of code. Because
766 we're using the text reporting here, and there's a whole directory full of
767 reports built for each source file, we'll just display a part of the
768 corresponding \fIwordmatch\-line.html\fR file, sufficient to give an idea of the
769 sort of output you can expect from this cool tool.
770 .PP
771 .Vb 1
772 \& $> html2text nytprof/wordmatch\-line.html
773 \&
774 \& Performance Profile \-\- \-block view\-.\-line view\-.\-sub view\-
775 \& For wordmatch
776 \& Run on Fri Sep 26 13:46:39 2008
777 \& Reported on Fri Sep 26 13:47:22 2008
778 \&
779 \& File wordmatch
780 \&
781 \& Subroutines \-\- ordered by exclusive time
782 \& |Calls |P|F|Inclusive|Exclusive|Subroutine |
783 \& | | | |Time |Time | |
784 \& |251215|5|1|13.09263 |10.47692 |main::|matches|
785 \& |260642|2|1|2.71199 |2.71199 |main::|debug |
786 \& |1 |1|1|0.21404 |0.21404 |main::|report |
787 \& |0 |0|0|0 |0 |main::|BEGIN |
788 \&
789 \&
790 \& |Line|Stmts.|Exclusive|Avg. |Code |
791 \& | | |Time | | |
792 \& |1 | | | |#!/usr/bin/perl |
793 \& |2 | | | | |
794 \& | | | | |use strict; |
795 \& |3 |3 |0.00086 |0.00029|# spent 0.00003s making 1 calls to strict:: |
796 \& | | | | |import |
797 \& | | | | |use warnings; |
798 \& |4 |3 |0.01563 |0.00521|# spent 0.00012s making 1 calls to warnings:: |
799 \& | | | | |import |
800 \& |5 | | | | |
801 \& |6 | | | |=head1 NAME |
802 \& |7 | | | | |
803 \& |8 | | | |filewords \- word analysis of input file |
804 \& <...snip...>
805 \& |62 |1 |0.00445 |0.00445|print report( %count ); |
806 \& | | | | |# spent 0.21404s making 1 calls to main::report|
807 \& |63 | | | | |
808 \& | | | | |# spent 23.56955s (10.47692+2.61571) within |
809 \& | | | | |main::matches which was called 251215 times, |
810 \& | | | | |avg 0.00005s/call: # 50243 times |
811 \& | | | | |(2.12134+0.51939s) at line 57 of wordmatch, avg|
812 \& | | | | |0.00005s/call # 50243 times (2.17735+0.54550s) |
813 \& |64 | | | |at line 56 of wordmatch, avg 0.00005s/call # |
814 \& | | | | |50243 times (2.10992+0.51797s) at line 58 of |
815 \& | | | | |wordmatch, avg 0.00005s/call # 50243 times |
816 \& | | | | |(2.12696+0.51598s) at line 55 of wordmatch, avg|
817 \& | | | | |0.00005s/call # 50243 times (1.94134+0.51687s) |
818 \& | | | | |at line 54 of wordmatch, avg 0.00005s/call |
819 \& | | | | |sub matches { |
820 \& <...snip...>
821 \& |102 | | | | |
822 \& | | | | |# spent 2.71199s within main::debug which was |
823 \& | | | | |called 260642 times, avg 0.00001s/call: # |
824 \& | | | | |251215 times (2.61571+0s) by main::matches at |
825 \& |103 | | | |line 74 of wordmatch, avg 0.00001s/call # 9427 |
826 \& | | | | |times (0.09628+0s) at line 50 of wordmatch, avg|
827 \& | | | | |0.00001s/call |
828 \& | | | | |sub debug { |
829 \& |104 |260642|0.58496 |2e\-06 |my $message = shift; |
830 \& |105 | | | | |
831 \& |106 |260642|1.09917 |4e\-06 |if ( $debug ) { |
832 \& |107 | | | |print STDERR "DBG: $message\en"; |
833 \& |108 | | | |} |
834 \& |109 | | | |} |
835 \& |110 | | | | |
836 \& |111 |1 |0.01501 |0.01501|exit 0; |
837 \& |112 | | | | |
838 .Ve
839 .PP
840 Oodles of very useful information in there \- this seems to be the way forward.
841 .PP
842 See also \f(CW\*(C`Devel::NYTProf::Apache\*(C'\fR which hooks \f(CW\*(C`Devel::NYTProf\*(C'\fR into
843 \&\f(CW\*(C`mod_perl\*(C'\fR.
844 .SH SORTING
845 .IX Header "SORTING"
846 Perl modules are not the only tools a performance analyst has at their
847 disposal, system tools like \f(CW\*(C`time\*(C'\fR should not be overlooked as the next
848 example shows, where we take a quick look at sorting. Many books, theses and
849 articles, have been written about efficient sorting algorithms, and this is not
850 the place to repeat such work, there's several good sorting modules which
851 deserve taking a look at too: \f(CW\*(C`Sort::Maker\*(C'\fR, \f(CW\*(C`Sort::Key\*(C'\fR spring to mind.
852 However, it's still possible to make some observations on certain Perl specific
853 interpretations on issues relating to sorting data sets and give an example or
854 two with regard to how sorting large data volumes can effect performance.
855 Firstly, an often overlooked point when sorting large amounts of data, one can
856 attempt to reduce the data set to be dealt with and in many cases \f(CWgrep()\fR can
857 be quite useful as a simple filter:
858 .PP
859 .Vb 1
860 \& @data = sort grep { /$filter/ } @incoming
861 .Ve
862 .PP
863 A command such as this can vastly reduce the volume of material to actually
864 sort through in the first place, and should not be too lightly disregarded
865 purely on the basis of its simplicity. The \f(CW\*(C`KISS\*(C'\fR principle is too often
866 overlooked \- the next example uses the simple system \f(CW\*(C`time\*(C'\fR utility to
867 demonstrate. Let's take a look at an actual example of sorting the contents of
868 a large file, an apache logfile would do. This one has over a quarter of a
869 million lines, is 50M in size, and a snippet of it looks like this:
870 .PP
871 # logfile
872 .PP
873 .Vb 10
874 \& 188.209\-65\-87.adsl\-dyn.isp.belgacom.be \- \- [08/Feb/2007:12:57:16 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
875 \& 188.209\-65\-87.adsl\-dyn.isp.belgacom.be \- \- [08/Feb/2007:12:57:16 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
876 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:41 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
877 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:42 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
878 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:43 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
879 \& 217.113.68.60 \- \- [08/Feb/2007:13:02:15 +0000] "GET / HTTP/1.1" 304 \- "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
880 \& 217.113.68.60 \- \- [08/Feb/2007:13:02:16 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
881 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
882 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
883 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
884 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:48 +0000] "GET / HTTP/1.0" 200 3309 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
885 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:58 +0000] "GET /data/css HTTP/1.0" 404 206 "http://www.rfi.net/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
886 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:59 +0000] "GET /favicon.ico HTTP/1.0" 404 209 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
887 \& crawl1.cosmixcorp.com \- \- [08/Feb/2007:13:27:57 +0000] "GET /robots.txt HTTP/1.0" 200 179 "\-" "voyager/1.0"
888 \& crawl1.cosmixcorp.com \- \- [08/Feb/2007:13:28:25 +0000] "GET /links.html HTTP/1.0" 200 3413 "\-" "voyager/1.0"
889 \& fhm226.internetdsl.tpnet.pl \- \- [08/Feb/2007:13:37:32 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
890 \& fhm226.internetdsl.tpnet.pl \- \- [08/Feb/2007:13:37:34 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
891 \& 80.247.140.134 \- \- [08/Feb/2007:13:57:35 +0000] "GET / HTTP/1.1" 200 3309 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
892 \& 80.247.140.134 \- \- [08/Feb/2007:13:57:37 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
893 \& pop.compuscan.co.za \- \- [08/Feb/2007:14:10:43 +0000] "GET / HTTP/1.1" 200 3309 "\-" "www.clamav.net"
894 \& livebot\-207\-46\-98\-57.search.live.com \- \- [08/Feb/2007:14:12:04 +0000] "GET /robots.txt HTTP/1.0" 200 179 "\-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
895 \& livebot\-207\-46\-98\-57.search.live.com \- \- [08/Feb/2007:14:12:04 +0000] "GET /html/oracle.html HTTP/1.0" 404 214 "\-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
896 \& dslb\-088\-064\-005\-154.pools.arcor\-ip.net \- \- [08/Feb/2007:14:12:15 +0000] "GET / HTTP/1.1" 200 3309 "\-" "www.clamav.net"
897 \& 196.201.92.41 \- \- [08/Feb/2007:14:15:01 +0000] "GET / HTTP/1.1" 200 3309 "\-" "MOT\-L7/08.B7.DCR MIB/2.2.1 Profile/MIDP\-2.0 Configuration/CLDC\-1.1"
898 .Ve
899 .PP
900 The specific task here is to sort the 286,525 lines of this file by Response
901 Code, Query, Browser, Referring Url, and lastly Date. One solution might be to
902 use the following code, which iterates over the files given on the
903 command-line.
904 .PP
905 # sort-apache-log
906 .PP
907 .Vb 1
908 \& #!/usr/bin/perl \-n
909 \&
910 \& use v5.36;
911 \&
912 \& my @data;
913 \&
914 \& LINE:
915 \& while ( <> ) {
916 \& my $line = $_;
917 \& if (
918 \& $line =~ m/^(
919 \& ([\ew\e.\e\-]+) # client
920 \& \es*\-\es*\-\es*\e[
921 \& ([^]]+) # date
922 \& \e]\es*"\ew+\es*
923 \& (\eS+) # query
924 \& [^"]+"\es*
925 \& (\ed+) # status
926 \& \es+\eS+\es+"[^"]*"\es+"
927 \& ([^"]*) # browser
928 \& "
929 \& .*
930 \& )$/x
931 \& ) {
932 \& my @chunks = split(/ +/, $line);
933 \& my $ip = $1;
934 \& my $date = $2;
935 \& my $query = $3;
936 \& my $status = $4;
937 \& my $browser = $5;
938 \&
939 \& push(@data, [$ip, $date, $query, $status, $browser, $line]);
940 \& }
941 \& }
942 \&
943 \& my @sorted = sort {
944 \& $a\->[3] cmp $b\->[3]
945 \& ||
946 \& $a\->[2] cmp $b\->[2]
947 \& ||
948 \& $a\->[0] cmp $b\->[0]
949 \& ||
950 \& $a\->[1] cmp $b\->[1]
951 \& ||
952 \& $a\->[4] cmp $b\->[4]
953 \& } @data;
954 \&
955 \& foreach my $data ( @sorted ) {
956 \& print $data\->[5];
957 \& }
958 \&
959 \& exit 0;
960 .Ve
961 .PP
962 When running this program, redirect \f(CW\*(C`STDOUT\*(C'\fR so it is possible to check the
963 output is correct from following test runs and use the system \f(CW\*(C`time\*(C'\fR utility
964 to check the overall runtime.
965 .PP
966 .Vb 1
967 \& $> time ./sort\-apache\-log logfile > out\-sort
968 \&
969 \& real 0m17.371s
970 \& user 0m15.757s
971 \& sys 0m0.592s
972 .Ve
973 .PP
974 The program took just over 17 wallclock seconds to run. Note the different
975 values \f(CW\*(C`time\*(C'\fR outputs, it's important to always use the same one, and to not
976 confuse what each one means.
977 .IP "Elapsed Real Time" 4
978 .IX Item "Elapsed Real Time"
979 The overall, or wallclock, time between when \f(CW\*(C`time\*(C'\fR was called, and when it
980 terminates. The elapsed time includes both user and system times, and time
981 spent waiting for other users and processes on the system. Inevitably, this is
982 the most approximate of the measurements given.
983 .IP "User CPU Time" 4
984 .IX Item "User CPU Time"
985 The user time is the amount of time the entire process spent on behalf of the
986 user on this system executing this program.
987 .IP "System CPU Time" 4
988 .IX Item "System CPU Time"
989 The system time is the amount of time the kernel itself spent executing
990 routines, or system calls, on behalf of this process user.
991 .PP
992 Running this same process as a \f(CW\*(C`Schwarzian Transform\*(C'\fR it is possible to
993 eliminate the input and output arrays for storing all the data, and work on the
994 input directly as it arrives too. Otherwise, the code looks fairly similar:
995 .PP
996 # sort-apache-log-schwarzian
997 .PP
998 .Vb 1
999 \& #!/usr/bin/perl \-n
1000 \&
1001 \& use v5.36;
1002 \&
1003 \& print
1004 \&
1005 \& map $_\->[0] =>
1006 \&
1007 \& sort {
1008 \& $a\->[4] cmp $b\->[4]
1009 \& ||
1010 \& $a\->[3] cmp $b\->[3]
1011 \& ||
1012 \& $a\->[1] cmp $b\->[1]
1013 \& ||
1014 \& $a\->[2] cmp $b\->[2]
1015 \& ||
1016 \& $a\->[5] cmp $b\->[5]
1017 \& }
1018 \& map [ $_, m/^(
1019 \& ([\ew\e.\e\-]+) # client
1020 \& \es*\-\es*\-\es*\e[
1021 \& ([^]]+) # date
1022 \& \e]\es*"\ew+\es*
1023 \& (\eS+) # query
1024 \& [^"]+"\es*
1025 \& (\ed+) # status
1026 \& \es+\eS+\es+"[^"]*"\es+"
1027 \& ([^"]*) # browser
1028 \& "
1029 \& .*
1030 \& )$/xo ]
1031 \&
1032 \& => <>;
1033 \&
1034 \& exit 0;
1035 .Ve
1036 .PP
1037 Run the new code against the same logfile, as above, to check the new time.
1038 .PP
1039 .Vb 1
1040 \& $> time ./sort\-apache\-log\-schwarzian logfile > out\-schwarz
1041 \&
1042 \& real 0m9.664s
1043 \& user 0m8.873s
1044 \& sys 0m0.704s
1045 .Ve
1046 .PP
1047 The time has been cut in half, which is a respectable speed improvement by any
1048 standard. Naturally, it is important to check the output is consistent with
1049 the first program run, this is where the Unix system \f(CW\*(C`cksum\*(C'\fR utility comes in.
1050 .PP
1051 .Vb 3
1052 \& $> cksum out\-sort out\-schwarz
1053 \& 3044173777 52029194 out\-sort
1054 \& 3044173777 52029194 out\-schwarz
1055 .Ve
1056 .PP
1057 BTW. Beware too of pressure from managers who see you speed a program up by 50%
1058 of the runtime once, only to get a request one month later to do the same again
1059 (true story) \- you'll just have to point out you're only human, even if you are a
1060 Perl programmer, and you'll see what you can do...
1061 .SH LOGGING
1062 .IX Header "LOGGING"
1063 An essential part of any good development process is appropriate error handling
1064 with appropriately informative messages, however there exists a school of
1065 thought which suggests that log files should be \fIchatty\fR, as if the chain of
1066 unbroken output somehow ensures the survival of the program. If speed is in
1067 any way an issue, this approach is wrong.
1068 .PP
1069 A common sight is code which looks something like this:
1070 .PP
1071 .Vb 2
1072 \& logger\->debug( "A logging message via process\-id: $$ INC: "
1073 \& . Dumper(\e%INC) )
1074 .Ve
1075 .PP
1076 The problem is that this code will always be parsed and executed, even when the
1077 debug level set in the logging configuration file is zero. Once the \fBdebug()\fR
1078 subroutine has been entered, and the internal \f(CW$debug\fR variable confirmed to
1079 be zero, for example, the message which has been sent in will be discarded and
1080 the program will continue. In the example given though, the \f(CW\*(C`\e%INC\*(C'\fR hash will
1081 already have been dumped, and the message string constructed, all of which work
1082 could be bypassed by a debug variable at the statement level, like this:
1083 .PP
1084 .Vb 2
1085 \& logger\->debug( "A logging message via process\-id: $$ INC: "
1086 \& . Dumper(\e%INC) ) if $DEBUG;
1087 .Ve
1088 .PP
1089 This effect can be demonstrated by setting up a test script with both forms,
1090 including a \f(CWdebug()\fR subroutine to emulate typical \f(CWlogger()\fR functionality.
1091 .PP
1092 # ifdebug
1093 .PP
1094 .Vb 1
1095 \& #!/usr/bin/perl
1096 \&
1097 \& use v5.36;
1098 \&
1099 \& use Benchmark;
1100 \& use Data::Dumper;
1101 \& my $DEBUG = 0;
1102 \&
1103 \& sub debug {
1104 \& my $msg = shift;
1105 \&
1106 \& if ( $DEBUG ) {
1107 \& print "DEBUG: $msg\en";
1108 \& }
1109 \& };
1110 \&
1111 \& timethese(100000, {
1112 \& \*(Aqdebug\*(Aq => sub {
1113 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) )
1114 \& },
1115 \& \*(Aqifdebug\*(Aq => sub {
1116 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) ) if $DEBUG
1117 \& },
1118 \& });
1119 .Ve
1120 .PP
1121 Let's see what \f(CW\*(C`Benchmark\*(C'\fR makes of this:
1122 .PP
1123 .Vb 5
1124 \& $> perl ifdebug
1125 \& Benchmark: timing 100000 iterations of constant, sub...
1126 \& ifdebug: 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU) @ 10000000.00/s (n=100000)
1127 \& (warning: too few iterations for a reliable count)
1128 \& debug: 14 wallclock secs (13.18 usr + 0.04 sys = 13.22 CPU) @ 7564.30/s (n=100000)
1129 .Ve
1130 .PP
1131 In the one case the code, which does exactly the same thing as far as
1132 outputting any debugging information is concerned, in other words nothing,
1133 takes 14 seconds, and in the other case the code takes one hundredth of a
1134 second. Looks fairly definitive. Use a \f(CW$DEBUG\fR variable BEFORE you call the
1135 subroutine, rather than relying on the smart functionality inside it.
1136 .SS "Logging if DEBUG (constant)"
1137 .IX Subsection "Logging if DEBUG (constant)"
1138 It's possible to take the previous idea a little further, by using a compile
1139 time \f(CW\*(C`DEBUG\*(C'\fR constant.
1140 .PP
1141 # ifdebug-constant
1142 .PP
1143 .Vb 1
1144 \& #!/usr/bin/perl
1145 \&
1146 \& use v5.36;
1147 \&
1148 \& use Benchmark;
1149 \& use Data::Dumper;
1150 \& use constant
1151 \& DEBUG => 0
1152 \& ;
1153 \&
1154 \& sub debug {
1155 \& if ( DEBUG ) {
1156 \& my $msg = shift;
1157 \& print "DEBUG: $msg\en";
1158 \& }
1159 \& };
1160 \&
1161 \& timethese(100000, {
1162 \& \*(Aqdebug\*(Aq => sub {
1163 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) )
1164 \& },
1165 \& \*(Aqconstant\*(Aq => sub {
1166 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) ) if DEBUG
1167 \& },
1168 \& });
1169 .Ve
1170 .PP
1171 Running this program produces the following output:
1172 .PP
1173 .Vb 5
1174 \& $> perl ifdebug\-constant
1175 \& Benchmark: timing 100000 iterations of constant, sub...
1176 \& constant: 0 wallclock secs (\-0.00 usr + 0.00 sys = \-0.00 CPU) @ \-7205759403792793600000.00/s (n=100000)
1177 \& (warning: too few iterations for a reliable count)
1178 \& sub: 14 wallclock secs (13.09 usr + 0.00 sys = 13.09 CPU) @ 7639.42/s (n=100000)
1179 .Ve
1180 .PP
1181 The \f(CW\*(C`DEBUG\*(C'\fR constant wipes the floor with even the \f(CW$debug\fR variable,
1182 clocking in at minus zero seconds, and generates a "warning: too few iterations
1183 for a reliable count" message into the bargain. To see what is really going
1184 on, and why we had too few iterations when we thought we asked for 100000, we
1185 can use the very useful \f(CW\*(C`B::Deparse\*(C'\fR to inspect the new code:
1186 .PP
1187 .Vb 1
1188 \& $> perl \-MO=Deparse ifdebug\-constant
1189 \&
1190 \& use Benchmark;
1191 \& use Data::Dumper;
1192 \& use constant (\*(AqDEBUG\*(Aq, 0);
1193 \& sub debug {
1194 \& use warnings;
1195 \& use strict \*(Aqrefs\*(Aq;
1196 \& 0;
1197 \& }
1198 \& use warnings;
1199 \& use strict \*(Aqrefs\*(Aq;
1200 \& timethese(100000, {\*(Aqsub\*(Aq, sub {
1201 \& debug "A $0 logging message via process\-id: $$" . Dumper(\e%INC);
1202 \& }
1203 \& , \*(Aqconstant\*(Aq, sub {
1204 \& 0;
1205 \& }
1206 \& });
1207 \& ifdebug\-constant syntax OK
1208 .Ve
1209 .PP
1210 The output shows the \fBconstant()\fR subroutine we're testing being replaced with
1211 the value of the \f(CW\*(C`DEBUG\*(C'\fR constant: zero. The line to be tested has been
1212 completely optimized away, and you can't get much more efficient than that.
1213 .SH POSTSCRIPT
1214 .IX Header "POSTSCRIPT"
1215 This document has provided several way to go about identifying hot-spots, and
1216 checking whether any modifications have improved the runtime of the code.
1217 .PP
1218 As a final thought, remember that it's not (at the time of writing) possible to
1219 produce a useful program which will run in zero or negative time and this basic
1220 principle can be written as: \fIuseful programs are slow\fR by their very
1221 definition. It is of course possible to write a nearly instantaneous program,
1222 but it's not going to do very much, here's a very efficient one:
1223 .PP
1224 .Vb 1
1225 \& $> perl \-e 0
1226 .Ve
1227 .PP
1228 Optimizing that any further is a job for \f(CW\*(C`p5p\*(C'\fR.
1229 .SH "SEE ALSO"
1230 .IX Header "SEE ALSO"
1231 Further reading can be found using the modules and links below.
1232 .SS PERLDOCS
1233 .IX Subsection "PERLDOCS"
1234 For example: \f(CW\*(C`perldoc \-f sort\*(C'\fR.
1235 .PP
1236 perlfaq4.
1237 .PP
1238 perlfork, perlfunc, perlretut, perlthrtut.
1239 .PP
1240 threads.
1241 .SS "MAN PAGES"
1242 .IX Subsection "MAN PAGES"
1243 \&\f(CW\*(C`time\*(C'\fR.
1244 .SS MODULES
1245 .IX Subsection "MODULES"
1246 It's not possible to individually showcase all the performance related code for
1247 Perl here, naturally, but here's a short list of modules from the CPAN which
1248 deserve further attention.
1249 .PP
1250 .Vb 10
1251 \& Apache::DProf
1252 \& Apache::SmallProf
1253 \& Benchmark
1254 \& DBIx::Profile
1255 \& Devel::AutoProfiler
1256 \& Devel::DProf
1257 \& Devel::DProfLB
1258 \& Devel::FastProf
1259 \& Devel::GraphVizProf
1260 \& Devel::NYTProf
1261 \& Devel::NYTProf::Apache
1262 \& Devel::Profiler
1263 \& Devel::Profile
1264 \& Devel::Profit
1265 \& Devel::SmallProf
1266 \& Devel::WxProf
1267 \& POE::Devel::Profiler
1268 \& Sort::Key
1269 \& Sort::Maker
1270 .Ve
1271 .SS URLS
1272 .IX Subsection "URLS"
1273 Very useful online reference material:
1274 .PP
1275 .Vb 1
1276 \& https://web.archive.org/web/20120515021937/http://www.ccl4.org/~nick/P/Fast_Enough/
1277 \&
1278 \& https://web.archive.org/web/20050706081718/http://www\-106.ibm.com/developerworks/library/l\-optperl.html
1279 \&
1280 \& https://perlbuzz.com/2007/11/14/bind_output_variables_in_dbi_for_speed_and_safety/
1281 \&
1282 \& http://en.wikipedia.org/wiki/Performance_analysis
1283 \&
1284 \& http://apache.perl.org/docs/1.0/guide/performance.html
1285 \&
1286 \& http://perlgolf.sourceforge.net/
1287 \&
1288 \& http://www.sysarch.com/Perl/sort_paper.html
1289 .Ve
1290 .SH AUTHOR
1291 .IX Header "AUTHOR"
1292 Richard Foley <[email protected]> Copyright (c) 2008