Debian Codegraph @

upstream/opensuse-tumbleweed/man1/perlperf.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "PERLPERF 1"
61 .TH PERLPERF 1 2026-03-09 "perl v5.42.1" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 perlperf \- Perl Performance and Optimization Techniques
68 .SH DESCRIPTION
69 .IX Header "DESCRIPTION"
70 This is an introduction to the use of performance and optimization techniques
71 which can be used with particular reference to perl programs. While many perl
72 developers have come from other languages, and can use their prior knowledge
73 where appropriate, there are many other people who might benefit from a few
74 perl specific pointers. If you want the condensed version, perhaps the best
75 advice comes from the renowned Japanese Samurai, Miyamoto Musashi, who said:
76 .PP
77 .Vb 1
78 \& "Do Not Engage in Useless Activity"
79 .Ve
80 .PP
81 in 1645.
82 .SH OVERVIEW
83 .IX Header "OVERVIEW"
84 Perhaps the most common mistake programmers make is to attempt to optimize
85 their code before a program actually does anything useful \- this is a bad idea.
86 There\*(Aqs no point in having an extremely fast program that doesn\*(Aqt work. The
87 first job is to get a program to \fIcorrectly\fR do something \fBuseful\fR, (not to
88 mention ensuring the test suite is fully functional), and only then to consider
89 optimizing it. Having decided to optimize existing working code, there are
90 several simple but essential steps to consider which are intrinsic to any
91 optimization process.
92 .SS "ONE STEP SIDEWAYS"
93 .IX Subsection "ONE STEP SIDEWAYS"
94 Firstly, you need to establish a baseline time for the existing code, which
95 timing needs to be reliable and repeatable. You\*(Aqll probably want to use the
96 \&\f(CW\*(C`Benchmark\*(C'\fR or \f(CW\*(C`Devel::NYTProf\*(C'\fR modules, or something similar, for this step,
97 or perhaps the Unix system \f(CW\*(C`time\*(C'\fR utility, whichever is appropriate. See the
98 base of this document for a longer list of benchmarking and profiling modules,
99 and recommended further reading.
100 .SS "ONE STEP FORWARD"
101 .IX Subsection "ONE STEP FORWARD"
102 Next, having examined the program for \fIhot spots\fR, (places where the code
103 seems to run slowly), change the code with the intention of making it run
104 faster. Using version control software, like \f(CW\*(C`subversion\*(C'\fR, will ensure no
105 changes are irreversible. It\*(Aqs too easy to fiddle here and fiddle there \-
106 don\*(Aqt change too much at any one time or you might not discover which piece of
107 code \fBreally\fR was the slow bit.
108 .SS "ANOTHER STEP SIDEWAYS"
109 .IX Subsection "ANOTHER STEP SIDEWAYS"
110 It\*(Aqs not enough to say: "that will make it run faster", you have to check it.
111 Rerun the code under control of the benchmarking or profiling modules, from the
112 first step above, and check that the new code executed the \fBsame task\fR in
113 \&\fIless time\fR. Save your work and repeat...
114 .SH "GENERAL GUIDELINES"
115 .IX Header "GENERAL GUIDELINES"
116 The critical thing when considering performance is to remember there is no such
117 thing as a \f(CW\*(C`Golden Bullet\*(C'\fR, which is why there are no rules, only guidelines.
118 .PP
119 It is clear that inline code is going to be faster than subroutine or method
120 calls, because there is less overhead, but this approach has the disadvantage
121 of being less maintainable and comes at the cost of greater memory usage \-
122 there is no such thing as a free lunch. If you are searching for an element in
123 a list, it can be more efficient to store the data in a hash structure, and
124 then simply look to see whether the key is defined, rather than to loop through
125 the entire array using \fBgrep()\fR for instance. \fBsubstr()\fR may be (a lot) faster
126 than \fBgrep()\fR but not as flexible, so you have another trade\-off to access. Your
127 code may contain a line which takes 0.01 of a second to execute which if you
128 call it 1,000 times, quite likely in a program parsing even medium sized files
129 for instance, you already have a 10 second delay, in just one single code
130 location, and if you call that line 100,000 times, your entire program will
131 slow down to an unbearable crawl.
132 .PP
133 Using a subroutine as part of your sort is a powerful way to get exactly what
134 you want, but will usually be slower than the built\-in \fIalphabetic\fR \f(CW\*(C`cmp\*(C'\fR and
135 \&\fInumeric\fR \f(CW\*(C`<=>\*(C'\fR sort operators. It is possible to make multiple
136 passes over your data, building indices to make the upcoming sort more
137 efficient, and to use what is known as the \f(CW\*(C`OM\*(C'\fR (Orcish Maneuver) to cache the
138 sort keys in advance. The cache lookup, while a good idea, can itself be a
139 source of slowdown by enforcing a double pass over the data \- once to setup the
140 cache, and once to sort the data. Using \f(CWpack()\fR to extract the required sort
141 key into a consistent string can be an efficient way to build a single string
142 to compare, instead of using multiple sort keys, which makes it possible to use
143 the standard, written in \f(CW\*(C`c\*(C'\fR and fast, perl \f(CWsort()\fR function on the output,
144 and is the basis of the \f(CW\*(C`GRT\*(C'\fR (Guttman Rossler Transform). Some string
145 combinations can slow the \f(CW\*(C`GRT\*(C'\fR down, by just being too plain complex for its
146 own good.
147 .PP
148 For applications using database backends, the standard \f(CW\*(C`DBIx\*(C'\fR namespace has
149 tried to help with keeping things nippy, not least because it tries to \fInot\fR
150 query the database until the latest possible moment, but always read the docs
151 which come with your choice of libraries. Among the many issues facing
152 developers dealing with databases should remain aware of is to always use
153 \&\f(CW\*(C`SQL\*(C'\fR placeholders and to consider pre\-fetching data sets when this might
154 prove advantageous. Splitting up a large file by assigning multiple processes
155 to parsing a single file, using say \f(CW\*(C`POE\*(C'\fR, \f(CW\*(C`threads\*(C'\fR or \f(CW\*(C`fork\*(C'\fR can also be a
156 useful way of optimizing your usage of the available \f(CW\*(C`CPU\*(C'\fR resources, though
157 this technique is fraught with concurrency issues and demands high attention to
158 detail.
159 .PP
160 Every case has a specific application and one or more exceptions, and there is
161 no replacement for running a few tests and finding out which method works best
162 for your particular environment, this is why writing optimal code is not an
163 exact science, and why we love using Perl so much \- TMTOWTDI.
164 .SH BENCHMARKS
165 .IX Header "BENCHMARKS"
166 Here are a few examples to demonstrate usage of Perl\*(Aqs benchmarking tools.
167 .SS "Assigning and Dereferencing Variables"
168 .IX Subsection "Assigning and Dereferencing Variables"
169 I\*(Aqm sure most of us have seen code which looks like, (or worse than), this:
170 .PP
171 .Vb 2
172 \& if ( $obj\->{_ref}\->{_myscore} >= $obj\->{_ref}\->{_yourscore} ) {
173 \& ...
174 .Ve
175 .PP
176 This sort of code can be a real eyesore to read, as well as being very
177 sensitive to typos, and it\*(Aqs much clearer to dereference the variable
178 explicitly. We\*(Aqre side\-stepping the issue of working with object\-oriented
179 programming techniques to encapsulate variable access via methods, only
180 accessible through an object. Here we\*(Aqre just discussing the technical
181 implementation of choice, and whether this has an effect on performance. We
182 can see whether this dereferencing operation, has any overhead by putting
183 comparative code in a file and running a \f(CW\*(C`Benchmark\*(C'\fR test.
184 .PP
185 # dereference
186 .PP
187 .Vb 1
188 \& #!/usr/bin/perl
189 \&
190 \& use v5.36;
191 \&
192 \& use Benchmark;
193 \&
194 \& my $ref = {
195 \& \*(Aqref\*(Aq => {
196 \& _myscore => \*(Aq100 + 1\*(Aq,
197 \& _yourscore => \*(Aq102 \- 1\*(Aq,
198 \& },
199 \& };
200 \&
201 \& timethese(1000000, {
202 \& \*(Aqdirect\*(Aq => sub {
203 \& my $x = $ref\->{ref}\->{_myscore} . $ref\->{ref}\->{_yourscore} ;
204 \& },
205 \& \*(Aqdereference\*(Aq => sub {
206 \& my $ref = $ref\->{ref};
207 \& my $myscore = $ref\->{_myscore};
208 \& my $yourscore = $ref\->{_yourscore};
209 \& my $x = $myscore . $yourscore;
210 \& },
211 \& });
212 .Ve
213 .PP
214 It\*(Aqs essential to run any timing measurements a sufficient number of times so
215 the numbers settle on a numerical average, otherwise each run will naturally
216 fluctuate due to variations in the environment, to reduce the effect of
217 contention for \f(CW\*(C`CPU\*(C'\fR resources and network bandwidth for instance. Running
218 the above code for one million iterations, we can take a look at the report
219 output by the \f(CW\*(C`Benchmark\*(C'\fR module, to see which approach is the most effective.
220 .PP
221 .Vb 1
222 \& $> perl dereference
223 \&
224 \& Benchmark: timing 1000000 iterations of dereference, direct...
225 \& dereference: 2 wallclock secs ( 1.59 usr + 0.00 sys = 1.59 CPU) @ 628930.82/s (n=1000000)
226 \& direct: 1 wallclock secs ( 1.20 usr + 0.00 sys = 1.20 CPU) @ 833333.33/s (n=1000000)
227 .Ve
228 .PP
229 The difference is clear to see and the dereferencing approach is slower. While
230 it managed to execute an average of 628,930 times a second during our test, the
231 direct approach managed to run an additional 204,403 times, unfortunately.
232 Unfortunately, because there are many examples of code written using the
233 multiple layer direct variable access, and it\*(Aqs usually horrible. It is,
234 however, minusculely faster. The question remains whether the minute gain is
235 actually worth the eyestrain, or the loss of maintainability.
236 .SS "Search and replace or tr"
237 .IX Subsection "Search and replace or tr"
238 If we have a string which needs to be modified, while a regex will almost
239 always be much more flexible, \f(CW\*(C`tr\*(C'\fR, an oft underused tool, can still be
240 useful. One scenario might be to replace all vowels with another character.
241 The regex solution might look like this:
242 .PP
243 .Vb 1
244 \& $str =~ s/[aeiou]/x/g
245 .Ve
246 .PP
247 The \f(CW\*(C`tr\*(C'\fR alternative might look like this:
248 .PP
249 .Vb 1
250 \& $str =~ tr/aeiou/xxxxx/
251 .Ve
252 .PP
253 We can put that into a test file which we can run to check which approach is
254 the fastest, using a global \f(CW$STR\fR variable to assign to the \f(CW\*(C`my $str\*(C'\fR
255 variable so as to avoid perl trying to optimize any of the work away by
256 noticing it\*(Aqs assigned only the once.
257 .PP
258 # regex\-transliterate
259 .PP
260 .Vb 1
261 \& #!/usr/bin/perl
262 \&
263 \& use v5.36;
264 \&
265 \& use Benchmark;
266 \&
267 \& my $STR = "$$\-this and that";
268 \&
269 \& timethese(1000000, {
270 \& \*(Aqsr\*(Aq => sub { my $str = $STR; $str =~ s/[aeiou]/x/g; return $str; },
271 \& \*(Aqtr\*(Aq => sub { my $str = $STR; $str =~ tr/aeiou/xxxxx/; return $str; },
272 \& });
273 .Ve
274 .PP
275 Running the code gives us our results:
276 .PP
277 .Vb 1
278 \& $> perl regex\-transliterate
279 \&
280 \& Benchmark: timing 1000000 iterations of sr, tr...
281 \& sr: 2 wallclock secs ( 1.19 usr + 0.00 sys = 1.19 CPU) @ 840336.13/s (n=1000000)
282 \& tr: 0 wallclock secs ( 0.49 usr + 0.00 sys = 0.49 CPU) @ 2040816.33/s (n=1000000)
283 .Ve
284 .PP
285 The \f(CW\*(C`tr\*(C'\fR version is a clear winner. One solution is flexible, the other is
286 fast \- and it\*(Aqs appropriately the programmer\*(Aqs choice which to use.
287 .PP
288 Check the \f(CW\*(C`Benchmark\*(C'\fR docs for further useful techniques.
289 .SH "PROFILING TOOLS"
290 .IX Header "PROFILING TOOLS"
291 A slightly larger piece of code will provide something on which a profiler can
292 produce more extensive reporting statistics. This example uses the simplistic
293 \&\f(CW\*(C`wordmatch\*(C'\fR program which parses a given input file and spews out a short
294 report on the contents.
295 .PP
296 # wordmatch
297 .PP
298 .Vb 1
299 \& #!/usr/bin/perl
300 \&
301 \& use v5.36;
302 \&
303 \& =head1 NAME
304 \&
305 \& filewords \- word analysis of input file
306 \&
307 \& =head1 SYNOPSIS
308 \&
309 \& filewords \-f inputfilename [\-d]
310 \&
311 \& =head1 DESCRIPTION
312 \&
313 \& This program parses the given filename, specified with C<\-f>, and
314 \& displays a simple analysis of the words found therein. Use the C<\-d>
315 \& switch to enable debugging messages.
316 \&
317 \& =cut
318 \&
319 \& use FileHandle;
320 \& use Getopt::Long;
321 \&
322 \& my $debug = 0;
323 \& my $file = \*(Aq\*(Aq;
324 \&
325 \& my $result = GetOptions (
326 \& \*(Aqdebug\*(Aq => \e$debug,
327 \& \*(Aqfile=s\*(Aq => \e$file,
328 \& );
329 \& die("invalid args") unless $result;
330 \&
331 \& unless ( \-f $file ) {
332 \& die("Usage: $0 \-f filename [\-d]");
333 \& }
334 \& my $FH = FileHandle\->new("< $file")
335 \& or die("unable to open file($file): $!");
336 \&
337 \& my $i_LINES = 0;
338 \& my $i_WORDS = 0;
339 \& my %count = ();
340 \&
341 \& my @lines = <$FH>;
342 \& foreach my $line ( @lines ) {
343 \& $i_LINES++;
344 \& $line =~ s/\en//;
345 \& my @words = split(/ +/, $line);
346 \& my $i_words = scalar(@words);
347 \& $i_WORDS = $i_WORDS + $i_words;
348 \& debug("line: $i_LINES supplying $i_words words: @words");
349 \& my $i_word = 0;
350 \& foreach my $word ( @words ) {
351 \& $i_word++;
352 \& $count{$i_LINES}{spec} += matches($i_word, $word,
353 \& \*(Aq[^a\-zA\-Z0\-9]\*(Aq);
354 \& $count{$i_LINES}{only} += matches($i_word, $word,
355 \& \*(Aq^[^a\-zA\-Z0\-9]+$\*(Aq);
356 \& $count{$i_LINES}{cons} += matches($i_word, $word,
357 \& \*(Aq^[(?i:bcdfghjklmnpqrstvwxyz)]+$\*(Aq);
358 \& $count{$i_LINES}{vows} += matches($i_word, $word,
359 \& \*(Aq^[(?i:aeiou)]+$\*(Aq);
360 \& $count{$i_LINES}{caps} += matches($i_word, $word,
361 \& \*(Aq^[(A\-Z)]+$\*(Aq);
362 \& }
363 \& }
364 \&
365 \& print report( %count );
366 \&
367 \& sub matches {
368 \& my $i_wd = shift;
369 \& my $word = shift;
370 \& my $regex = shift;
371 \& my $has = 0;
372 \&
373 \& if ( $word =~ /($regex)/ ) {
374 \& $has++ if $1;
375 \& }
376 \&
377 \& debug( "word: $i_wd "
378 \& . ($has ? \*(Aqmatches\*(Aq : \*(Aqdoes not match\*(Aq)
379 \& . " chars: /$regex/");
380 \&
381 \& return $has;
382 \& }
383 \&
384 \& sub report {
385 \& my %report = @_;
386 \& my %rep;
387 \&
388 \& foreach my $line ( keys %report ) {
389 \& foreach my $key ( keys $report{$line}\->%* ) {
390 \& $rep{$key} += $report{$line}{$key};
391 \& }
392 \& }
393 \&
394 \& my $report = qq|
395 \& $0 report for $file:
396 \& lines in file: $i_LINES
397 \& words in file: $i_WORDS
398 \& words with special (non\-word) characters: $i_spec
399 \& words with only special (non\-word) characters: $i_only
400 \& words with only consonants: $i_cons
401 \& words with only capital letters: $i_caps
402 \& words with only vowels: $i_vows
403 \& |;
404 \&
405 \& return $report;
406 \& }
407 \&
408 \& sub debug {
409 \& my $message = shift;
410 \&
411 \& if ( $debug ) {
412 \& print STDERR "DBG: $message\en";
413 \& }
414 \& }
415 \&
416 \& exit 0;
417 .Ve
418 .SS Devel::DProf
419 .IX Subsection "Devel::DProf"
420 This venerable module has been the de\-facto standard for Perl code profiling
421 for more than a decade, but has been replaced by a number of other modules
422 which have brought us back to the 21st century. Although you\*(Aqre recommended to
423 evaluate your tool from the several mentioned here and from the CPAN list at
424 the base of this document, (and currently Devel::NYTProf seems to be the
425 weapon of choice \- see below), we\*(Aqll take a quick look at the output from
426 Devel::DProf first, to set a baseline for Perl profiling tools. Run the
427 above program under the control of \f(CW\*(C`Devel::DProf\*(C'\fR by using the \f(CW\*(C`\-d\*(C'\fR switch on
428 the command\-line.
429 .PP
430 .Vb 1
431 \& $> perl \-d:DProf wordmatch \-f perl5db.pl
432 \&
433 \& <...multiple lines snipped...>
434 \&
435 \& wordmatch report for perl5db.pl:
436 \& lines in file: 9428
437 \& words in file: 50243
438 \& words with special (non\-word) characters: 20480
439 \& words with only special (non\-word) characters: 7790
440 \& words with only consonants: 4801
441 \& words with only capital letters: 1316
442 \& words with only vowels: 1701
443 .Ve
444 .PP
445 \&\f(CW\*(C`Devel::DProf\*(C'\fR produces a special file, called \fItmon.out\fR by default, and
446 this file is read by the \f(CW\*(C`dprofpp\*(C'\fR program, which is already installed as part
447 of the \f(CW\*(C`Devel::DProf\*(C'\fR distribution. If you call \f(CW\*(C`dprofpp\*(C'\fR with no options,
448 it will read the \fItmon.out\fR file in the current directory and produce a human
449 readable statistics report of the run of your program. Note that this may take
450 a little time.
451 .PP
452 .Vb 1
453 \& $> dprofpp
454 \&
455 \& Total Elapsed Time = 2.951677 Seconds
456 \& User+System Time = 2.871677 Seconds
457 \& Exclusive Times
458 \& %Time ExclSec CumulS #Calls sec/call Csec/c Name
459 \& 102. 2.945 3.003 251215 0.0000 0.0000 main::matches
460 \& 2.40 0.069 0.069 260643 0.0000 0.0000 main::debug
461 \& 1.74 0.050 0.050 1 0.0500 0.0500 main::report
462 \& 1.04 0.030 0.049 4 0.0075 0.0123 main::BEGIN
463 \& 0.35 0.010 0.010 3 0.0033 0.0033 Exporter::as_heavy
464 \& 0.35 0.010 0.010 7 0.0014 0.0014 IO::File::BEGIN
465 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::FindOption
466 \& 0.00 \- \-0.000 1 \- \- Symbol::BEGIN
467 \& 0.00 \- \-0.000 1 \- \- Fcntl::BEGIN
468 \& 0.00 \- \-0.000 1 \- \- Fcntl::bootstrap
469 \& 0.00 \- \-0.000 1 \- \- warnings::BEGIN
470 \& 0.00 \- \-0.000 1 \- \- IO::bootstrap
471 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::ConfigDefaults
472 \& 0.00 \- \-0.000 1 \- \- Getopt::Long::Configure
473 \& 0.00 \- \-0.000 1 \- \- Symbol::gensym
474 .Ve
475 .PP
476 \&\f(CW\*(C`dprofpp\*(C'\fR will produce some quite detailed reporting on the activity of the
477 \&\f(CW\*(C`wordmatch\*(C'\fR program. The wallclock, user and system, times are at the top of
478 the analysis, and after this are the main columns defining which define the
479 report. Check the \f(CW\*(C`dprofpp\*(C'\fR docs for details of the many options it supports.
480 .PP
481 See also \f(CW\*(C`Apache::DProf\*(C'\fR which hooks \f(CW\*(C`Devel::DProf\*(C'\fR into \f(CW\*(C`mod_perl\*(C'\fR.
482 .SS Devel::Profiler
483 .IX Subsection "Devel::Profiler"
484 Let\*(Aqs take a look at the same program using a different profiler:
485 \&\f(CW\*(C`Devel::Profiler\*(C'\fR, a drop\-in Perl\-only replacement for \f(CW\*(C`Devel::DProf\*(C'\fR. The
486 usage is very slightly different in that instead of using the special \f(CW\*(C`\-d:\*(C'\fR
487 flag, you pull \f(CW\*(C`Devel::Profiler\*(C'\fR in directly as a module using \f(CW\*(C`\-M\*(C'\fR.
488 .PP
489 .Vb 1
490 \& $> perl \-MDevel::Profiler wordmatch \-f perl5db.pl
491 \&
492 \& <...multiple lines snipped...>
493 \&
494 \& wordmatch report for perl5db.pl:
495 \& lines in file: 9428
496 \& words in file: 50243
497 \& words with special (non\-word) characters: 20480
498 \& words with only special (non\-word) characters: 7790
499 \& words with only consonants: 4801
500 \& words with only capital letters: 1316
501 \& words with only vowels: 1701
502 .Ve
503 .PP
504 \&\f(CW\*(C`Devel::Profiler\*(C'\fR generates a tmon.out file which is compatible with the
505 \&\f(CW\*(C`dprofpp\*(C'\fR program, thus saving the construction of a dedicated statistics
506 reader program. \f(CW\*(C`dprofpp\*(C'\fR usage is therefore identical to the above example.
507 .PP
508 .Vb 1
509 \& $> dprofpp
510 \&
511 \& Total Elapsed Time = 20.984 Seconds
512 \& User+System Time = 19.981 Seconds
513 \& Exclusive Times
514 \& %Time ExclSec CumulS #Calls sec/call Csec/c Name
515 \& 49.0 9.792 14.509 251215 0.0000 0.0001 main::matches
516 \& 24.4 4.887 4.887 260643 0.0000 0.0000 main::debug
517 \& 0.25 0.049 0.049 1 0.0490 0.0490 main::report
518 \& 0.00 0.000 0.000 1 0.0000 0.0000 Getopt::Long::GetOptions
519 \& 0.00 0.000 0.000 2 0.0000 0.0000 Getopt::Long::ParseOptionSpec
520 \& 0.00 0.000 0.000 1 0.0000 0.0000 Getopt::Long::FindOption
521 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::File::new
522 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::Handle::new
523 \& 0.00 0.000 0.000 1 0.0000 0.0000 Symbol::gensym
524 \& 0.00 0.000 0.000 1 0.0000 0.0000 IO::File::open
525 .Ve
526 .PP
527 Interestingly we get slightly different results, which is mostly because the
528 algorithm which generates the report is different, even though the output file
529 format was allegedly identical. The elapsed, user and system times are clearly
530 showing the time it took for \f(CW\*(C`Devel::Profiler\*(C'\fR to execute its own run, but
531 the column listings feel more accurate somehow than the ones we had earlier
532 from \f(CW\*(C`Devel::DProf\*(C'\fR. The 102% figure has disappeared, for example. This is
533 where we have to use the tools at our disposal, and recognise their pros and
534 cons, before using them. Interestingly, the numbers of calls for each
535 subroutine are identical in the two reports, it\*(Aqs the percentages which differ.
536 As the author of \f(CW\*(C`Devel::Profiler\*(C'\fR writes:
537 .PP
538 .Vb 6
539 \& ...running HTML::Template\*(Aqs test suite under Devel::DProf shows
540 \& output() taking NO time but Devel::Profiler shows around 10% of the
541 \& time is in output(). I don\*(Aqt know which to trust but my gut tells me
542 \& something is wrong with Devel::DProf. HTML::Template::output() is a
543 \& big routine that\*(Aqs called for every test. Either way, something needs
544 \& fixing.
545 .Ve
546 .PP
547 YMMV.
548 .PP
549 See also \f(CW\*(C`Devel::Apache::Profiler\*(C'\fR which hooks \f(CW\*(C`Devel::Profiler\*(C'\fR
550 into \f(CW\*(C`mod_perl\*(C'\fR.
551 .SS Devel::SmallProf
552 .IX Subsection "Devel::SmallProf"
553 The \f(CW\*(C`Devel::SmallProf\*(C'\fR profiler examines the runtime of your Perl program and
554 produces a line\-by\-line listing to show how many times each line was called,
555 and how long each line took to execute. It is called by supplying the familiar
556 \&\f(CW\*(C`\-d\*(C'\fR flag to Perl at runtime.
557 .PP
558 .Vb 1
559 \& $> perl \-d:SmallProf wordmatch \-f perl5db.pl
560 \&
561 \& <...multiple lines snipped...>
562 \&
563 \& wordmatch report for perl5db.pl:
564 \& lines in file: 9428
565 \& words in file: 50243
566 \& words with special (non\-word) characters: 20480
567 \& words with only special (non\-word) characters: 7790
568 \& words with only consonants: 4801
569 \& words with only capital letters: 1316
570 \& words with only vowels: 1701
571 .Ve
572 .PP
573 \&\f(CW\*(C`Devel::SmallProf\*(C'\fR writes its output into a file called \fIsmallprof.out\fR, by
574 default. The format of the file looks like this:
575 .PP
576 .Vb 1
577 \& <num> <time> <ctime> <line>:<text>
578 .Ve
579 .PP
580 When the program has terminated, the output may be examined and sorted using
581 any standard text filtering utilities. Something like the following may be
582 sufficient:
583 .PP
584 .Vb 1
585 \& $> cat smallprof.out | grep \ed*: | sort \-k3 | tac | head \-n20
586 \&
587 \& 251215 1.65674 7.68000 75: if ( $word =~ /($regex)/ ) {
588 \& 251215 0.03264 4.40000 79: debug("word: $i_wd ".($has ? \*(Aqmatches\*(Aq :
589 \& 251215 0.02693 4.10000 81: return $has;
590 \& 260643 0.02841 4.07000 128: if ( $debug ) {
591 \& 260643 0.02601 4.04000 126: my $message = shift;
592 \& 251215 0.02641 3.91000 73: my $has = 0;
593 \& 251215 0.03311 3.71000 70: my $i_wd = shift;
594 \& 251215 0.02699 3.69000 72: my $regex = shift;
595 \& 251215 0.02766 3.68000 71: my $word = shift;
596 \& 50243 0.59726 1.00000 59: $count{$i_LINES}{cons} =
597 \& 50243 0.48175 0.92000 61: $count{$i_LINES}{spec} =
598 \& 50243 0.00644 0.89000 56: my $i_cons = matches($i_word, $word,
599 \& 50243 0.48837 0.88000 63: $count{$i_LINES}{caps} =
600 \& 50243 0.00516 0.88000 58: my $i_caps = matches($i_word, $word, \*(Aq^[(A\-
601 \& 50243 0.00631 0.81000 54: my $i_spec = matches($i_word, $word, \*(Aq[^a\-
602 \& 50243 0.00496 0.80000 57: my $i_vows = matches($i_word, $word,
603 \& 50243 0.00688 0.80000 53: $i_word++;
604 \& 50243 0.48469 0.79000 62: $count{$i_LINES}{only} =
605 \& 50243 0.48928 0.77000 60: $count{$i_LINES}{vows} =
606 \& 50243 0.00683 0.75000 55: my $i_only = matches($i_word, $word, \*(Aq^[^a\-
607 .Ve
608 .PP
609 You can immediately see a slightly different focus to the subroutine profiling
610 modules, and we start to see exactly which line of code is taking the most
611 time. That regex line is looking a bit suspicious, for example. Remember that
612 these tools are supposed to be used together, there is no single best way to
613 profile your code, you need to use the best tools for the job.
614 .PP
615 See also \f(CW\*(C`Apache::SmallProf\*(C'\fR which hooks \f(CW\*(C`Devel::SmallProf\*(C'\fR into
616 \&\f(CW\*(C`mod_perl\*(C'\fR.
617 .SS Devel::FastProf
618 .IX Subsection "Devel::FastProf"
619 \&\f(CW\*(C`Devel::FastProf\*(C'\fR is another Perl line profiler. This was written with a view
620 to getting a faster line profiler, than is possible with for example
621 \&\f(CW\*(C`Devel::SmallProf\*(C'\fR, because it\*(Aqs written in \f(CW\*(C`C\*(C'\fR. To use \f(CW\*(C`Devel::FastProf\*(C'\fR,
622 supply the \f(CW\*(C`\-d\*(C'\fR argument to Perl:
623 .PP
624 .Vb 1
625 \& $> perl \-d:FastProf wordmatch \-f perl5db.pl
626 \&
627 \& <...multiple lines snipped...>
628 \&
629 \& wordmatch report for perl5db.pl:
630 \& lines in file: 9428
631 \& words in file: 50243
632 \& words with special (non\-word) characters: 20480
633 \& words with only special (non\-word) characters: 7790
634 \& words with only consonants: 4801
635 \& words with only capital letters: 1316
636 \& words with only vowels: 1701
637 .Ve
638 .PP
639 \&\f(CW\*(C`Devel::FastProf\*(C'\fR writes statistics to the file \fIfastprof.out\fR in the current
640 directory. The output file, which can be specified, can be interpreted by using
641 the \f(CW\*(C`fprofpp\*(C'\fR command\-line program.
642 .PP
643 .Vb 1
644 \& $> fprofpp | head \-n20
645 \&
646 \& # fprofpp output format is:
647 \& # filename:line time count: source
648 \& wordmatch:75 3.93338 251215: if ( $word =~ /($regex)/ ) {
649 \& wordmatch:79 1.77774 251215: debug("word: $i_wd ".($has ? \*(Aqmatches\*(Aq : \*(Aqdoes not match\*(Aq)." chars: /$regex/");
650 \& wordmatch:81 1.47604 251215: return $has;
651 \& wordmatch:126 1.43441 260643: my $message = shift;
652 \& wordmatch:128 1.42156 260643: if ( $debug ) {
653 \& wordmatch:70 1.36824 251215: my $i_wd = shift;
654 \& wordmatch:71 1.36739 251215: my $word = shift;
655 \& wordmatch:72 1.35939 251215: my $regex = shift;
656 .Ve
657 .PP
658 Straightaway we can see that the number of times each line has been called is
659 identical to the \f(CW\*(C`Devel::SmallProf\*(C'\fR output, and the sequence is only very
660 slightly different based on the ordering of the amount of time each line took
661 to execute, \f(CW\*(C`if ( $debug ) { \*(C'\fR and \f(CW\*(C`my $message = shift;\*(C'\fR, for example. The
662 differences in the actual times recorded might be in the algorithm used
663 internally, or it could be due to system resource limitations or contention.
664 .PP
665 See also the DBIx::Profile which will profile database queries running
666 under the \f(CW\*(C`DBIx::*\*(C'\fR namespace.
667 .SS Devel::NYTProf
668 .IX Subsection "Devel::NYTProf"
669 \&\f(CW\*(C`Devel::NYTProf\*(C'\fR is the \fBnext generation\fR of Perl code profiler, fixing many
670 shortcomings in other tools and implementing many cool features. First of all it
671 can be used as either a \fIline\fR profiler, a \fIblock\fR or a \fIsubroutine\fR
672 profiler, all at once. It can also use sub\-microsecond (100ns) resolution on
673 systems which provide \f(CWclock_gettime()\fR. It can be started and stopped even
674 by the program being profiled. It\*(Aqs a one\-line entry to profile \f(CW\*(C`mod_perl\*(C'\fR
675 applications. It\*(Aqs written in \f(CW\*(C`c\*(C'\fR and is probably the fastest profiler
676 available for Perl. The list of coolness just goes on. Enough of that, let\*(Aqs
677 see how it works \- just use the familiar \f(CW\*(C`\-d\*(C'\fR switch to plug it in and run
678 the code.
679 .PP
680 .Vb 1
681 \& $> perl \-d:NYTProf wordmatch \-f perl5db.pl
682 \&
683 \& wordmatch report for perl5db.pl:
684 \& lines in file: 9427
685 \& words in file: 50243
686 \& words with special (non\-word) characters: 20480
687 \& words with only special (non\-word) characters: 7790
688 \& words with only consonants: 4801
689 \& words with only capital letters: 1316
690 \& words with only vowels: 1701
691 .Ve
692 .PP
693 \&\f(CW\*(C`NYTProf\*(C'\fR will generate a report database into the file \fInytprof.out\fR by
694 default. Human readable reports can be generated from here by using the
695 supplied \f(CW\*(C`nytprofhtml\*(C'\fR (HTML output) and \f(CW\*(C`nytprofcsv\*(C'\fR (CSV output) programs.
696 We\*(Aqve used the Unix system \f(CW\*(C`html2text\*(C'\fR utility to convert the
697 \&\fInytprof/index.html\fR file for convenience here.
698 .PP
699 .Vb 1
700 \& $> html2text nytprof/index.html
701 \&
702 \& Performance Profile Index
703 \& For wordmatch
704 \& Run on Fri Sep 26 13:46:39 2008
705 \& Reported on Fri Sep 26 13:47:23 2008
706 \&
707 \& Top 15 Subroutines \-\- ordered by exclusive time
708 \& |Calls |P |F |Inclusive|Exclusive|Subroutine |
709 \& | | | |Time |Time | |
710 \& |251215|5 |1 |13.09263 |10.47692 |main:: |matches |
711 \& |260642|2 |1 |2.71199 |2.71199 |main:: |debug |
712 \& |1 |1 |1 |0.21404 |0.21404 |main:: |report |
713 \& |2 |2 |2 |0.00511 |0.00511 |XSLoader:: |load (xsub) |
714 \& |14 |14|7 |0.00304 |0.00298 |Exporter:: |import |
715 \& |3 |1 |1 |0.00265 |0.00254 |Exporter:: |as_heavy |
716 \& |10 |10|4 |0.00140 |0.00140 |vars:: |import |
717 \& |13 |13|1 |0.00129 |0.00109 |constant:: |import |
718 \& |1 |1 |1 |0.00360 |0.00096 |FileHandle:: |import |
719 \& |3 |3 |3 |0.00086 |0.00074 |warnings::register::|import |
720 \& |9 |3 |1 |0.00036 |0.00036 |strict:: |bits |
721 \& |13 |13|13|0.00032 |0.00029 |strict:: |import |
722 \& |2 |2 |2 |0.00020 |0.00020 |warnings:: |import |
723 \& |2 |1 |1 |0.00020 |0.00020 |Getopt::Long:: |ParseOptionSpec|
724 \& |7 |7 |6 |0.00043 |0.00020 |strict:: |unimport |
725 \&
726 \& For more information see the full list of 189 subroutines.
727 .Ve
728 .PP
729 The first part of the report already shows the critical information regarding
730 which subroutines are using the most time. The next gives some statistics
731 about the source files profiled.
732 .PP
733 .Vb 10
734 \& Source Code Files \-\- ordered by exclusive time then name
735 \& |Stmts |Exclusive|Avg. |Reports |Source File |
736 \& | |Time | | | |
737 \& |2699761|15.66654 |6e\-06 |line . block . sub|wordmatch |
738 \& |35 |0.02187 |0.00062|line . block . sub|IO/Handle.pm |
739 \& |274 |0.01525 |0.00006|line . block . sub|Getopt/Long.pm |
740 \& |20 |0.00585 |0.00029|line . block . sub|Fcntl.pm |
741 \& |128 |0.00340 |0.00003|line . block . sub|Exporter/Heavy.pm |
742 \& |42 |0.00332 |0.00008|line . block . sub|IO/File.pm |
743 \& |261 |0.00308 |0.00001|line . block . sub|Exporter.pm |
744 \& |323 |0.00248 |8e\-06 |line . block . sub|constant.pm |
745 \& |12 |0.00246 |0.00021|line . block . sub|File/Spec/Unix.pm |
746 \& |191 |0.00240 |0.00001|line . block . sub|vars.pm |
747 \& |77 |0.00201 |0.00003|line . block . sub|FileHandle.pm |
748 \& |12 |0.00198 |0.00016|line . block . sub|Carp.pm |
749 \& |14 |0.00175 |0.00013|line . block . sub|Symbol.pm |
750 \& |15 |0.00130 |0.00009|line . block . sub|IO.pm |
751 \& |22 |0.00120 |0.00005|line . block . sub|IO/Seekable.pm |
752 \& |198 |0.00085 |4e\-06 |line . block . sub|warnings/register.pm|
753 \& |114 |0.00080 |7e\-06 |line . block . sub|strict.pm |
754 \& |47 |0.00068 |0.00001|line . block . sub|warnings.pm |
755 \& |27 |0.00054 |0.00002|line . block . sub|overload.pm |
756 \& |9 |0.00047 |0.00005|line . block . sub|SelectSaver.pm |
757 \& |13 |0.00045 |0.00003|line . block . sub|File/Spec.pm |
758 \& |2701595|15.73869 | |Total |
759 \& |128647 |0.74946 | |Average |
760 \& | |0.00201 |0.00003|Median |
761 \& | |0.00121 |0.00003|Deviation |
762 \&
763 \& Report produced by the NYTProf 2.03 Perl profiler, developed by Tim Bunce and
764 \& Adam Kaplan.
765 .Ve
766 .PP
767 At this point, if you\*(Aqre using the \fIhtml\fR report, you can click through the
768 various links to bore down into each subroutine and each line of code. Because
769 we\*(Aqre using the text reporting here, and there\*(Aqs a whole directory full of
770 reports built for each source file, we\*(Aqll just display a part of the
771 corresponding \fIwordmatch\-line.html\fR file, sufficient to give an idea of the
772 sort of output you can expect from this cool tool.
773 .PP
774 .Vb 1
775 \& $> html2text nytprof/wordmatch\-line.html
776 \&
777 \& Performance Profile \-\- \-block view\-.\-line view\-.\-sub view\-
778 \& For wordmatch
779 \& Run on Fri Sep 26 13:46:39 2008
780 \& Reported on Fri Sep 26 13:47:22 2008
781 \&
782 \& File wordmatch
783 \&
784 \& Subroutines \-\- ordered by exclusive time
785 \& |Calls |P|F|Inclusive|Exclusive|Subroutine |
786 \& | | | |Time |Time | |
787 \& |251215|5|1|13.09263 |10.47692 |main::|matches|
788 \& |260642|2|1|2.71199 |2.71199 |main::|debug |
789 \& |1 |1|1|0.21404 |0.21404 |main::|report |
790 \& |0 |0|0|0 |0 |main::|BEGIN |
791 \&
792 \&
793 \& |Line|Stmts.|Exclusive|Avg. |Code |
794 \& | | |Time | | |
795 \& |1 | | | |#!/usr/bin/perl |
796 \& |2 | | | | |
797 \& | | | | |use strict; |
798 \& |3 |3 |0.00086 |0.00029|# spent 0.00003s making 1 calls to strict:: |
799 \& | | | | |import |
800 \& | | | | |use warnings; |
801 \& |4 |3 |0.01563 |0.00521|# spent 0.00012s making 1 calls to warnings:: |
802 \& | | | | |import |
803 \& |5 | | | | |
804 \& |6 | | | |=head1 NAME |
805 \& |7 | | | | |
806 \& |8 | | | |filewords \- word analysis of input file |
807 \& <...snip...>
808 \& |62 |1 |0.00445 |0.00445|print report( %count ); |
809 \& | | | | |# spent 0.21404s making 1 calls to main::report|
810 \& |63 | | | | |
811 \& | | | | |# spent 23.56955s (10.47692+2.61571) within |
812 \& | | | | |main::matches which was called 251215 times, |
813 \& | | | | |avg 0.00005s/call: # 50243 times |
814 \& | | | | |(2.12134+0.51939s) at line 57 of wordmatch, avg|
815 \& | | | | |0.00005s/call # 50243 times (2.17735+0.54550s) |
816 \& |64 | | | |at line 56 of wordmatch, avg 0.00005s/call # |
817 \& | | | | |50243 times (2.10992+0.51797s) at line 58 of |
818 \& | | | | |wordmatch, avg 0.00005s/call # 50243 times |
819 \& | | | | |(2.12696+0.51598s) at line 55 of wordmatch, avg|
820 \& | | | | |0.00005s/call # 50243 times (1.94134+0.51687s) |
821 \& | | | | |at line 54 of wordmatch, avg 0.00005s/call |
822 \& | | | | |sub matches { |
823 \& <...snip...>
824 \& |102 | | | | |
825 \& | | | | |# spent 2.71199s within main::debug which was |
826 \& | | | | |called 260642 times, avg 0.00001s/call: # |
827 \& | | | | |251215 times (2.61571+0s) by main::matches at |
828 \& |103 | | | |line 74 of wordmatch, avg 0.00001s/call # 9427 |
829 \& | | | | |times (0.09628+0s) at line 50 of wordmatch, avg|
830 \& | | | | |0.00001s/call |
831 \& | | | | |sub debug { |
832 \& |104 |260642|0.58496 |2e\-06 |my $message = shift; |
833 \& |105 | | | | |
834 \& |106 |260642|1.09917 |4e\-06 |if ( $debug ) { |
835 \& |107 | | | |print STDERR "DBG: $message\en"; |
836 \& |108 | | | |} |
837 \& |109 | | | |} |
838 \& |110 | | | | |
839 \& |111 |1 |0.01501 |0.01501|exit 0; |
840 \& |112 | | | | |
841 .Ve
842 .PP
843 Oodles of very useful information in there \- this seems to be the way forward.
844 .PP
845 See also \f(CW\*(C`Devel::NYTProf::Apache\*(C'\fR which hooks \f(CW\*(C`Devel::NYTProf\*(C'\fR into
846 \&\f(CW\*(C`mod_perl\*(C'\fR.
847 .SH SORTING
848 .IX Header "SORTING"
849 Perl modules are not the only tools a performance analyst has at their
850 disposal, system tools like \f(CW\*(C`time\*(C'\fR should not be overlooked as the next
851 example shows, where we take a quick look at sorting. Many books, theses and
852 articles, have been written about efficient sorting algorithms, and this is not
853 the place to repeat such work, there\*(Aqs several good sorting modules which
854 deserve taking a look at too: \f(CW\*(C`Sort::Maker\*(C'\fR, \f(CW\*(C`Sort::Key\*(C'\fR spring to mind.
855 However, it\*(Aqs still possible to make some observations on certain Perl specific
856 interpretations on issues relating to sorting data sets and give an example or
857 two with regard to how sorting large data volumes can effect performance.
858 Firstly, an often overlooked point when sorting large amounts of data, one can
859 attempt to reduce the data set to be dealt with and in many cases \f(CWgrep()\fR can
860 be quite useful as a simple filter:
861 .PP
862 .Vb 1
863 \& @data = sort grep { /$filter/ } @incoming
864 .Ve
865 .PP
866 A command such as this can vastly reduce the volume of material to actually
867 sort through in the first place, and should not be too lightly disregarded
868 purely on the basis of its simplicity. The \f(CW\*(C`KISS\*(C'\fR principle is too often
869 overlooked \- the next example uses the simple system \f(CW\*(C`time\*(C'\fR utility to
870 demonstrate. Let\*(Aqs take a look at an actual example of sorting the contents of
871 a large file, an apache logfile would do. This one has over a quarter of a
872 million lines, is 50M in size, and a snippet of it looks like this:
873 .PP
874 # logfile
875 .PP
876 .Vb 10
877 \& 188.209\-65\-87.adsl\-dyn.isp.belgacom.be \- \- [08/Feb/2007:12:57:16 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
878 \& 188.209\-65\-87.adsl\-dyn.isp.belgacom.be \- \- [08/Feb/2007:12:57:16 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
879 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:41 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
880 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:42 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
881 \& 151.56.71.198 \- \- [08/Feb/2007:12:57:43 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en\-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
882 \& 217.113.68.60 \- \- [08/Feb/2007:13:02:15 +0000] "GET / HTTP/1.1" 304 \- "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
883 \& 217.113.68.60 \- \- [08/Feb/2007:13:02:16 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
884 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
885 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
886 \& debora.to.isac.cnr.it \- \- [08/Feb/2007:13:03:58 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "\-" "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.0 (like Gecko)"
887 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:48 +0000] "GET / HTTP/1.0" 200 3309 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
888 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:58 +0000] "GET /data/css HTTP/1.0" 404 206 "http://www.rfi.net/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
889 \& 195.24.196.99 \- \- [08/Feb/2007:13:26:59 +0000] "GET /favicon.ico HTTP/1.0" 404 209 "\-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
890 \& crawl1.cosmixcorp.com \- \- [08/Feb/2007:13:27:57 +0000] "GET /robots.txt HTTP/1.0" 200 179 "\-" "voyager/1.0"
891 \& crawl1.cosmixcorp.com \- \- [08/Feb/2007:13:28:25 +0000] "GET /links.html HTTP/1.0" 200 3413 "\-" "voyager/1.0"
892 \& fhm226.internetdsl.tpnet.pl \- \- [08/Feb/2007:13:37:32 +0000] "GET /suse\-on\-vaio.html HTTP/1.1" 200 2858 "http://www.linux\-on\-laptops.com/sony.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
893 \& fhm226.internetdsl.tpnet.pl \- \- [08/Feb/2007:13:37:34 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net/suse\-on\-vaio.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
894 \& 80.247.140.134 \- \- [08/Feb/2007:13:57:35 +0000] "GET / HTTP/1.1" 200 3309 "\-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
895 \& 80.247.140.134 \- \- [08/Feb/2007:13:57:37 +0000] "GET /data/css HTTP/1.1" 404 206 "http://www.rfi.net" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
896 \& pop.compuscan.co.za \- \- [08/Feb/2007:14:10:43 +0000] "GET / HTTP/1.1" 200 3309 "\-" "www.clamav.net"
897 \& livebot\-207\-46\-98\-57.search.live.com \- \- [08/Feb/2007:14:12:04 +0000] "GET /robots.txt HTTP/1.0" 200 179 "\-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
898 \& livebot\-207\-46\-98\-57.search.live.com \- \- [08/Feb/2007:14:12:04 +0000] "GET /html/oracle.html HTTP/1.0" 404 214 "\-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
899 \& dslb\-088\-064\-005\-154.pools.arcor\-ip.net \- \- [08/Feb/2007:14:12:15 +0000] "GET / HTTP/1.1" 200 3309 "\-" "www.clamav.net"
900 \& 196.201.92.41 \- \- [08/Feb/2007:14:15:01 +0000] "GET / HTTP/1.1" 200 3309 "\-" "MOT\-L7/08.B7.DCR MIB/2.2.1 Profile/MIDP\-2.0 Configuration/CLDC\-1.1"
901 .Ve
902 .PP
903 The specific task here is to sort the 286,525 lines of this file by Response
904 Code, Query, Browser, Referring Url, and lastly Date. One solution might be to
905 use the following code, which iterates over the files given on the
906 command\-line.
907 .PP
908 # sort\-apache\-log
909 .PP
910 .Vb 1
911 \& #!/usr/bin/perl \-n
912 \&
913 \& use v5.36;
914 \&
915 \& my @data;
916 \&
917 \& LINE:
918 \& while ( <> ) {
919 \& my $line = $_;
920 \& if (
921 \& $line =~ m/^(
922 \& ([\ew\e.\e\-]+) # client
923 \& \es*\-\es*\-\es*\e[
924 \& ([^]]+) # date
925 \& \e]\es*"\ew+\es*
926 \& (\eS+) # query
927 \& [^"]+"\es*
928 \& (\ed+) # status
929 \& \es+\eS+\es+"[^"]*"\es+"
930 \& ([^"]*) # browser
931 \& "
932 \& .*
933 \& )$/x
934 \& ) {
935 \& my @chunks = split(/ +/, $line);
936 \& my $ip = $1;
937 \& my $date = $2;
938 \& my $query = $3;
939 \& my $status = $4;
940 \& my $browser = $5;
941 \&
942 \& push(@data, [$ip, $date, $query, $status, $browser, $line]);
943 \& }
944 \& }
945 \&
946 \& my @sorted = sort {
947 \& $a\->[3] cmp $b\->[3]
948 \& ||
949 \& $a\->[2] cmp $b\->[2]
950 \& ||
951 \& $a\->[0] cmp $b\->[0]
952 \& ||
953 \& $a\->[1] cmp $b\->[1]
954 \& ||
955 \& $a\->[4] cmp $b\->[4]
956 \& } @data;
957 \&
958 \& foreach my $data ( @sorted ) {
959 \& print $data\->[5];
960 \& }
961 \&
962 \& exit 0;
963 .Ve
964 .PP
965 When running this program, redirect \f(CW\*(C`STDOUT\*(C'\fR so it is possible to check the
966 output is correct from following test runs and use the system \f(CW\*(C`time\*(C'\fR utility
967 to check the overall runtime.
968 .PP
969 .Vb 1
970 \& $> time ./sort\-apache\-log logfile > out\-sort
971 \&
972 \& real 0m17.371s
973 \& user 0m15.757s
974 \& sys 0m0.592s
975 .Ve
976 .PP
977 The program took just over 17 wallclock seconds to run. Note the different
978 values \f(CW\*(C`time\*(C'\fR outputs, it\*(Aqs important to always use the same one, and to not
979 confuse what each one means.
980 .IP "Elapsed Real Time" 4
981 .IX Item "Elapsed Real Time"
982 The overall, or wallclock, time between when \f(CW\*(C`time\*(C'\fR was called, and when it
983 terminates. The elapsed time includes both user and system times, and time
984 spent waiting for other users and processes on the system. Inevitably, this is
985 the most approximate of the measurements given.
986 .IP "User CPU Time" 4
987 .IX Item "User CPU Time"
988 The user time is the amount of time the entire process spent on behalf of the
989 user on this system executing this program.
990 .IP "System CPU Time" 4
991 .IX Item "System CPU Time"
992 The system time is the amount of time the kernel itself spent executing
993 routines, or system calls, on behalf of this process user.
994 .PP
995 Running this same process as a \f(CW\*(C`Schwarzian Transform\*(C'\fR it is possible to
996 eliminate the input and output arrays for storing all the data, and work on the
997 input directly as it arrives too. Otherwise, the code looks fairly similar:
998 .PP
999 # sort\-apache\-log\-schwarzian
1000 .PP
1001 .Vb 1
1002 \& #!/usr/bin/perl \-n
1003 \&
1004 \& use v5.36;
1005 \&
1006 \& print
1007 \&
1008 \& map $_\->[0] =>
1009 \&
1010 \& sort {
1011 \& $a\->[4] cmp $b\->[4]
1012 \& ||
1013 \& $a\->[3] cmp $b\->[3]
1014 \& ||
1015 \& $a\->[1] cmp $b\->[1]
1016 \& ||
1017 \& $a\->[2] cmp $b\->[2]
1018 \& ||
1019 \& $a\->[5] cmp $b\->[5]
1020 \& }
1021 \& map [ $_, m/^(
1022 \& ([\ew\e.\e\-]+) # client
1023 \& \es*\-\es*\-\es*\e[
1024 \& ([^]]+) # date
1025 \& \e]\es*"\ew+\es*
1026 \& (\eS+) # query
1027 \& [^"]+"\es*
1028 \& (\ed+) # status
1029 \& \es+\eS+\es+"[^"]*"\es+"
1030 \& ([^"]*) # browser
1031 \& "
1032 \& .*
1033 \& )$/xo ]
1034 \&
1035 \& => <>;
1036 \&
1037 \& exit 0;
1038 .Ve
1039 .PP
1040 Run the new code against the same logfile, as above, to check the new time.
1041 .PP
1042 .Vb 1
1043 \& $> time ./sort\-apache\-log\-schwarzian logfile > out\-schwarz
1044 \&
1045 \& real 0m9.664s
1046 \& user 0m8.873s
1047 \& sys 0m0.704s
1048 .Ve
1049 .PP
1050 The time has been cut in half, which is a respectable speed improvement by any
1051 standard. Naturally, it is important to check the output is consistent with
1052 the first program run, this is where the Unix system \f(CW\*(C`cksum\*(C'\fR utility comes in.
1053 .PP
1054 .Vb 3
1055 \& $> cksum out\-sort out\-schwarz
1056 \& 3044173777 52029194 out\-sort
1057 \& 3044173777 52029194 out\-schwarz
1058 .Ve
1059 .PP
1060 BTW. Beware too of pressure from managers who see you speed a program up by 50%
1061 of the runtime once, only to get a request one month later to do the same again
1062 (true story) \- you\*(Aqll just have to point out you\*(Aqre only human, even if you are a
1063 Perl programmer, and you\*(Aqll see what you can do...
1064 .SH LOGGING
1065 .IX Header "LOGGING"
1066 An essential part of any good development process is appropriate error handling
1067 with appropriately informative messages, however there exists a school of
1068 thought which suggests that log files should be \fIchatty\fR, as if the chain of
1069 unbroken output somehow ensures the survival of the program. If speed is in
1070 any way an issue, this approach is wrong.
1071 .PP
1072 A common sight is code which looks something like this:
1073 .PP
1074 .Vb 2
1075 \& logger\->debug( "A logging message via process\-id: $$ INC: "
1076 \& . Dumper(\e%INC) )
1077 .Ve
1078 .PP
1079 The problem is that this code will always be parsed and executed, even when the
1080 debug level set in the logging configuration file is zero. Once the \fBdebug()\fR
1081 subroutine has been entered, and the internal \f(CW$debug\fR variable confirmed to
1082 be zero, for example, the message which has been sent in will be discarded and
1083 the program will continue. In the example given though, the \f(CW\*(C`\e%INC\*(C'\fR hash will
1084 already have been dumped, and the message string constructed, all of which work
1085 could be bypassed by a debug variable at the statement level, like this:
1086 .PP
1087 .Vb 2
1088 \& logger\->debug( "A logging message via process\-id: $$ INC: "
1089 \& . Dumper(\e%INC) ) if $DEBUG;
1090 .Ve
1091 .PP
1092 This effect can be demonstrated by setting up a test script with both forms,
1093 including a \f(CWdebug()\fR subroutine to emulate typical \f(CWlogger()\fR functionality.
1094 .PP
1095 # ifdebug
1096 .PP
1097 .Vb 1
1098 \& #!/usr/bin/perl
1099 \&
1100 \& use v5.36;
1101 \&
1102 \& use Benchmark;
1103 \& use Data::Dumper;
1104 \& my $DEBUG = 0;
1105 \&
1106 \& sub debug {
1107 \& my $msg = shift;
1108 \&
1109 \& if ( $DEBUG ) {
1110 \& print "DEBUG: $msg\en";
1111 \& }
1112 \& };
1113 \&
1114 \& timethese(100000, {
1115 \& \*(Aqdebug\*(Aq => sub {
1116 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) )
1117 \& },
1118 \& \*(Aqifdebug\*(Aq => sub {
1119 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) ) if $DEBUG
1120 \& },
1121 \& });
1122 .Ve
1123 .PP
1124 Let\*(Aqs see what \f(CW\*(C`Benchmark\*(C'\fR makes of this:
1125 .PP
1126 .Vb 5
1127 \& $> perl ifdebug
1128 \& Benchmark: timing 100000 iterations of constant, sub...
1129 \& ifdebug: 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU) @ 10000000.00/s (n=100000)
1130 \& (warning: too few iterations for a reliable count)
1131 \& debug: 14 wallclock secs (13.18 usr + 0.04 sys = 13.22 CPU) @ 7564.30/s (n=100000)
1132 .Ve
1133 .PP
1134 In the one case the code, which does exactly the same thing as far as
1135 outputting any debugging information is concerned, in other words nothing,
1136 takes 14 seconds, and in the other case the code takes one hundredth of a
1137 second. Looks fairly definitive. Use a \f(CW$DEBUG\fR variable BEFORE you call the
1138 subroutine, rather than relying on the smart functionality inside it.
1139 .SS "Logging if DEBUG (constant)"
1140 .IX Subsection "Logging if DEBUG (constant)"
1141 It\*(Aqs possible to take the previous idea a little further, by using a compile
1142 time \f(CW\*(C`DEBUG\*(C'\fR constant.
1143 .PP
1144 # ifdebug\-constant
1145 .PP
1146 .Vb 1
1147 \& #!/usr/bin/perl
1148 \&
1149 \& use v5.36;
1150 \&
1151 \& use Benchmark;
1152 \& use Data::Dumper;
1153 \& use constant
1154 \& DEBUG => 0
1155 \& ;
1156 \&
1157 \& sub debug {
1158 \& if ( DEBUG ) {
1159 \& my $msg = shift;
1160 \& print "DEBUG: $msg\en";
1161 \& }
1162 \& };
1163 \&
1164 \& timethese(100000, {
1165 \& \*(Aqdebug\*(Aq => sub {
1166 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) )
1167 \& },
1168 \& \*(Aqconstant\*(Aq => sub {
1169 \& debug( "A $0 logging message via process\-id: $$" . Dumper(\e%INC) ) if DEBUG
1170 \& },
1171 \& });
1172 .Ve
1173 .PP
1174 Running this program produces the following output:
1175 .PP
1176 .Vb 5
1177 \& $> perl ifdebug\-constant
1178 \& Benchmark: timing 100000 iterations of constant, sub...
1179 \& constant: 0 wallclock secs (\-0.00 usr + 0.00 sys = \-0.00 CPU) @ \-7205759403792793600000.00/s (n=100000)
1180 \& (warning: too few iterations for a reliable count)
1181 \& sub: 14 wallclock secs (13.09 usr + 0.00 sys = 13.09 CPU) @ 7639.42/s (n=100000)
1182 .Ve
1183 .PP
1184 The \f(CW\*(C`DEBUG\*(C'\fR constant wipes the floor with even the \f(CW$debug\fR variable,
1185 clocking in at minus zero seconds, and generates a "warning: too few iterations
1186 for a reliable count" message into the bargain. To see what is really going
1187 on, and why we had too few iterations when we thought we asked for 100000, we
1188 can use the very useful \f(CW\*(C`B::Deparse\*(C'\fR to inspect the new code:
1189 .PP
1190 .Vb 1
1191 \& $> perl \-MO=Deparse ifdebug\-constant
1192 \&
1193 \& use Benchmark;
1194 \& use Data::Dumper;
1195 \& use constant (\*(AqDEBUG\*(Aq, 0);
1196 \& sub debug {
1197 \& use warnings;
1198 \& use strict \*(Aqrefs\*(Aq;
1199 \& 0;
1200 \& }
1201 \& use warnings;
1202 \& use strict \*(Aqrefs\*(Aq;
1203 \& timethese(100000, {\*(Aqsub\*(Aq, sub {
1204 \& debug "A $0 logging message via process\-id: $$" . Dumper(\e%INC);
1205 \& }
1206 \& , \*(Aqconstant\*(Aq, sub {
1207 \& 0;
1208 \& }
1209 \& });
1210 \& ifdebug\-constant syntax OK
1211 .Ve
1212 .PP
1213 The output shows the \fBconstant()\fR subroutine we\*(Aqre testing being replaced with
1214 the value of the \f(CW\*(C`DEBUG\*(C'\fR constant: zero. The line to be tested has been
1215 completely optimized away, and you can\*(Aqt get much more efficient than that.
1216 .SH POSTSCRIPT
1217 .IX Header "POSTSCRIPT"
1218 This document has provided several ways to go about identifying hot\-spots, and
1219 checking whether any modifications have improved the runtime of the code.
1220 .PP
1221 As a final thought, remember that it\*(Aqs not (at the time of writing) possible to
1222 produce a useful program which will run in zero or negative time and this basic
1223 principle can be written as: \fIuseful programs are slow\fR by their very
1224 definition. It is of course possible to write a nearly instantaneous program,
1225 but it\*(Aqs not going to do very much, here\*(Aqs a very efficient one:
1226 .PP
1227 .Vb 1
1228 \& $> perl \-e 0
1229 .Ve
1230 .PP
1231 Optimizing that any further is a job for \f(CW\*(C`p5p\*(C'\fR.
1232 .SH "SEE ALSO"
1233 .IX Header "SEE ALSO"
1234 Further reading can be found using the modules and links below.
1235 .SS PERLDOCS
1236 .IX Subsection "PERLDOCS"
1237 For example: \f(CW\*(C`perldoc \-f sort\*(C'\fR.
1238 .PP
1239 perlfaq4.
1240 .PP
1241 perlfork, perlfunc, perlretut, perlthrtut.
1242 .PP
1243 threads.
1244 .SS "MAN PAGES"
1245 .IX Subsection "MAN PAGES"
1246 \&\f(CW\*(C`time\*(C'\fR.
1247 .SS MODULES
1248 .IX Subsection "MODULES"
1249 It\*(Aqs not possible to individually showcase all the performance related code for
1250 Perl here, naturally, but here\*(Aqs a short list of modules from the CPAN which
1251 deserve further attention.
1252 .PP
1253 .Vb 10
1254 \& Apache::DProf
1255 \& Apache::SmallProf
1256 \& Benchmark
1257 \& DBIx::Profile
1258 \& Devel::AutoProfiler
1259 \& Devel::DProf
1260 \& Devel::DProfLB
1261 \& Devel::FastProf
1262 \& Devel::GraphVizProf
1263 \& Devel::NYTProf
1264 \& Devel::NYTProf::Apache
1265 \& Devel::Profiler
1266 \& Devel::Profile
1267 \& Devel::Profit
1268 \& Devel::SmallProf
1269 \& Devel::WxProf
1270 \& POE::Devel::Profiler
1271 \& Sort::Key
1272 \& Sort::Maker
1273 .Ve
1274 .SS URLS
1275 .IX Subsection "URLS"
1276 Very useful online reference material:
1277 .PP
1278 .Vb 1
1279 \& https://web.archive.org/web/20120515021937/http://www.ccl4.org/~nick/P/Fast_Enough/
1280 \&
1281 \& https://web.archive.org/web/20050706081718/http://www\-106.ibm.com/developerworks/library/l\-optperl.html
1282 \&
1283 \& https://perlbuzz.com/2007/11/14/bind_output_variables_in_dbi_for_speed_and_safety/
1284 \&
1285 \& http://en.wikipedia.org/wiki/Performance_analysis
1286 \&
1287 \& http://apache.perl.org/docs/1.0/guide/performance.html
1288 \&
1289 \& http://perlgolf.sourceforge.net/
1290 \&
1291 \& http://www.sysarch.com/Perl/sort_paper.html
1292 .Ve
1293 .SH AUTHOR
1294 .IX Header "AUTHOR"
1295 Richard Foley <[email protected]> Copyright (c) 2008