Debian Codegraph @

README

raw
1 -------------------------------------------------
2 - README file for the LTP GCOV extension (LCOV) -
3 - Last changes: 2024-12-25
4 -------------------------------------------------
5
6 Description
7 -----------
8 LCOV is a tool to manipulate and display information about what parts of a
9 program are actually executed (i.e. "covered") while running a particular test
10 case or set of testcases. LCOV consists of a set of Perl scripts which build on
11 the text output of various coverage tools - e.g., gcov, llvm-cov, Coverage.py,
12 Cobertura, Devel::Cover, Jacoco, etc. - to implement the following enhanced
13 functionality:
14
15 * HTML based output: coverage rates are indicated using bar
16 graphs and specific colors in a hyperlinked coverage report, intended
17 to enable the user to quickly diagnose and address coverage issues.
18
19 * Support for large projects: overview pages allow quick browsing of
20 coverage data by providing a hierarchical directory structure
21 view, a flat list of all source files in the project, or a three-level
22 detail view: directory, file and source code view.
23
24 * Support for multiple languages - including C/C++, Perl, and Python.
25
26 LCOV was initially designed to support Linux kernel coverage measurements,
27 but works as well for coverage measurements on standard user space
28 applications.
29
30 LCOV supports differential coverage, as well as date- and owner-binning.
31 See:
32 https://arxiv.org/abs/2008.07947
33 or
34 https://ieeexplore.ieee.org/document/9438597
35 for a detailed explanation of the concepts and several possible use models.
36
37 A video presentation of the basic ideas can be found at
38 http://doi.org/10.5281/zenodo.4653252
39
40 In addition, several other features and capabilities are available. See
41 section 6, below, for a brief description - and also see the man pages and
42 the test cases.
43
44
45 Further README contents
46 -----------------------
47 1. Included files
48 2. Installing LCOV
49 3. Dependencies
50 4. An example of how to access kernel coverage data
51 5. An example of how to access coverage data for a user space program
52 6. LCOV features
53 7. Questions and Comments
54 8. Filing a new issue
55
56
57
58 1. Important files
59 ------------------
60 README - This README file
61 CHANGES - List of changes between releases
62 bin/lcov - Tool for capturing LCOV coverage data
63 bin/genhtml - Tool for creating HTML output from LCOV data
64 bin/gendesc - Tool for creating description files as used by genhtml
65 bin/perl2lcov - Tool to translate Perl Devel::Cover data to lcov format
66 bin/llvm2lcov - Tool to translate LLVM 'llvm-cov' JSON data to LCOV format
67 bin/py2lcov - Tool to translate Python Coverage.py to lcov format
68 bin/xml2lcov - Tool to translate Cobertura-like XML coverage data
69 to lcov format
70 bin/geninfo - Internal tool (creates LCOV data files)
71 bin/genpng - Internal tool (creates png overviews of source files)
72 lcovrc - LCOV configuration file
73 man - Directory containing man pages for included tools
74 example - Directory containing an example to demonstrate LCOV
75 tests - Directory containing lcov regression tests
76 Makefile - Makefile providing 'install' and 'uninstall' targets
77
78
79 2. Installing LCOV
80 ------------------
81 The LCOV package is available as either RPM or tarball from:
82
83 https://github.com/linux-test-project/lcov/releases
84
85 To install the tarball, unpack it to a directory and run:
86
87 make install
88
89 Use Git for the most recent (but possibly unstable) version:
90
91 git clone https://github.com/linux-test-project/lcov.git
92
93 Change to the resulting lcov directory and type:
94
95 make install
96
97 The default install location is /usr/local. Note that you may need to
98 have superuser permissions to write into system directories.
99
100 To install in a different location - for example, your home directory, run:
101
102 make PREFIX=$HOME/my_lcov install
103
104 your PREFIX should be an absolute path.
105
106 To run the LCOV regression test suite on your installation:
107
108 $ cp -r $LCOV_HOME/share/test path/to/myTestDir
109 $ cd path/to/myTestDir
110 $ make [COVERAGE=1]
111
112 If desired, you can collect coverage data for the LCOV module by setting
113 the COVERAGE makefile variable.
114 Note that the Devel::Cover package must be installed if COVERAGE is enabled
115 or if you want to use the perl2lcov utility.
116 To view the collected coverage information, point your browser to
117 .../lcov_coverage/index.html after running the tests.
118
119 Note that the testcases are primarily intended to test LCOV functionality
120 and not to be easily readable tutorial examples.
121
122 3. Dependencies:
123 ----------------
124
125 The lcov module is implemented primarily in Perl - and requires both a
126 moderately up-to-date Perl installation and multiple Perl packages.
127
128 These perl packages include:
129
130 - Capture::Tiny
131 - DateTime
132 - Devel::Cover
133 - Digest::MD5
134 - File::Spec
135 - at least one flavor of JSON module.
136 In order of performance/preference:
137 - JSON::XS
138 - Cpanel::JSON::XS
139 - JSON::PP
140 - JSON
141 - Memory::Process
142 - Module::Load::Conditional
143 - Scalar::Util
144 - Time::HiRes
145 - TimeDate
146
147 If your system is missing any of these, then you may be able to install them
148 via:
149
150 $ perl -MCPAN -e 'install(<packageName>)'
151
152 You will very likely need superuser access to be able to install Perl
153 modules.
154
155 Some of the applications provided with the lcov module are written
156 in Python - and may require additional Python packages.
157 In particular, 'xlsxwriter' is required in order to generate any
158 of the spreadsheet reports.
159
160 To measure Python code coverage, users will need Python packages:
161
162 - Coverage.py
163
164 In addition, contributors will need:
165
166 - perltidy
167
168 Your platform may support other mechanisms to install and/or update
169 required packages.
170
171
172
173 4. An example of how to access Linux kernel coverage data
174 ---------------------------------------------------------
175 Requirements: Follow the Linux kernel coverage setup instructions at:
176
177 https://docs.kernel.org/dev-tools/gcov.html
178
179 As root, do the following:
180
181 a) Resetting counters
182
183 lcov --zerocounters
184
185 b) Capturing the current coverage state to a file
186
187 lcov --capture --output-file kernel.info
188
189 c) Getting HTML output
190
191 genhtml kernel.info
192
193 Point the web browser of your choice to the resulting index.html file.
194
195
196 5. An example of how to access coverage data for a user space program
197 ---------------------------------------------------------------------
198
199 a) Capture current coverage state to a file:
200
201 i) C/C++ code:
202
203 Compile your program using the '--coverage' GCC or LLVM
204 option. During linking, make sure to specify '--coverage':
205
206 $ gcc -o myTest --coverage simple.c
207 OR
208 $ gcc -c file1.c file2.c ... --coverage
209 $ gcc -o myOtherTest --coverage file1.o file2.o ....
210
211 Alternately, LLVM users can use the 'profdata path' (rather than the
212 'gcov path') to collect coverage data from their C/C++ code. See
213 https://github.com/linux-test-project/lcov/discussions/234 for more
214 information.
215
216 Run your testcase at least once:
217
218 $ path/to/my/testcase/myTest
219
220 Capture the current coverage state to a file:
221
222 $ lcov --directory path/to/my/testcase --capture --output-file app.info
223
224 (LLVM users using the 'profdata path' will use a somewhat different
225 command for this step - see the discussion referenced above.)
226
227 If you want to collect Modified Condition / Decision Coverage (MD/DC)
228 date, then:
229 - you must use gcc/14.2 (or newer), or LLVM/18 (or newer)
230 - your GCC compile- and link command line must include flag
231 '-fcondition-coverage'.
232 - LLVM users must use the 'profdata path' for coverage data collection,
233 and your compile command line must include
234 '-fprofile-inst-generate -fcoverage-mapping -fcoverage-mcdc'.
235 See the above referenced discussion for details.
236 - your lcov and genhtml command line must include flag
237 '--mcdc-coverage'
238 See the '--mcdc-coverage' section in the genhtml and geninfo man pages.
239
240 Note that runtime coverage data exists only after the application has
241 been started and stopped at least once. Otherwise, no data will be found
242 and lcov will abort with an error mentioning that there are no
243 data/.gcda files.
244
245 The coverage runtime emits data (the .gcda files) in an atexit
246 callback. If your application exits abnormally or crashes before
247 the callback is executed, then no coverage data will be available.
248
249 For further information on the gcc profiling mechanism, please
250 consult the gcov man page.
251
252 See 'man lcov' for more information - especially if your build/test
253 environment is not trivial.
254
255 ii) Python code:
256
257 - install the Coverage.py module
258
259 - execute your testcase to produce python coverage data:
260
261 $ COVERAGE_FILE=./pycov.dat coverage run --append --branch \
262 myPythonScript [my script args]
263
264 - translate Python coverage data to LCOV format:
265
266 $ py2lcov -o pycov.info [py2lcov_options] pycov.dat [x.dat]+
267
268 See 'py2lcov --help' and the Coverage.py documentation for more
269 information.
270
271 iii) Perl code:
272
273 - install the Devel::Cover module
274
275 - execute your testcase to produce perl coverage data:
276
277 $ perl -MDevel::Cover=-db,perlcov_db,-coverage,statement,branch,condition,subroutine,-silent,1 myPerlTest.pl [my script args]
278
279 - translate Perl coverage data to LCOV format:
280
281 $ perl2lcov --output perlcov.info perlcov_db [perl2lcov options]
282
283 See 'perl2lcov --help' and the Devel::Cover documentation for more
284 information.
285
286 iv) XML data (for example, generated by Cobertura):
287
288 - translate XM coverage data to LCOV format:
289
290 $ xml2lcov --output myData.info coverage.xml [xml2lcov options]
291
292 See 'xml2lcov --help' and the Cobertura documentation for more
293 information.
294
295 b) Generate an HTML coverage report:
296
297 Generate an HTML report, combining all of your LCOV data files:
298
299 $ genhtml -o html_report app.info pycov.info perlcov.info
300
301 Point the web browser of your choice to the resulting file:
302 html_report/index.html.
303
304 See 'man genhtml' for more details.
305
306 c) Generate a differential coverage report:
307
308 See the example in .../example (run "make test_differential")
309 as well as the examples in .../tests/gendiffcov.
310
311
312 6. LCOV Features:
313 -----------------
314
315 LCOV features and capabilities fall into 7 major categories:
316
317 a) Categorization
318
319 This refers primarily to differential coverage categorization as
320 well as date- and owner-binning. See https://arxiv.org/abs/2008.07947
321 or https://ieeexplore.ieee.org/document/9438597 for a detailed
322 description of the concepts.
323
324 Differential categorization and binning are orthogonal in the sense
325 that you can generate differential report without binning as well
326 as 'vanilla' coverage reports with binning. See the above papers
327 and the genhtml man page for details.
328
329 Related options:
330 --baseline-file, --diff-file, --annotate-script, --select-script
331 --date-bins, --date-labels --new-file-as-baseline,
332 --elide-path-mismatch
333
334 b) Error handling
335
336 A generic - but very simple - error handler has been added to the
337 lcov tool suite. The error handler is used to report exceptions,
338 and provides a mechanism for the user to ignore the particular
339 message if desired. Note that ignoring certain errors can cause
340 subsequent errors and/or can result in inconsistent or confusing
341 coverage reports.
342 See the genhtml/lcov/geninfo man pages for details.
343
344 Note that some errors are unrecoverable - and cannot be suppressed or
345 ignored.
346
347 Related options:
348 --ignore-error, --expect-message-count, --keep-going, --msg-log
349
350 c) Navigation and display:
351
352 Navigation aids such as hyperlinks to the first uncovered region,
353 to the next uncovered region, etc. have been implemented. Similarly,
354 new tables, new columns, and new links between tables enable the
355 user to identify the author of particular code (covered or not
356 covered), as well as the time period when the code was written.
357
358 Collectively, these features help the user to quickly identify the
359 cause of code coverage issues, and to then decide what to do.
360
361 An option to generate a 'hierarchical' coverage report (which follows
362 the source code directory structure) or 'flat' (all files in top level
363 of two-level report) as well as various other small features (tooltip
364 popups, user-specified HTML header, footer, and table labels, etc.) are
365 also available.
366
367 See the genhtml man page for some details, as well as the
368 'gendiffcov/simple' testcases for some examples.
369
370 Related options:
371 --baseline-title, --baseline-date, --current-date,
372 --flat, --hierarchical,
373 --show-owners, --show-noncode, --show-navigation, --show-proportion,
374 --suppress-aliases
375
376 d) Data manipulation
377
378 Filters are used to suppress or remove certain coverage artifacts -
379 for example, branches generated by the compiler (e.g., for exception
380 handling). These artifacts can overwhelm the user code and obscure
381 coverage features that are interesting to the user.
382
383 Other options are used to focus on or to exclude certain sections
384 of code, as well as to do regexp replacement of file names - possibly
385 using case-insensitive comparison.
386 (Path munging is useful primarily when the build structure does
387 not exactly match the layout in your revision control system; this
388 is common in large projects with reusable components.)
389
390 During coverage data capture, the --build-directory option can be used
391 to specify a search path, to find the .gcno (compile-time coverage data)
392 file corresponding to a particular .gcda (runtime coverage data) file.
393 Similarly, the --source-directory option can be used to specify a
394 search path for source files.
395
396 See the lcov/geninfo/genhtml man pages for a detailed description of
397 the available filters and manipulation features.
398
399 Related options:
400 --include, --exclude, --erase-functions, --omit-lines,
401 --substitute, --filter
402 --build-directory --source-directory
403
404 e) Callbacks/customization
405
406 The user can supply callbacks which are used to:
407
408 i) interface with the revision control system
409 Sample scripts:
410 - Perforce: see 'p4diff', 'p4annotate.pm', 'p4annotate'
411 - Git: see 'gitdiff', 'gitblame.pm', 'gitblame'
412 ii) verify that source code versions are compatible, and
413 Sample scripts: see 'get_signature', 'P4version.pm', 'getp4version',
414 'gitversion', 'gitversion.pm', and 'batchGitVersion.pm'
415 iii) enforce a desired code coverage criteria
416 Sample script: criteria.pm/criteria
417 iv) find source files in more complicated environments - where
418 simple substitutions become complicated or unweildy.
419 v) select a subset of coverage data to display - e.g., to
420 use in a code review which wants to concentrate on only
421 the changes caused by a particular commit or range of commits,
422 or to review changes in a particular release.
423 Sample script: select.pm
424 vi) keep track of environment and other settings - to aid
425 infrastructure debugging in more complicated use cases.
426
427 The callback may be any desired script or executable - but there
428 may be performance advantages if it is written as a Perl module.
429
430 See the genhtml/lcov/geninfo man pages for details.
431
432 Note that the various sample scripts are found in the source code
433 'scripts' directory, but are installed in the
434 $LCOV_HOME/share/lcov/support-scripts directory of the release.
435
436 Related options:
437 --annotate-script, --criteria-script, --version-script
438 --resolve-script, --select-script, --context-script
439
440 f) Performance
441
442 lcov/genhtml/geninfo have been refactored to parallelize computation
443 across multiple cores, if requested.
444 In general, this provides speedup that is nearly linear in the number
445 of cores.
446 There is also an option to throttle parallelism to not exceed peak
447 memory consumption constraints, as well as options to enable simple
448 profile data collection - so you can see where time is going and
449 thus to hint at potential optimizations. The 'spreadsheet.py'
450 script can be used to view generated profile data.
451
452 There are several configuration file options which can be used to
453 tweak certain parallelization parameters to optimize performance
454 for your environment in cases that the default behaviour is suboptimal.
455 See the lcovrc man page for more information.
456
457 See the genhtml/lcov/geninfo man pages for details
458
459 Related options: --parallel, --memory, --profile
460
461 g) Language/tool support
462
463 Added 'llvm2lcov', 'py2lcov', 'perl2lcov' and 'xml2lcov' scripts.
464
465 - llvm2lcov:
466
467 translates JSON coverage data generated by 'llvm-cov export -format=text ...'
468 to lcov format.
469
470 See "llvm2lcov --help" for brief instruction on how to use the
471 translator. Note that llvm2lcov uses a similar set of command line
472 and configuration file options as lcov, genhtml, and geninfo.
473
474 - py2lcov:
475
476 translates python Coverage.py XML data to lcov format.
477
478 See the Coverage.py documentation at https://coverage.readthedocs.io,
479 as well as ".../py2lcov --help"
480
481 - perl2lcov
482
483 translates Perl Devel::Cover data to lcov format.
484
485 See the Devel::Cover documentation at
486 https://metacpan.org/pod/Devel::Cover
487 to find out how to generate coverage data for Perl code.
488
489 See "perl2lcov --help" for brief instructions on how to
490 use the translator.
491 Note that perl2lcov uses a similar set of command line and
492 config file options as lcov, genhtml, and geninfo.
493
494 - xml2lcov
495
496 translates XML coverage data to lcov format.
497 The XML data may come from Cobertura or similar tools.
498
499 See "xml2lcov --help" for brief instructions on how to use
500 the translator.
501 See the Coburtura documentation for directions on how to
502 generate XML data.
503
504 Other languages can be integrated using a similar approach.
505
506 In general, the new features and options are implemented uniformly in lcov,
507 genhtml, and geninfo. Most of the features can be enabled/disabled
508 using either command line options or by setting defaults in your 'lcovrc'
509 file. See the lcovrc man page for details.
510
511
512 7. Questions and comments
513 -------------------------
514 See the included man pages for more information on how to use the LCOV tools.
515
516 In case of further questions, feel free to open a new issue or discussion using
517 the issue tracker on the LCOV code repository site at:
518
519 https://github.com/linux-test-project/lcov
520
521
522 8. Filing a new issue
523 ---------------------
524 Before filing a new issue - and if you are using an LCOV release (as opposed
525 to using a clone of the github repo) - please verify whether the issue is
526 still present in the LCOV master version. See section 2, above for
527 directions on how to clone and install the most up-to-date LCOV version.
528
529 If possible, please include a testcase which illustrates the problem
530 when you file an issue.
531 Please describe your environment (platform, compiler, perl, and python
532 versions, etc.). Please include a detailed description of the issue:
533 what you were trying to do (your goal - not the mechanics of your
534 procedure), what you did (the mechanics of your procedure), the result
535 you wanted to see vs. what actually happened.
536 Depending on the issue, your testcase may need to include source code and
537 compile/link command lines, directions for how to run your example, the
538 command lines used to capture and generate your lcov reports, etc.
539 In other cases, the captured '.info' files may be sufficient to reproduce
540 the issue.
541 When in doubt: more is better than less.
542
543 If you cannot include a testcase - e.g., because you feel that it is
544 senstitive or proprietary - then your detailed description is even more
545 important.
546 Note that, without an example, it may be difficult or impossible to
547 diagnose or fix the problem.
548
549 Bear in mind that you are asking for help from volunteers. Your
550 priority might not be their priority. Civility, consideration and politeness
551 go a long way.
552
553 Please check back and to verify the fix and close the issue once it has
554 been addressed.
555 Again: remember that you are asking for help from volunteers.
556 Make sure that you are doing your part.