Debian Codegraph @

cmake/tribits/doc/guides/TribitsGuidesBody.rst

raw
1 Background
2 ==========
3
4 In order to easily find the most appropriate documentation, see the `TriBITS
5 Developer and User Roles`_ guide. This guide describes the different roles
6 that users of TriBITS may play and offers links to relevant sections of the
7 documentation. Additionally, the reader may wish to review the `CMake Language
8 Overview and Gotchas`_ section which is meant for users that are new to both
9 CMake and TriBITS. This section gives a brief overview of getting started with
10 CMake and provides some warnings about non-obvious CMake behavior that often
11 trips up new users of TriBITS.
12
13
14 TriBITS Developer and User Roles
15 --------------------------------
16
17 There are approximately five different types roles related to TriBITS. These
18 different roles require different levels of expertise and knowledge of CMake
19 and knowledge of the TriBITS system. The primary roles are 1) `TriBITS
20 Project User`_, 2) `TriBITS Project Developer`_, 3) `TriBITS Project
21 Architect`_, 4) `TriBITS System Developer`_, and 5) `TriBITS System
22 Architect`_. Each of these roles builds on the necessary knowledge of the
23 lower-level roles.
24
25 .. _TriBITS Project User:
26 .. _TriBITS Project Users:
27
28 The first role is that of a **TriBITS Project User** who only needs to be able
29 to configure, build, and test a project that uses TriBITS as its build system.
30 A person acting in this role needs to know little about CMake other than
31 basics about how to run the ``cmake`` and ``ctest`` executables, how to set
32 CMake cache variables, and the basics of building software by typing ``make``
33 and running tests with ``ctest``. The proper reference for a TriBITS Project
34 User is the `Project-Specific Build Reference`_. The `TriBITS
35 Overview`_ document may also be of some help. A TriBITS project user may also
36 need to consult `Package Dependencies and Enable/Disable Logic`_.
37
38 .. _TriBITS Project Developer:
39 .. _TriBITS Project Developers:
40
41 A **TriBITS Project Developer** is someone who contributes to a software
42 project that uses TriBITS. They will add source files, libraries and
43 executables , test executables and define tests run with ``ctest``. They have
44 to configure and build the project code in order to be able to develop and run
45 tests and therefore this role includes all of the necessary knowledge and
46 functions of a TriBITS Project User. A casual TriBITS Project Developer
47 typically does not need to know a lot about CMake and really only needs to
48 know a subset of the `TriBITS Macros and Functions`_ defined in the document
49 `TriBITS Users Guide and Reference`_ in addition to the genetic `TriBITS Build
50 Reference <TribitsBuildReference.html>`_ document. A slightly more
51 sophisticated TriBITS Project Developer will also add new packages, add new
52 package dependencies, and define new external packages/TPLs. The `TriBITS
53 Users Guide and Reference`_ should supply everything such a developer needs to
54 know and more. However, only a smaller part of that document needs to be
55 understood and accessed by people assuming this role.
56
57 .. jfrye: Add links to parts of document above?
58
59 .. _TriBITS Project Architect:
60 .. _TriBITS Project Architects:
61
62 The next level of roles is a **TriBITS Project Architect**. This is someone
63 (perhaps only one person on a project development team) that knows the usage
64 and functioning of TriBITS in great detail. They understand how to set up a
65 TriBITS project from scratch, how to set up automated testing using the
66 TriBITS system, and know how to use TriBITS to implement the overall software
67 development process. A person in this role is also likely to be the one who
68 makes the initial technical decision for their project to adopt TriBITS for
69 its native build and test system. The document `TriBITS Users Guide and
70 Reference`_, detailed CMake/CTest/CDash documentation provided by Kitware, and
71 great books like `Professional CMake`_ should provide most of what a person in
72 this role needs to know. A person assuming this role is the primary audience
73 for a lot of the more advanced material in that document.
74
75 .. _TriBITS System Developer:
76 .. _TriBITS System Developers:
77 .. _TriBITS System Architect:
78 .. _TriBITS System Architects:
79
80 The last two roles **TriBITS System Developer** and **TriBITS System
81 Architect** are for those individuals that actually extend and modify the
82 TriBITS system itself. A TriBITS System Developer needs to know how to add
83 new TriBITS functionality while maintaining backward compatibility, know how
84 to add new unit tests for the TriBITS system (see `The TriBITS Test
85 Package`_), and perform other related tasks. Such a developer needs to be
86 very knowledgeable of the basic functioning of CMake and know how TriBITS is
87 implemented in the CMake language. A TriBITS System Architect is someone who
88 must be consulted on almost all non-trivial changes or additions to the
89 TriBITS system. A *TriBITS System Architect* in addition needs to know the
90 entire TriBITS system, the design philosophy that provides the foundation for
91 TriBITS and be an expert in CMake, CTest, and CDash. Much of what needs to be
92 known by a TriBITS System Developer and a TriBITS System Architect is
93 contained in the document `TriBITS Maintainers Guide and Reference`_. The
94 rest of the primary documentation for these roles will be in the TriBITS
95 CMake source code and various unit tests itself defined in `The TriBITS Test
96 Package`_. At the time of this writing, there is currently there is only one
97 TriBITS System Architect (who also happens to be the primary author of this
98 document).
99
100 An explicit goal of the document `TriBITS Users Guide and Reference`_ is to
101 foster the creation of new `TriBITS Project Architects`_ (i.e. those who would
102 make the decision to adopt TriBITS for their projects). An explicit goal of
103 the document `TriBITS Maintainers Guide and Reference`_ is to foster the
104 creation of new `TriBITS System Developers`_ to help extend and maintain the
105 TriBITS package itself.
106
107 Depending on the particular role that a reader falls into, this document
108 may not be necessary and the `TriBITS Overview`_ or the
109 `<Project>BuildReference`_ documents may be more appropriate. Hopefully the
110 above roles and discussion help the reader select the right document to
111 start with.
112
113 NOTE: Before getting started with TriBITS, if a reader is unfamiliar with
114 CMake, please review the `CMake Language Overview and Gotchas`_. Once those
115 CMake basics and common gotchas have been reviewed, we now get into the meat
116 of TriBITS starting with software engineering principles that lie at the
117 foundation of TriBITS. That is followed by with and overall of the structure
118 of a TriBITS project.
119
120
121 Software Engineering Packaging Principles
122 -----------------------------------------
123
124 The design of TriBITS takes into account standard software engineering
125 packaging principles. In his book [`Agile Software Development, 2003`_],
126 Robert Martin defines several software engineering principles related to
127 packaging software which are listed below:
128
129 * *Package Cohesion OO Principles*:
130
131 1) *REP (Release-Reuse Equivalency Principle)*: The granule of reuse is the
132 granule of release.
133
134 2) *CCP (Common Closure Principle)*: The classes in a package should be
135 closed together against the same kinds of changes. A change that affects
136 a closed package affects all the classes in that package and no other
137 packages.
138
139 3) *CRP (Common Reuse Principle)*: The classes in a package are used
140 together. If you reuse one of the classes in a package, you reuse them
141 all.
142
143 * *Package Coupling OO Principles*:
144
145 4) *ADP (Acyclic Dependencies Principle)*: Allow no cycles in the package
146 dependency graph.
147
148 5) *SDP (Stable Dependencies Principle)*: Depend in the direction of
149 stability.
150
151 6) SAP (Stable Abstractions Principle): A package should be as abstract as
152 it is stable.
153
154 Any of these six OO packaging principles (and other issues) may be considered
155 when deciding how to partition software into different `TriBITS Packages`_.
156
157 NOTE: The purpose of this TriBITS Developers Guide document is not teach
158 basic software engineering so these various principles will not be expanded on
159 further. However, interested readers are strongly encouraged to read
160 [`Agile Software Development, 2003`_] as one of the better software
161 engineering books out there (see https://bartlettroscoe.github.io/reading-list/#most_recommended_se_books).
162
163
164 TriBITS Project Structure
165 =========================
166
167 TriBITS is a Framework, implemented in CMake, to create CMake projects. As a
168 `Software Framework`_, TriBITS defines the overall structure of a CMake build
169 system for a project and it processes the various project-, repository-, and
170 package-specific files in a specified order. Almost all of this processing
171 takes place in the `tribits_project()`_ macro (or macros and functions it
172 calls). The following subsections define the essence of the TriBITS framework
173 in some detail. Later sections cover specific topics and the various sections
174 link to each other. Within this section, the subsection `TriBITS Structural
175 Units`_ defines the basic units `TriBITS Project`_, `TriBITS Repository`_,
176 `TriBITS Package`_, `TriBITS External Package/TPL`_ and other related
177 structural units. The subsection `Processing of TriBITS Files: Ordering and
178 Details`_ defines exactly what files TriBITS processes and in what order. It
179 also shows how to get TriBITS to show exactly what files it is processing to
180 help in debugging issues. The subsection `Coexisting Projects, Repositories,
181 and Packages`_ gives some of the rules and constrains for how the different
182 structure units can co-exist in the same directories. The last two
183 subsections in this section cover `Standard TriBITS TPLs`_ and `Common TriBITS
184 TPLs`_.
185
186 TriBITS Structural Units
187 ------------------------
188
189 A CMake project that uses TriBITS as its build and test system is composed of
190 a single `TriBITS Project`_, one or more `TriBITS Repositories`_ and one or
191 more `TriBITS Packages`_. In addition, a TriBITS Package can be broken up
192 into `TriBITS Subpackages`_. Together, the collection of TriBITS Packages and
193 TriBITS Subpackages are called *TriBITS Software Engineering Packages*, or
194 `TriBITS Packages`_ for short.
195
196 First, to better establish the basic nomenclature, the key structural TriBITS
197 units are:
198
199 .. _TriBITS Top-Level Package:
200 .. _TriBITS Top-Level Packages:
201
202 * `TriBITS Top-Level Package`_: A collection of related software that
203 typically includes one or more source files built into one or more libraries
204 and has associated tests to help define and protect the functionality
205 provided by the software. A top-level package (or just "package" if no
206 ambiguity) also typically defines a single integrated unit of documentation
207 and testing (see `TriBITS Automated Testing`_). A top-level TriBITS package
208 may or may not be broken down into multiple subpackages. Examples of TriBITS
209 packages in `TribitsExampleProject`_ include ``SimpleCXX``, ``MixedLang``
210 and ``WithSubpackages``. (Don't confuse a TriBITS "Package" with a raw
211 CMake "Package". A raw CMake "Package" actually maps to a `TriBITS External
212 Package/TPL`_; see `Why a TriBITS Package is not a CMake Package`_.)
213
214 .. _TriBITS Subpackages:
215
216 * `TriBITS Subpackage`_: A part of a parent `TriBITS Package`_ that also
217 typically has source files built into libraries and tests but is documented
218 and tested along with the other subpackages in the parent package. The primary
219 purpose for supporting subpackages is to provide finer-grained control of
220 software dependencies. In `TribitsExampleProject`_, ``WithSubpackages`` is
221 an example of a package with subpackages ``'A'``, ``'B'``, and ``'C'``. The
222 full subpackage name is prefixed by the parent package name (e.g. the full
223 name for subpackage ``'A'`` is ``WithSubpackagesA``). The parent package is
224 always implicitly dependent on its subpackages (e.g. parent package
225 ``WithSubpackages`` depends on its subpackages ``WithSubpackagesA``,
226 ``WithSubpackagesB``, and ``WithSubpackagesC``).
227
228 .. _TriBITS TPL:
229 .. _TriBITS TPLs:
230 .. _TriBITS External Package/TPL:
231 .. _TriBITS External Packages/TPLs:
232
233 * `TriBITS External Package/TPL`_: The specification for a particular external
234 dependency that is required or can be used in one or more `TriBITS
235 Packages`_. A modern TriBITS external package/TPL (Third Party Library) is
236 typically just a small file ``FindTPL<tplName>.cmake`` that calls
237 ``find_package(<externalPkg>)`` and defines the ``<tplName>::all_libs``
238 target. More generally, an external package/TPL can be specified as a list
239 list of libraries and/or include directories for header files. Examples of
240 basic external packages/TPLs include ``BLAS``, ``LAPACK``, and ``Boost``.
241
242 .. _TriBITS Package:
243 .. _TriBITS Packages:
244
245 * **TriBITS Package**: The combined set of `TriBITS Top-Level Packages`_,
246 `TriBITS Subpackages`_, and `TriBITS External Packages/TPLs`_ that
247 constitute the basic *Software Engineering* packages of a TriBITS project
248 (see `Software Engineering Packaging Principles`_): Packages are the basis
249 for setting dependencies in the TriBITS system. For example, the Packages
250 provided by the example top-level package ``WithSubpackages`` (in order of
251 increasing dependencies) are ``WithSubpackagesA``, ``WithSubpackagesB``,
252 ``WithSubpackagesC``, and ``WithSubpackages`` (see
253 `TribitsExampleProject`_).
254
255 .. _TriBITS Repositories:
256
257 * `TriBITS Repository`_: A collection of one or more `TriBITS Packages`_
258 specified in a `<repoDir>/PackagesList.cmake`_ file and zero or more
259 external package/TPL declarations specified in a `<repoDir>/TPLsList.cmake`_
260 file. As discussed below, Repositories can include Native Repositories and
261 Version Control (VC) Repositories.
262
263 .. _TriBITS Projects:
264
265 * `TriBITS Project`_: A collection of `TriBITS Repositories`_ and `TriBITS
266 Packages`_ that defines a complete CMake ``PROJECT`` defining software which can be
267 directly configured with ``cmake`` and then be built, tested, installed,
268 etc.
269
270 .. _TriBITS Meta-Project:
271 .. _TriBITS Meta-Projects:
272
273 * **TriBITS Meta-Project**: A `TriBITS Project`_ that contains no native
274 `TriBITS packages`_ or `TriBITS external packages/TPLs`_ but is composed out
275 packages from other `TriBITS Repositories`_.
276
277 In this document, dependencies are described as either being *upstream* or
278 *downstream/forward* defined as:
279
280 .. _upstream:
281 .. _upstream dependency:
282 .. _upstream dependencies:
283
284 * If unit "B" requires (or can use, or comes before) unit "A", then "A" is an
285 **upstream dependency** of "B".
286
287 .. _downstream:
288 .. _downstream dependency:
289 .. _downstream dependencies:
290
291 * If unit "B" requires (or can use, or comes before) unit "A", then "B" is a
292 **downstream dependency** or a **forward dependency** of "A".
293
294 The following subsections define the major structural units of a TriBITS
295 project in more detail. Each structural unit is described along with the
296 files and directories associated with each. In addition, a key set of TriBITS
297 CMake variables for each are defined as well.
298
299 In the next major section following this one, some `Example TriBITS Projects`_
300 are described. For those who just want to jump in and learn best by example,
301 these example projects are a good way to start. These example projects will
302 be referenced in the more detailed descriptions given in this document.
303
304 The last issue to touch on before getting into the detailed descriptions of
305 the different TriBITS structural units is the issue of how CMake variables are
306 defined and used by TriBITS. The CMake variables described in the TriBITS
307 structural units below fall into one of two major types:
308
309 * *Local Fixed-Name Variables* are used temporarily in the processing of a
310 TriBITS unit. These include variables such as ``PROJECT_NAME``,
311 ``REPOSITORY_NAME``, ``PACKAGE_NAME``, and ``PARENT_PACKAGE_NAME``. These
312 are distinguished by having a non-namespaced fixed/constant name. They are
313 typically part of TriBITS reflection system, allowing subordinate units to
314 determine the encapsulating unit in which they are participating. For,
315 example, a TriBITS subpackage can determine its name, its parent package's
316 name and directories, its parent repository name and directories, and the
317 enclosing project's name and directories without having to refer to any
318 specific names.
319
320 * *Globally Scoped Namespaced Variables* are used to refer to properties of a
321 named TriBITS unit that are seen globally by the entire TriBITS project.
322 These include variables such as ``${REPOSITORY_NAME}_SOURCE_DIR``
323 (e.g. ``TribitsExProj_SOURCE_DIR``) and ``${PACKAGE_NAME}_BINARY_DIR``
324 (e.g. ``SimpleCXX_BINARY_DIR``). They are available after processing the
325 unit, for use by `downstream`_ or subordinate units. They are part of the
326 TriBITS dependency system, allowing downstream units to access properties of
327 their known `upstream dependencies`_.
328
329 More information about these various files is described in section `Processing
330 of TriBITS Files: Ordering and Details`_.
331
332
333 TriBITS Project
334 +++++++++++++++
335
336 A TriBITS Project:
337
338 * Defines the ``PROJECT_NAME`` CMake variable (defined in
339 `<projectDir>/ProjectName.cmake`_)
340 * Defines a complete CMake project which calls ``project(${PROJECT_NAME}
341 ...)`` and can be directly configured, built, tested, installed, etc.
342 * Consists of one or more `TriBITS Repositories`_ (and may itself be a
343 `TriBITS Repository`_) which can include native and extra repositories.
344 * Allows for extra Repositories to be added on before or after the set of
345 native Repositories (specified in
346 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ or by CMake cache
347 variables)
348 * Defines a default CDash server and default project name on the server (the
349 project name on the CDash server must be the same as ``${PROJECT_NAME}``).
350 * Defines pre-push testing standard builds (using `checkin-test.py`_)
351
352 For more details on the definition of a TriBITS Project, see:
353
354 * `TriBITS Project Core Files`_
355 * `TriBITS Project Core Variables`_
356
357
358 TriBITS Project Core Files
359 ..........................
360
361 The core files making up a TriBITS Project (where ``<projectDir> =
362 ${PROJECT_SOURCE_DIR}``) are::
363
364 <projectDir>/
365 ProjectName.cmake # Defines PACKAGE_NAME
366 CMakeLists.txt # Base project CMakeLists.txt file
367 CTestConfig.cmake # [Optional] Needed for CDash submits
368 Version.cmake # [Optional] Dev mode, Project version, VC branch
369 project-checkin-test-config.py # [Optional] checkin-test.py config
370 cmake/
371 NativeRepositoriesList.cmake # [Optional] Rarely used
372 ExtraRepositoriesList.cmake # [Optional] Lists repos and VC URLs
373 ProjectCiFileChangeLogic.py # [Optional] CI global change/test logic
374 ProjectCompilerPostConfig.cmake # [Optional] Override/tweak build flags
375 ProjectDependenciesSetup.cmake # [Optional] Project deps overrides
376 CallbackDefineProjectPackaging.cmake # [Optional] CPack settings
377 tribits/ # [Optional] Or provide ${PROJECT_NAME}_TRIBITS_DIR
378 ctest/
379 CTestCustom.cmake.in # [Optional] Custom ctest settings
380
381 These TriBITS Project files are documented in more detail below:
382
383 * `<projectDir>/ProjectName.cmake`_
384 * `<projectDir>/CMakeLists.txt`_
385 * `<projectDir>/CTestConfig.cmake`_
386 * `<projectDir>/Version.cmake`_
387 * `<projectDir>/project-checkin-test-config.py`_
388 * `<projectDir>/cmake/NativeRepositoriesList.cmake`_
389 * `<projectDir>/cmake/ExtraRepositoriesList.cmake`_
390 * `<projectDir>/cmake/ProjectCiFileChangeLogic.py`_
391 * `<projectDir>/cmake/ProjectCompilerPostConfig.cmake`_
392 * `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_
393 * `<projectDir>/cmake/CallbackDefineProjectPackaging.cmake`_
394 * `<projectDir>/cmake/tribits/`_
395 * `<projectDir>/cmake/ctest/CTestCustom.cmake.in`_
396
397
398 .. _<projectDir>/ProjectName.cmake:
399
400 **<projectDir>/ProjectName.cmake**: [Required] At a minimum provides a
401 ``set()`` statement to set the local variable ``PROJECT_NAME``. This file is
402 the first file that is read by a number of tools in order to get the TriBITS
403 project's name. This file is read first in every context that involves
404 processing the TriBITS project's files, including processes and tools that
405 just need to build the package dependency tree (see `Reduced Package
406 Dependency Processing`_). Being this is the first file read in for a TriBITS
407 project and that it is read in first at the top level scope in every context,
408 this is a good file to put in other universal static project options. Note
409 that this is a project, not a repository file so no general
410 repository-specific settings should go in this file. A simple example of this
411 file is `TribitsExampleProject`_/``ProjectName.cmake``:
412
413 .. include:: ../../examples/TribitsExampleProject/ProjectName.cmake
414 :literal:
415
416 A meta-project's ``ProjectName.cmake`` file might have a number of other
417 variables set that define how the various different TriBITS repos are cobbled
418 together into a single TriBITS meta-project (usually because the different
419 TriBITS repos and packages are a bit messy and have other issues). For
420 example, the CASL VERA TriBITS meta-project at one point had a very extensive
421 collection of set statements in this file.
422
423
424 .. _<projectDir>/CMakeLists.txt:
425
426 **<projectDir>/CMakeLists.txt**: [Required] The top-level CMake project file.
427 This is the first file that the ``cmake`` executable processes that starts
428 everything off and is the base-level scope for local (non-cache) CMake
429 variables. Due to a few CMake limitations and quirks, a project's top-level
430 ``CMakeLists.txt`` file is not quit as clean as one might otherwise hope would
431 be but it is not too bad. A simple, but representative, example is
432 `TribitsExampleProject`_/``CMakeLists.txt``:
433
434 .. include:: ../../examples/TribitsExampleProject/CMakeLists.txt
435 :literal:
436
437 A couple of CMake and TriBITS quirks that the above example
438 ``CMakeLists.txt`` addresses are worth some discussion. First, to avoid
439 duplication, the project's ``ProjectName.cmake`` file is read in with an
440 ``include()`` that defines the local variable ``PROJECT_NAME``. Right after
441 this initial include, the built-in CMake command ``project(${PROJECT_NAME}
442 NONE)`` is run. This command must be explicitly called with ``NONE`` so as to
443 avoid default CMake behavior for defining compilers. The definition of
444 compilers comes later as part of the TriBITS system inside of the
445 `tribits_project()`_ command (see `Full Processing of TriBITS Project
446 Files`_).
447
448 As noted in the above example file, the only project defaults that should be
449 set in this top-level ``CMakeLists.txt`` file are those that do not impact the
450 list of package enables/disables. The latter type of defaults should set in
451 other files (see below).
452
453 In this example project, a CMake cache variable `${PROJECT_NAME}_TRIBITS_DIR`_
454 must be set by the user to define where the base ``tribits`` source directory
455 is located. With this variable set (i.e. passed into ``cmake`` command-line
456 use ``-DTribitsExProj_TRIBITS_DIR=<someDir>``), one just includes a single
457 file to pull in the TriBITS system::
458
459 include("${${PROJECT_NAME}_TRIBITS_DIR}/TriBITS.cmake")
460
461 With the ``TriBITS.cmake`` file included, the configuration of the project
462 using TriBITS occurs with a single call to `tribits_project()`_.
463
464 Some projects, like Trilinos, actually snapshot the `TriBITS/tribits/`_
465 directory into their source tree `<projectDir>/cmake/tribits/`_ and therefore
466 don't need to have this variable set. In Trilinos, the include line is just::
467
468 include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/tribits/TriBITS.cmake)
469
470 The minimum CMake version must also be declared in the top-level
471 ``CMakeLists.txt`` file as shown. Explicitly setting the minimum CMake
472 version avoids strange errors that can occur when someone tries to build the
473 project using a version of CMake that is too old. The project should set the
474 minimum CMake version based on the CMake features used in that project's own
475 CMake files. The minimum CMake version required by TriBITS is defined in
476 the variable ``TRIBITS_CMAKE_MINIMUM_REQUIRED`` (the current minimum version
477 of CMake required by TriBITS is given at in `Getting set up to use CMake`_) .
478 For example, the ``VERA/CMakeLists.txt`` file lists as its first line::
479
480 set(VERA_TRIBITS_CMAKE_MINIMUM_REQUIRED 3.23.0)
481 cmake_minimum_required(VERSION ${VERA_TRIBITS_CMAKE_MINIMUM_REQUIRED}
482 FATAL_ERROR)
483
484
485 .. _<projectDir>/CTestConfig.cmake:
486
487 **<projectDir>/CTestConfig.cmake**: [Optional] Specifies the CDash site and
488 project to submit results to when doing an automated build driven by the CTest
489 driver function `tribits_ctest_driver()`_ (see `TriBITS CTest/CDash Driver`_).
490 This file is also required to use the TriBITS-generated ``dashboard`` target
491 (see `Dashboard Submissions`_). An example of this file is
492 `TribitsExampleProject`_/``CTestConfig.cmake``:
493
494 .. include:: ../../examples/TribitsExampleProject/CTestConfig.cmake
495 :literal:
496
497 Most of the variables set in this file are directly understood by raw
498 ``ctest`` and those variables not be explained here further (see documentation
499 for the standard CMake module ``CTest``). The usage of the function
500 `set_default_and_from_env()`_ allows the variables to be overridden both as
501 CMake cache variables and in the environment. The latter is needed when
502 running using ``ctest`` as the driver (since older versions of ``ctest`` did
503 not support ``-D<var-name>:<type>=<value>`` command-line arguments like for
504 ``cmake``). Given that all of these variables are nicely namespaced,
505 overriding them in the shell environment is not as dangerous as might
506 otherwise be the case but this is what had to be done to get around
507 limitations for older versions of CMake/CTest.
508
509 NOTE: One can also set::
510
511 set_default_and_from_env(TRIBITS_2ND_CTEST_DROP_SITE ...)
512 set_default_and_from_env(TRIBITS_2ND_CTEST_DROP_LOCATION ...)
513
514 in this file in order to submit to a second CDash site/location. For details,
515 see `Dashboard Submissions`_. This is useful when considering a CDash upgrade
516 and/or implementing new CDash features or tweaks.
517
518
519 .. _<projectDir>/Version.cmake:
520
521 **<projectDir>/Version.cmake**: If defined, gives the project's version and
522 determines development/release mode (see `Project and Repository Versioning
523 and Release Mode`_). This file is read in (using ``include()``) in the
524 project's base-level ``<projectDir>/CMakeLists.txt`` file scope so local
525 variables set in this file are seen by the entire CMake project. For example,
526 `TribitsExampleProject`_/``Version.cmake``, looks like:
527
528 .. include:: ../../examples/TribitsExampleProject/Version.cmake
529 :literal:
530
531 When this file exists in the base project, these will be used to create
532 standard SOVERSION symlinks to shared libs. For example, on Linux, in
533 addition to the real shared lib ``lib<libname>.so``, the standard SOVERSION
534 symlinks are created like::
535
536 lib<libname>.so.01
537 lib<libname>.so.1.1
538
539 When this file exists at the repository level, the prefix
540 ``${REPOSITORY_NAME}_`` is used instead of hard-coding the project name. This
541 is so that the same ``Version.txt`` file can be used as the
542 `<repoDir>/Version.cmake`_ file and have the repository name be flexible.
543 TriBITS sets ``REPOSITORY_NAME = ${PROJECT_NAME}`` when it reads in this file
544 at the project-level scope.
545
546 It is strongly recommended that every TriBITS project contain a
547 ``Version.cmake`` file, even if a release has never occurred. Otherwise, the
548 project needs to define the variable
549 `${PROJECT_NAME}_ENABLE_DEVELOPMENT_MODE_DEFAULT`_ at the global project scope
550 (perhaps in ``<projectDir>/ProjectName.cmake``) to get right development mode
551 behavior.
552
553
554 .. _<projectDir>/project-checkin-test-config.py:
555
556 **<projectDir>/project-checkin-test-config.py**: [Optional] Used to define the
557 ``--default-builds`` and other project-level configuration options for the
558 project's usage of the `checkin-test.py`_ tool. Machine or package-specific
559 options should **not** be placed in this file. An example of this file for
560 `TribitsExampleProject`_/``project-checkin-test-config.py`` is shown below:
561
562 .. include:: ../../examples/TribitsExampleProject/project-checkin-test-config.py
563 :literal:
564
565 The contents of the file ``project-checkin-test-config.py`` show above are
566 pretty self explanatory. This file defines a single python dictionary
567 data-structure called ``configuration`` which gives some default arguments in
568 ``defaults``, and then cmake options that define the projects
569 ``--default-builds``. For more details, see the section `Pre-push Testing
570 using checkin-test.py`_.
571
572
573 .. _<projectDir>/cmake/NativeRepositoriesList.cmake:
574
575 **<projectDir>/cmake/NativeRepositoriesList.cmake**: [Deprecated] If present,
576 this file gives the list of native repositories for the TriBITS project. The
577 file must contain a ``set()`` statement defining the variable
578 ``${PROJECT_NAME}_NATIVE_REPOSITORIES`` which is just a flat list of repository
579 names that must also be directory names under ``<projectDir>/``. For example,
580 if this file contains::
581
582 set(${PROJECT_NAME}_NATIVE_REPOSITORIES Repo0 Repo1)
583
584 then the directories ``<projectDir>/Repo0/`` and ``<projectDir>/Repo1/`` must
585 exist and must be valid TriBITS repositories (see `TriBITS Repository`_).
586
587 There are no examples for the usage of this file in any of the TriBITS
588 examples or test projects. However, support for this file is maintained for
589 backward compatibility since there may be some TriBITS projects that still use
590 it. It is recommended instead to define multiple repositories using the
591 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file as it allows for more
592 flexibility in how extra repositories are specified and how they are accessed.
593 The latter file allows the various tools to perform version control (VC)
594 activities with these repos while "native repositories" do not.
595
596 If this file ``NativeRepositoriesList.cmake`` does not exist, then TriBITS
597 sets ``${PROJECT_NAME}_NATIVE_REPOSITORIES`` equal to ".", or the base project
598 directory (i.e. ``<projectDir>/.``). In this case, the file
599 ``<projectDir>/PackagesList.cmake`` and ``<projectDir>/TPLsList.cmake`` must
600 exist. However, if the project has no native packages or external
601 packages/TPLs, then these files can be set up with empty lists. This is the
602 case for meta-projects like CASL VERA that have only extra repositories
603 specified in the file `<projectDir>/cmake/ExtraRepositoriesList.cmake`_.
604
605 .. _<projectDir>/cmake/ExtraRepositoriesList.cmake:
606
607 **<projectDir>/cmake/ExtraRepositoriesList.cmake**: [Optional] If present,
608 this file defines a list of extra repositories that are added on to the
609 project's native repositories. The list of repositories is defined using the
610 macro `tribits_project_define_extra_repositories()`_. For example, the extra
611 repos file:
612
613 .. include:: ExtraReposList.cmake
614 :literal:
615
616 shows the specification of both TriBITS Repositories and non-TriBITS VC
617 Repositories. In the above file, the repositories ``ExtraRepo1``,
618 ``ExtraRepo3``, and ``ExtraRepo4`` are both TriBITS and VC repositories that
619 are cloned into directories under ``<projectDir>`` of the same names from the
620 URLs ``someurl.com:/ExtraRepo1``, ``someurl3.com:/ExtraRepo3``, and
621 ``someurl4.com:/ExtraRepo4``, respectively. However, the repository
622 ``ExtraRepo2`` is **not** a `TriBITS Repository`_ because it is marked as
623 ``NOPACKAGES``. In this case, it gets cloned as the directory::
624
625 <projectDir>/packages/SomePackage/Blah
626
627 However, the code in the tools `checkin-test.py`_ and
628 `tribits_ctest_driver()`_ will consider non-TriBITS VC repos like
629 ``ExtraRepo2`` and any changes to this repository will be listed as changes to
630 ``somePackage`` (see `Pre-push Testing using checkin-test.py`_).
631
632 NOTE: This file can be overridden by setting the cache variable
633 `<Project>_EXTRAREPOS_FILE`_.
634
635 .. ToDO: I am not really sure what repos get cloned the first time based on
636 .. the list of extra repos. From looking at TribitsCTestDriverCore.cmake, it
637 .. looks like only the selected repos will be cloned. I need to add some unit
638 .. tests that really show what the real behavior is and then document that
639 .. behavior here.
640
641
642 .. _<projectDir>/cmake/ProjectCiFileChangeLogic.py:
643
644 **<projectDir>/cmake/ProjectCiFileChangeLogic.py**: [Optional] If present,
645 then this Python module is imported and the Python class defined there
646 ProjectCiFileChangeLogic there is used to determine which files need to
647 trigger a global rebuild of the project enabling all packages.
648
649 An example of this given in the file
650 ``TribitsExampleProject/cmake/ProjectCiFileChangeLogic.py``:
651
652 .. include:: ../../examples/TribitsExampleProject/cmake/ProjectCiFileChangeLogic.py
653 :literal:
654
655 This logic is used in all code that is used in CI testing including
656 `checkin-test.py`_, `tribits_ctest_driver()`_ and
657 `get-tribits-packages-from-files-list.py`_. If this file does not exist, then
658 TriBITS has some default logic which may or may not be sufficient for the
659 needs of a given project.
660
661
662 .. _<projectDir>/cmake/ProjectCompilerPostConfig.cmake:
663
664 **<projectDir>/cmake/ProjectCompilerPostConfig.cmake**: [Optional] If present,
665 then this file is read using ``include()`` at the top-level CMakeLists.txt
666 file scope right after the compilers for the languages ``<LANG>`` = ``C``,
667 ``CXX``, and ``Fortran`` are determined and checked using
668 ``enable_language(<LANG>)`` but before any other checks are performed. This
669 file can contain logic for the project to adjust the flags set in
670 ``CMAKE_<LANG>_FLAGS`` and changes to other aspects of the build flags
671 (including link flags, etc.).
672
673 One example of the usage of this file is the Trilinos project where this file
674 is (or was) used to apply specialized logic implemented in the Kokkos build
675 system to select compiler options and to determine how C++11 and OpenMP flags
676 are set. This file in Trilinos looked like::
677
678 if (${Trilinos_ENABLE_Kokkos})
679
680 ...
681
682 include(${Kokkos_GEN_DIR}/kokkos_generated_settings.cmake)
683
684 if (NOT KOKKOS_ARCH STREQUAL "None")
685
686 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${KOKKOS_CXX_FLAGS}")
687
688 message("-- " "Skip adding flags for OpenMP because Kokkos flags does that ...")
689 set(OpenMP_CXX_FLAGS_OVERRIDE " ")
690
691 endif()
692
693 endif()
694
695 The exact context where this file is processed (if it exists) is described in
696 `Full Processing of TriBITS Project Files`_ and `TriBITS Environment Probing
697 and Setup`_.
698
699
700 .. _<projectDir>/cmake/ProjectDependenciesSetup.cmake:
701
702 **<projectDir>/cmake/ProjectDependenciesSetup.cmake**: [Optional] If present,
703 this file is included a single time as part of the generation of the project's
704 dependency data-structure (see `Reduced Package Dependency Processing`_). It
705 gets included at the top project level scope after all of the
706 `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_ files have been included
707 but before all of the package `<packageDir>/cmake/Dependencies.cmake`_ files
708 are included. Any local variables set in this file have project-wide scope.
709 The primary purpose for this file is to set variables that will impact the
710 processing of project's package ``Dependencies.cmake`` files.
711
712 The typical usage of this file is to set the default CDash email address for
713 all packages or override the email addresses for all of a repository's package
714 CDash regression email addresses (see `CDash regression email addresses`_).
715 For example, to set the default email address for all of the packages, one
716 would set in this file::
717
718 set_default(${PROJECT_NAME}_PROJECT_MASTER_EMAIL_ADDRESS
719 [email protected])
720
721 The repository email address variables
722 `${REPOSITORY_NAME}_REPOSITORY_EMAIL_URL_ADDRESS_BASE`_ and
723 `${REPOSITORY_NAME}_REPOSITORY_MASTER_EMAIL_ADDRESS`_ possibly set in the just
724 processed `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_ files can also
725 be overridden in this file. The CASL VERA meta-project uses this file to
726 override several of the repository-specific email addresses for its
727 constituent repositories.
728
729 In general, variables that affect how package dependencies are defined or
730 affect package enable/disable logic for *only this particular project* should
731 be defined in this file.
732
733 .. _<projectDir>/cmake/CallbackDefineProjectPackaging.cmake:
734
735 **<projectDir>/cmake/CallbackDefineProjectPackaging.cmake**: [Optional] If
736 exists, defines the CPack settings for the project (see `Official CPack
737 Documentation <http://www.cmake.org/cmake/help/documentation.html>`_ and
738 `Online CPack Wiki <http://www.cmake.org/Wiki/CMake:Packaging_With_CPack>`_).
739 This file must define a macro called ``tribits_project_define_packaging()``
740 which is then invoked by TriBITS. The file:
741
742 `TribitsExampleProject`_/``cmake/CallbackDefineProjectPackaging.cmake``
743
744 provides a good example which is:
745
746 .. include:: ../../examples/TribitsExampleProject/cmake/CallbackDefineProjectPackaging.cmake
747 :literal:
748
749 The CPack variables show above that should be defined at the project-level are
750 described in the `Official CPack Documentation`_.
751
752 Settings that are general for all distributions (like non-package repository
753 files to exclude from the tarball) should be set at the in the file
754 `<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake`_. See `Creating
755 Source Distributions` for more details.
756
757 .. _<projectDir>/cmake/tribits/:
758
759 **<projectDir>/cmake/tribits/**: [Optional] This is the typical location of
760 the `TriBITS/tribits/`_ source tree for projects that choose to snapshot
761 TriBITS into their source tree. In fact, TriBITS assumes this is the default
762 location for the TriBITS source tree if ``${PROJECT_NAME}_TRIBITS_DIR`` is not
763 otherwise specified. Trilinos, for example, currently snapshots the TriBITS
764 source tree into this directory. See `TriBITS directory snapshotting`_ for
765 more details.
766
767 .. _<projectDir>/cmake/ctest/CTestCustom.cmake.in:
768
769 **<projectDir>/cmake/ctest/CTestCustom.cmake.in**: [Optional] If this file
770 exists, it is processed using a ``configure_file()`` command to write the file
771 ``CTestCustom.cmake`` in the project base build directory
772 ```${PROJECT_BINARY_DIR}/``. This file is picked up automatically by
773 ``ctest`` (see `CTest documentation`_). This file is typically used to change
774 the maximum size of test output. For example, the
775 `TribitsExampleProject`_/``cmake/ctest/CTestCustom.cmake.in`` looks like:
776
777 .. include:: ../../examples/TribitsExampleProject/cmake/ctest/CTestCustom.cmake.in
778 :literal:
779
780 which sets the output size for each test submitted to CDash be unlimited
781 (which is not really recommended). These variables used by Trilinos at one
782 time were::
783
784 set(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE 50000)
785 set(CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE 5000000)
786
787 which sets the max output for passed and failed tests to 50000k and 5000000k,
788 respectively.
789
790 For documentation of the options one can change for CTest, see online `CTest
791 documentation`_.
792
793
794 TriBITS Project Core Variables
795 ..............................
796
797 The following `local variables` are defined in the top-level Project
798 ``CMakeLists.txt`` file scope and are therefore accessible by all files
799 processed by TriBITS:
800
801 .. _PROJECT_NAME:
802
803 ``PROJECT_NAME``
804
805 The name of the TriBITS Project. This exists to support, among other
806 things, the ability for subordinate units (Repositories and Packages) to
807 determine the Project in which is participating. This is typically read
808 from a ``set()`` statement in the project's
809 `<projectDir>/ProjectName.cmake`_ file.
810
811 .. _PROJECT_SOURCE_DIR:
812
813 ``PROJECT_SOURCE_DIR``
814
815 The absolute path to the base Project source directory. This is set
816 automatically by TriBITS given the directory passed into ``cmake`` at
817 configure time at the beginning of the `tribits_project()`_ macro.
818
819 .. _PROJECT_BINARY_DIR:
820
821 ``PROJECT_BINARY_DIR``
822
823 The absolute path to the base Project binary/build directory. This is set
824 automatically by TriBITS and is the directory where ``cmake`` is run from
825 and is set at the beginning of the `tribits_project()`_ macro.
826
827 .. _${PROJECT_NAME}_SOURCE_DIR:
828
829 ``${PROJECT_NAME}_SOURCE_DIR``
830
831 Set to the same directory as ``${PROJECT_SOURCE_DIR}`` automatically by
832 the built-in ``project()`` command called in the top-level
833 `<projectDir>/CMakeLists.txt`_ file..
834
835 .. _${PROJECT_NAME}_BINARY_DIR:
836
837 ``${PROJECT_NAME}_BINARY_DIR``
838
839 Set to the same directory as ``${PROJECT_BINARY_DIR}`` automatically by
840 the built-in ``project()`` command called in the top-level
841 `<projectDir>/CMakeLists.txt`_ file..
842
843 The following `cache variables` are defined for every TriBITS project:
844
845 .. _${PROJECT_NAME}_TRIBITS_DIR:
846
847 ``${PROJECT_NAME}_TRIBITS_DIR``
848
849 CMake cache variable that gives the path to the TriBITS implementation
850 directory. When set to a relative path (set as type ``STRING``, see
851 below), this is taken relative to ``${CMAKE_CURRENT_SOURCE_DIR}/`` (the
852 project base source dir). When an absolute path is given, it is used
853 without modification. If this variable is not set in the
854 `<projectDir>/CMakeLists.txt`_ file, then it will be automatically set as
855 a ``PATH`` cache variable by the include of ``TriBITS.cmake`` by the
856 statement ::
857
858 set( ${PROJECT_NAME}_TRIBITS_DIR
859 "${CMAKE_CURRENT_SOURCE_DIR}/cmake/tribits" CACHE PATH "...")
860
861 Therefore, projects that snapshot TriBITS into
862 `<projectDir>/cmake/tribits/`_ don't need to explicitly set
863 ``${PROJECT_NAME}_TRIBITS_DIR``. In addition, one can also point to a
864 different TriBITS implementation just by setting the absolute path::
865
866 -D <Project>_TRIBITS_DIR=<some-abs-dir>
867
868 or to a relative path using, for example::
869
870 -D <Project>_TRIBITS_DIR:STRING=TriBITS/tribits
871
872 Note that when the ``TriBITS`` git repo itself is cloned by a TriBITS
873 project, then ``${PROJECT_NAME}_TRIBITS_DIR`` should be set to the
874 directory ``TriBITS/tribits`` (see `TriBITS/tribits/`_) as shown above.
875
876 .. _${PROJECT_NAME}_ENABLE_TESTS:
877
878 ``${PROJECT_NAME}_ENABLE_TESTS``
879
880 CMake cache variable that if set to ``ON``, then tests for all explicitly
881 enabled packages will be turned on. This has a default value of ``OFF``.
882 This is used in logic to enable individual package tests (see
883 `<Project>_ENABLE_TESTS only enables explicitly enabled package
884 tests`_).
885
886 ``${PACKAGE_NAME}_ENABLE_EXAMPLES``
887
888 CMake cache variable that if set to ``ON``, then examples for all
889 explicitly enabled packages will be turned on. This has a default value
890 of ``OFF``.
891
892 The following `internal project-scope local` (non-cache) CMake variables are
893 defined by TriBITS giving the project's TriBITS repositories.:
894
895 .. _${PROJECT_NAME}_NATIVE_REPOSITORIES:
896
897 ``${PROJECT_NAME}_NATIVE_REPOSITORIES``
898
899 The list of Native Repositories for a given TriBITS project
900 (i.e. Repositories that are always present when configuring the Project
901 and are managed in the same VC repo typically). This variable is set in
902 the file `<projectDir>/cmake/NativeRepositoriesList.cmake`_ if it exists.
903 If the file ``NativeRepositoriesList.cmake`` does not exist, then the
904 project is assumed to also be a repository and the list of native
905 repositories is just the local project directory
906 ``${PROJECT_SOURCE_DIR}/.``. In this case, the
907 ``${PROJECT_SOURCE_DIR}/`` must contain at a minimum a
908 ``PackagesList.cmake`` file, and a ``TPLsList.cmake`` file (see `TriBITS
909 Repository`_).
910
911 .. _${PROJECT_NAME}_EXTRA_REPOSITORIES:
912
913 ``${PROJECT_NAME}_EXTRA_REPOSITORIES``
914
915 The list of Extra Repositories that the project is being configured with.
916 This list of repositories either comes from processing the project's
917 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file or comes from the
918 CMake cache variable ``${PROJECT_NAME}_EXTRA_REPOSITORIES``. See
919 `Enabling extra repositories with add-on packages`_ for details.
920
921 .. _${PROJECT_NAME}_ALL_REPOSITORIES:
922
923 ``${PROJECT_NAME}_ALL_REPOSITORIES``
924
925 Concatenation of all the repos listed in
926 `${PROJECT_NAME}_NATIVE_REPOSITORIES`_ and
927 `${PROJECT_NAME}_EXTRA_REPOSITORIES`_ in the order they are processed.
928
929
930 TriBITS Repository
931 ++++++++++++++++++
932
933 A TriBITS Repository is the basic unit of ready-made composition between
934 different collections of software that use the TriBITS CMake build and system.
935
936 In short, a TriBITS Repository:
937
938 * Is a named collection of related TriBITS Packages and external packages/TPLs
939 (defined in `<repoDir>/PackagesList.cmake`_ and `<repoDir>/TPLsList.cmake`_
940 respectively)
941 * Defines the base source and binary directories for the Repository
942 ``${REPOSITORY_NAME}_SOURCE_DIR`` and ``${REPOSITORY_NAME}_BINARY_DIR``.
943 * Defines a common set of initializations and other hooks for all the
944 packages in the repository.
945 * Typically maps to a VC (i.e. git) repository and therefore represents a unit
946 of integration, versioning and reuse. (But core TriBITS has no dependency
947 on any VC tool.)
948
949 For more details on the definition of a TriBITS Repository, see:
950
951 * `TriBITS Repository Core Files`_
952 * `TriBITS Repository Core Variables`_
953
954 TriBITS Repository Core Files
955 .............................
956
957 The core files making up a TriBITS Repository (where ``<repoDir> =
958 ${${REPOSITORY_NAME}_SOURCE_DIR}``) are::
959
960 <repoDir>/
961 PackagesList.cmake
962 TPLsList.cmake
963 Copyright.txt # [Optional] Only needed if creating version header file
964 Version.cmake # [Optional] Info inserted into ${REPO_NAME}_version.h
965 cmake/
966 RepositoryDependenciesSetup.cmake # [Optional] CDash email addresses?
967 CallbackSetupExtraOptions.cmake # [Optional] Called after main options
968 CallbackDefineRepositoryPackaging.cmake # [Optional] CPack packaging
969
970 These TriBITS Repository files are documented in more detail below:
971
972 * `<repoDir>/PackagesList.cmake`_
973 * `<repoDir>/TPLsList.cmake`_
974 * `<repoDir>/Copyright.txt`_
975 * `<repoDir>/Version.cmake`_
976 * `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_
977 * `<repoDir>/cmake/CallbackSetupExtraOptions.cmake`_
978 * `<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake`_
979
980
981 .. _<repoDir>/PackagesList.cmake:
982
983 **<repoDir>/PackagesList.cmake**: [Required] Provides the list of top-level
984 packages defined by the repository. This file typically just calls the macro
985 `tribits_repository_define_packages()`_ to define the list of packages along
986 with their directories and other properties. For example, the file
987 `TribitsExampleProject`_/``PackagesList.cmake`` looks like:
988
989 .. include:: ../../examples/TribitsExampleProject/PackagesList.cmake
990 :literal:
991
992 Other commands that are appropriate to use in this file include
993 `tribits_disable_package_on_platforms()`_ and
994 `tribits_allow_missing_external_packages()`_. Also, if the binary directory
995 for any package ``<packageName>`` needs to be changed from the default, then
996 the variable ``<packageName>_SPECIFIED_BINARY_DIR`` can be set. (see `TriBITS
997 Package == TriBITS Repository == TriBITS Project`_).
998
999 It is perfectly legal for a TriBITS repository to define no packages at all
1000 with::
1001
1002 tribits_repository_define_packages()
1003
1004 and this would be the case for a TriBITS meta-project that has no native
1005 packages, only extra repositories.
1006
1007 .. ToDo: Point to example meta-project.
1008
1009
1010 .. _<repoDir>/TPLsList.cmake:
1011
1012 **<repoDir>/TPLsList.cmake**: [Required] Provides the list of external
1013 packages/TPLs that are referenced as dependencies in the repository's
1014 package's `<packageDir>/cmake/Dependencies.cmake`_ files (see `TriBITS
1015 External Package/TPL`_). This file typically just calls the macro
1016 `tribits_repository_define_tpls()`_ to define the TPLs along with their find
1017 modules and other properties. An example is
1018 `ReducedMockTrilinos`_/``TPLsList.cmake`` which shows:
1019
1020 .. include:: ../../examples/ReducedMockTrilinos/TPLsList.cmake
1021 :literal:
1022
1023 See `TriBITS External Package/TPL`_ for details on what gets defined for each
1024 TriBITS TPL once this file is processed.
1025
1026 It is perfectly fine to specify no TPLs at all for a repository with::
1027
1028 tribits_repository_define_tpls()
1029
1030 but the macro ``tribits_repository_define_tpls()`` has to be called, even if
1031 there are no TPLs. See `tribits_repository_define_tpls()`_ for further
1032 details and constraints.
1033
1034
1035 .. _<repoDir>/Copyright.txt:
1036
1037 **<repoDir>/Copyright.txt**: [Optional] Gives the default copyright and
1038 license declaration for all of the software in the TriBITS repository
1039 directory ``<repoDir>/``. This file is read into a string and then used to
1040 configure the repository's version header file (see `Project and Repository
1041 Versioning and Release Mode`_). Even if a repository version header file is
1042 not produced, it is a good idea for every TriBITS repository to define this
1043 file, just for legal purposes. For a good open-source license, one should
1044 consider copying the ``TriBITS/Copyright.txt`` file which is a simple 3-clause
1045 BSD-like license like:
1046
1047 .. include:: ../../Copyright.txt
1048 :literal:
1049
1050 .. _<repoDir>/Version.cmake:
1051
1052 **<repoDir>/Version.cmake**: [Optional] Contains version information for the
1053 repository (and the project also if this is also the base project). For
1054 example, `TribitsExampleProject`_/``Version.cmake``, this looks like:
1055
1056 .. include:: ../../examples/TribitsExampleProject/Version.cmake
1057 :literal:
1058
1059 Note that the prefix ``${REPOSITORY_NAME}_`` is used instead of hard-coding
1060 the repository's name to allow flexibility in what a meta-project names a
1061 given TriBITS repository.
1062
1063 The local variables in these set statements are processed in the base project
1064 directory's local scope and are therefore seen by the entire CMake project.
1065 When this file is read in repository mode, the variable
1066 ``${REPOSITORY_NAME}_ENABLE_DEVELOPMENT_MODE_DEFAULT`` is ignored.
1067
1068 .. _<repoDir>/cmake/RepositoryDependenciesSetup.cmake:
1069
1070 **<repoDir>/cmake/RepositoryDependenciesSetup.cmake**: [Optional] If present,
1071 this file is included a single time as part of the generation of the project
1072 dependency data-structure (see `Reduced Package Dependency Processing`_). It
1073 gets included in the order listed in `${PROJECT_NAME}_ALL_REPOSITORIES`_. Any
1074 local variables set in this file have project-wide scope. The primary purpose
1075 for this file is to set variables that will impact the processing of project's
1076 package's ``Dependencies.cmake`` files and take care of other enable/disable
1077 issues that are not otherwise cleanly handled by the TriBITS system
1078 automatically.
1079
1080 The typical usage of this file is to set the default CDash email address for
1081 all of the defined packages (see `CDash regression email addresses`_). For
1082 example, to set the default email address for all of the packages in this
1083 repository, one would set in this file::
1084
1085 set_default(${REPOSITORY_NAME}_REPOSITORY_MASTER_EMAIL_ADDRESS
1086 [email protected])
1087
1088 Note that the prefix ``${REPOSITORY_NAME}_`` is used instead of hard-coding
1089 the repo name to allow greater flexibility in how meta-projects refer to this
1090 TriBITS repo.
1091
1092 .. _<repoDir>/cmake/CallbackSetupExtraOptions.cmake:
1093
1094 **<repoDir>/cmake/CallbackSetupExtraOptions.cmake**: [Optional] If defined,
1095 this file is processed (included) for each repo in order right after the basic
1096 TriBITS options are defined in the macro
1097 ``tribits_define_global_options_and_define_extra_repos()``. This file must
1098 define the macro ``tribits_repository_setup_extra_options()`` which is then
1099 called by the TriBITS system. This file is only processed when doing a basic
1100 configuration of the project and **not** when it is just building up the
1101 dependency data-structures (i.e. it is **not** processed in the `Reduced
1102 Package Dependency Processing`_). Any local variables set have project-wide
1103 scope.
1104
1105 A few additional variables are defined by the time this file is processed and
1106 can be used in the logic in these files. Some of the variables that should
1107 already be defined (in addition to all of the basic user TriBITS cache
1108 variables set in ``tribits_define_global_options_and_define_extra_repos()``)
1109 include ``CMAKE_HOST_SYSTEM_NAME``, ``${PROJECT_NAME}_HOSTNAME``, and
1110 ``Python3_EXECUTABLE`` (see `Python Support`_). The types of commands and
1111 logic to put in this file include:
1112
1113 * Setting additional user cache variable options that are used by multiple
1114 packages in the TriBITS Repository. For example, Trilinos defines a
1115 ``Trilinos_DATA_DIR`` user cache variable that several Trilinos packages use
1116 to get extra test data to define extra tests.
1117 * Disabling packages in the TriBITS Repository when conditions will not allow
1118 them to be enabled. For example, Trilinos disables the package
1119 ``ForTrilinos`` when ``Fortran`` is disabled and disables the package
1120 ``PyTrilinos`` when Python support is disabled.
1121
1122 An example of this file is:
1123
1124 `TribitsExampleProject`_/``/cmake/CallbackSetupExtraOptions.cmake``
1125
1126 which currently looks like:
1127
1128 .. include:: ../../examples/TribitsExampleProject/cmake/CallbackSetupExtraOptions.cmake
1129 :literal:
1130
1131 .. _<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake:
1132
1133 **<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake**: [Optional] If
1134 this file exists, then it defines extra CPack-related options that are
1135 specific to this TriBITS Repository. This file must define the macro
1136 ``tribits_repository_define_packaging()`` which is called by TriBITS. This
1137 file is processed as the top project-level scope so any local variables set
1138 have project-wide effect. This file is processed after the project's
1139 `<projectDir>/cmake/CallbackDefineProjectPackaging.cmake`_ file so any project
1140 CPACK variables are defined for the repository-level options and commands are
1141 created. This file typically just sets extra excludes to remove files from
1142 the tarball. The file:
1143
1144 `TribitsExampleProject`_/``cmake/CallbackDefineRepositoryPackaging.cmake``
1145
1146 provides a good example which is:
1147
1148 .. include:: ../../examples/TribitsExampleProject/cmake/CallbackDefineRepositoryPackaging.cmake
1149 :literal:
1150
1151 As shown in the above example, it is important to prefix the excluded files
1152 and directories with the repository base directory
1153 ``${${REPOSITORY_NAME}_SOURCE_DIR}/`` since these are interpreted by CPack as
1154 regular-expressions.
1155
1156
1157 TriBITS Repository Core Variables
1158 .................................
1159
1160 The following temporary local variables are defined automatically by TriBITS
1161 before processing a given TriBITS repository's files
1162 (e.g. ``PackagesList.cmake``, ``TPLsList.cmake``, etc.):
1163
1164 .. _REPOSITORY_NAME:
1165
1166 ``REPOSITORY_NAME``
1167
1168 The name of the current TriBITS repository. This name will be the
1169 repository name listed in
1170 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file or if this
1171 repository directory is the project base directory, ``REPOSITORY_NAME``
1172 will be set to ``${PROJECT_NAME}``.
1173
1174 ``REPOSITORY_DIR``
1175
1176 Path of the current Repository *relative* to the Project's base source
1177 directory `${PROJECT_NAME}_SOURCE_DIR`_.. This is typically just the
1178 repository name but can be an arbitrary directory if specified through the
1179 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file.
1180
1181 The following project-scope (non-cache) local variables are set once the list
1182 of TriBITS repositories is processed and before any of the repository's files
1183 are processed:
1184
1185 ``${REPOSITORY_NAME}_SOURCE_DIR``
1186
1187 The absolute path to the base of a given TriBITS Repository's source
1188 directory. CMake code, for example in a packages' ``CMakeLists.txt``
1189 file, typically refers to this by the raw name like ``RepoX_SOURCE_DIR``.
1190 This makes such CMake code independent of where the various TriBITS repos
1191 are in relation to each other or the TriBITS Project (but does hard-code
1192 the repository name which is not ideal).
1193
1194 ``${REPOSITORY_NAME}_BINARY_DIR``
1195
1196 The absolute path to the base of a given TriBITS Repository's binary
1197 directory. CMake code, for example in packages, refer to this by the raw
1198 name like ``RepoX_SOURCE_DIR``. This makes such CMake code independent of
1199 where the various TriBITS repos are in relation to each other or the
1200 Project.
1201
1202 The following project-level local variables can be defined by the project or
1203 the user to help define the what packages from the repository
1204 ``${REPOSITORY_NAME}`` contribute to the primary meta-project packages (PMPP):
1205
1206 .. _${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES:
1207
1208 ``${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES``
1209
1210 If set to ``TRUE``, then the package's in the TriBITS repository are not
1211 considered to be part of the primary meta-project packages. This affects
1212 what packages get enabled by default when enabling all packages with
1213 setting ``${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON`` and what tests and
1214 examples get enabled by default when setting
1215 ``${PROJECT_NAME}_ENABLE_TESTS=ON``. See `TriBITS Dependency Handling
1216 Behaviors`_ for more details.
1217
1218 .. _${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES_EXCEPT:
1219
1220 ``${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES_EXCEPT``
1221
1222 When the above variable is set to ``TRUE``, this variable is read by
1223 TriBITS to find the list of TriBITS packages selected packages in the
1224 repository ``${REPOSITORY_NAME}`` which are considered to be part of the
1225 set of the project's primary meta-project package when the above variable
1226 is set to ``ON``. NOTE: It is not necessary to list all of the
1227 subpackages in a given parent package. Only the parent package need be
1228 listed and it will be equivalent to listing all of the subpackages. See
1229 `TriBITS Dependency Handling Behaviors`_ for more details.
1230
1231 The above primary meta-project variables should be set in the meta-project's
1232 `<projectDir>/ProjectName.cmake`_ file so that they will be set in all
1233 situations.
1234
1235
1236 TriBITS Package
1237 +++++++++++++++
1238
1239 A TriBITS Package:
1240
1241 * Is the fundamental TriBITS structural unit of software partitioning and
1242 aggregation.
1243
1244 * Must have a unique package name (``PACKAGE_NAME``) that is globally unique
1245 (see `Globally unique TriBITS package names`_).
1246
1247 * Typically defines a set of libraries and/or header files and/or executables
1248 and/or tests with CMake build targets for building these for which TriBITS
1249 exports the list of include directories, libraries, and targets that are
1250 created (along with CMake dependencies).
1251
1252 * Is declared in its parent repository's `<repoDir>/PackagesList.cmake`_ file.
1253
1254 * Declares dependencies on `upstream`_ packages by just naming the
1255 dependencies in the file `<packageDir>/cmake/Dependencies.cmake`_.
1256
1257 * Can optionally have subpackages listed in the argument
1258 `SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS`_ to the macro call
1259 `tribits_package_define_dependencies()`_.
1260
1261 * Is the unit of testing as driven by `tribits_ctest_driver()`_ and displayed
1262 on CDash.
1263
1264 .. _Globally unique TriBITS package names:
1265
1266 **WARNING:** As noted above, one must be very careful to pick **globally
1267 unique TriBITS package names**. This name must be unique not only within its
1268 defined TriBITS repository but also across all packages in all TriBITS
1269 repositories that ever might be cobbled together into a single TriBITS (meta)
1270 project! Choosing a good package name is the single most important decision
1271 when it comes to defining a TriBITS package. One must be careful **not** to
1272 pick names like "Debug" or "StandardUtils" that have a high chance of clashing
1273 with poorly named TriBITS packages from other TriBITS repositories.
1274
1275 For more details on the definition of a TriBITS Package (or subpackage), see:
1276
1277 * `TriBITS Package Core Files`_
1278 * `TriBITS Package Core Variables`_
1279
1280 TriBITS Package Core Files
1281 ..........................
1282
1283 The core files that make up a *TriBITS Package* (where ``<packageDir> =
1284 ${${PACKAGE_NAME}_SOURCE_DIR}``) are::
1285
1286 <packageDir>/
1287 CMakeLists.txt # Only processed if the package is enabled
1288 cmake/
1289 Dependencies.cmake # Always processed if its repo is processed
1290 <packageName>_config.h.in # [Optional], name is not fixed
1291
1292 There are a few simple rules for the location and the contents of the
1293 ``<packageDir>/`` directory:
1294
1295 * The directory ``<packageDir>/`` must not be a subdirectory of the package
1296 directory of any other package (e.g. not ``pkga/pkgb``).
1297 * All of the source files, test files, etc. for the package should be included
1298 under ``<packageDir>/``.
1299
1300 The above rules are not needed for basic building and testing but are needed
1301 for extended features like automatically detecting when a package has changed
1302 by looking at what files have changed (see `Pre-push Testing using
1303 checkin-test.py`_) and for creating source tarballs correctly (see `Creating
1304 Source Distributions`_). Therefore, it would be wise to abide by the above
1305 rules when defining packages.
1306
1307 The following TriBITS Package files are documented in more detail below:
1308
1309 * `<packageDir>/cmake/Dependencies.cmake`_
1310 * `<packageDir>/cmake/<packageName>_config.h.in`_
1311 * `<packageDir>/CMakeLists.txt`_
1312
1313 .. _<packageDir>/cmake/Dependencies.cmake:
1314
1315 **<packageDir>/cmake/Dependencies.cmake**: [Required] Defines the dependencies
1316 for a given TriBITS package using the macro
1317 `tribits_package_define_dependencies()`_. This file is processed at the
1318 top-level project scope (using an ``include()``) so any local variables set
1319 will be seen by the entire project. This file is always processed, including
1320 when just building the project's dependency data-structure (see `Reduced
1321 Package Dependency Processing`_).
1322
1323 An example of a ``Dependencies.cmake`` file for a package with optional and
1324 required dependencies is for the mock ``Panzer`` package in `MockTrilinos`_:
1325
1326 .. include:: ../../examples/MockTrilinos/packages/panzer/cmake/Dependencies.cmake
1327 :literal:
1328
1329 .. _with_subpackages/cmake/Dependencies.cmake:
1330
1331 An example of a package with subpackages is ``WithSubpackages`` which has the
1332 dependencies file:
1333
1334 `TribitsExampleProject`_/``packages/with_subpackages/cmake/Dependencies.cmake``
1335
1336 which is:
1337
1338 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/cmake/Dependencies.cmake
1339 :literal:
1340
1341 ``WithSubpackages`` defines three subpackages which creates three new packages
1342 with names ``WithSubpackagesA``, ``WithSubpackagesB``, and
1343 ``WithSubpackagesC``.
1344
1345 if a TriBITS Package or Subpackage has no dependencies, it still has to call
1346 ``tribits_package_define_dependencies()`` but it is called with no arguments
1347 such as with:
1348
1349 `TribitsHelloWorld`_/``hello_world/cmake/Dependencies.cmake:``
1350
1351 which contains:
1352
1353 .. include:: ../../examples/TribitsHelloWorld/hello_world/cmake/Dependencies.cmake
1354 :literal:
1355
1356 Other TriBITS macros/functions that can be called in this file include
1357 `tribits_tpl_tentatively_enable()`_ and
1358 `tribits_allow_missing_external_packages()`_.
1359
1360 .. _<packageName>_config.h.in:
1361 .. _<packageDir>/cmake/<packageName>_config.h.in:
1362
1363 **<packageDir>/cmake/<packageName>_config.h.in**: [Optional] The package's
1364 configured header file. This file will contain placeholders for variables
1365 that will be substitute at configure time with `tribits_configure_file()`_.
1366 This includes usage of ``#cmakedefine <varName>`` and other standard CMake
1367 file configuration features used by CMake's ``configure_file()`` command.
1368
1369 An example of this file is shown in:
1370
1371 `TribitsExampleProject`_/``packages/simple_cxx/cmake/SimpleCxx_config.h.in``
1372
1373 which is:
1374
1375 .. include:: ../../examples/TribitsExampleProject/packages/simple_cxx/cmake/SimpleCxx_config.h.in
1376 :literal:
1377
1378 The variable ``HAVE_SIMPLECXX___INT64`` is set up in the base file
1379 ``SimpleCxx/CMakeLists.txt`` (see `<packageDir>/CMakeLists.txt`_ below). For
1380 an explanation of ``HAVE_SIMPLECXX_DEBUG``, see `tribits_add_debug_option()`_.
1381 For an explanation of ``HAVE_SIMPLECXX_SIMPLETPL``, see `How to add a new
1382 TriBITS Package dependency`_. For an explanation of
1383 ``@SIMPLECXX_DEPRECATED_DECLARATIONS@``, see `Setting up support for
1384 deprecated code handling`_.
1385
1386 **NOTE:** The file name ``<packageName>_config.h.in`` is not at all fixed and
1387 the package can call this file anything it wants. Also, a package can
1388 configure multiple header files in different directories for different
1389 purposes using `tribits_configure_file()`_ or even calls to the raw CMake
1390 function `configure_file()`_.
1391
1392 .. _<packageDir>/CMakeLists.txt:
1393
1394 **<packageDir>/CMakeLists.txt**: [Required] The package's top-level
1395 ``CMakeLists.txt`` file that defines the libraries, include directories, and
1396 contains the tests for the package.
1397
1398 The basic structure of this file for a **package without subpackages** is
1399 shown in:
1400
1401 `TribitsExampleProject`_/``packages/simple_cxx/CMakeLists.txt``
1402
1403 which is:
1404
1405 .. include:: ../../examples/TribitsExampleProject/packages/simple_cxx/CMakeLists.txt
1406 :literal:
1407
1408 The first command at the top of the file is a call to `tribits_package()`_
1409 which takes the package name (``SimpleCxx`` in this case) in addition to a few
1410 other options. While TriBITS obviously already knows the package name (since
1411 it read it from the `<repoDir>/PackagesList.cmake`_ file), the purpose for
1412 repeating it in this call is as documentation for the developer's sake (and
1413 this name is checked against the expected package name). Then a set of
1414 configure-time tests are typically performed (if the package needs any of
1415 these). In this example, the existence of the C++ ``__int64`` data-type is
1416 checked using the module ``CheckFor__int64.cmake`` (which is in the ``cmake/``
1417 directory of this package. (CMake has great support `Configure-time System
1418 Tests`_.) This is followed by package-specific options. In this case, the
1419 standard TriBITS options for debug checking and deprecated warnings are added
1420 using the standard macros `tribits_add_debug_option()`_ and
1421 `tribits_add_show_deprecated_warnings_option()`_. After all of this up-front
1422 stuff is complete (which will be present in any moderately complex
1423 CMake-configured project) the source and the test sub-directories are added
1424 that actually define the library and the tests. In this case, the standard
1425 `tribits_add_test_directories()`_ macro is used which only conditionally adds
1426 the tests for the package.
1427
1428 The final command in the package's base ``CMakeLists.txt`` file must always be
1429 `tribits_package_postprocess()`_. This is needed in order to perform some
1430 necessary post-processing by TriBITS.
1431
1432 It is also possible for the package's top-level ``CMakeLists.txt`` to be the
1433 only ``CMakeLists.txt`` file for a package. Such an example can be seen in the
1434 example project `TribitsHelloWorld`_ in the ``HelloWorld`` package.
1435
1436 When a TriBITS package is broken up into subpackages (see `TriBITS
1437 Subpackage`_), its ``CMakeLists.txt`` file looks a little different from a
1438 package with no subpackages as shown above. The basic structure of this file
1439 for a **package with subpackages** is shown in:
1440
1441 `TribitsExampleProject`_/``packages/with_subpackages/CMakeLists.txt``
1442
1443 which contains:
1444
1445 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/CMakeLists.txt
1446 :literal:
1447
1448 What is different about ``CMakeLists.txt`` files for packages without
1449 subpackages is that the `tribits_package()`_ command is broken up into two
1450 parts `tribits_package_decl()`_ and `tribits_package_def()`_. In between
1451 these two commands, the parent package can define the common package options
1452 and then calls the command `tribits_process_subpackages()`_ which fully
1453 processes the packages. If the parent package has libraries and/or
1454 tests/example of its own, it can define those after calling
1455 `tribits_package_def()`_, just like with a regular package. However, it is
1456 rare for a package broken up into subpackages to have its own libraries and/or
1457 tests and examples. As always, the final command called inside of a package's
1458 base ``CMakeLists.txt`` file is `tribits_package_postprocess()`_.
1459
1460 NOTE: The package's base ``CMakeLists.txt`` file only gets processed if the
1461 package is actually enabled
1462 (i.e. ``${PROJECT_NAME}_ENABLE_${PACKAGE_NAME}=ON``). This is an important
1463 design feature of TriBITS in that the contents of non-enabled packages can't
1464 damage the configure, build, and test of the enabled packages based on errors
1465 in non-enabled packages. This is critical to allow experimental `EX`_
1466 test-group packages and lower-maturity packages to exist in the same source
1467 repositories safely with higher-maturity and more important packages.
1468
1469
1470 TriBITS Package Core Variables
1471 ..............................
1472
1473 A packages' core variables are broken down into the following categories:
1474
1475 * `TriBITS Package Local Variables`_
1476 * `TriBITS Package Top-Level Local Variables`_
1477 * `TriBITS Package Cache Variables`_
1478 * `TriBITS Package Optional Dependency Macro Variables`_
1479
1480 .. _TriBITS Package Local Variables:
1481
1482 The following locally scoped **TriBITS Package Local Variables** are defined
1483 when the files for a given TriBITS Package (or any package for that matter)
1484 are being processed:
1485
1486 ``PACKAGE_NAME``
1487
1488 The name of the current TriBITS package. This is set automatically by
1489 TriBITS before the packages' ``CMakeLists.txt`` file is processed.
1490 **WARNING:** This name must be globally unique across the entire project
1491 (see `Globally unique TriBITS package names`_).
1492
1493 ``PACKAGE_SOURCE_DIR``
1494
1495 The absolute path to the package's base source directory. This is
1496 set automatically by TriBITS in the macro `tribits_package()`_.
1497
1498 ``PACKAGE_BINARY_DIR``
1499
1500 The absolute path to the package's base binary/build directory. This is
1501 set automatically by TriBITS in the macro `tribits_package()`_.
1502
1503 ``PACKAGE_NAME_UC``
1504
1505 This is set to the upper-case version of ``${PACKAGE_NAME}``. This is set
1506 automatically by TriBITS in the macro `tribits_package()`_.
1507
1508 .. _TriBITS Package Top-Level Local Variables:
1509
1510 Once all of the TriBITS package's ``Dependencies.cmake`` files have been
1511 processed, the following **TriBITS Package Top-Level Local Variables** are
1512 defined:
1513
1514 .. _${PACKAGE_NAME}_SOURCE_DIR:
1515
1516 ``${PACKAGE_NAME}_SOURCE_DIR``
1517
1518 The absolute path to the package's base source directory. CMake code, for
1519 example in other packages, refer to this by the raw name like
1520 ``PackageX_SOURCE_DIR``. This makes such CMake code independent of where
1521 the package is in relation to other packages. NOTE: This variable is
1522 defined for all declared packages that exist, independent of whether they
1523 are enabled or not. This variable is set as soon as it is known if the
1524 given package exists or not.
1525
1526 .. _${PACKAGE_NAME}_REL_SOURCE_DIR:
1527
1528 ``${PACKAGE_NAME}_REL_SOURCE_DIR``
1529
1530 The **relative path** to the package's base source directory, relative to
1531 the projects base source directory `${PROJECT_NAME}_SOURCE_DIR`_. This is
1532 used in various contexts such as processing the packages
1533 `<packageDir>/CMakeLists.txt`_ file and generating the projects
1534 `<Project>PackageDependencies.xml`_ file where relative paths are needed.
1535
1536 .. _${PACKAGE_NAME}_BINARY_DIR:
1537
1538 ``${PACKAGE_NAME}_BINARY_DIR``
1539
1540 The absolute path to the package's base binary directory. CMake code, for
1541 example in other packages, refer to this by the raw name like
1542 ``PackageX_BINARY_DIR``. This makes such CMake code independent of where
1543 the package is in relation to other packages. NOTE: This variable is
1544 **only** defined if the package is actually enabled!
1545
1546 .. _${PACKAGE_NAME}_PARENT_REPOSITORY:
1547
1548 ``${PACKAGE_NAME}_PARENT_REPOSITORY``
1549
1550 The name of the package's parent repository. This can be used by a
1551 package to access information about its parent repository. For example,
1552 the variable ``${${PACKAGE_NAME}_PARENT_REPOSITORY}_SOURCE_DIR`` can be
1553 dereferenced and read of needed (but it is not recommended that packages
1554 be aware of their parent repository in general)..
1555
1556 .. _${PACKAGE_NAME}_TESTGROUP:
1557
1558 ``${PACKAGE_NAME}_TESTGROUP``
1559
1560 Defines the `Package Test Group`_ for the package. This determines in
1561 what contexts the package is enabled or not for testing-related purposes
1562 (see `Nested Layers of TriBITS Project Testing`_)
1563
1564 .. _${PACKAGE_NAME}_SUBPACKAGES:
1565
1566 ``${PACKAGE_NAME}_SUBPACKAGES``
1567
1568 Defines the list of subpackage names for a top-level parent package. This
1569 gives the unique subpackage name without the parent package prefix. For
1570 example, the `ReducedMockTrilinos`_ package ``Thyra`` has the subpackages
1571 ``CoreLibs``, ``GoodStuff``, etc. (which forms the full package names
1572 ``ThyraCoreLibs``, ``ThyraGoodStuff``, etc.). If a top-level package is
1573 not broken down into subpackages, then this list is empty.
1574
1575 .. _TriBITS Package Cache Variables:
1576
1577 In addition, the following user-settable **TriBITS Package Cache Variables**
1578 are defined before a Package's ``CMakeLists.txt`` file is processed:
1579
1580 .. _${PROJECT_NAME}_ENABLE_${PACKAGE_NAME}:
1581
1582 ``${PROJECT_NAME}_ENABLE_${PACKAGE_NAME}``
1583
1584 Set to ``ON`` if the package is enabled and is to be processed or will be
1585 set to ``ON`` or ``OFF`` automatically during enable/disable logic. For a
1586 parent package that is not directly enabled but where one of its
1587 subpackages is enabled, this will get set to ``ON`` (but that is not the
1588 same as the parent package being directly enabled and therefore does not
1589 imply that all of the required subpackages will be enabled, only that the
1590 parent package will be processed).
1591
1592 .. _${PACKAGE_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}:
1593
1594 ``${PACKAGE_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}``
1595
1596 Set to ``ON`` if support for the optional `upstream`_ dependent package
1597 ``${UPSTREAM_PACKAGE_NAME}`` is enabled in package
1598 ``${PACKAGE_NAME}``. Here ``${UPSTREAM_PACKAGE_NAME}`` corresponds to
1599 each optional upstream package listed in the ``LIB_OPTIONAL_PACKAGES``
1600 and ``TEST_OPTIONAL_PACKAGES`` arguments to the
1601 `tribits_package_define_dependencies()`_ macro.
1602
1603 **NOTE:** It is important that the CMake code in the package's
1604 ``CMakeLists.txt`` files key off of this variable and **not** the
1605 project-level variable
1606 ``${PROJECT_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}`` because the
1607 package-level variable
1608 ``${PACKAGE_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}`` can be explicitly
1609 turned off by the user even through the packages ``${PACKAGE_NAME}`` and
1610 ``${UPSTREAM_PACKAGE_NAME}`` are both enabled at the project level!
1611 See `Support for optional package can be explicitly disabled`_.
1612
1613 **NOTE:** This variable will also be set for required dependencies as well
1614 to allow for uniform processing such as when looping over the items in
1615 `${PACKAGE_NAME}_LIB_DEFINED_DEPENDENCIES`_ or
1616 `${PACKAGE_NAME}_TEST_DEFINED_DEPENDENCIES`_.
1617
1618 **NOTE:** The value of this variable also determines the value of the
1619 macro define variable name
1620 `HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>`_.
1621
1622 .. _${PACKAGE_NAME}_ENABLE_TESTS:
1623
1624 ``${PACKAGE_NAME}_ENABLE_TESTS``
1625
1626 Set to ``ON`` if the package's tests are to be enabled. This will enable
1627 a package's tests and all of its subpackage's tests.
1628
1629 .. _${PACKAGE_NAME}_ENABLE_EXAMPLES:
1630
1631 ``${PACKAGE_NAME}_ENABLE_EXAMPLES``
1632
1633 Set to ``ON`` if the package's examples are to be enabled. This will
1634 enable a package's examples and all of its subpackage's examples.
1635
1636 The above global cache variables can be explicitly set by the user or may be
1637 set automatically as part of the `Package Dependencies and Enable/Disable
1638 Logic`_.
1639
1640 .. _TriBITS Package Optional Dependency Macro Variables:
1641
1642 The following local **TriBITS Package Optional Dependency Macro Variables**
1643 are defined in the top-level project scope before a Package's
1644 ``CMakeLists.txt`` file is processed:
1645
1646 .. _HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>:
1647
1648 ``HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>``
1649
1650 Set to ``ON`` if support for optional upstream package
1651 ``${UPSTREAM_PACKAGE_NAME`` is enabled in downstream package
1652 ``${PACKAGE_NAME}``
1653 (i.e. `${PACKAGE_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}`_ = ``ON``) and is
1654 set to ``FALSE`` otherwise. Here, ``<PACKAGE_NAME_UC>`` and
1655 ``<UPSTREAM_PACKAGE_NAME_UC>`` are the upper-case names for the packages
1656 ``${PACKAGE_NAME}`` and ``${UPSTREAM_PACKAGE_NAME}``, respectively.
1657 For example, if optional support for upstream package ``Triutils`` is
1658 enabled in downstream package ``EpetraExt`` in `ReducedMockTrilinos`_,
1659 then ``EpetraExt_ENABLE_TriUtils=ON`` and ``HAVE_EPETRAEXT_TRIUTILS=ON``.
1660 This variable is meant to be used in::
1661
1662 #cmakedefine HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>
1663
1664 in configured header files
1665 (e.g. `<packageDir>/cmake/<packageName>_config.h.in`_). For example, for
1666 the ``EpetraExt`` and ``Triutils`` example, this would be::
1667
1668 #cmakedefine HAVE_EPETRAEXT_TRIUTILS
1669
1670 NOTE: TriBITS automatically sets this variable depending on the value of
1671 `${PACKAGE_NAME}_ENABLE_${UPSTREAM_PACKAGE_NAME}`_ during the step
1672 "Adjust package and TPLs enables and disables" in `Full Processing of
1673 TriBITS Project Files`_. And tweaking this variable after that must be
1674 done carefully as described in `How to tweak downstream TriBITS "ENABLE"
1675 variables during package configuration`_.
1676
1677 Currently, a Package can refer to its containing Repository and refer to its
1678 source and binary directories. This is so that it can refer to
1679 repository-level resources (e.g. like the ``Trilinos_version.h`` file for
1680 Trilinos packages). However, this may be undesirable because it will make it
1681 hard to pull a package out of one TriBITS repository and place it in another
1682 repository for a different use. However, a package can indirectly refer to
1683 its own repository without loss of generality by reading the variable
1684 ``${PACKAGE_NAME}_PARENT_REPOSITORY``. The problem is referring to other
1685 TriBITS repositories explicitly.
1686
1687
1688 TriBITS Subpackage
1689 ++++++++++++++++++
1690
1691 A TriBITS Subpackage:
1692
1693 * Is a compartmentalization of a parent `TriBITS Package`_ according to
1694 `Software Engineering Packaging Principles`_.
1695 * Typically defines a set of libraries and/or header files and/or executables
1696 and/or tests with CMake build targets for building these for which TriBITS
1697 exports the list of include directories, libraries, and targets that are
1698 created (along with CMake dependencies).
1699 * Is declared in its parent package's `<packageDir>/cmake/Dependencies.cmake`_
1700 file in a call to `tribits_package_define_dependencies()`_ using the
1701 argument `SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS`_.
1702 * Defines dependencies on `upstream`_ packages by just naming them in the file
1703 `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_ using the macro
1704 `tribits_package_define_dependencies()`_.
1705 * Can **NOT** have its own subpackages defined (only top-level packages can
1706 have subpackages).
1707 * Is enabled or disabled along with all other subpackages in the parent
1708 package automatically if it's parent package is enabled or disabled with
1709 ``${PROJECT_NAME}_ENABLE_${PARENT_PACKAGE_NAME}`` set to ``ON`` or ``OFF``
1710 respectively (see `Enable/disable of parent package is enable/disable for
1711 subpackages`_ and `PARENT_PACKAGE_NAME`_).
1712 * Has tests turned on automatically if
1713 ``${PARENT_PACKAGE_NAME}_ENABLE_TESTS=ON``.
1714
1715 The contents of a TriBITS Subpackage are almost identical to those of a
1716 TriBITS Package. The differences are described below and in `How is a TriBITS
1717 Subpackage different from a TriBITS Package?`_.
1718
1719 For more details on the definition of a TriBITS Package (or subpackage), see:
1720
1721 * `TriBITS Subpackage Core Files`_
1722 * `TriBITS Subpackage Core Variables`_
1723
1724
1725 TriBITS Subpackage Core Files
1726 ..............................
1727
1728 The set of core files for a subpackage are identical to the `TriBITS Package
1729 Core Files`_ and are::
1730
1731 <packageDir>/<spkgDir>/
1732 CMakeLists.txt # Only processed if this subpackage is enabled
1733 cmake/
1734 Dependencies.cmake # Always processed if the parent package
1735 # is listed in the enclosing Repository
1736
1737 (where ``<packageDir> = ${${PARENT_PACKAGE_NAME}_SOURCE_DIR}`` and
1738 ``<spkgDir>`` is the subpackage directory listed in the
1739 `SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS`_ to
1740 `tribits_package_define_dependencies()`_).
1741
1742 There are a few simple rules for the location and the contents of the
1743 ``<spkgDir>/`` directory:
1744
1745 * The relative directory ``<spkgDir>/`` should be a strict subdirectory of
1746 ``<packageDir>/`` (e.g. not ``../../somewhereelse``).
1747 * The directory ``<spkgDir>/`` must not be a subdirectory of the package
1748 directory of any other subpackage (e.g. not ``spkga/spkgb``).
1749 * All of the source files, test files, etc. for the subpackage should be
1750 included under ``<spkgDir>/``.
1751
1752 The above rules are not needed for basic building and testing but are needed
1753 for extended features like automatically detecting when a package has changed
1754 by looking at what files have changed (see `Pre-push Testing using
1755 checkin-test.py`_) and for creating source tarballs correctly (see `Creating
1756 Source Distributions`_). Therefore, it would be wise to abide by the above
1757 rules when defining subpackages.
1758
1759 These TriBITS Subpackage files are documented in more detail below:
1760
1761 * `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_
1762 * `<packageDir>/<spkgDir>/CMakeLists.txt`_
1763
1764 .. _<packageDir>/<spkgDir>/cmake/Dependencies.cmake:
1765
1766 **<packageDir>/<spkgDir>/cmake/Dependencies.cmake**: [Required] The contents
1767 of this file for subpackages are identical as for top-level packages. It just
1768 contains a call to the macro `tribits_package_define_dependencies()`_ to
1769 define this package's `upstream`_ package dependencies. A simple example is
1770 for the example subpackage ``WithSubpackagesB`` (declared in
1771 `with_subpackages/cmake/Dependencies.cmake`_) with the file:
1772
1773 `TribitsExampleProject`_/``packages/with_subpackages/b/cmake/Dependencies.cmake``
1774
1775 which is:
1776
1777 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/b/cmake/Dependencies.cmake
1778 :literal:
1779
1780 What this shows is that subpackages must list their dependencies on each other
1781 (if such dependencies exist) using the full package name
1782 ``${PARENT_PACKAGE_NAME}${SUBPACKAGE_NAME}`` or in this case::
1783
1784 'WithSubpackagesA' = 'WithSubpackages' + 'A'
1785
1786 Note that the parent package depends on its subpackages, not the other way
1787 around. For example, the ``WithSubpackages`` parent package automatically
1788 depends its subpackages ``WithSubpackagesA``, ``WithSubpackagesC``, and
1789 ``WithSubpackagesC``. As such all (direct) dependencies for a subpackage must
1790 be listed in its own ``Dependencies.cmake`` file. For example, the
1791 ``WithSubpackages`` subpackage ``A`` depends on the ``SimpleCxx`` package and
1792 is declared as such as shown in:
1793
1794 `TribitsExampleProject`_/``packages/with_subpackages/a/cmake/Dependencies.cmake``
1795
1796 which is:
1797
1798 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/a/cmake/Dependencies.cmake
1799 :literal:
1800
1801 What this means is that any package dependencies listed in the parent
1802 package's `<packageDir>/cmake/Dependencies.cmake`_ file are **NOT**
1803 dependencies of its subpackages. For example, if
1804 `with_subpackages/cmake/Dependencies.cmake`_ where changed to be::
1805
1806 tribits_package_define_dependencies(
1807 LIB_REQUIRED_TPLS Boost
1808 SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS
1809 A A PT REQUIRED
1810 ...
1811 )
1812
1813 then the ``Boost`` TPL would **NOT** be a dependency of the package
1814 ``WithSubpackagesA`` but instead would be listed as a
1815 dependency of the parent package ``WithSubpackages``. (And in this
1816 case, this TPL dependency is pretty worthless since the package
1817 ``WithSubpackages`` does not even define any libraries or tests of its
1818 own.)
1819
1820 .. _<packageDir>/<spkgDir>/CMakeLists.txt:
1821
1822 **<packageDir>/<spkgDir>/CMakeLists.txt**: [Required] The subpackage's
1823 top-level ``CMakeLists.txt`` file that defines the libraries, include
1824 directories, and contains the tests for the subpackage. The contents of a
1825 subpackage's top-level ``CMakeLists.txt`` file are almost identical to a
1826 top-level package's `<packageDir>/CMakeLists.txt`_ file. The primary
1827 difference is that the commands `tribits_package()`_ and
1828 `tribits_package_postprocess()`_ and replaced with `tribits_subpackage()`_ and
1829 `tribits_subpackage_postprocess()`_ as shown in the file:
1830
1831
1832 `TribitsExampleProject`_/``packages/with_subpackages/a/CMakeLists.txt``
1833
1834 which contains:
1835
1836 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/a/CMakeLists.txt
1837 :literal:
1838
1839 Unlike `tribits_package()`_, `tribits_subpackage()`_ does not take any extra
1840 arguments. Those extra settings are assumed to be defined by the top-level
1841 parent package. Like top-level packages, subpackages are free to define
1842 user-settable options and configure-time tests but typically don't. The idea
1843 is that subpackages should be lighter weight than top-level packages. Other
1844 than using `tribits_subpackage()`_ and `tribits_subpackage_postprocess()`_, a
1845 subpackage can be laid out just like any other package and can call on any
1846 other commands to add libraries, add executables, add test, etc.
1847
1848
1849 TriBITS Subpackage Core Variables
1850 .................................
1851
1852 The core variables associated with a subpackage are identical to the `TriBITS
1853 Package Core Variables`_. In addition, a subpackage may need to refer to its
1854 top-level parent package where a top-level package does not have a parent
1855 package. These additional variables that are defined for subpackages are
1856 broken down into the following categories:
1857
1858 * `TriBITS Subpackage Local Variables`_
1859 * `TriBITS Subpackage Top-Level Local Variables`_
1860
1861 .. _TriBITS Subpackage Local Variables:
1862
1863 In addition to the `TriBITS Package Local Variables`_, the following locally
1864 scoped **TriBITS Subpackage Local Variables** are defined when the files for a
1865 given TriBITS Subpackage are being processed:
1866
1867 .. _PARENT_PACKAGE_NAME:
1868
1869 ``PARENT_PACKAGE_NAME``
1870
1871 The name of the parent package.
1872
1873 ``PARENT_PACKAGE_SOURCE_DIR``
1874
1875 The absolute path to the parent package's base source directory.
1876
1877 ``PARENT_PACKAGE_BINARY_DIR``
1878
1879 The absolute path to the parent package's base binary directory.
1880
1881
1882 .. _TriBITS Subpackage Top-Level Local Variables:
1883
1884 In addition to the `TriBITS Package Top-Level Local Variables`_, once all of a
1885 TriBITS subpackage's ``Dependencies.cmake`` files have been processed, the
1886 following **TriBITS Subpackage Top-Level Local Variables** are defined:
1887
1888 .. _${PACKAGE_NAME}_PARENT_PACKAGE:
1889
1890 ``${PACKAGE_NAME}_PARENT_PACKAGE``
1891
1892 The name of the parent package. (NOTE: If this is empty "", then
1893 ``${PACKAGE_NAME}`` is actually a parent package and not a subpackage.)
1894
1895
1896 How is a TriBITS Subpackage different from a TriBITS Package?
1897 .............................................................
1898
1899 A common question this is natural to ask is how a TriBITS Subpackage is
1900 different from a TriBITS Package? They contain the same basic files (i.e. a
1901 ``cmake/Dependencies.cmake`` file, a top-level ``CMakeLists.txt`` file, source
1902 files, test files, etc.). They both are included in the list of `TriBITS
1903 Packages`_ and therefore can both be enabled/disabled by the user or in
1904 automatic dependency logic (see `Package Dependencies and Enable/Disable
1905 Logic`_). The primary difference is that a subpackage is meant to involve
1906 less overhead and is to be used to partition the parent package's software
1907 into chunks according to `Software Engineering Packaging Principles`_. Also,
1908 the dependency logic treats a parent package's subpackages as part of itself
1909 so when the parent package is explicitly enabled or disabled, it is identical
1910 to explicitly enabling or disabling all of its subpackages (see
1911 `Enable/disable of parent package is enable/disable for subpackages`_). Also,
1912 subpackages are tested along with their peer subpackages with the parent
1913 package as part of `TriBITS CTest/CDash Driver`_. This effectively means that
1914 if a build failure is detected in any subpackage, then that will effectively
1915 disable the parent package and all of its other subpackages in downstream
1916 testing. This is a type of "all for one and one for all" when it comes to the
1917 relationship between the subpackages within a single parent package. These
1918 are some of the issues to consider when breaking up software into packages and
1919 subpackages that will be mentioned in other sections as well.
1920
1921
1922 TriBITS External Package/TPL
1923 ++++++++++++++++++++++++++++
1924
1925 A *TriBITS External Package/TPL*:
1926
1927 * Provides access to a set of pre-built/installed libraries and/or header
1928 files and/or executables and/or some other resources through a standard
1929 interface to one or more downstream TriBITS Packages.
1930 * Has a globally unique name ``<tplName>`` (see `Globally unique TriBITS TPL
1931 names`_) that is declared in a `<repoDir>/TPLsList.cmake`_ file.
1932 * Has as `FindTPL<tplName>.cmake`_ module (for non-`TriBITS-Compliant External
1933 Packages`_) that finds the pieces of an external package/TPL and provides
1934 them to downstream packages through a required INTERFACE target
1935 ``<tplName>::all_libs`` (which gives the libraries, include directories, and
1936 other usage requirements, see `TriBITS-Compliant External Package`_).
1937 * Is listed as an explicit optional or required dependency in one or more
1938 downstream TriBITS packages' `<packageDir>/cmake/Dependencies.cmake`_ files.
1939 * Can be enabled automatically or can trigger the disable of dependent
1940 downstream packages (see `TriBITS Dependency Handling Behaviors`_).
1941
1942 The TriBITS external package/TPL mechanism provides a uniform way to find and
1943 provide access to any type of external resource no matter how it might be
1944 installed or ways to provide access to it. Using a TriBITS external
1945 package/TPL is to be preferred over using raw CMake
1946 ``find_package(<externalPkg>)`` call because the TriBITS system guarantees
1947 that only a single unique version of an external package/TPL of the same
1948 version will be used all of the downstream packages that uses it. Also, by
1949 defining a TriBITS TPL, automatic enable/disable logic will be applied as
1950 described in `Package Dependencies and Enable/Disable Logic`_. For example,
1951 if an external package/TPL is explicitly disabled, all of the downstream
1952 packages that depend on it will be automatically disabled as well (see
1953 `Package disable triggers auto-disables of downstream dependencies`_).
1954
1955 NOTE: The TriBITS TPL system implements a mechanism to turn external
1956 dependencies into both `TriBITS-compliant packages`_ for consumption by
1957 downstream TriBITS internal packages and also writes ``<tplName>Config.cmake``
1958 files that are `TriBITS-compliant external packages`_ for consumption by
1959 downstream ``<Package>Config.cmake`` files (which are `TriBITS-compliant
1960 external packages`_) generated by `TriBITS-compliant internal packages`_.
1961
1962
1963 .. _Globally unique TriBITS TPL names:
1964
1965 **WARNING:** One must be very careful to pick **Globally unique TriBITS
1966 External Package/TPL names** ``<tplName>`` across all TPLs in all TriBITS
1967 repositories that ever might be cobbled together into a single TriBITS (meta)
1968 project! However, choosing TPL names is usually much easier and less risky
1969 than choosing `Globally unique TriBITS package names`_ because widely used
1970 TPLs tend to already be uniquely named. For example, the external package/TPL
1971 names ``BLAS`` and ``LAPACK`` are well defined in the applied math and
1972 computational science community and are not likely to clash.
1973
1974
1975 TriBITS External Package/TPL Core Files
1976 .......................................
1977
1978 The core files that define a *TriBITS External Package/TPL* are::
1979
1980 <tplDefsDir>/
1981 FindTPL<tplName>.cmake # The name is not fixed (see <tplName>_FINDMOD)
1982 FindTPL<tplName>Dependencies.cmake # [Optional], defines upstream dependencies
1983
1984 Above, ``<tplDefsDir>/`` can be a subdirectory under a parent TriBITS
1985 repository ``<repoDir>/`` (e.g. ``<repoDir>/cmake/tpls/``) or can be under a
1986 TriBITS package directory ``<packageDir>/``
1987 (e.g. ``<packageDir>/cmake/tpls/``).
1988
1989 The following TriBITS External Package/TPL files are documented in more detail
1990 below:
1991
1992 * `<tplDefsDir>/FindTPL<tplName>.cmake`_
1993 * `<tplDefsDir>/FindTPL<tplName>Dependencies.cmake`_
1994
1995
1996 .. _FindTPL<tplName>.cmake:
1997 .. _<tplDefsDir>/FindTPL<tplName>.cmake:
1998
1999 **<tplDefsDir>/FindTPL<tplName>.cmake**: [Required] *TriBITS TPL find module*
2000 that defines how a TriBITS external package/TPL is found and provided for
2001 usage by a downstream TriBITS package. This module must provide the
2002 ``<tplName>::all_libs`` target and must create a `TriBITS-compliant external
2003 package`_ wrapper package config file ``<tplName>Config.cmake``. (See the
2004 requirements for a ``FindTPL<tplName>.cmake`` file in `Requirements for
2005 FindTPL<tplName>.cmake modules`_).
2006
2007 The form of a simple ``FindTPL<tplName>.cmake`` file that uses an internal
2008 call to ``find_package(<externalPkg>)`` which provides modern IMPORTED CMake
2009 targets can use the
2010 `tribits_extpkg_create_imported_all_libs_target_and_config_file()`_ function
2011 and looks like::
2012
2013 find_package(<externalPkg> REQUIRED)
2014 tribits_extpkg_create_imported_all_libs_target_and_config_file(
2015 <tplName>
2016 INNER_FIND_PACKAGE_NAME <externalPkg>
2017 IMPORTED_TARGETS_FOR_ALL_LIBS <importedTarget0> <importedTarget1> ... )
2018
2019 In this case, the purpose for the ``FindTPL<tplName>.cmake`` file (as opposed
2020 to a direct call to ``find_package(<externalPkg>)``) is to ensure the
2021 definition of the complete target ``<tplName>::all_libs`` which contains all
2022 usage requirements for the external package/TPL (i.e. all of the libraries,
2023 include directories, etc.) and this also generates the wrapper package config
2024 file ``<tplName>Config.cmake``.
2025
2026 The form of a simple ``FindTPL<tplName>.cmake`` file that just provides a list
2027 of required header files and libraries that does **not** use an internal call
2028 to ``find_package()`` and instead uses the function
2029 `tribits_tpl_find_include_dirs_and_libraries()`_ looks like::
2030
2031 tribits_tpl_find_include_dirs_and_libraries( <tplName>
2032 REQUIRED_HEADERS <header0> <header1> ...
2033 REQUIRED_LIBS_NAMES <libname0> <libname1> ...
2034 MUST_FIND_ALL_LIBS
2035 )
2036
2037 An example concrete file is ``tribits/common_tpls/FindTPLPETSC.cmake``:
2038
2039 .. include:: ../../common_tpls/FindTPLPETSC.cmake
2040 :literal:
2041
2042 For complete details, see `Creating the FindTPL<tplName>.cmake file`_.
2043
2044
2045 .. _FindTPL<tplName>Dependencies.cmake:
2046 .. _<tplDefsDir>/FindTPL<tplName>Dependencies.cmake:
2047
2048 **<tplDefsDir>/FindTPL<tplName>Dependencies.cmake**: [Optional]
2049 Declares dependencies on upstream external packages/TPLs for the external
2050 package/TPL ``<tplName>``. Many external packages/TPLs defined with a
2051 `FindTPL<tplName>.cmake`_ file do not have any upstream dependencies or have
2052 internal mechanisms to get those (such as when using
2053 ``find_package(<externalPkg>)`` where the ``<externalPkg>Config.cmake`` file
2054 which recursively uses ``find_dependency()`` to get its upstream
2055 dependencies). But for ``FindTPL<tplName>.cmake`` files that just use
2056 `tribits_tpl_find_include_dirs_and_libraries()`_ (see `Creating a
2057 FindTPL<tplName>.cmake module without find_package()`_), TriBITS needs to be told
2058 about any upstream external packages/TPLs that it may depend on so it can add
2059 the dependencies between the created IMPORTED target libraries.
2060
2061 The file ``FindTPL<tplName>Dependencies.cmake`` is typically just a single
2062 call to `tribits_extpkg_define_dependencies()`_ and takes the form::
2063
2064 tribits_extpkg_define_dependencies( <tplName>
2065 DEPENDENCIES <upstreamTpl_0> <upstreamTpl_1> ... )
2066
2067 This defines all of the TPLs that ``<tplName>`` could directly depends on but
2068 only dependencies for enabled upstream TPLs will be added to the IMPORTED
2069 targets.
2070
2071 NOTE: TPL-to-TPL dependencies are optional. Therefore, in the above example,
2072 enabling the TPL ``<tplName>`` will not auto-enable a dependent upstream TPL
2073 ``<upstreamTpl_i>``. Likewise, disabling an upstream TPL ``<upstreamTpl_i>``
2074 will not auto-disable a dependent downstream TPL ``<tplName>``.
2075
2076
2077 TriBITS External Package/TPL Core Variables
2078 ...........................................
2079
2080 Once the `<repoDir>/TPLsList.cmake`_ files are all processed, then each
2081 defined TPL ``TPL_NAME`` is assigned the following global non-cache variables:
2082
2083 .. _${PACKAGE_NAME}_FINDMOD:
2084 .. _<tplName>_FINDMOD:
2085 .. _${TPL_NAME}_FINDMOD:
2086
2087 ``${TPL_NAME}_FINDMOD``
2088
2089 For a **non-** `TriBITS-compliant external package`_, this is the relative
2090 path (w.r.t. ``<projectDir>``) or absolute path for the *TriBITS TPL find
2091 module* (typically named `FindTPL<tplName>.cmake`_). This is set using the
2092 ``FINDMOD`` field in the call to `tribits_repository_define_tpls()`_. The
2093 final value of the variable is defined by the **last**
2094 `<repoDir>/TPLsList.cmake`_ file that is processed that declares the TPL
2095 ``TPL_NAME``. For example, if ``Repo1/TPLsList.cmake`` and
2096 ``Repo2/TPLsList.cmake`` both list the TPL ``SomeTpl``, then if ``Repo2``
2097 is processed after ``Repo1``, then ``SomeTpl_FINDMOD`` is determined by
2098 ``Repo2/TPLsList.cmake`` and the find module listed in
2099 ``Repo1/TPLsList.cmake`` is ignored. NOTE: for a `TriBITS-compliant
2100 external package`_, the special value ``TRIBITS_PKG`` is also recognized.
2101 (Any pre-installed TriBITS package is a `TriBITS-compliant external
2102 package`_.)
2103
2104 .. _<tplName>_DEPENDENCIES_FILE:
2105 .. _${TPL_NAME}_DEPENDENCIES_FILE:
2106
2107 ``${TPL_NAME}_DEPENDENCIES_FILE``
2108
2109 Relative path (w.r.t. ``<projectDir>``) or absolute path for the external
2110 package/TPL's dependencies file (typically named
2111 `FindTPL<tplName>Dependencies.cmake`_). This is always beside the find
2112 module `${TPL_NAME}_FINDMOD`_. (In fact, for a **non-**
2113 `TriBITS-compliant external package`_, ``${TPL_NAME}_DEPENDENCIES_FILE``
2114 is constructed from ``${TPL_NAME}_FINDMOD``). NOTE: A `TriBITS-compliant
2115 external package`_ with dependencies will also have this file set and the
2116 path will be specified independent of the path to the non-existent
2117 ``FindTPL<tplName>.cmake`` file (see the ``FINDMOD`` field in the call to
2118 `tribits_repository_define_tpls()`_).
2119
2120 .. _<tplName>_TESTGROUP:
2121 .. _${TPL_NAME}_TESTGROUP:
2122
2123 ``${TPL_NAME}_TESTGROUP``
2124
2125 TPL's `Package Test Group`_: This is set using the ``CLASSIFICATION``
2126 field in the call to `tribits_repository_define_tpls()`_. If multiple
2127 repos define a given TPL, then the *first* `<repoDir>/TPLsList.cmake`_
2128 file that is processed that declares the TPL ``TPL_NAME`` specifies the
2129 test group. For example, if ``Repo1/TPLsList.cmake`` and
2130 ``Repo2/TPLsList.cmake`` both list the TPL ``SomeTpl``, then if ``Repo2``
2131 is processed after ``Repo1``, then ``SomeTpl_TESTGROUP`` is determined by
2132 ``Repo1/TPLsList.cmake`` and the test group in ``Repo2/TPLsList.cmake`` is
2133 ignored. However, if ``${TPL_NAME}_TESTGROUP`` is already set before the
2134 `<repoDir>/TPLsList.cmake`_ files are processed, then that test group will
2135 be used. Therefore, the project can override the test group for a given
2136 TPL if desired by setting ``${TPL_NAME}_TESTGROUP`` before the first
2137 `<repoDir>/TPLsList.cmake`_ file gets processed.
2138
2139 .. _${TPL_NAME}_TPLS_LIST_FILE:
2140
2141 ``${TPL_NAME}_TPLS_LIST_FILE``
2142
2143 Absolute path of the (last) `<repoDir>/TPLsList.cmake`_ file that declared
2144 this external package/TPL.
2145
2146 Note, the ``<findmod>`` field path in the call to
2147 `tribits_repository_define_tpls()`_ is relative to the TriBITS repository dir
2148 ``<repoDir>`` but a relative path in for the variable `<tplName>_FINDMOD`_ is
2149 relative to the project dir ``<projectDir>``. There is a translation of the
2150 ``<findmod>`` field to the variable ``<tplName>_FINDMOD`` that takes place
2151 when the `<repoDir>/TPLsList.cmake`_ file is processed to make this so.
2152
2153 As noted above, it is allowed for the same TPL to be listed in multiple
2154 `<repoDir>/TPLsList.cmake`_ files. In this case, the rules for overrides of
2155 the find module and the test group as described above.
2156
2157 The specification given in `Enabling support for an optional Third-Party
2158 Library (TPL)`_ and `Creating the FindTPL<tplName>.cmake file`_ describe how
2159 to create a ``FindTPL<tplName>.cmake`` module. However, all that is required
2160 is that some CMake file fragment exist such that, once included, will define
2161 the target ``<tplName>::all_libs`` and create the ``<tplName>Config.cmake``
2162 file in the correct location (see `Requirements for FindTPL<tplName>.cmake modules`_).
2163
2164
2165 Processing of TriBITS Files: Ordering and Details
2166 --------------------------------------------------
2167
2168 One of the most important things to know about TriBITS is what files it
2169 processes, in what order, and in what context. This is critical to being able
2170 to understand what impact (if any) setting a variable or otherwise changing
2171 the CMake run-time state will have on configuring a CMake project which uses
2172 TriBITS. While the different files that make up a `TriBITS Project`_,
2173 `TriBITS Repository`_, `TriBITS Package`_, `TriBITS Subpackage`_, and `TriBITS
2174 TPL`_ were defined in the section `TriBITS Structural Units`_, that material
2175 did not fully describe the context and in what order these files are processed
2176 by the TriBITS framework.
2177
2178 The TriBITS system processes the project's files in one of two general use
2179 cases. The first use case is in the basic configuration of the project with a
2180 standard ``cmake`` command invocation in order to set up the build files in
2181 the binary directory (see `Full TriBITS Project Configuration`_). The second
2182 use case is in reading the project's dependency-related files in order to
2183 build the package dependency data-structure (e.g. the
2184 `<Project>PackageDependencies.xml`_ file, see `Reduced Package Dependency
2185 Processing`_). The second use case of reading the project's dependency files
2186 is largely a subset of the first.
2187
2188 Another factor that is important to understand is the scoping in which the
2189 various files are processed (with ``include()`` or ``add_subdirectory()``).
2190 This scoping has a large impact on the configuration of the project and what
2191 effect the processing of files and setting variables have on the project as a
2192 whole. Some of the strange scoping rules for CMake are discussed in `CMake
2193 Language Overview and Gotchas`_ and should be understood before trying to
2194 debug issues with processing. Many of the basic files are processed
2195 (included) in the base project `<projectDir>/CMakeLists.txt`_ scope and
2196 therefore any local variables set in these files are accessible to the entire
2197 CMake project (after the file is processed, of course). Other files get
2198 processed inside of functions which have their own local scope and therefore
2199 only impact the rest of the project in more purposeful ways. And of course
2200 all of the package `<packageDir>/CMakeLists.txt`_ files that are processed
2201 using ``add_subdirectory()`` create a new local scope for that given package.
2202
2203
2204 Full TriBITS Project Configuration
2205 ++++++++++++++++++++++++++++++++++
2206
2207 The first use case to describe is the full processing of all of the TriBITS
2208 project's files starting with the base `<projectDir>/CMakeLists.txt`_ file.
2209 This begins with the invocation of the following CMake command to generate the
2210 project's build files::
2211
2212 $ cmake [options] <projectDir>
2213
2214 Below, is a short pseudo-code algorithm for the TriBITS framework processing
2215 and callbacks that begins in the `<projectDir>/CMakeLists.txt`_ file and
2216 proceeds through the call to `tribits_project()`_.
2217
2218 .. _Full Processing of TriBITS Project Files:
2219
2220 **Full Processing of TriBITS Project Files:**
2221
2222 | 1. Read `<projectDir>/ProjectName.cmake`_ (sets `PROJECT_NAME`_)
2223 | 2. Call ``project(${PROJECT_NAME} NONE)`` (sets `${PROJECT_NAME}_SOURCE_DIR`_
2224 | and `${PROJECT_NAME}_BINARY_DIR`_)
2225 | 3. Call `tribits_project()`_:
2226 | 1) Set `PROJECT_SOURCE_DIR`_ and `PROJECT_BINARY_DIR`_
2227 | 2) For each ``<optFile>`` in ${`${PROJECT_NAME}_CONFIGURE_OPTIONS_FILE`_}
2228 | then in ${`${PROJECT_NAME}_CONFIGURE_OPTIONS_FILE_APPEND`_}
2229 :
2230 | * ``include(<optFile>)``
2231 | 3) Set variables ``CMAKE_HOST_SYSTEM_NAME`` and ``${PROJECT_NAME}_HOSTNAME``
2232 | (both of these can be overridden in the cache by the user)
2233 | 4) Find some optional command-line tools:
2234 | a) Find Python (sets ``Python3_EXECUTABLE``, see `Python Support`_)
2235 | b) Find Git (sets ``GIT_EXECUTABLE`` and ``GIT_VERSION_STRING``)
2236 | 5) ``include(`` `<projectDir>/Version.cmake`_ ``)``
2237 | 6) Define primary TriBITS options and read in the list of extra repositories
2238 | (calls ``tribits_define_global_options_and_define_extra_repos()``)
2239 | * ``include(`` `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ ``)``
2240 | 7) For each ``<repoDir>`` in all defined TriBITS repositories:
2241 | * ``include(`` `<repoDir>/cmake/CallbackSetupExtraOptions.cmake`_ ``)``
2242 | * Call macro ``tribits_repository_setup_extra_options()``
2243 | 9) Call `tribits_read_all_project_deps_files_create_deps_graph()`_:
2244 | a) For each ``<repoDir>`` in all defined TriBITS repositories:
2245 | * ``include(`` `<repoDir>/TPLsList.cmake`_ ``)`` and process list
2246 | * ``include(`` `<repoDir>/PackagesList.cmake`_ ``)`` and process list
2247 | b) For each ``<repoDir>`` in all defined TriBITS repositories:
2248 | * ``include(`` `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_ ``)``
2249 | c) ``include(`` `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_ ``)``
2250 | d) For each ``<packageDir>`` in all defined top-level packages:
2251 | * ``include(`` `<packageDir>/cmake/Dependencies.cmake`_ ``)``
2252 | - Sets all package-specific options (see `TriBITS Package Cache Variables`_)
2253 | * For each ``<spkgDir>`` in all subpackages for this package:
2254 | * ``include(`` `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_ ``)``
2255 | - Sets all subpackage-specific options
2256 | 10) Adjust package and TPLs enables and disables
2257 | (see `Package Dependencies and Enable/Disable Logic`_)
2258 | 11) `Probe and set up the environment`_ (finds MPI, compilers, etc.)
2259 | (see `TriBITS Environment Probing and Setup`_)
2260 | * ``include(`` `<projectDir>/cmake/ProjectCompilerPostConfig.cmake`_ ``)``
2261 | 12) For each ``<tplName>`` in the set of enabled TPLs:
2262 | * ``include(${<tplName>_FINDMOD})`` (see `TriBITS TPL`_)
2263 | 13) For each ``<repoDir>`` in all defined TriBITS repositories:
2264 | * Read `<repoDir>/Copyright.txt`_
2265 | * ``include(`` `<repoDir>/Version.cmake`_ ``)``
2266 | (see `Project and Repository Versioning and Release Mode`_)
2267 | 14) For each ``<packageDir>`` in all enabled top-level packages
2268 | * ``add_subdirectory(`` `<packageDir>/CMakeLists.txt`_ ``)``
2269 | * For each ``<spkgDir>`` in all enabled subpackages for this package:
2270 | * ``add_subdirectory(`` `<packageDir>/<spkgDir>/CMakeLists.txt`_ ``)``
2271 | 16) ``include(`` `<projectDir>/cmake/CallbackDefineProjectPackaging.cmake`_ ``)``
2272 | * Call ``tribits_project_define_packaging()``
2273 | 16) For each ``<repoDir>`` in all defined TriBITS repositories:
2274 | * ``include(`` `<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake`_ ``)``
2275 | * Call ``tribits_repository_define_packaging()``
2276
2277 The TriBITS Framework obviously does a lot more than what is described above
2278 but the basic trace of major operations and ordering and the processing of
2279 project, repository, package, and subpackage files should be clear. All of
2280 this information should also be clear when enabling `File Processing
2281 Tracing`_ and watching the output from the ``cmake`` configure STDOUT.
2282
2283 Reduced Package Dependency Processing
2284 ++++++++++++++++++++++++++++++++++++++
2285
2286 In addition to the full processing that occurs as part of the `Full TriBITS
2287 Project Configuration`_, there are also TriBITS tools that only process as
2288 subset of project's files. This reduced processing is performed in order to
2289 build up the project's package dependencies data-structure and to write the
2290 file `<Project>PackageDependencies.xml`_. For example, the tool
2291 `checkin-test.py`_ and the function `tribits_ctest_driver()`_ both drive this
2292 type of processing. In particular, the CMake -P script
2293 `TribitsDumpDepsXmlScript.cmake`_ reads all of the project's
2294 dependency-related files and dumps out the `<Project>PackageDependencies.xml`_
2295 file (see `TriBITS Project Dependencies XML file and tools`_). This reduced
2296 processing (e.g. as executed in ``cmake -P TribitsDumpDepsXmlScript.cmake``)
2297 is described below.
2298
2299 .. _Reduced Dependency Processing of TriBITS Project Files:
2300
2301 **Reduced Dependency Processing of TriBITS Project:**
2302
2303 | 1. Read `<projectDir>/ProjectName.cmake`_ (sets `PROJECT_NAME`_)
2304 | 2. ``include(`` `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ ``)``
2305 | 3. Call ``tribits_read_all_project_deps_files_create_deps_graph()``:
2306 | a) For each ``<repoDir>`` in all defined TriBITS repositories:
2307 | * ``include(`` `<repoDir>/TPLsList.cmake`_ ``)``
2308 | * ``include(`` `<repoDir>/PackagesList.cmake`_ ``)``
2309 | b) For each ``<repoDir>`` in all defined TriBITS repositories:
2310 | * ``include(`` `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_ ``)``
2311 | c) ``include(`` `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_ ``)``
2312 | d) For each ``<packageDir>`` in all defined top-level packages:
2313 | * ``include(`` `<packageDir>/cmake/Dependencies.cmake`_ ``)``
2314 | - Sets all package-specific options (see `TriBITS Package Cache Variables`_)
2315 | * For each ``<spkgDir>`` in all subpackages for this package:
2316 | * ``include(`` `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_ ``)``
2317 | - Sets all subpackage-specific options
2318 | e) Write the file `<Project>PackageDependencies.xml`_
2319 | (specified using `${PROJECT_NAME}_DEPS_XML_OUTPUT_FILE`_)
2320
2321 When comparing the above reduced dependency processing to the `Full Processing
2322 of TriBITS Project Files`_ it is important to note that that several files are
2323 **not** processed in the reduced algorithm shown above. The files that are
2324 **not** processed include `<projectDir>/Version.cmake`_,
2325 `<repoDir>/Version.cmake`_ and
2326 `<repoDir>/cmake/CallbackSetupExtraOptions.cmake`_ (in addition to not
2327 processing any of the ``CMakeLists.txt`` files obviously). Therefore, one
2328 cannot put anything in these non-processed files that would impact the
2329 definition of TriBITS repositories, packages, TPLs, etc. Anything that would
2330 affect the dependencies data-structure that gets written out as
2331 `<Project>PackageDependencies.xml`_ must be contained in the files that are
2332 processed shown in the reduced processing above.
2333
2334 Debugging issues with `Reduced Dependency Processing of TriBITS Project
2335 Files`_ is more difficult because one cannot easily turn on `File Processing
2336 Tracing`_ like one can when doing the full CMake configure. However, options
2337 may be added to the various tools to show this file processing and help debug
2338 problems.
2339
2340 File Processing Tracing
2341 +++++++++++++++++++++++
2342
2343 In order to aid in debugging problems with `Full TriBITS Project
2344 Configuration`_ and `Reduced Package Dependency Processing`_, TriBITS defines
2345 the CMake cache option `${PROJECT_NAME}_TRACE_FILE_PROCESSING`_. When
2346 enabled, TriBITS will print out when any of the project-related,
2347 repository-related, or package-related file is being processed by TriBITS.
2348 When ``${PROJECT_NAME}_TRACE_FILE_PROCESSING=ON``, lines starting with ``"--
2349 File Trace:"`` are printed to ``cmake`` stdout for files that TriBITS
2350 automatically processes where there may be any confusion about what files are
2351 processed and when.
2352
2353 For example, for `TribitsExampleProject`_, the configure file trace for the
2354 configure command::
2355
2356 $ cmake \
2357 -DTribitsExProj_TRIBITS_DIR=<tribitsDir> \
2358 -DTribitsExProj_ENABLE_MPI=ON \
2359 -DTribitsExProj_ENABLE_ALL_PACKAGES=ON \
2360 -DTribitsExProj_ENABLE_TESTS=ON \
2361 -DTribitsExProj_TRACE_FILE_PROCESSING=ON \
2362 -DTribitsExProj_ENABLE_CPACK_PACKAGING=ON \
2363 -DTribitsExProj_DUMP_CPACK_SOURCE_IGNORE_FILES=ON \
2364 <tribitsDir>/doc/TribitsExampleProject \
2365 | grep "^-- File Trace:"
2366
2367 looks something like::
2368
2369 -- File Trace: PROJECT INCLUDE [...]/Version.cmake
2370 -- File Trace: REPOSITORY INCLUDE [...]/cmake/CallbackSetupExtraOptions.cmake
2371 -- File Trace: REPOSITORY INCLUDE [...]/PackagesList.cmake
2372 -- File Trace: REPOSITORY INCLUDE [...]/TPLsList.cmake
2373 -- File Trace: PACKAGE INCLUDE [...]/packages/simple_cxx/cmake/Dependencies.cmake
2374 -- File Trace: PACKAGE INCLUDE [...]/packages/mixed_lang/cmake/Dependencies.cmake
2375 -- File Trace: PACKAGE INCLUDE [...]/packages/with_subpackages/cmake/Dependencies.cmake
2376 -- File Trace: PACKAGE INCLUDE [...]/packages/with_subpackages/a/cmake/Dependencies.cmake
2377 -- File Trace: PACKAGE INCLUDE [...]/packages/with_subpackages/b/cmake/Dependencies.cmake
2378 -- File Trace: PACKAGE INCLUDE [...]/packages/with_subpackages/c/cmake/Dependencies.cmake
2379 -- File Trace: PACKAGE INCLUDE [...]/packages/wrap_external/cmake/Dependencies.cmake
2380 -- File Trace: PROJECT CONFIGURE [...]/cmake/ctest/CTestCustom.cmake.in
2381 -- File Trace: REPOSITORY READ [...]/Copyright.txt
2382 -- File Trace: REPOSITORY INCLUDE [...]/Version.cmake
2383 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/simple_cxx/CMakeLists.txt
2384 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/simple_cxx/test/CMakeLists.txt
2385 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/mixed_lang/CMakeLists.txt
2386 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/mixed_lang/test/CMakeLists.txt
2387 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/CMakeLists.txt
2388 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/a/CMakeLists.txt
2389 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/a/tests/CMakeLists.txt
2390 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/b/CMakeLists.txt
2391 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/b/tests/CMakeLists.txt
2392 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/c/CMakeLists.txt
2393 -- File Trace: PACKAGE ADD_SUBDIR [...]/packages/with_subpackages/c/tests/CMakeLists.txt
2394 -- File Trace: PROJECT INCLUDE [...]/cmake/CallbackDefineProjectPackaging.cmake
2395 -- File Trace: REPOSITORY INCLUDE [...]/cmake/CallbackDefineRepositoryPackaging.cmake
2396
2397 However, every file that TriBITS processes is not printed in this file trace
2398 if it should be obvious that the file is being processed. For example, the
2399 package's configured header file created using `tribits_configure_file()`_
2400 does not result in a file trace print statement because this is an
2401 unconditional command that is explicitly called in one of the package's
2402 ``CMakeLists.txt`` files so it should be clear that this file is being
2403 processed and exactly when it is processed.
2404
2405
2406 Coexisting Projects, Repositories, and Packages
2407 -----------------------------------------------
2408
2409 Certain simplifications are allowed when defining TriBITS projects,
2410 repositories and packages. The known allowed simplifications are described
2411 below.
2412
2413 **TriBITS Repository Dir == TriBITS Project Dir**: It is allowed for a TriBITS
2414 Project and a TriBITS Repository to be the same source directory and in fact
2415 this is the default for every TriBITS project (unless the
2416 `<projectDir>/cmake/NativeRepositoriesList.cmake`_ is defined). In this case,
2417 the repository name `REPOSITORY_NAME`_ and the project name `PROJECT_NAME`_
2418 are the same as well. This is quite common and is in fact the default that
2419 every TriBITS Project is also a TriBITS repository (and therefore must contain
2420 `<repoDir>/PackagesList.cmake`_ and `<repoDir>/TPLsList.cmake`_ files). This
2421 is the case, for example, with the `Trilinos`_ and the
2422 `TribitsExampleProject`_ projects and repositories. In this case, the
2423 Project's and the Repository's ``Version.cmake`` and ``Copyright.txt`` files
2424 are also one and the same, as they should be (see `Project and Repository
2425 Versioning and Release Mode`_).
2426
2427 **TriBITS Package Dir == TriBITS Repository Dir**: It is also allowed for a
2428 TriBITS Repository to have only one package and to have that package be the
2429 base repository directory. The TriBITS Repository and the single TriBITS
2430 Package would typically have the same name in this case (but that is actually
2431 not required but it is confusing if they are not the same). For example, in
2432 the TriBITS test project `MockTrilinos`_, the repository and package
2433 ``extraRepoOnePackage`` are the same directory. In this case, the file
2434 ``extraRepoOnePackage/PackagesList.cmake`` looks like:
2435
2436 .. include:: ../../examples/MockTrilinos/extraRepoOnePackage/PackagesList.cmake
2437 :literal:
2438
2439 (Note the dot ``'.'`` for the package directory.)
2440
2441 This is also how the real TriBITS repository and package `DataTransferKit`_ is
2442 set up (at least that is the way it was when this document was first written).
2443
2444 .. _DataTransferKit: https://github.com/CNERG/DataTransferKit
2445
2446 However, to maximize flexibility, it is recommended that a TriBITS package and
2447 its TriBITS repository **not** share the same directory or the same name.
2448 This allows a TriBITS repository to define more packages later.
2449
2450 .. _TriBITS Package == TriBITS Repository == TriBITS Project:
2451
2452 **TriBITS Package Dir == TriBITS Repository Dir == TriBITS Project Dir**: In
2453 the extreme, it is possible to collapse a single TriBITS package, repository,
2454 and project into the same base source directory. They can also share the same
2455 name for the package, repository and package. One example of this is the
2456 ``TriBITS`` project and `The TriBITS Test Package`_ themselves, which are both
2457 rooted in the base ``TriBITS/`` source directory of the stand-alone TriBITS
2458 repository. There are a few restrictions and modifications needed to get this
2459 to work:
2460
2461 * The base ``CMakeLists.txt`` file must be modified to allow it to be
2462 processed both as the base project ``CMakeLists.txt`` file and as the
2463 package's base ``CMakeLists.txt`` file. In the case of
2464 ``TriBITS/CMakeLists.txt``, a big if statement is used.
2465
2466 .. ToDo: Create a new macro called tribits_project_and_package() that will
2467 .. automatically take care of the details of a TriBITS package also being a
2468 .. TriBITS project.
2469
2470 Other than that simple modification to the top-level ``CMakeLists.txt`` file,
2471 a TriBITS project, repository, and package can all be rooted in the same
2472 source directory.
2473
2474 The primary use case for collapsing a project, repository, and package into a
2475 single base source directory would be to support the stand-alone build of a
2476 TriBITS package as its own entity that uses an independent installation of the
2477 TriBITS (or a minimal snapshot of TriBITS). If a given TriBITS package has no
2478 required `upstream`_ TriBITS package dependencies and minimal external
2479 package/TPL dependencies (or only uses `Standard TriBITS TPLs`_ or `Common
2480 TriBITS TPLs`_ already defined in the ``tribits/core/std_tpls/`` or
2481 ``tribits/common_tpls/`` directories), then creating a stand-alone project
2482 build of a single TriBITS package requires fairly little extra overhead or
2483 duplication.
2484
2485
2486 Standard and Common TPLs
2487 ------------------------
2488
2489 While a TriBITS Repository can define their own external packages/TPLs and
2490 their own TPL find modules (see `TriBITS External Package/TPL`_), the TriBITS
2491 source tree contains TriBITS find modules for a few different standard TPLs
2492 and common TPLs. `Standard TriBITS TPLs`_ are integral to the TriBITS system
2493 itself while `Common TriBITS TPLs`_ are TPL that are used in several different
2494 TriBITS Repositories and are contained in TriBITS for convenience and
2495 uniformity.
2496
2497
2498 Standard TriBITS TPLs
2499 +++++++++++++++++++++
2500
2501 TriBITS contains find modules for a few standard TPLs integral to the TriBITS
2502 system. The standard TriBITS TPLs are contained under the directory::
2503
2504 tribits/core/std_tpls/
2505
2506 The current list of standard TriBITS TPL find modules is:
2507
2508 .. include:: TribitsStandardTPLsList.txt
2509 :literal:
2510
2511 The TPLs ``MPI`` and ``CUDA`` are standard because they are special in that
2512 they define compilers and other special tools that are used in
2513 `tribits_add_library()`_, `tribits_add_executable()`_, `tribits_add_test()`_
2514 and other commands.
2515
2516 These standard TPLs are used in a `<repoDir>/TPLsList.cmake`_ file as::
2517
2518 tribits_repository_define_tpls(
2519 MPI "${${PROJECT_NAME}_TRIBITS_DIR}/core/std_tpls/" PT
2520 CUDA "${${PROJECT_NAME}_TRIBITS_DIR}/core/std_tpls/" ST
2521 ...
2522 )
2523
2524
2525 Common TriBITS TPLs
2526 +++++++++++++++++++
2527
2528 TriBITS also contains find modules for several TPLs that are used across many
2529 independent TriBITS repositories. The goal of maintaining these under TriBITS
2530 is to enforce conformity in case these independent repositories are combined
2531 into a single meta-project.
2532
2533 The common TriBITS TPLs are contained under the directory::
2534
2535 tribits/common_tpls/
2536
2537 The current list of common TriBITS TPL find modules is:
2538
2539 .. include:: TribitsCommonTPLsList.txt
2540 :literal:
2541
2542 Common TPLs are used in a `<repoDir>/TPLsList.cmake`_ file as::
2543
2544 tribits_repository_define_tpls(
2545 BLAS "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/" PT
2546 LAPACK "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/" PT
2547 ...
2548 )
2549
2550 By using a standard TPL definition, it is guaranteed that the TPL used will be
2551 consistent with all of the TriBITS packages that depend on these TPLs in case
2552 they are combined into a single project.
2553
2554 Note that just because packages in two different TriBITS repositories
2555 reference the same TPL does not necessarily mean that it needs to be moved
2556 into the TriBITS source tree under ``tribits/common_tpls/``. For example, if
2557 the TPL ``QT`` is defined in an `upstream`_ repository (e.g. Trilinos), then a
2558 package in a `downstream`_ repository can list a dependency on the TPL ``QT``
2559 without having to define its own ``QT`` TPL in its repository's
2560 `<repoDir>/TPLsList.cmake`_ file. For more details, see `TriBITS TPL`_.
2561
2562 .. Where to set variables?
2563 .. -----------------------
2564
2565 .. A TriBITS project, repository, package have a number of files in which
2566 variables can be set and commands can be called in order to affect how the
2567 project is defined and configured. With so many files that can be included by
2568 TriBITS, it can be difficult to know where the right place is to set a given
2569 set of variables. The primary considerations for where to set a variable
2570 depend on:
2571
2572 .. ToDo: Describe considerations on where to set variables ...
2573
2574
2575 .. _TriBITS-Compliant Package:
2576
2577 TriBITS-Compliant Packages
2578 ==========================
2579
2580 At the CMake build-system level, there are just a few key requirements that a
2581 TriBITS package has for its upstream dependent packages when it is being
2582 configured to be built. These requirements apply whether the upstream package
2583 is defined internally in the current CMake project or provided externally and
2584 pulled in through ``find_package(<Package>)``.
2585
2586 The common requirements for both internal and external **TriBITS-compliant
2587 packages** as imposed by downstream TriBITS internal packages are:
2588
2589 * Provides the (INTERFACE) target ``<Package>::all_libs`` which provides all
2590 usage requirements for the libraries of ``<Package>`` through the target
2591 properties:
2592
2593 * ``INTERFACE_LINK_LIBRARIES``: The library files needed link against (or
2594 upstream library targets including ``<UpstreamPackage>::all_libs`` for all
2595 its upstream packages)
2596
2597 * ``INTERFACE_INCLUDE_DIRECTORIES``: Include directories to all public header files
2598
2599 * ``INTERFACE_COMPILE_OPTIONS``: Required compiler options
2600
2601 * ``INTERFACE_COMPILE_DEFINITIONS``: Required compiler/macro definitions
2602
2603 * ``INTERFACE_LINK_OPTIONS``: Required compiler/macro definitions
2604
2605 * Any other ``INTERFACE_XXX`` or ``IMPORTED_XXX`` target property needed to
2606 correctly use the libraries for package ``<Package>``.
2607
2608 * Provides namespaced variables ``<Package>_ENABLE_<UpstreamPackage>`` set to
2609 ``TRUE`` or ``FALSE`` for all of the upstream required and optional
2610 dependencies for the package ``<Package>``.
2611
2612 * [Optional] Provides namespaced variables of the form
2613 ``<Package>_<SOME_INFO>`` for any other information about the configuration
2614 of package ``<Package>`` that may need to be known by a downstream TriBITS
2615 package.
2616
2617 * [Optional] Provides any (namespaced by ``<package>_`` or ``<Package>_``)
2618 CMake macros or functions that downstream CMake packages may need to use the
2619 upstream package ``<Package>``.
2620
2621 * [Optional] All of the upstream dependencies (listed in the
2622 ``INTERFACE_LINK_LIBRARIES`` property recursively) are also
2623 `TriBITS-compliant packages`_
2624
2625 The TriBITS system will also set the variable:
2626
2627 * ``<Package>_IS_TRIBITS_COMPLIANT``: Set to ``TRUE``
2628
2629 for all packages that are determined to be TriBITS-compliant packages and
2630 satisfy the above criteria.
2631
2632 The above are all that is needed by downstream TriBITS packages to build and
2633 link against their upstream dependencies.
2634
2635 Additional requirements are placed on TriBITS-compliant packages depending on
2636 if they are defined as internal CMake packages (i.e. `TriBITS-compliant
2637 internal packages`_) or are pulled in as external pre-built/pre-installed
2638 packages (i.e. `TriBITS-compliant external packages`_).
2639
2640
2641 .. _TriBITS-Compliant Internal Packages:
2642
2643 TriBITS-Compliant Internal Packages
2644 -----------------------------------
2645
2646 For TriBITS packages that are defined, built, and installed from a TriBITS
2647 CMake project, there are an additional set of requirements for them to
2648 behavior correctly with respect to other TriBITS packages.
2649
2650 The requirements for **TriBITS-compliant internal packages** are:
2651
2652 * All of the requirements for a `TriBITS-Compliant Package`_.
2653
2654 * At the end of configuration and generation, writes out a `TriBITS-Compliant
2655 External Package`_ file ``<Package>Config.cmake`` and supporting files under
2656 the build directory ``<buildDir>/cmake_packages/<Package>/`` allowing the
2657 built (but not installed) package to be used by downstream CMake
2658 packages/projects.
2659
2660 * Provides an install target to create a `TriBITS-Compliant External Package`_
2661 file ``<Package>Config.cmake`` and supporting files under the install
2662 directory ``<installDir>/lib/cmake/<Package>/`` allowing the installed
2663 package to be used by downstream CMake packages/projects.
2664
2665 * [Optional] All of the upstream dependencies (recursively) are also
2666 `TriBITS-compliant packages`_.
2667
2668 If a TriBITS package provides any CTest tests/examples, then it must also
2669 satisfy the following requirements:
2670
2671 * Test names must be prefixed with the package name ``<Package>_``.
2672
2673 * The package name ``<Package>`` must be appended to the ``LABELS`` test
2674 property list.
2675
2676 * Tests should only be added if the variable ``<Package>_ENABLE_TESTS`` is
2677 true.
2678
2679 * Examples (that run as CTest tests) should only be added if the variable
2680 ``<Package>_ENABLE_EXAMPLES`` is true.
2681
2682 * The ``PROCESSORS`` test property and other test properties must be set in a
2683 way consistent with `tribits_add_test()`_ so as to run in parallel with
2684 other tests and not overwhelm the computing resources on the machine.
2685
2686 * The test ``<fullTestName>`` must not be added if the cache variable
2687 ``<fullTestName>_DISABLE`` is set to ``TRUE`` or if the cache variable
2688 ``<fullTestName>_SET_DISABLED_AND_MSG`` is set to non-empty (and the message
2689 string should be printed to STDOUT).
2690
2691 TriBITS internal packages that are defined using the TriBITS framework using
2692 the TriBITS-provided macros and functions such as `tribits_add_library()`_ and
2693 have tests defined using the functions `tribits_add_test()`_ and
2694 `tribits_add_advanced_test()`_ are automatically `TriBITS-compliant internal
2695 packages`_. And when these TriBITS-implemented internal packages are
2696 installed, they automatically provide `TriBITS-compliant external packages`_.
2697 But it is possible for a CMake package to write its own raw CMake code to
2698 satisfy these basic requirements for both internal and (installed) external
2699 packages.
2700
2701
2702 .. _TriBITS-Compliant External Package:
2703
2704 TriBITS-Compliant External Packages
2705 -----------------------------------
2706
2707 For packages that are installed on the system and not built in the current
2708 CMake project, a streamlined type of `TriBITS External Package/TPL`_ is a
2709 *TriBITS-compliant external package*. These special types of external
2710 package's don't need to provide a `FindTPL<tplName>.cmake`_ find module.
2711 Instead, they are fully defined by calling ``find_package(<Package>)`` or
2712 ``include(<someBaseDir>/<Package>Config.cmake)`` to load their
2713 ``<Package>Config.cmake`` package config file.
2714
2715 The requirements for **TriBITS-compliant external packages** are:
2716
2717 * All of the requirements for a `TriBITS-Compliant Package`_.
2718
2719 * Defined by an installed ``<Package>Config.cmake`` file that provides
2720 IMPORTED targets and ``set()`` statements for all of the needed variables.
2721
2722 * Provides CMake variables:
2723
2724 * ``<Package>_CONFIG`` or
2725 ``<Package>_TRIBITS_COMPLIANT_PACKAGE_CONFIG_FILE``: Points to the file
2726 ``<Package>Config.cmake`` (i.e. ``${CMAKE_CURRENT_LIST_FILE}``)
2727
2728 * [Optional] All of the upstream dependencies (recursively) are also provided
2729 as `TriBITS-compliant external packages`_ with
2730 ``<UpstreamPackage>Config.cmake`` files (see above) and all of the targets
2731 and variables for a TriBITS-compliant external package are defined when the
2732 ``<Package>Config.cmake`` file is included (or pulled in with
2733 ``find_package()`` or ``find_dependency()``).
2734
2735 NOTE: TriBITS-compliant external packages that provide TriBITS-compliant
2736 external packages for all of their upstream dependencies are said to be *fully
2737 TriBITS-compliant external packages* while those that support the minimal
2738 requirements are said to be *minimally TriBITS-compliant external packages*.
2739 The TriBITS external package/TPL system is robust enough to deal with
2740 minimally TriBITS-compliant external packages. Any TriBITS external
2741 packages/TPLs upstream from a minimally TriBITS-compliant external package
2742 will be found again in the current TriBITS project. (In these cases, it is up
2743 to the user to make sure that the same upstream packages are found.)
2744
2745
2746 Example TriBITS Projects
2747 =========================
2748
2749 In this section, a few different example TriBITS projects and packages are
2750 previewed. Most of these examples exist in the TriBITS source directory
2751 ``tribits`` itself so they are available to all users of TriBITS. These
2752 examples also provide a means to test the TriBITS system itself (see `The
2753 TriBITS Test Package`_).
2754
2755 The first example covered is the bare bones `TribitsHelloWorld`_ example
2756 project. The second example covered in detail is `TribitsExampleProject`_.
2757 This example covers all the basics for setting up a simple multi-package
2758 TriBITS project. The third example outlined is `MockTrilinos`_ which mostly
2759 exists to test the TriBITS system itself but also contains some nice examples
2760 of a few different TriBITS features and behaviors. The forth example is the
2761 `ReducedMockTrilinos`_ project which is used to demonstrate TriBITS behavior
2762 in this document. Also mentioned is the `Trilinos`_ project itself which can
2763 be a useful example of the usage of TriBITS (see disclaimers in the section
2764 `Trilinos`_). The last example mentioned is `The TriBITS Test Package`_
2765 itself which allows the TriBITS system to be tested and installed from any
2766 TriBITS project that lists it, including the ``TriBITS`` project itself
2767 (see `Coexisting Projects, Repositories, and Packages`_).
2768
2769 The directory ``tribits/examples/`` contains some other example TriBITS
2770 projects and repositories as well that are referred to in this and other
2771 documents.
2772
2773
2774 TribitsHelloWorld
2775 -----------------
2776
2777 ``TribitsHelloWorld`` is about the simplest possible TriBITS project that you
2778 can imagine and is contained under the directory::
2779
2780 tribits/examples/TribitsHelloWorld/
2781
2782 This example project contains only a single TriBITS package and no frills at
2783 all (does not support MPI or Fortran). However, it does show how minimal a
2784 `TriBITS Project`_ (which is also a `TriBITS Repository`_) and a `TriBITS
2785 Package`_ can be and still demonstrates some of the value of TriBITS over raw
2786 CMake. The simple ``HelloWorld`` package is used to compare with the raw
2787 CMakeLists.txt file in the ``RawHeloWorld`` example project in the `TriBITS
2788 Overview`_ document.
2789
2790 The directory structure for this example his shown below:
2791
2792 .. include:: TribitsHelloWorldDirAndFiles.txt
2793 :literal:
2794
2795 This has all of the required `TriBITS Project Core Files`_, `TriBITS
2796 Repository Core Files`_, and `TriBITS Package Core Files`_. It just builds a
2797 simple library, a simple executable, a test executable, and the tests them as
2798 shown by the file ``TribitsHelloWorld/hello_world/CMakeLists.txt`` which is:
2799
2800 .. include:: ../../examples/TribitsHelloWorld/hello_world/CMakeLists.txt
2801 :literal:
2802
2803 The build and test of this simple project is tested in the `The TriBITS Test
2804 Package`_ file::
2805
2806 TriBITS/test/core/ExamplesUnitTests/CMakeLists.txt
2807
2808 Note that this little example is a fully functional `TriBITS Repository`_ and
2809 can be embedded in to a larger TriBITS meta-project and be seamlessly built
2810 along with any other such TriBITS-based software.
2811
2812 .. ToDo: Put in reference to the example meta-project.
2813
2814
2815 TribitsExampleProject
2816 ----------------------
2817
2818 ``TribitsExampleProject`` in an example `TriBITS Project`_ and `TriBITS
2819 Repository`_ contained in the TriBITS source tree under::
2820
2821 tribits/examples/TribitsExampleProject/
2822
2823 When this is used as the base TriBITS project, this is the directory
2824 corresponds to ``<projectDir>`` and ``<repoDir>`` referenced in `TriBITS
2825 Project Core Files`_ and `TriBITS Repository Core Files`_, respectively.
2826
2827 Several files from this project are used as examples in the section `TriBITS
2828 Project Structure`_. Here, a fuller description is given of this project and
2829 a demonstration of how TriBITS works. From this simple example project, one
2830 can quickly see how the basic structural elements of a TriBITS project,
2831 repository, and package (and subpackage) are pulled together.
2832
2833 This simple project shows how what is listed in files:
2834
2835 * `<repoDir>/PackagesList.cmake`_,
2836 * `<packageDir>/cmake/Dependencies.cmake`_, and
2837 * `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_
2838
2839 are used to specify the packages and packages in a TriBITS project and
2840 repository. More details about the contents of the ``Dependencies.cmake``
2841 files is described in the section `Package Dependencies and Enable/Disable
2842 Logic`_.
2843
2844 The name of this project ``PROJECT_NAME`` is given in its
2845 ``TribitsExampleProject/ProjectName.cmake`` file:
2846
2847 .. include:: ../../examples/TribitsExampleProject/ProjectName.cmake
2848 :literal:
2849
2850 The variable ``PROJECT_NAME=TribitsExProj`` is used to prefix (using
2851 ``"${PROJECT_NAME}_"``) all of the project's global TriBITS variables like
2852 ``TribitsExProj_ENABLE_TESTS``, ``TribitsExProj_ENABLE_ALL_PACKAGES``, etc.
2853
2854 .. _TribitsExampleProject Files and Directories:
2855
2856 The directory structure and key files for this example project are shown in
2857 the partial list of `TribitsExampleProject Files and Directories`_ below::
2858
2859 TribitsExampleProject/
2860 CMakeLists.txt
2861 Copyright.txt
2862 PackagesList.cmake
2863 ProjectName.cmake
2864 project-checkin-test-config.py
2865 TPLsList.cmake
2866 Version.cmake
2867 ...
2868 cmake/
2869 CallbackDefineProjectPackaging.cmake
2870 CallbackDefineRepositoryPackaging.cmake
2871 CallbackSetupExtraOptions.cmake
2872 packages/
2873 simple_cxx/
2874 CMakeLists.txt
2875 cmake/
2876 CheckFor__int64.cmake
2877 Dependencies.cmake
2878 SimpleCxx_config.h.in
2879 src/
2880 CMakeLists.txt
2881 SimpleCxx_HelloWorld.cpp
2882 SimpleCxx_HelloWorld.hpp
2883 test/
2884 CMakeLists.txt
2885 SimpleCxx_HelloWorld_Tests.cpp
2886 mixed_lang/ ...
2887 with_subpackages/
2888 CMakeLists.txt
2889 cmake/
2890 Dependencies.cmake
2891 A/
2892 CMakeLists.txt
2893 cmake/
2894 Dependencies.cmake
2895 ...
2896 B/ ...
2897 C/ ...
2898 wrap_external/ ...
2899
2900 Above, the sub-directories under ``packages/`` are sorted according to the
2901 order listed in the ``TribitsExampleProject/PackagesList.cmake`` file:
2902
2903 .. include:: ../../examples/TribitsExampleProject/PackagesList.cmake
2904 :literal:
2905
2906 From this file, we get the list of top-level packages ``SimpleCxx``,
2907 ``MixedLang``, ``WithSubpackages``, and ``WrapExternal`` (and their base
2908 package directories and testing group, see `<repoDir>/PackagesList.cmake`_).
2909 (NOTE: By default the package ``InsertedPkg`` is not defined because its
2910 directory is missing, see `How to insert a package into an upstream repo`_.)
2911
2912 A full listing of package files in `TribitsExampleProject Files and
2913 Directories`_ is only shown for the ``SimpleCxx`` package directory
2914 ``packages/simple_cxx/``. For this package, ``<packageDir> =
2915 <repoDir>/packages/simple_cxx`` and ``PACKAGE_NAME = SimpleCxx``. As
2916 explained in `TriBITS Package Core Files`_, the files
2917 `<packageDir>/cmake/Dependencies.cmake`_ and `<packageDir>/CMakeLists.txt`_
2918 must exist for every package directory listed in
2919 `<repoDir>/PackagesList.cmake`_ and we see these files under in the directory
2920 ``packages/simple_cxx/``. The package ``SimpleCxx`` does not have any
2921 `upstream`_ package dependencies.
2922
2923 Now consider the example top-level package ``WithSubpackages`` which,
2924 as the name suggests, is broken down into subpackages. The
2925 ``WithSubpackages`` dependencies file::
2926
2927 TribitsExampleProject/packages/with_subpackages/cmake/Dependencies.cmake
2928
2929 with contents:
2930
2931 .. include:: ../../examples/TribitsExampleProject/packages/with_subpackages/cmake/Dependencies.cmake
2932 :literal:
2933
2934 references the three subpackages with sub-directories ``<spkgDir>`` = ``A``,
2935 ``B``, and ``C`` under the parent package directory
2936 ``packages/package_with_packages/`` which are shown in `TribitsExampleProject
2937 Files and Directories`_. This gives another set of three packages
2938 ``WithSubpackagesA``, ``WithSubpackagesB``,
2939 and ``WithSubpackagesC``. Combining ``<packageDir> =
2940 packages/package_with_packages`` and ``<spkgDir>`` for each subpackage gives
2941 the subpackage directories::
2942
2943 TribitsExampleProject/packages/with_subpackages/a/
2944 TribitsExampleProject/packages/with_subpackages/b/
2945 TribitsExampleProject/packages/with_subpackages/c/
2946
2947 Together with the top-level parent package ``WithSubpackages``
2948 itself, this top-level package provides four `TriBITS Packages`_ giving the
2949 final list of packages provided by this TriBITS repo as::
2950
2951 SimpleCxx MixedLang WithSubpackagesA WithSubpackagesB WithSubpackagesC \
2952 WithSubpackages WrapExternal 7
2953
2954 The above list of packages is printed (with the number of packages printed
2955 at the end) by TriBITS to the ``cmake`` stdout on the line starting with
2956 ``"Final set of non-enabled packages:"`` when no packages are enabled (see
2957 `Selecting the list of packages to enable`_). (Note that TriBITS does not put
2958 in line-brakes with continuation characters ``"\"`` as shown above.) TriBITS
2959 defines enable/disable cache variables for each of these defined packages
2960 like ``TribitsExProj_ENABLE_SimpleCxx`` and
2961 ``TribitsExProj_ENABLE_WithSubpackagesA``, and defines all
2962 the variables listed in `TriBITS Package Cache Variables`_ that are settable
2963 by the users or by the dependency logic described in section `Package
2964 Dependencies and Enable/Disable Logic`_.
2965
2966 When starting a new TriBITS project, repository, or package, one should
2967 consider basing these on the examples in ``TribitsExampleProject``. In fact,
2968 the skeletons for any of the
2969
2970 * `TriBITS Project Core Files`_,
2971 * `TriBITS Repository Core Files`_,
2972 * `TriBITS Package Core Files`_, or
2973 * `TriBITS Subpackage Core Files`_
2974
2975 should be copied from this example project as they represent best practice
2976 when using TriBITS for the typical use cases.
2977
2978
2979 TribitsExampleProject2
2980 ----------------------
2981
2982 ``TribitsExampleProject2`` in an example `TriBITS Project`_ and `TriBITS
2983 Repository`_ contained in the TriBITS source tree under::
2984
2985 tribits/examples/TribitsExampleProject2/
2986
2987 This example TriBITS project provides some examples for a few other features
2988 and testing scenarios. It contains three internal packages ``Package1``,
2989 ``Package2``, and ``Package3`` as shown in its ``PackagesList.cmake`` file:
2990
2991 .. include:: ../../examples/TribitsExampleProject2/PackagesList.cmake
2992 :literal:
2993
2994 and supports four external packages/TPLs ``Tpl1``, ``Tpl2``, ``Tpl3``, and
2995 ``Tpl4`` as shown in its ``TPLsList.cmake`` file:
2996
2997 .. include:: ../../examples/TribitsExampleProject2/TPLsList.cmake
2998 :literal:
2999
3000
3001 MockTrilinos
3002 -------------
3003
3004 The TriBITS project ``MockTrilinos`` is contained under the directory::
3005
3006 tribits/examples/MockTrilinos/
3007
3008 This TriBITS project is not a full TriBITS project (i.e. it does not build
3009 anything). Instead, it is primarily used to test the TriBITS system using
3010 tests defined in the `The TriBITS Test Package`_. The ``MockTrilinos``
3011 project is actually given the name ``PROJECT_NAME = Trilinos`` and contains a
3012 subset of Trilinos packages with slightly modified dependencies from a real
3013 version of the Trilinos project from May 2009. The list of packages in::
3014
3015 tribits/examples/MockTrilinos/PackagesList.cmake
3016
3017 is:
3018
3019 .. include:: ../../examples/MockTrilinos/PackagesList.cmake
3020 :literal:
3021
3022 All of the package directories listed above have
3023 `<packageDir>/cmake/Dependencies.cmake`_ files but generally do not have
3024 `<packageDir>/CMakeLists.txt`_ files since most of usage of ``MockTrilinos``
3025 just involves the testing of the algorithms and behaviors described in the
3026 section `Package Dependencies and Enable/Disable Logic`_.
3027
3028 ``MockTrilinos`` also contains a number of extra TriBITS repositories used in
3029 various tests. These extra repositories offer examples of different types of
3030 TriBITS repositories like:
3031
3032 * ``extraRepoOnePackage``: Contains just the single package
3033 ``extraRepoOnePackage`` which is defined in the base repository directory.
3034
3035 * ``extraRepoOnePackageThreeSubpackages``: Contains just the single package
3036 ``extraRepoOnePackageThreeSubpackages`` which is defined in the base
3037 repository directory but is broken up into subpackages.
3038
3039 * ``extraRepoTwoPackages``: Contains just two packages but provides an example
3040 of defining multiple repositories with possible missing required and
3041 optional `upstream`_ packages (see `Multi-Repository Support`_).
3042
3043 * ``extraTrilinosRepo``: Just a typical extra repo with add-on packages and
3044 external packages/TPLs that depends on a few upstream ``MockTrilinos``
3045 packages.
3046
3047 New test extra repositories are added when new types of tests are needed that
3048 would require new package and TPL dependency structures since existing
3049 dependency tests based on ``MockTrilinos`` are expensive to change by their
3050 very nature.
3051
3052 The primary reason that the ``MockTrilinos`` test project is mentioned in this
3053 developers guide is because it contains a variety of packages, subpackages,
3054 and TPLs with a variety of different types of dependencies. This variety is
3055 is needed to more fully test the TriBITS system but this project and the tests
3056 also serve as examples and extra documentation for the behavior of the TriBITS
3057 system. Several of the dependency-related examples referenced in this
3058 document come from ``MockTrilinos``.
3059
3060 Most of the dependency tests involving ``MockTrilinos`` are specified in::
3061
3062 TriBITS/test/core/DependencyUnitTests/CMakeLists.txt
3063
3064 A great deal about the current behavior of TriBITS `Package Dependencies and
3065 Enable/Disable Logic`_ can be learned from inspecting these tests. There are
3066 also some faster-running unit tests involving ``MockTrilinos`` defined in the
3067 file::
3068
3069 TriBITS/test/core/TribitsAdjustPackageEnables_UnitTests.cmake
3070
3071
3072 ReducedMockTrilinos
3073 -------------------
3074
3075 The TriBITS project ``ReducedMockTrilinos`` is contained under the directory::
3076
3077 tribits/examples/ReducedMockTrilinos/
3078
3079 It is a scaled-down version of the `MockTrilinos`_ test project with just a
3080 handful of packages and some modified dependencies. Its primary purpose for
3081 this example project is to be used for examples in the section `Package
3082 Dependencies and Enable/Disable Logic`_ and to test a few features of the
3083 TriBITS system not covered in other tests.
3084
3085 The list of packages in::
3086
3087 tribits/examples/ReducedMockTrilinos/PackagesList.cmake
3088
3089 is:
3090
3091 .. include:: ../../examples/ReducedMockTrilinos/PackagesList.cmake
3092 :literal:
3093
3094 All of the listed packages are standard TriBITS packages except for the mock
3095 ``Thyra`` package which is broken down into subpackages. More details of this
3096 example project are described in `Package Dependencies and Enable/Disable
3097 Logic`_.
3098
3099 Trilinos
3100 --------
3101
3102 The real Trilinos project and repository itself is an advanced example for the
3103 usage of TriBITS. Almost every single-repository use case for TriBITS is
3104 demonstrated somewhere in Trilinos. While some of the usage of TriBITS in
3105 Trilinos may not be not exemplary (e.g., because it represents old usage, or
3106 was written by CMake/TriBITS beginners) it does represent real working usage.
3107 Given that Trilinos is a widely available software repository, anyone should be
3108 able to access a newer version of Trilinos and mine it for CMake and TriBITS
3109 examples.
3110
3111
3112 The TriBITS Test Package
3113 ------------------------
3114
3115 The last TriBITS example mentioned here is the TriBITS test package named
3116 (appropriately) ``TriBITS`` itself defined in the ``TriBITS`` repository. The
3117 directory for the ``TriBITS`` test package is the base TriBITS source
3118 directory ``tribits``. This allows any TriBITS project to add testing for the
3119 TriBITS system by just listing the TriBITS repository in its
3120 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file. Trilinos lists the
3121 TriBITS repository in its ``ExtraRepositoriesList.cmake`` file as:
3122
3123 tribits_project_define_extra_repositories(
3124 TriBITS "" GIT https://github.com/TriBITSPub/TriBITS "" Continuous
3125 ...
3126 )
3127
3128 No `downstream`_ TriBITS packages list a dependency on ``TriBITS`` in their
3129 `<packageDir>/cmake/Dependencies.cmake`_ files. Defining the ``TriBITS``
3130 TriBITS package in only done for running the TriBITS tests.
3131
3132 Once the ``TriBITS`` test package is added to the list of project/repository
3133 packages, it can be enabled just like any other package by adding the
3134 following to the ``cmake`` command-line options::
3135
3136 -D <Project>_ENABLE_TriBITS=ON \
3137 -D <Project>_ENABLE_TESTS=ON
3138
3139 One can then inspect the added tests prefixed by ``"TriBITS_"`` to see what
3140 tests are defined and how they are run. There is a wealth of information
3141 about the TriBITS system embedded in these tests and where documentation and
3142 these tests disagreed, believe the tests!
3143
3144
3145 Package Dependencies and Enable/Disable Logic
3146 =============================================
3147
3148 Arguably, the more important feature/aspect of the TriBITS system is the
3149 partitioning of a large software project into packages and managing the
3150 dependencies between these packages to support building, testing, and
3151 deploying different pieces as needed (see discussion of `Software Engineering
3152 Packaging Principles`_). This is especially useful in incremental CI testing
3153 of large projects. However, maintaining such dependencies is also a critical
3154 component in creating and maintaining Self-Sustaining Software (see the
3155 `TriBITS Lifecycle Model`_). The fundamental mechanism for breaking up a
3156 large software into manageable pieces is to partition the software into
3157 different `TriBITS Packages`_ and then define the dependencies between these
3158 packages (which are defined inside of the
3159 `<packageDir>/cmake/Dependencies.cmake`_ files for each package).
3160
3161 Note that the basic idea of breaking up a large set of software into pieces,
3162 defining dependencies between the pieces, and then applying algorithms to
3163 manipulate the dependency data-structures is nothing new. If fact, nearly
3164 every binary package deployment system provided in various Linux OS
3165 distributions have the concept of packages and dependencies and will
3166 automatically install all of the necessary `upstream dependencies`_ when a
3167 `downstream dependency`_ install is requested. The main difference (and the
3168 added complexity) with TriBITS is that it can handle both required and
3169 optional dependencies since it can build from source. A binary package
3170 installation system, however, typically can't support optional dependencies
3171 because only pre-built binary libraries and tools are available to install.
3172
3173 This section is organized and broken-down as follows. First, the subsection
3174 `Example ReducedMockTrilinos Project Dependency Structure`_ presents the
3175 `ReducedMockTrilinos`_ example project, describes its dependency structure,
3176 and uses it to begin to describe how TriBITS sets up and manages package
3177 dependencies. The packages in this `ReducedMockTrilinos`_ example project are
3178 used in the following subsections so one will be constantly referred back to
3179 this subsection. The following subsection `TriBITS Dependency Handling
3180 Behaviors`_ defines and describes the nitty-gritty details of the TriBITS
3181 package dependency structure and the algorithms that manipulate the various
3182 package and test enables and disables. Specific examples for the TriBITS
3183 dependency handling algorithms are given in the subsection `Example
3184 Enable/Disable Use Cases`_. Finally, the subsection
3185 `<Project>PackageDependencies.xml`_ describes the standard XML output
3186 data-structure that gets created by TriBITS that defines a project's package
3187 dependencies.
3188
3189
3190 Example ReducedMockTrilinos Project Dependency Structure
3191 --------------------------------------------------------
3192
3193 To demonstrate the TriBITS package dependency handling system, the small
3194 simple `ReducedMockTrilinos`_ project is used. The list of packages for this
3195 project is defined in the file ``ReducedMockTrilinos/PackagesList.cmake`` (see
3196 `<repoDir>/PackagesList.cmake`_) which contents:
3197
3198 .. include:: ../../examples/ReducedMockTrilinos/PackagesList.cmake
3199 :literal:
3200
3201 All of the listed packages are standard TriBITS packages except for the mock
3202 ``Thyra`` package which is broken down into subpackages as shown in
3203 ``packages/thyra/cmake/Dependnecies.cmake`` (see
3204 `<packageDir>/cmake/Dependencies.cmake`_) which is:
3205
3206 .. include:: ../../examples/ReducedMockTrilinos/packages/thyra/cmake/Dependencies.cmake
3207 :literal:
3208
3209 This gives the full list of top-level TriBITS packages::
3210
3211 Teuchos RTOp Epetra Triutils EpetraExt Thyra
3212
3213 Adding in the subpackages defined in the top-level ``Thyra`` package, the full
3214 set of `TriBITS Packages`_ for this project is::
3215
3216 Teuchos RTOp Epetra Triutils EpetraExt ThyraCoreLibs ThyraGoodStuff \
3217 ThyraCrazyStuff ThyraEpetra ThyraEpetraExt Thyra
3218
3219 Note that one can see this full list of top-level packages and packages in
3220 the ``cmake`` configure output lines starting with::
3221
3222 Final set of non-enabled top-level packages:
3223 Final set of non-enabled packages:
3224
3225 respectively, when configuring with no package enables as shown in the example
3226 `Default configure with no packages enabled on input`_.
3227
3228 The list of `TriBITS External Packages/TPLs`_ for this example project given
3229 in the file ``ReducedMockTrilinos/TPLsList.cmake`` (see
3230 `<repoDir>/TPLsList.cmake`_) which is:
3231
3232 .. include:: ../../examples/ReducedMockTrilinos/TPLsList.cmake
3233 :literal:
3234
3235 Take note of the `Package Test Group`_ (i.e. `PT`_, `ST`_, or `EX`_) assigned
3236 to each package as it plays a significant role in how the TriBITS dependency
3237 system handles enables and disables.
3238
3239 The dependency structure of this simple TriBITS project is shown below in
3240 `ReducedMockTrilinos Dependencies`_.
3241
3242 .. _ReducedMockTrilinos Dependencies:
3243
3244 **ReducedMockTrilinos Dependencies:**
3245
3246 .. include:: ReducedMockTrilinosOutput/ExpectedDependencies.txt
3247 :literal:
3248
3249 The above dependency structure printout is produced by configuring with
3250 ``${PROJECT_NAME}_DUMP_PACKAGE_DEPENDENCIES=ON`` (which also results in more
3251 dependency information than what is shown above, e.g. like computed forward
3252 package dependencies). Note that the top-level package ``Thyra`` is shown
3253 to depend on its subpackages (not the other way around). (Many people are
3254 confused about the nature of the dependencies between packages and
3255 subpackages. See `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_ for more
3256 discussion.)
3257
3258 .. ToDo: Show diagram with this dependency structure.
3259
3260 A number of user-settable CMake cache variables determine what packages and
3261 what tests and examples get enabled. These cache variables are described in
3262 `Selecting the list of packages to enable`_ and are described below. Also,
3263 the assigned `Package Test Group`_ (i.e. `PT`_, `ST`_, and `EX`_) also affects
3264 what packages get enabled or disabled.
3265
3266 Any of these packages can be enabled or disabled with
3267 ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=(ON|OFF)`` (the default enable is
3268 typically empty ``""``, see `PT/ST packages given default unset
3269 enable/disable state`_). For ``ReducedMockTrilinos``, this gives the
3270 enable/disable cache variables (with the initial default values)::
3271
3272 Trilinos_ENABLE_Teuchos=""
3273 Trilinos_ENABLE_RTOp""
3274 Trilinos_ENABLE_Epetra=""
3275 Trilinos_ENABLE_Triutils=""
3276 Trilinos_ENABLE_EpetraExt=""
3277 Trilinos_ENABLE_ThyraCore=""
3278 Trilinos_ENABLE_ThyraGoodStuff=""
3279 Trilinos_ENABLE_ThyraCrazyStuff="OFF" # Because it is 'EX'
3280 Trilinos_ENABLE_ThyraEpetra=""
3281 Trilinos_ENABLE_ThyraEpetraExt=""
3282
3283 Every TriBITS package is assumed to have tests and/or examples so TriBITS
3284 defines the following cache variables as well (with the initial default
3285 values)::
3286
3287 Teuchos_ENABLE_TESTS=""
3288 RTOp_ENABLE_TESTS=""
3289 Epetra_ENABLE_TESTS=""
3290 Triutils_ENABLE_TESTS=""
3291 EpetraExt_ENABLE_TESTS=""
3292 ThyraCoreLibs_ENABLE_TESTS=""
3293 ThyraGoodStuff_ENABLE_TESTS=""
3294 ThyraEpetra_ENABLE_TESTS=""
3295 ThyraEpetraExt_ENABLE_TESTS=""
3296 Thyra_ENABLE_TESTS=""
3297
3298 NOTE: TriBITS only sets the variables ``<TRIBITS_PACKAGE>_ENABLE_TESTS`` into
3299 the cache if the package ``<TRIBITS_PACKAGE>`` becomes enabled at some
3300 point. This cuts down the clutter in the CMake cache for large projects with
3301 lots of packages where the user only enables a subset of the packages.
3302
3303 NOTE: TriBITS also defines the cache variables
3304 ``<TRIBITS_PACKAGE>_ENABLE_EXAMPLES`` for each enabled TriBITS package which
3305 is handled the same way as the ``<TRIBITS_PACKAGE>_ENABLE_TEST`` variables.
3306
3307 Also, every defined external package/TPL is given its own
3308 ``TPL_ENABLE_<TRIBITS_TPL>`` enable/disable cache variable. For the TPLs in
3309 ``ReducedMockTrilinos``, this gives the enable/disable cache variables (with
3310 default values)::
3311
3312 TPL_ENABLE_MPI=""
3313 TPL_ENABLE_BLAS=""
3314 TPL_ENABLE_LAPACK=""
3315 TPL_ENABLE_Boost=""
3316 TPL_ENABLE_UMFPACK=""
3317 TPL_ENABLE_AMD=""
3318 TPL_ENABLE_PETSC=""
3319
3320 In addition, for every optional package dependency, TriBITS defines a cache
3321 variable ``<TRIBITS_PACKAGE>_ENABLE_<OPTIONAL_DEP>``. For the optional
3322 dependencies shown in `ReducedMockTrilinos Dependencies`_, that gives the
3323 additional cache variables (with default values)::
3324
3325 Teuchos_ENABLE_Boost=""
3326 Teuchos_ENABLE_MPI=""
3327 Teuchos_ENABLE_Boost=""
3328 Epetra_ENABLE_MPI=""
3329 EpetraExt_ENABLE_Triutils=""
3330 EpetraExt_ENABLE_UMFPACK=""
3331 EpetraExt_ENABLE_AMD=""
3332 EpetraExt_ENABLE_PETSC=""
3333 Thyra_ENABLE_ThyraGoodStuff=""
3334 Thyra_ENABLE_ThyraCrazyStuff=""
3335 Thyra_ENABLE_ThyraEpetra=""
3336 Thyra_ENABLE_ThyraEpetraExt=""
3337
3338 The above optional package-specific cache variables allow one to control
3339 whether or not support for `upstream`_ dependency X is turned on in package Y
3340 independent of whether or not X and Y are themselves both enabled. For
3341 example, if the packages ``Triutils`` and ``EpetraExt`` are both enabled, one
3342 can explicitly disable support for the optional dependency ``Triutils`` in
3343 ``EpetraExt`` by setting ``EpetraExt_ENABLE_Triutils=OFF``. One may want to
3344 do this for several reasons but the bottom line is that this gives the user
3345 more detailed control over package dependencies. See the `TriBITS Dependency
3346 Handling Behaviors`_ and `Explicit disable of an optional package
3347 dependency`_ for more discussion and examples.
3348
3349 Before getting into specific `Example Enable/Disable Use Cases`_, some of the
3350 `TriBITS Dependency Handling Behaviors`_ are first defined below.
3351
3352
3353 TriBITS Dependency Handling Behaviors
3354 -------------------------------------
3355
3356 Below, some of the rules and behaviors of the TriBITS dependency management
3357 system are described. Examples refer to the `Example ReducedMockTrilinos
3358 Project Dependency Structure`_. More detailed examples of these behaviors are
3359 given in the section `Example Enable/Disable Use Cases`_.
3360
3361 In brief, the rules/behaviors of the TriBITS package dependency management
3362 system are:
3363
3364 0) `No circular dependencies of any kind are allowed`_
3365 1) `PT/ST packages given default unset enable/disable state`_
3366 2) `EX packages disabled by default`_
3367 3) `Package enable triggers auto-enables of upstream dependencies`_
3368 4) `Package disable triggers auto-disables of downstream dependencies`_
3369 5) `PT/ST TPLs given default unset enable/disable state`_
3370 6) `EX TPLs given default unset enable/disable state`_
3371 7) `Required TPLs are auto-enabled for enabled packages`_
3372 8) `Optional TPLs only enabled explicitly by the user`_
3373 9) `Disables trump enables where there is a conflict`_
3374 10) `Enable/disable of parent package is enable/disable for subpackages`_
3375 11) `Enable/disable of parent package tests/examples is enable/disable for subpackages tests/examples`_
3376 12) `Subpackage enable does not auto-enable the parent package`_
3377 13) `Support for optional package/TPL is enabled by default`_
3378 14) `Support for optional package can be explicitly disabled`_
3379 15) `Explicit enable of optional package/TPL support auto-enables package/TPL`_
3380 16) `ST packages only auto-enabled if ST code is enabled`_
3381 17) `<Project>_ENABLE_ALL_FORWARD_DEP_PACKAGES enables downstream packages/tests`_
3382 18) `<Project>_ENABLE_ALL_PACKAGES enables all PT (cond. ST) packages`_
3383 19) `<Project>_ENABLE_TESTS only enables explicitly enabled package tests`_
3384 20) `If no packages are enabled, nothing will get built`_
3385 21) `TriBITS prints all enables and disables to stdout`_
3386 22) `TriBITS auto-enables/disables done using non-cache local variables`_
3387
3388 In more detail, these rules/behaviors are:
3389
3390 .. _No circular dependencies of any kind are allowed:
3391
3392 0) **No circular dependencies of any kind are allowed**: The zeroth rule of
3393 the TriBITS dependency management system is that no TriBITS package (or
3394 its tests) can declare a dependency on a `downstream`_ package, period!
3395 This is one of the most basic `Software Engineering Packaging Principles`_
3396 stated as the *ADP (Acyclic Dependencies Principle)*, i.e. "Allow no cycles
3397 in the package dependency graph". By default, the TriBITS system will
3398 check for circular dependencies and will fail the configure if any are
3399 found. For a more detailed discussion of circular package dependencies, see
3400 `Design Considerations for TriBITS`_.
3401
3402 .. _PT/ST packages given default unset enable/disable state:
3403
3404 1) **PT/ST packages given default unset enable/disable state**: A package
3405 ``<TRIBITS_PACKAGE>`` with testing group ``PT`` or ``ST`` is given an
3406 **unset enable/disable state by default**
3407 (i.e. ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=""``). For example, the
3408 ``PT`` package ``Teuchos`` is not enabled or disabled by default and is
3409 given the initial value ``Trilinos_ENABLE_Teuchos=""``. For an example,
3410 see `Default configure with no packages enabled on input`_. This allows
3411 ``PT``, and ``ST`` packages to be enabled or disabled using other logic
3412 defined by TriBITS which is described below. TriBITS defines persistent
3413 cache variables for these with the default value of empty "". Therefore,
3414 if the user or other CMake code does not hard enable or disable one of
3415 these variables, then on future configures it will be defined but have the
3416 value of empty "".
3417
3418 .. _EX packages disabled by default:
3419
3420 2) **EX packages disabled by default**: An package ``<TRIBITS_PACKAGE>``
3421 with testing group ``EX`` is **disabled by default**
3422 (i.e. ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=OFF``). For an example,
3423 see `Default configure with no packages enabled on input`_. This results
3424 in all required `downstream`_ packages to be disabled by default.
3425 However, the user can explicitly set
3426 ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=ON`` for an ``EX`` package and
3427 it will be enabled (unless one of its required dependencies are not enabled
3428 for some reason). In this case, the cache variable is given the cache
3429 value of ``OFF``.
3430
3431 .. _Package enable triggers auto-enables of upstream dependencies:
3432
3433 3) **Package enable triggers auto-enables of upstream dependencies**: Any
3434 package ``<TRIBITS_PACKAGE>`` can be explicitly **enabled** by the user
3435 by setting the cache variable
3436 ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=ON``
3437 (e.g. ``Trilinos_ENABLE_EpetraExt=ON``). When an package is enabled in
3438 this way, the TriBITS system will try to enable all of the required
3439 upstream packages and TPLs defined by the package (specified in its
3440 ``Dependencies.cmake`` file). If an enabled package can't be enabled
3441 and has to be disabled, either a warning is printed or processing will stop
3442 with an error (depending on the value of
3443 ``${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES``, see `Disables
3444 trump enables where there is a conflict`_). In addition, if
3445 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=ON``, then TriBITS will try
3446 to enable all of the specified optional `PT`_ packages as well (and also
3447 optional upstream `ST`_ packages if
3448 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON``). For an example, see
3449 `Explicit enable of a package, its tests, an optional TPL, with ST
3450 enabled`_.
3451
3452 .. _Package disable triggers auto-disables of downstream dependencies:
3453
3454 4) **Package disable triggers auto-disables of downstream dependencies**: Any
3455 package ``<TRIBITS_PACKAGE>`` can be explicitly **disabled** by the user by
3456 setting the cache variable ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=OFF``
3457 (or ``TPL_ENABLE_<TRIBITS_PACKAGE>=OFF`` for an external package/TPL)
3458 (e.g. ``Trilinos_ENABLE_Teuchos=OFF``, ``TPL_ENABLE_BLAS=OFF``). When an
3459 package is explicitly disabled, it will result in the disable of all
3460 dependent `downstream`_ external packages/TPLs and internal packages that
3461 have required dependency on it. It will also disable optional support for
3462 the disabled packages in downstream packages that list it as an optional
3463 dependency. For an example, see `Explicit disable of a package`_.
3464
3465 .. _PT/ST TPLs given default unset enable/disable state:
3466
3467 5) **PT/ST TPLs given default unset enable/disable state**: A TriBITS TPL
3468 ``<TRIBITS_TPL>`` with testing group ``PT`` or ``ST`` is given an **unset
3469 enable/disable state by default** (i.e. ``TPL_ENABLE_<TRIBITS_TPL>=""``).
3470 For example, the ``PT`` TPL ``BLAS`` is not enabled or disabled by default
3471 (i.e. ``TPL_ENABLE_BLAS=""``). For an example, see `Default configure with
3472 no packages enabled on input`_. This allows ``PT``, and ``ST`` TPLs to be
3473 enabled or disabled using other logic.
3474
3475 .. _EX TPLs given default unset enable/disable state:
3476
3477 6) **EX TPLs given default unset enable/disable state**: A TriBITS TPL
3478 ``<TRIBITS_TPL>`` with testing group ``EX``, is given an **unset
3479 enable/disable state by default** (i.e. ``TPL_ENABLE_<TRIBITS_TPL>=""``,
3480 same as for ``PT`` and ``EX`` TPLs). For an example, see `Default
3481 configure with no packages enabled on input`_. This is different behavior
3482 than for ``EX`` packages described above which provides an initial hard
3483 disable. However, since TriBITS will never automatically enable an
3484 optional TPL (see `Optional TPLs only enabled explicitly by the user`_) and
3485 since only `downstream`_ ``EX`` packages are allowed to have a required
3486 dependencies on an ``EX`` TPL, there is no need to set the default enable
3487 for an ``EX`` TPL to ``OFF``. If an ``EX`` package has a required
3488 dependency on an ``EX`` TPL, just enabling the ``EX`` package should
3489 automatically enable the ``EX`` TPL as described in `Required TPLs are
3490 auto-enabled for enabled packages`_.
3491
3492 .. _Required TPLs are auto-enabled for enabled packages:
3493
3494 7) **Required TPLs are auto-enabled for enabled packages**: All TPLs listed
3495 as required TPL dependencies for the final set of enabled packages are
3496 **set to enabled** (i.e. ``TPL_ENABLE_<TRIBITS_TPL>=ON``), unless the
3497 listed TPLs are already explicit disabled (in which case the package
3498 would be disabled or an error would occur, see `Disables trump enables
3499 where there is a conflict`_). For example, if the ``Teuchos`` package is
3500 enabled, then that will trigger the enable of its required TPLs ``BLAS``
3501 and ``LAPACK``. For an example, see `Explicit enable of a package and its
3502 tests`_.
3503
3504 .. _Optional TPLs only enabled explicitly by the user:
3505
3506 8) **Optional TPLs only enabled explicitly by the user**: Optional TPLs,
3507 regardless of their testing group ``PT``, ``ST`` or ``EX``, will only be
3508 enabled if they are explicitly enabled by the user. For example, just
3509 because the package ``Teuchos`` is enabled, the optional TPLs ``Boost`` and
3510 ``MPI`` will **not** be enabled by default, even if
3511 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=ON``. To enable the
3512 optional TPL ``Boost``, for example, and enable support for ``Boost`` in
3513 the ``Teuchos`` package, the user must explicitly set
3514 ``TPL_ENABLE_Boost=ON``. For an example, see `Explicit enable of a
3515 package, its tests, an optional TPL, with ST enabled`_.
3516
3517 .. _Disables trump enables where there is a conflict:
3518
3519 9) **Disables trump enables where there is a conflict** and TriBITS will
3520 never override a disable in order to satisfy some dependency. For
3521 example, if the user sets ``Trilinos_ENABLE_Teuchos=OFF`` and
3522 ``Trilinos_ENABLE_RTOp=ON``, then TriBITS will **not** override the
3523 disable of ``Teuchos`` in order to satisfy the required dependency of
3524 ``RTOp``. In cases such as this, the behavior of the TriBITS dependency
3525 adjustment system will depend on the setting of the top-level user cache
3526 variable `${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES`_:
3527
3528 .. _${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON:
3529
3530 * If ``${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON``, then
3531 TriBITS will disable the explicit enable and continue on. In the above
3532 example, TriBITS will override ``Trilinos_ENABLE_RTOp=ON`` and set
3533 ``Trilinos_ENABLE_RTOp=OFF`` and print a verbose warning to the
3534 ``cmake`` stdout.
3535
3536 .. _${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=OFF:
3537
3538 * If ``${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=OFF``, then
3539 TriBITS will generate a detailed error message printed to ``cmake``
3540 stdout and then abort processing. In the above example, TriBITS will
3541 report that ``RTOp`` is enabled but the required package ``Teuchos``
3542 is disabled and therefore ``RTOp`` can't be enabled and processing must
3543 stop.
3544
3545 For an example of both behaviors, see `Conflicting explicit enable and
3546 disable`_.
3547
3548 .. _Enable/disable of parent package is enable/disable for subpackages:
3549
3550 10) **Enable/disable of parent package is enable/disable for subpackages**: An
3551 explicit enable/disable of a top-level parent package with subpackages
3552 with ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>=(ON|OFF)`` is equivalent
3553 to the explicit enable/disable of all of the parent package's subpackages.
3554 For example, explicitly setting ``Trilinos_ENABLE_Thyra=ON`` is equivalent
3555 to explicitly setting::
3556
3557 Trilinos_ENABLE_ThyraCoreLibs=ON
3558 Trilinos_ENABLE_ThyraGoodStuff=ON # Only if enabling ST code!
3559 Trilinos_ENABLE_ThyraEpetra=ON
3560 Trilinos_ENABLE_ThyraEpetraExt=ON # Only if enabling ST code!
3561
3562 (Note that ``Trilinos_ENABLE_ThyraCrazyStuff`` is **not** set to ``ON``
3563 because it is already set to ``OFF`` by default, see `EX packages
3564 disabled by default`_.) Likewise, explicitly setting
3565 ``Trilinos_ENABLE_Thyra=OFF`` is equivalent to explicitly setting all of
3566 the ``Thyra`` subpackages to ``OFF`` at the outset. For a ``PT`` example,
3567 see `Explicit enable of a package and its tests`_. For a ``ST`` example,
3568 see `Explicit enable of a package, its tests, an optional TPL, with ST
3569 enabled`_.
3570
3571 .. _Enable/disable of parent package tests/examples is enable/disable for subpackages tests/examples:
3572
3573 11) **Enable/disable of parent package tests/examples is enable/disable for
3574 subpackages tests/examples**: Setting
3575 ``<TRIBITS_PACKAGE>_ENABLE_TESTS=[ON|OFF]`` is equivalent to setting the
3576 default for ``<TRIBITS_PACKAGE><SP>_ENABLE_TESTS=[ON|OFF]`` for each
3577 subpackage ``<SP>`` of the parent package ``<TRIBITS_PACKAGE>`` (if
3578 ``<TRIBITS_PACKAGE>`` has subpackages). Same is true for
3579 ``<TRIBITS_PACKAGE>_ENABLE_EXAMPLES=[ON|OFF]`` setting the default for
3580 ``<TRIBITS_PACKAGE><SP>_ENABLE_EXAMPLES=[ON|OFF]``. In addition, setting
3581 ``<TRIBITS_PACKAGE>_ENABLE_TESTS=[ON|OFF]`` will set
3582 ``<TRIBITS_PACKAGE>_ENABLE_EXAMPLES=[ON|OFF]`` by default as well (but not
3583 vice versa).
3584
3585 .. _Subpackage enable does not auto-enable the parent package:
3586
3587 12) **Subpackage enable does not auto-enable the parent package**: Enabling an
3588 package that is a subpackage does **not** automatically enable the
3589 parent package (except for at the very end, mostly just for show). For
3590 example, enabling the package ``ThyraEpetra`` does not result in enable
3591 of the parent ``Thyra`` package, (except when
3592 ``${PROJECT_NAME}_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON`` for course). For
3593 an example, see `Explicit enable of a subpackage`_. This means that if a
3594 `downstream`_ package declares a dependency on the package
3595 ``ThyraEpetra``, but not the parent package ``Thyra``, then the ``Thyra``
3596 package (and its other subpackages and their dependencies) will not get
3597 auto-enabled. Also note that enabling a subset of the required
3598 subpackages for a parent package does not require the enable of the other
3599 required subpackages. For example, if both ``ThyraEpetra`` and
3600 ``ThyraCore`` were both required subpackages of the parent package
3601 ``Thyra``, then enabling ``ThyraCore`` does not require the enabling of
3602 ``ThyraEpetra``. In these cases where one of a parent package's
3603 subpackages is enabled for some reason, the final value of
3604 ``${PROJECT_NAME}_ENABLE_${PACKAGE_NAME}`` will be set to ``ON`` (but this
3605 does imply that all of the required subpackages will be enabled, only that
3606 the parent package will be processed.)
3607
3608 .. _Support for optional package/TPL is enabled by default:
3609
3610 13) **Support for optional package/TPL is enabled by default**: For an package
3611 ``<TRIBITS_PACKAGE>`` with an optional dependency on an `upstream`_
3612 package or TPL ``<TRIBITS_DEP_PACKAGE_OR_TPL>``, TriBITS will
3613 automatically set the intra-enable variable
3614 ``<TRIBITS_PACKAGE>_ENABLE_<TRIBITS_DEP_PACKAGE_OR_TPL>=ON`` if
3615 ``<TRIBITS_PACKAGE>`` and ``<TRIBITS_DEP_PACKAGE_OR_TPL>`` are **both
3616 enabled**. For example, if the packages ``Triutils`` and ``EpetraExt``
3617 are both enabled, then TriBITS will automatically set
3618 ``EpetraExt_ENABLE_Triutils=ON`` by default and enables support for
3619 ``Triutils`` in ``EpetraExt``. Likewise, if ``Teuchos`` and the optional
3620 TPL ``Boost`` are both enabled, then TriBITS will automatically set
3621 ``Teuchos_ENABLE_Boost=ON`` by default. This is obviously the logical
3622 behavior. See the full context for these examples in `Explicit enable of
3623 a package, its tests, an optional TPL, with ST enabled`_.
3624
3625 .. _Support for optional package can be explicitly disabled:
3626
3627 14) **Support for optional package can be explicitly disabled:** Even
3628 though TriBITS will automatically set
3629 ``<TRIBITS_PACKAGE>_ENABLE_<TRIBITS_DEP_PACKAGE_OR_TPL>=ON`` by default if
3630 ``<TRIBITS_PACKAGE>`` and ``<TRIBITS_DEP_PACKAGE_OR_TPL>`` are both
3631 enabled at the project level (as described above), the user can explicitly
3632 set ``<TRIBITS_PACKAGE>_ENABLE_<TRIBITS_DEP_PACKAGE_OR_TPL>=OFF`` which
3633 will turn off optional package-level support for the upstream package
3634 or TPL ``<TRIBITS_DEP_PACKAGE_OR_TPL>`` in the downstream package
3635 ``<TRIBITS_PACKAGE>``. For example, the user can enable ``EpetraExt`` and
3636 ``Triutils`` at the project level, but set
3637 ``EpetraExt_ENABLE_Triutils=OFF`` which will turn off package-level
3638 support for ``Triutils`` in the ``EpetraExt`` package. Likewise, the user
3639 can enable ``Teuchos``, ``Epetra`` and the optional TPL ``Boost``, but set
3640 ``Epetra_ENABLE_Boost=ON``. This would provide support for ``Boost`` in
3641 ``Teuchos`` but not in ``Epetra``. For examples, see `Explicit disable of
3642 an optional package dependency`_ and `Explicit disable of an optional TPL
3643 dependency`_.
3644
3645 .. _Explicit enable of optional package/TPL support auto-enables package/TPL:
3646
3647 15) **Explicit enable of optional package/TPL support auto-enables
3648 package/TPL**: If the user explicitly enables the TriBITS package
3649 ``<TRIBITS_PACKAGE>`` and explicitly sets
3650 ``<TRIBITS_PACKAGE>_ENABLE_<TRIBITS_DEP_PACKAGE_OR_TPL>=ON`` on input,
3651 then that will automatically enable the package or TPL
3652 ``<TRIBITS_DEP_PACKAGE_OR_TPL>`` (and all of its upstream dependencies
3653 accordingly). For example, if the user sets
3654 ``Trilinos_ENABLE_EpetraExt=ON`` and ``EpetraExt_ENABLE_Triutils=ON``,
3655 then that will result in the auto-enable of
3656 ``Trilinos_ENABLE_Triutils=ON`` regardless of the value of
3657 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES`` or
3658 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE``. This true even if the optional
3659 package or TPL is ``EX``. For example, setting
3660 ``Thyra_ENABLE_ThyraCrazyStuff=ON`` will result in the enabling of the
3661 ``EX`` package ``ThyraCrazyStuff``. However, always remember that
3662 `Disables trump enables where there is a conflict`_. For example, see
3663 `Explicit enable of an optional package dependency`_.
3664
3665 .. _ST packages only auto-enabled if ST code is enabled:
3666
3667 16) **ST packages only auto-enabled if ST code is enabled**: TriBITS will
3668 only enable an optional ``ST`` package when
3669 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=ON`` if
3670 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON`` is also set. Otherwise, when
3671 an optional ``ST`` `upstream`_ dependent package is not enabled due to
3672 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=OFF``, then a one-line warning is
3673 printed to stdout. For example, if the ``EpetraExt`` package is enabled
3674 but ``ST`` code is not enabled, then the optional package ``Triutils``
3675 will not be enabled (for an example, see `Explicit enable of a package and
3676 its tests`_). However, when ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON``
3677 and ``EpetraExt`` is enabled, then ``Triutils`` will be enabled too (for an
3678 example, see `Explicit enable of a package, its tests, an optional TPL,
3679 with ST enabled`_). The TriBITS default is
3680 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=ON``. This helps to avoid
3681 problems when users try to set a permutation of enables/disables which is
3682 not regularly tested.
3683
3684 .. _<Project>_ENABLE_ALL_FORWARD_DEP_PACKAGES enables downstream packages/tests:
3685
3686 17) **<Project>_ENABLE_ALL_FORWARD_DEP_PACKAGES enables downstream
3687 packages/tests**: Setting the user cache-variable
3688 ``${PROJECT_NAME}_ENABLE_ALL_FORWARD_PACKAGES=ON`` will result in the
3689 `downstream`_ ``PT`` internal packages and tests to be enabled (and all
3690 ``PT`` and ``ST`` packages and tests when
3691 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON``) for all explicitly enabled
3692 internal packages. For example, in the mock Trilinos project, configuring
3693 with ``Trilinos_ENABLE_Epetra=ON``, ``Trilinos_ENABLE_TESTS=ON``, and
3694 ``Trilinos_ENABLE_ALL_FORWARD_PACKAGES=ON`` will result the package
3695 enables (and test and example enables) for the downstream packages
3696 ``Triutils``, ``EpetraExt``, ``ThyraCoreLibs``, ``ThyraEpetra`` and
3697 ``Thyra``. For an example, see `Explicit enable of a package and
3698 downstream packages and tests`_. Note that when setting this option, the
3699 enable of an external package/TPL will **not** result in the auto-enable
3700 of downstream internal packages. For example, setting
3701 ``Trilinos_ENABLE_BLAS=ON`` will not result in the auto-enable of any
3702 internal packages that depend on ``BLAS`` like ``Teuchos`` (in the mock
3703 Trilinos project).
3704
3705 .. _${PROJECT_NAME}_ENABLE_ALL_PACKAGES:
3706
3707 .. _<Project>_ENABLE_ALL_PACKAGES enables all PT (cond. ST) packages:
3708
3709 18) **<Project>_ENABLE_ALL_PACKAGES enables all PT (cond. ST) packages**:
3710 Setting the user cache-variable ``${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON``
3711 will result in the enable of all ``PT`` packages when
3712 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=OFF`` and all ``PT`` and ``ST``
3713 packages when ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON``. For an
3714 example, see `Enable all packages`_. When the project is a meta-project,
3715 this will only enable the project's primary meta-project packages (PMPP).
3716 That is, a package will only be enabled due to
3717 ``${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON`` when its parent repository does
3718 not have `${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES`_ set to
3719 ``TRUE``. However, if::
3720
3721 ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES=TRUE
3722
3723 then the package may be enabled if it (or its parent package) is listed
3724 in `${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES_EXCEPT`_.
3725
3726 .. _<Project>_ENABLE_TESTS only enables explicitly enabled package tests:
3727
3728 19) **<Project>_ENABLE_TESTS only enables explicitly enabled package
3729 tests**: Setting ``${PROJECT_NAME}_ENABLE_TESTS=ON`` will **only enable
3730 tests for explicitly enabled packages** on input. For example,
3731 configuring with ``Trilinos_ENABLE_RTOp=ON`` and
3732 ``Trilinos_ENABLE_TESTS=ON`` will only result in the enable of tests for
3733 ``RTOp``, not ``Teuchos`` (even through TriBITS will enable ``Teuchos``
3734 because it is a required dependency of ``RTOp``). See an example, see
3735 `Explicit enable of a package and its tests`_. When the project is a
3736 meta-project, this will only enable tests for the project's primary
3737 meta-project packages (PMPP). The uses the same logic as for
3738 ``${PROJECT_NAME}_ENABLE_ALL_PACKAGES``, see
3739 `<Project>_ENABLE_ALL_PACKAGES enables all PT (cond. ST) packages`_.
3740
3741 .. _If no packages are enabled, nothing will get built:
3742
3743 20) **If no packages are enabled, nothing will get built**: Most TriBITS
3744 projects are set up such that if the user does not explicitly enable at
3745 least one package in some way, then nothing will be enabled or built.
3746 In this case, when ``${PROJECT_NAME}_ALLOW_NO_PACKAGES=TRUE`` a warning
3747 will be printed and configuration will complete. However, if
3748 ``${PROJECT_NAME}_ALLOW_NO_PACKAGES=FALSE``, then the configure will die
3749 with an error message. For example, the ``checkin-test.py`` tool sets
3750 ``${PROJECT_NAME}_ALLOW_NO_PACKAGES=OFF`` to make sure that something gets
3751 enabled and tested in order to accept the results of the test and allow a
3752 push. For an example, see `Default configure with no packages enabled on
3753 input`_.
3754
3755 .. _TriBITS prints all enables and disables to stdout:
3756
3757 21) **TriBITS prints all enables and disables to stdout**: TriBITS prints out
3758 (to ``cmake`` stdout) the initial set of enables/disables on input, prints
3759 a line whenever it sets (or overrides) an enable or disable, and prints
3760 out the final set of enables/disables. Therefore, the user just needs to
3761 grep the ``cmake`` stdout to find out why any particular package or TPL
3762 is enabled or disabled in the end. In addition, will print out when
3763 tests/examples for a given package gets enabled and when support for
3764 optional packages and TPLs is enabled or not. Examples of this output
3765 is given in all of the below examples but a detailed description of this
3766 output is given in `Explicit enable of a package and its tests`_.
3767
3768 .. _TriBITS auto-enables/disables done using non-cache local variables:
3769
3770 22) **TriBITS auto-enables/disables done using non-cache local variables**:
3771 TriBITS setting (or overrides) of enable/disable cache variables are done
3772 by setting local non-cache variables at the top project-level scope
3773 (i.e. the ``<projectDir>/CMakeLists.txt`` file scope) and does **not**
3774 touch the value of the cache variables that may be set by the user on
3775 input or the cache variables with documentation set by TriBITS. This is
3776 done so they don't get set in the cache and so that the same dependency
3777 enable/disable logic is redone, from scratch, with each re-configure.
3778 This results in the same enable/disable logic output as for the initial
3779 configure. This is to avoid confusion by the user about why some packages
3780 and TPLs are enabled and some are not on subsequent reconfigures. This is
3781 also desirable behavior as it preserves the user's input values for these
3782 variables to document what was set by the user. However, this
3783 implementation choice (and the tricky relationship between cache and
3784 non-cache CMake variables) must be clearly understood when one wants to go
3785 about tweaking these TriBITS enable/disable variables as described in `How
3786 to check for and tweak TriBITS "ENABLE" cache variables`_ and `How to
3787 tweak downstream TriBITS "ENABLE" variables during package
3788 configuration`_.
3789
3790 TriBITS prints out a lot of information about the enable/disable logic as it
3791 applies the above rules/behaviors. For a large TriBITS project with lots of
3792 packages, this can produce a lot of output to stdout. One just needs to
3793 understand what TriBITS is printing out and where to look in the output for
3794 different information. The examples in the section `Example Enable/Disable
3795 Use Cases`_ show what this output looks like for the various enable/disable
3796 scenarios and tries to explain in more detail the reasons for why the given
3797 behavior is implemented the way that it is. Given this output, the rule
3798 definitions given above, and the detailed `Example Enable/Disable Use Cases`_,
3799 one should always be able to figure out exactly why the final set of
3800 enables/disables is the way it is, even in the largest and most complex of
3801 TriBITS projects. (NOTE: The same can *not* be said for many other large
3802 software configuration and deployment systems where basic decisions about what
3803 to enable and disable are hidden from the user and can be very difficult to
3804 track down and debug.)
3805
3806 The above behaviors address the majority of the functionality of the TriBITS
3807 dependency management system. However, when dealing with TriBITS projects
3808 with multiple repositories, some other behaviors are supported through the
3809 definition of a few more variables. The following TriBITS repository-related
3810 variables alter what packages in a given TriBITS repository get enabled
3811 implicitly or not by TriBITS:
3812
3813 ``${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE``
3814
3815 If set to ``ON``, then the packages in Repository ``${REPOSITORY_NAME}``
3816 will not be implicitly enabled in any of the package adjustment logic.
3817
3818 ``${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE_EXCEPT``
3819
3820 List of packages in the Repository ``${REPOSITORY_NAME}`` that will be
3821 allowed to be implicitly enabled. Only checked if
3822 ``${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE`` is true.
3823
3824 The above variables typically are defined in the outer TriBITS Project's CTest
3825 driver scripts or even in top-level project files in order to adjust how
3826 packages in its listed repositories are handled. What these variable do is to
3827 allow a large project to turn off the auto-enable of optional packages in a
3828 given TriBITS repository to provide more detailed control of what gets used
3829 from a given TriBITS repository. This, for example, is used in the CASL VERA
3830 project to manage some of its extra repositories and packages to further reduce
3831 the number of packages that get auto-enabled.
3832
3833 .. ToDo: We should likely change this so that only tests and example don't get
3834 .. enabled by default, not the packages itself. This is mainly used to avoid
3835 .. certain packages from getting enabled in the CI server when updates are
3836 .. received. This needs to be explained in the section on the CI server.
3837
3838
3839 Example Enable/Disable Use Cases
3840 ---------------------------------
3841
3842 Below, a few of the standard enable/disable use cases for a TriBITS project
3843 are given using the `Example ReducedMockTrilinos Project Dependency
3844 Structure`_ that demonstrate the `TriBITS Dependency Handling Behaviors`_.
3845
3846 The use cases covered are:
3847
3848 * `Default configure with no packages enabled on input`_
3849 * `Explicit enable of a package and its tests`_
3850 * `Explicit enable of a package, its tests, an optional TPL, with ST enabled`_
3851 * `Explicit disable of a package`_
3852 * `Conflicting explicit enable and disable`_
3853 * `Explicit enable of an optional TPL`_
3854 * `Explicit disable of an optional TPL`_
3855 * `Explicit disable of a required TPL`_
3856 * `Explicit enable of a subpackage`_
3857 * `Explicit enable of an optional package dependency`_
3858 * `Explicit disable of an optional package dependency`_
3859 * `Explicit enable of an optional TPL dependency`_
3860 * `Explicit disable of an optional TPL dependency`_
3861 * `Explicit enable of a package and downstream packages and tests`_
3862 * `Enable all packages`_
3863
3864 All of these use cases and more can be easily run from the command-line by
3865 first setting:
3866
3867 $ export REDUCED_MOCK_TRILINOS=<base-dir>/tribits/examples/ReducedMockTrilinos
3868
3869 and then copy and pasting the ``cmake`` commands shown below. Just make sure
3870 to run these in a temp directory because this actually configures a CMake
3871 project in the local directory. Just make sure and run::
3872
3873 $ rm -r CMake*
3874
3875 before each run to clear the CMake cache.
3876
3877 These use cases are now described in detail below.
3878
3879 .. _Default configure with no packages enabled on input:
3880
3881 **Default configure with no packages enabled on input**
3882
3883 The first use-case to consider is the configure of a TriBITS project without
3884 enabling any packages. For the ``ReducedMockTrilinos`` project, this is done
3885 with::
3886
3887 $ cmake ${REDUCED_MOCK_TRILINOS}
3888
3889 which produces the relevant dependency-related output:
3890
3891 .. include:: ReducedMockTrilinosOutput/NoEnables.txt
3892 :literal:
3893
3894 The above example demonstrates the following behaviors of the TriBITS
3895 dependency handling system:
3896
3897 * `PT/ST packages given default unset enable/disable state`_
3898 * `EX packages disabled by default`_ (i.e., the ``EX`` package
3899 ``ThyraCrazyStuff`` is set to ``OFF`` by default at the very beginning).
3900 * `PT/ST TPLs given default unset enable/disable state`_
3901 * `EX TPLs given default unset enable/disable state`_
3902 * `If no packages are enabled, nothing will get built`_
3903
3904 .. _Explicit enable of a package and its tests:
3905
3906 **Explicit enable of a package and its tests**
3907
3908 One of the most typical use cases is for the user to explicitly enable one or
3909 more top-level TriBITS package and enable its tests. This configuration would
3910 be used to drive local development on a specific set of packages (i.e. tests
3911 do not need to be enabled for packages not being changed).
3912
3913 Consider the configure of the ``ReducedMockTrilinos`` project enabling the
3914 top-level ``Thyra`` package and its tests with::
3915
3916 $ cmake -DTrilinos_ENABLE_Thyra:BOOL=ON \
3917 -DTrilinos_ENABLE_TESTS:BOOL=ON \
3918 ${REDUCED_MOCK_TRILINOS}
3919
3920 which produces the relevant dependency-related output:
3921
3922 .. include:: ReducedMockTrilinosOutput/EnableThyra_EnableTests.txt
3923 :literal:
3924
3925 This is a configuration that a developer would use to develop on the ``Thyra``
3926 package and its subpackages for example. There is no need to be enabling the
3927 tests and examples for upstream packages unless those packages are going to be
3928 changed as well.
3929
3930 This case demonstrates a number of TriBITS dependency handling behaviors that
3931 are worth some discussion.
3932
3933 First, note that enabling the parent package ``Thyra`` with
3934 ``Trilinos_ENABLE_Thyra=ON`` right away results in the auto-enable of its
3935 ``PT`` subpackages ``ThyraCoreLibs`` and ``ThyraEpetra`` which demonstrates
3936 the behavior `Enable/disable of parent package is enable/disable for
3937 subpackages`_. Note that the ``ST`` subpackages ``ThyraGoodStuff`` and
3938 ``ThyraEpetraExt`` where *not* enabled because
3939 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=OFF`` (which is off by default) which
3940 demonstrates the behavior `ST packages only auto-enabled if ST code is
3941 enabled`_.
3942
3943 Second, note the auto-enable of required upstream packages ``Epetra``,
3944 ``RTOp`` and ``Teuchos`` shown in lines like::
3945
3946 -- Setting Trilinos_ENABLE_Teuchos=ON because ThyraCoreLibs has a required dependence on Teuchos
3947
3948 Lastly, note that the final set of enabled packages, packages, tests/examples
3949 and external packages/TPLs can be clearly seen when processing the external
3950 packages/TPLs and top-level packages in the lines::
3951
3952 Getting information for all enabled external packages/TPLs ...
3953
3954 -- Processing enabled external package/TPL: BLAS
3955 -- Processing enabled external package/TPL: LAPACK
3956
3957 Configuring individual enabled Trilinos packages ...
3958
3959 Processing enabled top-level package: Teuchos (Libs)
3960 Processing enabled top-level package: RTOp (Libs)
3961 Processing enabled top-level package: Epetra (Libs)
3962 Processing enabled top-level package: Thyra (CoreLibs, Epetra, Tests, Examples)
3963
3964 Note that subpackage enables are listed with their parent packages along with
3965 if the tests and/or examples are enabled. Top-level packages that don't have
3966 subpackages just show ``Libs`` and ``Tests`` and ``Examples`` if they have
3967 been enabled as well.
3968
3969 .. _Explicit enable of a package, its tests, an optional TPL, with ST enabled:
3970
3971 **Explicit enable of a package, its tests, an optional TPL, with ST enabled**
3972
3973 An extended use case shown here is for the explicit enable of a package and its
3974 tests along with the enable of an optional TPL and with ``ST`` code enabled.
3975 This is a configuration that would be used to support the local development of
3976 a TriBITS package that involves modifying ``ST`` software.
3977
3978 Consider the configure of the ``ReducedMockTrilinos`` project with::
3979
3980 $ cmake -DTPL_ENABLE_Boost:BOOL=ON \
3981 -DTrilinos_ENABLE_Thyra:BOOL=ON \
3982 -DTrilinos_ENABLE_TESTS:BOOL=ON \
3983 -DTrilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=ON \
3984 -DTrilinos_ENABLE_TESTS:BOOL=ON \
3985 ${REDUCED_MOCK_TRILINOS}
3986
3987 which produces the relevant dependency-related output:
3988
3989 .. include:: ReducedMockTrilinosOutput/EnableThyra_EnableTests_EnableBoost_ST.txt
3990 :literal:
3991
3992 A few more behaviors of the TriBITS system that this particular configuration
3993 use-case shows are described below.
3994
3995 First, note the enable of the ``ST`` ``Thyra`` subpackages in lines like::
3996
3997 -- Setting subpackage enable Trilinos_ENABLE_ThyraGoodStuff=ON because parent package \
3998 Trilinos_ENABLE_Thyra=ON
3999
4000 Second, note the auto-enable of support for optional packages in lines
4001 like::
4002
4003 -- Setting EpetraExt_ENABLE_Triutils=ON since Trilinos_ENABLE_EpetraExt=ON \
4004 AND Trilinos_ENABLE_Triutils=ON
4005
4006 Third, note the auto-enable of support for the optional TPL ``Boost`` in the
4007 line::
4008
4009 -- Setting Teuchos_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON
4010
4011 .. _Explicit disable of a package:
4012
4013 **Explicit disable of a package**
4014
4015 Another common use case is to enable a package but to disable an optional
4016 upstream package. This type of configuration would be used as part of a
4017 "black list" approach to enabling only a subset of packages and optional
4018 support. The "black list" approach is to enable a package with
4019 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=ON`` (the TriBITS default) but
4020 then to turn off a specific set of packages that you don't want. This is
4021 contrasted with a "white list" approach where you would configure with
4022 ``${PROJECT_NAME}_ENABLE_ALL_OPTIONAL_PACKAGES=OFF`` and then have to manually
4023 enable all of the optional packages you want. Experience with projects like
4024 Trilinos show that the "black list" approach is generally to be preferred for
4025 a few reasons.
4026
4027 Consider the configure of the ``ReducedMockTrilinos`` project enabling ``Thyra`` but disabling ``Epetra`` with::
4028
4029 $ cmake -DTrilinos_ENABLE_Thyra:BOOL=ON \
4030 -DTrilinos_ENABLE_Epetra:BOOL=OFF \
4031 -DTrilinos_ENABLE_TESTS:BOOL=ON \
4032 ${REDUCED_MOCK_TRILINOS}
4033
4034 which produces the relevant dependency-related output:
4035
4036 .. include:: ReducedMockTrilinosOutput/EnableThyra_DisableEpetra_EnableTests.txt
4037 :literal:
4038
4039 Note how the disable of ``Epetra`` wipes out all of the required and optional
4040 packages and intra-package dependencies that depend on ``Epetra``. What is
4041 left is only the ``ThyraCoreLibs`` and its upstream dependencies that don't
4042 depend on ``Epetra`` (which is only ``RTOp`` and ``Teuchos``).
4043
4044 .. _Conflicting explicit enable and disable:
4045
4046 **Conflicting explicit enable and disable**
4047
4048 One use case that occasionally comes up is when a set of inconsistent enables
4049 and disables are set. While this seems illogical that anyone would ever do
4050 this, when it comes to larger more complex projects with lots of packages and
4051 lots of dependencies, this can happen very easily. In some cases, someone is
4052 enabling a set of packages they want and is trying to weed out as many of the
4053 (what they think) are optional dependencies that they don't need and
4054 accidentally disables a package that is an indirect required dependency of one
4055 of the packages they want (in which case the configure should likely fail and
4056 provide a good error message). The other use case where conflicting
4057 enables/disables can occur is in CTest drivers using `tribits_ctest_driver()`_
4058 where an upstream package has failed and is explicitly disabled (in which case
4059 it should gracefully disable downstream dependent packages). TriBITS can
4060 either be set up to have the disable override the explicit enable or stop the
4061 configure in error depending on the value of the cache variable
4062 `${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES`_ (see `Disables trump
4063 enables where there is a conflict`_).
4064
4065 For example, consider what happens with the ``ReducedMockTrilinos`` project if
4066 someone tries to enable the ``RTOp`` package and disable the ``Teuchos``
4067 package. This is not consistent because ``RTOp`` has a required dependency on
4068 ``Teuchos``. The default behavior of TriBITS is this case is shown in the
4069 below configure::
4070
4071 $ cmake -DTrilinos_ENABLE_Epetra:BOOL=ON \
4072 -DTrilinos_ENABLE_RTOp:BOOL=ON \
4073 -DTrilinos_ENABLE_Teuchos:BOOL=OFF \
4074 ${REDUCED_MOCK_TRILINOS}
4075
4076 which produces the relevant dependency-related output:
4077
4078 .. include:: ReducedMockTrilinosOutput/EnableEpetra_EnableRTOp_DisableTeuchos.txt
4079 :literal:
4080
4081 As shown above, the TriBITS default (which is
4082 ``${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=OFF``) results in a
4083 configure-time error with a good error message.
4084
4085 However, if one sets
4086 ``${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON`` and configures
4087 with::
4088
4089 $ cmake -DTrilinos_ENABLE_Epetra:BOOL=ON \
4090 -DTrilinos_ENABLE_RTOp:BOOL=ON \
4091 -DTrilinos_ENABLE_Teuchos:BOOL=OFF \
4092 -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES:BOOL=ON \
4093 ${REDUCED_MOCK_TRILINOS}
4094
4095 then the disable trumps the enable and results in a successful configure as
4096 shown in the following relevant dependency-related output:
4097
4098 .. include:: ReducedMockTrilinosOutput/EnableEpetra_EnableRTOp_DisableTeuchos_DisableEnabledFwdDepPackages.txt
4099 :literal:
4100
4101 As shown above, what you end up with is just the enabled package ``Epetra``
4102 which does not have a required dependency on the disabled package ``Teuchos``.
4103 Developers of large complex TriBITS projects would be wise to set the default
4104 for `${PROJECT_NAME}_DISABLE_ENABLED_FORWARD_DEP_PACKAGES`_ to ``ON``,
4105 especially in automated builds and testing.
4106
4107 .. _Explicit enable of an optional TPL:
4108
4109 **Explicit enable of an optional TPL**:
4110
4111 ToDo: Set Trilinos_ENABLE_Thyra=ON and TPL_ENABLE_MPI=ON
4112
4113 .. _Explicit disable of an optional TPL:
4114
4115 **Explicit disable of an optional TPL**:
4116
4117 ToDo: Set Trilinos_ENABLE_Thyra=ON and TPL_ENABLE_MPI=OFF
4118
4119 .. _Explicit disable of a required TPL:
4120
4121 **Explicit disable of a required TPL**
4122
4123 ToDo: Set Trilinos_ENABLE_Epetra=ON and Trilinos_ENABLE_BLAS=OFF
4124
4125 .. _Explicit enable of a subpackage:
4126
4127 **Explicit enable of a subpackage**
4128
4129 ToDo: Enable ThyraEpetra and show how it enables other packages and at the
4130 end, enables the Thyra package (just for show).
4131
4132 .. _Explicit enable of an optional package dependency:
4133
4134 **Explicit enable of an optional package dependency**
4135
4136 ToDo: Set Trilinos_ENABLE_EpetraExt=ON and EpetraExt_ENABLE_Triutils=ON and
4137 shows how it enables Trilinos_ENABLE_Triutils=ON even through ST code is not
4138 enabled.
4139
4140 .. _Explicit disable of an optional package dependency:
4141
4142 **Explicit disable of an optional package dependency**
4143
4144 ToDo: Set Trilinos_ENABLE_EpetraExt=ON, Trilinos_ENABLE_Triutils=ON, and
4145 EpetraExt_ENABLE_Triutils=OFF. Discuss how EpetraExt's and ThyraEpetraExt's
4146 CMakeLists.txt files might turn off some features if they detects that
4147 EpetraExt/Triutils support is turned off.
4148
4149 .. _Explicit enable of an optional TPL dependency:
4150
4151 **Explicit enable of an optional TPL dependency**
4152
4153 ToDo: The current ReducedMockTrilinos is not set up to give a good example of
4154 this. We should add an optional Boost dependency to say, Epetra. Then we
4155 could show the enable of Teuchos and Epetra and Epetra_ENABLE_Boost=ON. That
4156 would enable Boost and enable support for Boost in Epetra but would not
4157 provide support for Boost in Teuchos.
4158
4159 .. _Explicit disable of an optional TPL dependency:
4160
4161 **Explicit disable of an optional TPL dependency**
4162
4163 ToDo: The current ReducedMockTrilinos is not set up to give a good example of
4164 this. We should add an optional Boost dependency to say, Epetra. Then we
4165 could show the enable of Teuchos and Epetra and TPL_ENABLE_Boost=ON but set
4166 Epetra_ENABLE_Boost=OFF. That would provide support for Boost in Teuchos but
4167 not in Epetra.
4168
4169 .. _Explicit enable of a package and downstream packages and tests:
4170
4171 **Explicit enable of a package and downstream packages and tests**
4172
4173 ToDo: Set Trilinos_ENABLE_RTOp=ON,
4174 Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON, and Trilinos_ENABLE_TESTS=ON and
4175 show what packages and tests/examples get enabled. This is the use case for
4176 the checkin-test.py tool for PT enabled code.
4177
4178 .. _Enable all packages:
4179
4180 **Enable all packages**
4181
4182 The last use case to consider is enabling all defined packages. This
4183 configuration would be used for either doing a full test of all of the
4184 packages defined or to create a distribution of the project.
4185
4186 Enabling all ``PT`` packages the with::
4187
4188 $ cmake -DTrilinos_ENABLE_ALL_PACKAGES:BOOL=ON \
4189 -DTrilinos_DUMP_PACKAGE_DEPENDENCIES:BOOL=ON \
4190 ${REDUCED_MOCK_TRILINOS}
4191
4192 produces the relevant dependency-related output:
4193
4194 .. include:: ReducedMockTrilinosOutput/EnableAllPackages.txt
4195 :literal:
4196
4197 As shown above, only the ``PT`` packages get enabled. To also enable the
4198 ``ST`` packages as well, one additionally set
4199 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON`` at configure time.
4200
4201 .. _<Project>PackageDependencies.xml:
4202
4203
4204 TriBITS Project Dependencies XML file and tools
4205 -----------------------------------------------
4206
4207 The TriBITS CMake configure system can write out the project's package
4208 dependencies into a file ``<Project>Dependencies.xml`` (or any name one wants
4209 to give it). This file is used by a number of the TriBITS SE-related tools.
4210 The structure of this XML file, showing one of the more interesting mock
4211 packages from the `MockTrilinos`_ project is shown below::
4212
4213 <PackageDependencies project="Trilinos">
4214 ...
4215 <Package name="Amesos" dir="packages/amesos" type="PT">
4216 <LIB_REQUIRED_DEP_PACKAGES value="Teuchos,Epetra"/>
4217 <LIB_OPTIONAL_DEP_PACKAGES value="EpetraExt"/>
4218 <TEST_REQUIRED_DEP_PACKAGES/>
4219 <TEST_OPTIONAL_DEP_PACKAGES value="Triutils,Galeri"/>
4220 <LIB_REQUIRED_DEP_TPLS/>
4221 <LIB_OPTIONAL_DEP_TPLS value="SuperLUDist,ParMETIS,UMFPACK,SuperLU,MUMPS"/>
4222 <TEST_REQUIRED_DEP_TPLS/>
4223 <TEST_OPTIONAL_DEP_TPLS/>
4224 <EmailAddresses>
4225 <Regression address="[email protected]"/>
4226 </EmailAddresses>
4227 <ParentPackage value=""/>
4228 </Package>
4229 ...
4230 </PackageDependencies>
4231
4232 This XML file contains the names, directories, `Test Test Category`_
4233 (i.e. ``type``), CDash email address, and all of the package and TPL
4234 dependencies for every package in the TriBITS project (including add-on
4235 repositories if specified). There are several python tools under
4236 ``tribits/ci_support/`` that read in this file and use the created
4237 data-structure for various tasks. This file and these tools are used by
4238 `checkin-test.py`_ and `tribits_ctest_driver()`_. But these tools can also be
4239 used to construct other workflows and tools. These tools require a Python3
4240 installation and for the `python3` executable to be installed.
4241
4242 .. _TribitsDumpDepsXmlScript.cmake:
4243
4244 A TriBITS project configure can create this file as a byproduct of
4245 configuration by setting the configure option (see `Outputting package
4246 dependency information`_), or the CMake -P script
4247 **TribitsDumpDepsXmlScript.cmake** can be used to create this file on the fly
4248 without having to configure a TriBITS project. To create this file outside of
4249 configuration, one can run::
4250
4251 cmake \
4252 [-D PROJECT_SOURCE_DIR=<projectSourceDir>] \
4253 [-D <Project>_PRE_REPOSITORIES=<prepo0>,<prepo1>,...] \
4254 [-D <Project>_EXTRA_REPOSITORIES=<erepo0>,<erepo1>,...] \
4255 -D <Project>_DEPS_XML_OUTPUT_FILE=<projectDepsFileOut> \
4256 -P <tribitsDir>/ci_support/TribitsDumpDepsXmlScript.cmake
4257
4258 If TriBITS is snashotted into the project in the standard location
4259 ``<projectDir>/cmake/tribits`` or the entire TriBITS repo is cloned under
4260 ``<projectDir>/TriBITS`` (so that the ``tribits`` dir is
4261 ``<projectDir>/TriBITS/tribits``) then one can leave off
4262 ``-DPROJECT_SOURCE_DIR=<projectSourceDir>`` and (if not wanting to include
4263 extra repos) just run::
4264
4265 cmake \
4266 -D <Project>_DEPS_XML_OUTPUT_FILE=<projectDepsFileOut> \
4267 -P <projectSourceDir>/cmake/tribits/ci_support/TribitsDumpDepsXmlScript.cmake
4268
4269 Once the XML file ``<projectDepsFileOut>`` is created, it can be used in
4270 various types of analysis and used with different tools and commands.
4271
4272 The tool `get-tribits-packages-from-files-list.py`_ can be used to determine
4273 the list of TriBITS packages that need to be tested given a list of changed
4274 files (e.g. as returned from ``git diff --name-only <from>..<to> >
4275 changed-files.txt``). This is used in the `checkin-test.py`_ tool and the
4276 `tribits_ctest_driver()`_ function to determine what TriBITS packages need to
4277 be tested based on what files have been changed.
4278
4279 The tool `get-tribits-packages-from-last-tests-failed.py`_ can be used to
4280 extract the list of TriBITS packages that correspond to the failings tests
4281 listed in the CTest-generated
4282 ``<build-dir>/Testing/Temporary/LastTestsFailed*.log`` file. This tool is
4283 used in the `tribits_ctest_driver()`_ function in CI-testing mode to determine
4284 what packages must be re-tested if they failed in the last CI iteration.
4285
4286 The tool `filter-packages-list.py`_ takes in a list of TriBITS package
4287 names and then filters the list according the `Test Test Category`_ of the
4288 packages. This is used in testing workflows that only test a subset of
4289 packages according to the Test Test Category at different stages in the
4290 workflow. For example, the `checkin-test.py`_ tool and the
4291 `tribits_ctest_driver()`_ function use this filtering to only test Primary
4292 Tested (PT) or Secondary Tested (ST) packages for a given set of changed files
4293 in a continuous integration workflow (see `Nested Layers of TriBITS Project
4294 Testing`_).
4295
4296
4297 TriBITS Automated Testing
4298 =========================
4299
4300 Much of the value provided by the TriBITS system is support for testing of
4301 complex projects. Many different types of testing are required in a
4302 complex project and development effort. A large project with lots of
4303 repositories and packages provides a number of testing and development
4304 challenges but also provides a number of opportunities to do testing in an
4305 efficient way; especially pre-push and post-push continuous integration (CI)
4306 testing. In addition, a number of post-push automated nightly test cases must
4307 be managed. TriBITS takes full advantage of the features of raw CMake, CTest,
4308 and CDash in support of testing and where gaps exist, TriBITS provides tools
4309 and customizations.
4310
4311 The following subsections describe several aspects to the TriBITS support for
4312 testing. The subsection `Test Classifications for Repositories, Packages, and
4313 Tests`_ defines the different types of test-related classifications that are
4314 defined by TriBITS. These different test classifications are then used to
4315 define a number of different standard `Nested Layers of TriBITS Project
4316 Testing`_ which include different types of CI testing as well as nightly and
4317 other tests. One of the most important types of CI testing, pre-push testing,
4318 is then described in more detail in the subsection `Pre-push Testing using
4319 checkin-test.py`_. The subsection `TriBITS CTest/CDash Driver`_ describes the usage
4320 of the advanced `tribits_ctest_driver()`_ function to do incremental project
4321 testing of a projects using advanced ``ctest -S`` scripts. The final
4322 subsection `TriBITS CDash Customizations`_ describes how projects can use a
4323 CDash server to more effectively display test results and provide
4324 notifications for failures that are compartmentalized for a large project.
4325
4326
4327 Test Classifications for Repositories, Packages, and Tests
4328 ----------------------------------------------------------
4329
4330 TriBITS defines a few different testing-related classifications for a TriBITS
4331 project. These different classifications are used to select subsets of the
4332 project's repositories, packages (and code within these packages), and tests
4333 to be included in a given project build and test definition. These different
4334 classification are:
4335
4336 * `Repository Test Classification`_
4337 * `Package Test Group`_
4338 * `Test Test Category`_
4339
4340 These different test-related classifications are used to defined several
4341 different `Nested Layers of TriBITS Project Testing`_. First, the `Repository
4342 Test Classification`_ determines what repositories are even processed in order
4343 for their packages to even consider being enabled. Second, if a repository
4344 is selected, then the `Package Test Group`_ determines what packages
4345 (and optional code in those packages) are even enabled such that their
4346 `<packageDir>/CMakeLists.txt`_ files are even processed (i.e. according to
4347 `TriBITS Dependency Handling Behaviors`_). Lastly, if an package gets
4348 enabled, then the `Test Test Category`_ determines what test executables and
4349 test cases get defined using the functions `tribits_add_executable()`_,
4350 `tribits_add_test()`_ and `tribits_add_advanced_test()`_.
4351
4352 More detailed descriptions of `Repository Test Classifications`_, `Package
4353 Test Groups`_, and `Test Test Categories`_ are given in the following
4354 subsections.
4355
4356 .. _Repository Test Classification:
4357 .. _Repository Test Classifications:
4358
4359 **Repository Test Classification**
4360
4361 The first type of test-related classification is for extra repositories
4362 defined in the file `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ (pulled
4363 in through the `${PROJECT_NAME}_EXTRAREPOS_FILE`_ cache variable) using the
4364 ``REPO_CLASSIFICATION`` field in the macro call
4365 `tribits_project_define_extra_repositories()`_. These classifications map to
4366 the standard CTest dashboard types ``Continuous``, ``Nightly``, and
4367 ``Experimental`` (see `CTest documentation`_ and `TriBITS CTest/CDash Driver`_
4368 for details).
4369
4370 .. _Repository Test Continuous:
4371
4372 * Repositories marked **Continuous** match the standard CTest dashboard type
4373 ``Continuous``. These repositories are pulled in when
4374 ``${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Continuous``, or
4375 ``Nightly``. Repositories marked as ``Continuous`` are cloned, updated, and
4376 processed by default in all project automated testing described in `Nested
4377 Layers of TriBITS Project Testing`_. NOTE: One should **not** confuse this
4378 with the `Test Test Category CONTINUOUS`_.
4379
4380 .. _Repository Test Nightly:
4381
4382 * Repositories marked **Nightly** match the standard CTest dashboard type
4383 ``Nightly``. These repositories are pulled in by default when
4384 ``${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly``. Repositories
4385 marked as ``Nightly`` are not processed by default as part of either
4386 `Pre-Push CI Testing`_ or `Post-Push CI Testing`_. One would mark a
4387 repository as ``Nightly`` for few reasons. First, an extra repo may be
4388 marked as ``Nightly`` if it may not be available to all developers to clone
4389 and only the nightly testing processes and machines may have access. Also,
4390 an extra repo may also be marked as ``Nightly`` if it does not contain any
4391 packages that the project wants to pay the cost to include in even
4392 `Post-Push CI Testing`_. NOTE: One should **not** confuse this with the
4393 `Test Test Category NIGHTLY`_.
4394
4395 .. _Repository Test Experimental:
4396
4397 * Repositories marked **Experimental** match the standard CTest dashboard type
4398 ``Experimental``. Repositories marked as ``Experimental`` are not processed
4399 by default as part of any automated testing described in the subsection
4400 `Nested Layers of TriBITS Project Testing`_ (except for perhaps some
4401 experimental builds). The main reason that an extra repo may be marked as
4402 ``Experimental`` is that it may only contain ``EX`` packages and
4403 therefore none of these packages would be enabled by default anyway. Also,
4404 a repo may be marked as ``Experimental`` if it is developed in a very sloppy
4405 way such that one cannot even assume that the repository's
4406 `<repoDir>/PackagesList.cmake`_, `<repoDir>/TPlsList.cmake`_, and
4407 `<packageDir>/cmake/Dependencies.cmake`_ files are not without errors.
4408 Since these files are always processed if the repository is included, then
4409 any errors in these files will cause the entire configure to fail!
4410
4411 .. ToDo: Determine, add unit tests for, and document exactly how
4412 .. 'Experimental' repos are handled
4413 .. w.r.t. ${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE. Currently it
4414 .. looks like there are not processed if
4415 .. ${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Experimental. I am not
4416 .. sure why we bother with Experimental repos other than to just ignore them.
4417 .. However, it does seem they get cloned no matter what their classification
4418 .. in the TribitsCTestDriverCore.cmake script.
4419
4420 .. _Package Test Group:
4421 .. _Package Test Groups:
4422
4423 **Package Test Group**
4424
4425 Once a set of TriBITS repositories are selected in accordance with their
4426 `Repository Test Classification`_, that determines the set of packages defined
4427 for the TriBITS project. Given the set of defined packages, the set of
4428 packages that get enabled is determined by the **Package Test Group** which is
4429 defined and described here.
4430
4431 Every `TriBITS Package`_ is assigned a test group. These test groups are for
4432 *Primary Tested* (`PT`_) code, *Secondary Tested* (`ST`_) code, and
4433 *Experimental* (`EX`_) code. The test group defines *what* package get
4434 selected (or are excluded from being selected) to include in a given build for
4435 testing-related purposes. packages may also conditionally build in additional
4436 code based on the testing group. The detailed rules for when an package is
4437 selected or excluded from the build based on the test group is given in
4438 `TriBITS Dependency Handling Behaviors`_. We only summarize those rules here.
4439
4440 More detailed descriptions of the test groups are given below.
4441
4442 .. _PT:
4443
4444 * **Primary Tested (PT)** Code is of the highest priority to keep working for
4445 the current development effort. Packages may be selected to be ``PT`` for a
4446 number of reasons. First, if the capability provided by the code is mature
4447 and if a regression would cause major harm to a customer, the code should
4448 likely be marked as ``PT``. Also, if the build and correct functioning of
4449 the code is needed by other development team members to support their
4450 day-to-day development activities, then the code should be marked as ``PT``
4451 as well. An external package/TPL, on the other hand, is marked as ``PT`` if
4452 it is required by a downstream ``PT`` package. Every project developer is
4453 expected to have every ``PT`` TPL installed on every machine where they do
4454 development on and from which they push to the global repo (see
4455 `checkin-test.py`_ tool). ``PT`` packages are the foundation for `Pre-Push
4456 CI Testing`_.
4457
4458 .. _ST:
4459
4460 * **Secondary Tested (ST)** Code is still very important code for the project
4461 and represents important capability to maintain but is excluded from the
4462 ``PT`` set of code for one of a few different reasons. First, code may be
4463 marked as ``ST`` if is not critical to drive most day-to-day development
4464 activities. If ``ST`` code breaks, it usually will not cause immediate and
4465 major harm to most developers. Also, code may be marked as ``ST`` if it has
4466 required dependencies on ``ST`` external packages/TPLs which are either hard
4467 to install or may not be available on all platforms where developers do
4468 their development and from where they push changes to the global repo. In
4469 addition, code may be marked as ``ST`` if the project is just too big and
4470 developers can't be expected to build and test all of this code with every
4471 push (so a decision is made to only make some code as ``PT`` so that pushes
4472 don't take too long). ``ST`` code can be included in the TriBITS
4473 auto-enable algorithms by setting the variable
4474 ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON`` (see `TriBITS Dependency
4475 Handling Behaviors`_). Otherwise, ``ST`` code **is not** enabled by
4476 auto-enable algorithms. Typically, ``ST`` code is excluded from the default
4477 builds in `Pre-Push CI Testing`_ but ``ST`` code **is** typically tested in
4478 `Post-Push CI Testing`_, `Nightly Testing`_ as well as in other testing
4479 cases.
4480
4481 .. _EX:
4482
4483 * **Experimental (EX)** Code is usually too unstable, buggy, or non-portable
4484 to be maintained as part of the project's automated testing processes. Or
4485 the code may not be important enough to the project to bother paying the
4486 cost to test it in the project's automated testing processes. The ability
4487 to mark some code as ``EX`` allows the developers of that code to include
4488 their code in the project's VC repos along with the rest of the project's
4489 code and be able to take advantage of all of the development tools provided
4490 by the project (using TriBITS) but at the same time not having to "sign up"
4491 for all of the responsibilities of maintaining working software that every
4492 developer has to take a part in helping to keep working.
4493
4494 The test group for each type of entity is assigned in the following places:
4495
4496 * The top-level `TriBITS Package`_'s test group is assigned using the
4497 ``CLASSIFICATION`` field in the macro call
4498 `tribits_repository_define_packages()`_ in its parent repository's
4499 `<repoDir>/PackagesList.cmake`_ file.
4500
4501 * A `TriBITS Subpackage`_'s test group is assigned using the
4502 ``CLASSIFICATIONS`` field of the `SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS`_
4503 argument in the macro call `tribits_package_define_dependencies()`_ in its
4504 parent package's `<packageDir>/cmake/Dependencies.cmake`_ file.
4505
4506 * A `TriBITS External Package/TPL`_'s test group is assigned using the
4507 ``CLASSIFICATION`` field in the macro call
4508 `tribits_repository_define_tpls()`_ in its parent repository's
4509 `<repoDir>/TPLsList.cmake`_ file.
4510
4511 After these files are processed, the variable `${PACKAGE_NAME}_TESTGROUP`_
4512 gives the test group for each defined Package while the variable
4513 ``${TPL_NAME}_TESTGROUP`` gives the test group for each defined TPL.
4514
4515 Note that the test group classification ``PT``/``ST``/``EX`` is *not* to be
4516 confused with the *maturity level* of the package as discussed in the
4517 `TriBITS Lifecycle Model`_. The test group classification in no way implies
4518 the maturity of the given TriBITS Package or piece of code. Instead, the
4519 test group is just used to sub-select packages (and pieces of code within
4520 those packages) that are the most important to sustain for the various current
4521 development group's activities. While more-mature code would typically never
4522 be classified as ``EX`` (Experimental), there are cases were immature packages
4523 may be classified as ``ST`` or even ``PT``. For example, a very important
4524 research project may be driving the development of a very new algorithm with
4525 the low *maturity level* of *Research Stable* (``RS``) or even *Exploratory*
4526 (``EP``) because keeping that code working may be critical to keeping the
4527 research project on track.
4528
4529 In addition to just selecting ``PT`` and ``ST`` packages as a whole, a
4530 TriBITS ``PT`` package can also contain conditional code and test
4531 directories that get enabled when ``${PROJECT_NAME}_SECONDARY_TESTED_CODE=ON``
4532 and therefore represents more ``ST`` code. The package's
4533 `<packageDir>/CMakeLists.txt`_ files can contain simple if statements and can
4534 use the `tribits_set_st_for_dev_mode()`_ function to automatically select
4535 extra code to enable when ``ST`` is enabled or when the project is in release
4536 mode.
4537
4538 .. ToDo: Provide examples in TribitsExampleProject of ST and EX packages
4539 .. and of conditional ST and EX code in an PT package. This is important
4540 .. to round out the examples.
4541
4542 .. _Test Test Category:
4543 .. _Test Test Categories:
4544
4545 **Test Test Category**
4546
4547 Once a package is even defined (due to its parent repository's selection
4548 consistent with its `Repository Test Classification`_) and is the package is
4549 enabled (consistent with its `Package Test Group`_) then the set of
4550 individual test executables and test cases that are included or not in that
4551 package depends on the ``CATEGORIES`` argument in the functions
4552 `tribits_add_executable()`_, `tribits_add_test()`_ and
4553 `tribits_add_advanced_test()`_, and the `${PROJECT_NAME}_TEST_CATEGORIES`_
4554 variable. This **Test Test Category** defines the last "knob" that the
4555 development team has in controlling what tests get run in a particular test
4556 scenario as described in the section `Nested Layers of TriBITS Project
4557 Testing`_.
4558
4559 The currently allowed values for the *Test Test Category* are ``BASIC``,
4560 ``CONTINUOUS``, ``NIGHTLY``, ``HEAVY``, and ``PERFORMANCE``. Tests are
4561 enabled based on their assigned test test category matching the categories set
4562 in the CMake cache variable `${PROJECT_NAME}_TEST_CATEGORIES`_. The test test
4563 categories ``BASIC``, ``CONTINUOUS``, ``NIGHTLY``, and ``HEAVY`` are subsets
4564 of each other. That is, a ``BASIC`` test is automatically included in the set
4565 of ``CONTINUOUS``, ``NIGHTLY``, and ``HEAVY`` tests (as set using
4566 ``${PROJECT_NAME}_TEST_CATEGORIES``).
4567
4568 The different test test categories are described below in more detail:
4569
4570 .. _Test Test Category BASIC:
4571
4572 * Tests marked **BASIC** represent key functionality that is needed by nearly
4573 every developer that works on the project and so must be protected at all
4574 times and are therefore included in `Pre-Push CI Testing`_. Tests marked as
4575 ``BASIC`` are enabled for the values of ``${PROJECT_NAME}_TEST_CATEGORIES``
4576 of ``BASIC``, ``CONTINUOUS``, ``NIGHT``, and ``HEAVY``. The category
4577 ``BASIC`` is the default test test category given to all test executables
4578 and tests that don't specify the ``CATEGORIES`` argument.
4579
4580 .. _Test Test Category CONTINUOUS:
4581
4582 * Tests marked **CONTINUOUS** also represent importantly functionality but are
4583 typically not run in `Pre-Push CI Testing`_ testing but instead are run in
4584 `Post-Push CI Testing`_, `Nightly Testing`_, and other types of testing.
4585 Tests marked as ``CONTINUOUS`` are enabled for the values of
4586 ``${PROJECT_NAME}_TEST_CATEGORIES`` equal to ``CONTINUOUS``, ``NIGHT``, and
4587 ``HEAVY``. A test may be marked ``CONTINUOUS`` and not ``BASIC`` for a few
4588 different reasons. For example, the code needed to run the test may take
4589 too long to build or the test itself may take too long to run in order to
4590 afford including it in `Pre-Push CI Testing`_.
4591
4592 .. _Test Test Category NIGHTLY:
4593
4594 * Tests marked **NIGHTLY** usually take even longer to build and/or run than
4595 ``CONTINUOUS`` tests and therefore are too expensive to include in
4596 `Post-Push CI Testing`_. Tests may also be marked as ``NIGHTLY`` even if
4597 they might run relatively fast if there is a desire to not cause the CI
4598 server to fail if these tests fail. In this case, the decision is to take
4599 the testing and maintenance of these tests and the capabilities they
4600 represent "offline" so that they don't influence the daily development cycle
4601 for the project but instead are addressed in a "secondary feedback loop".
4602 Tests marked as ``NIGHTLY`` are enabled for the values of
4603 ``${PROJECT_NAME}_TEST_CATEGORIES`` equal to ``NIGHT``, and ``HEAVY``.
4604
4605 .. _Test Test Category HEAVY:
4606
4607 * Tests marked **HEAVY** are usually reserved for very expensive tests that
4608 are too expensive to run nightly. ``HEAVY`` tests require more testing
4609 resources and therefore may only be run on a fully optimized build and/or
4610 run less frequently. Tests marked as ``HEAVY`` are enabled only for the
4611 value of ``${PROJECT_NAME}_TEST_CATEGORIES`` equal to ``HEAVY``.
4612
4613 .. _Test Test Category PERFORMANCE:
4614
4615 * Tests marked **PERFORMANCE** are a special category of tests that are
4616 specially designed to measure the serial (non-parallel) run-time performance
4617 of parts of the software (see `Performance Testing`_). Tests marked as
4618 ``PERFORMANCE`` are enabled only for the value of
4619 ``${PROJECT_NAME}_TEST_CATEGORIES`` equal to ``PERFORMANCE``.
4620
4621 Every TriBITS project has a default setting for
4622 ``${PROJECT_NAME}_TEST_CATEGORIES`` that is set for a basic ``cmake``
4623 configure of the project (see `${PROJECT_NAME}_TEST_CATEGORIES_DEFAULT`_ for
4624 more details). In addition, the different testing processes described in the
4625 section `Nested Layers of TriBITS Project Testing`_ set this to different
4626 values.
4627
4628
4629 Nested Layers of TriBITS Project Testing
4630 ----------------------------------------
4631
4632 Now that the different types of `Test Classifications for Repositories,
4633 Packages, and Tests`_ have been defined, this section describes how these
4634 different test-related classifications are used to select repositories,
4635 packages (and code) and tests to run in the standard project testing
4636 processes. More than any other section in this document, this section will
4637 describe and assume a certain class of software development processes (namely
4638 agile processes) where testing and *continuous integration* (CI) are critical
4639 components. However, detailed descriptions of these processes are deferred to
4640 the later sections `Pre-push Testing using checkin-test.py`_ and `TriBITS
4641 CTest/CDash Driver`_.
4642
4643 The standard TriBITS-supported project testing processes are:
4644
4645 * `Pre-Push CI Testing`_
4646 * `Post-Push CI Testing`_
4647 * `Nightly Testing`_
4648 * `Heavy Testing`_
4649 * `Performance Testing`_
4650
4651 .. ToDo: Discuss why we would want to create standardized test cases? The
4652 .. answer is that it standardized how testing is done across multiple TriBITS
4653 .. projects so that larger meta-projects and run tests in a consistent way.
4654
4655 .. ToDo: Show a diagram for these different testing processes and show how
4656 .. they nest inside of each other for the most part.
4657
4658 These standard testing processes are outlined in more detail below and show
4659 how the different test-related categories are used to define each of these.
4660
4661 .. _Pre-Push CI Testing:
4662
4663 **Pre-Push CI Testing**
4664
4665 The first level of testing is *Pre-Push CI Testing* that is performed before
4666 changes to the project are pushed to the master branch(es) in the global
4667 repository(s). With TriBITS, this type of testing and the following push is
4668 typically done using the `checkin-test.py`_ tool. This category of testing is
4669 described in much more detail in `Pre-push Testing using checkin-test.py`_.
4670 All of the "default builds" used with the ``checkin-test.py`` tool select
4671 repositories, packages and code, and individual tests using the following
4672 test-related classifications:
4673
4674 ========================= ================== ====================================
4675 Classification Type Classification (See Reference)
4676 ========================= ================== ====================================
4677 Repository Test Classif. ``Continuous`` (`Repository Test Continuous`_)
4678 Package Test Group ``PT`` (`PT`_)
4679 Test Test Category ``BASIC`` (`Test Test Category BASIC`_)
4680 ========================= ================== ====================================
4681
4682 Typically a TriBITS project will define a "standard development environment"
4683 which is comprised of a standard compiler (e.g. GCC 8.3.0), external
4684 package/TPL versions (e.g. OpenMPI 4.0.5, Boost 4.9, etc.), and other tools
4685 (e.g. cmake 3.23.0, git 2.10.1, etc.). This standard development environment
4686 is expected to be used to test changes to the project's code before any push.
4687 By using a standard development environment, if the code builds and all the
4688 tests pass for the "default" pre-push builds for one developer, then that
4689 maximizes the probability that the code will also build and all tests will
4690 pass for every other developer using the same development environment. This
4691 is critical to keep the development team maximally productive. Portability is
4692 also important for most projects but portability testing is best done in a
4693 secondary feedback look using `Nightly Testing`_ builds. TriBITS has some
4694 support for helping to set up a standard software development environment as
4695 described in section `TriBITS Development Toolset`_.
4696
4697 The basic assumption of all CI processes (including the one described here) is
4698 that if anyone pulls the project's development sources at any time, then all
4699 of the code will build and all of the tests will pass for the "default" build
4700 cases. For a TriBITS project, this means that the project's
4701 ``--default-builds`` (see above) will all pass for every ``PT`` package. All
4702 of these software development processes make this basic assumption and agile
4703 software development methods fall apart if this is not true.
4704
4705 .. _Post-Push CI Testing:
4706
4707 **Post-Push CI Testing**
4708
4709 After changes are pushed to the master branch(es) in the global repository(s),
4710 *Post-Push CI Testing* is performed where a CI server detects the changes and
4711 immediately fires off a CI build using CTest to test the changes and the
4712 results are posted to a CDash server (in the "Continuous" section on the
4713 project's dashboard page). This process is driven by CTest driver code that
4714 calls `tribits_ctest_driver()`_ as described in the section `TriBITS
4715 CTest/CDash Driver`_. Various types of specific CI builds can be constructed
4716 and run (see `CTest/CDash CI Server`_) but these post-push CI builds typically
4717 select repositories, packages and code, and individual tests using the
4718 following test-related classifications:
4719
4720 ========================= ================== ====================================
4721 Classification Type Classification (See Reference)
4722 ========================= ================== ====================================
4723 Repository Test Classif. ``Continuous`` (`Repository Test Continuous`_)
4724 Package Test Group ``PT`` & ``ST`` (`PT`_ and `ST`_)
4725 Test Test Category ``CONTINUOUS`` (`Test Test Category CONTINUOUS`_)
4726 ========================= ================== ====================================
4727
4728 Post-push CI testing would assume to use the same standard development
4729 environment as used for `Pre-Push CI Testing`_. Also, the project may also
4730 choose to run additional automated post-push CI builds that exactly match the
4731 pre-push CI default builds to help check on the health of these builds
4732 continuously and not just rely on the development team to always perform the
4733 pre-push CI builds correctly before pushing.
4734
4735 .. _Nightly Testing:
4736
4737 **Nightly Testing**
4738
4739 In addition to pre-push and post-push CI testing, a typical TriBITS project
4740 will set up multiple *Nightly Testing* builds (or once-a-day builds, they
4741 don't need to only run at night). These builds are also driven by CTest
4742 driver scripts described in the section `TriBITS CTest/CDash Driver`_ and post
4743 results to the project's CDash server (in the "Nightly" section on the
4744 project's dashboard page). Nightly builds don't run in a continuous loop but
4745 instead are run once a day (e.g. driven by a cron job) and there tends to be
4746 many different nightly build cases that test the project using different
4747 compilers (e.g. GCC, Intel, Microsoft, etc., and different versions of each),
4748 different external package/TPL versions (e.g. different OpenMPI versions,
4749 different MPICH versions, etc.), different platforms (e.g. Linux, Windows,
4750 etc.), and varying many other options and settings on these different
4751 platforms. What all nightly builds have in common is that they tend to select
4752 repositories, packages and code, and individual tests using the following
4753 test-related classifications:
4754
4755 ========================= ================== ====================================
4756 Classification Type Classification (See Reference)
4757 ========================= ================== ====================================
4758 Repository Test Classif. ``Nightly`` (`Repository Test Nightly`_)
4759 Package Test Group ``PT`` & ``ST`` (`PT`_ and `ST`_)
4760 Test Test Category ``NIGHTLY`` (`Test Test Category NIGHTLY`_)
4761 ========================= ================== ====================================
4762
4763 The nightly builds comprise the basic "heart beat" for the project.
4764
4765 .. _Heavy Testing:
4766
4767 **Heavy Testing**
4768
4769 *Heavy Testing* builds are just an extension to the `Nightly Testing`_ builds
4770 that add on more expensive tests marked using the `Test Test Category HEAVY`_.
4771 For projects that define heavy tests and heavy builds, individual test cases
4772 may be allowed to take 24 hours or longer to run so they can't even be run
4773 every day in nightly testing. What standard heavy builds have in common is
4774 that they tend to select repositories, packages and code, and individual
4775 tests using the following test-related classifications:
4776
4777 ========================= ================== ====================================
4778 Classification Type Classification (See Reference)
4779 ========================= ================== ====================================
4780 Repository Test Classif. ``Nightly`` (`Repository Test Nightly`_)
4781 Package Test Group ``PT`` & ``ST`` (`PT`_ and `ST`_)
4782 Test Test Category ``HEAVY`` (`Test Test Category HEAVY`_)
4783 ========================= ================== ====================================
4784
4785 Project developer teams should strive to limit the number of test cases that
4786 are marked as ``HEAVY`` since these tests will typically *not* get run in very
4787 may builds or may not be run every day and developers will tend to never
4788 enable them when doing more extensive testing using ``--st-extra-builds`` with
4789 the `checkin-test.py`_ tool in extended pre-push testing.
4790
4791 .. _Performance Testing:
4792
4793 **Performance Testing**
4794
4795 *Performance Testing* builds are a special class of builds that have tests
4796 that are specifically designed to test the run-time performance of a particular
4797 piece of code or algorithm. These tests tend to be sensitive to loads on the
4798 machine and therefore typically need to be run on an unloaded machine for
4799 reliable results. Details on how to write good performance tests with hard
4800 pass/fail time limits is beyond the scope of this document. All TriBITS does
4801 is to define the special `Test Test Category PERFORMANCE`_ to allow TriBITS
4802 packages to declare these tests in a consistent way so that they can be run
4803 along with performance tests defined in other TriBITS packages. From a
4804 TriBITS standpoint, all performance testing builds would tend to select
4805 repositories, packages and code, and individual tests using the following
4806 test-related classifications:
4807
4808 ========================= ================== ====================================
4809 Classification Type Classification (See Reference)
4810 ========================= ================== ====================================
4811 Repository Test Classif. ``Nightly`` (`Repository Test Nightly`_)
4812 Package Test Group ``PT`` & ``ST`` (`PT`_ and `ST`_)
4813 Test Test Category ``PERFORMANCE`` (`Test Test Category PERFORMANCE`_)
4814 ========================= ================== ====================================
4815
4816 .. ToDo: I need to set up automated testing for TriBITS to use as the example
4817 .. for all of these types of testing. There is no better example that one
4818 .. that actually works. It would also be nice to have a snapshot repo of
4819 .. TribitsExampleProject that also had this testing enabled for it but I am
4820 .. not sure that really makes sense.
4821
4822
4823 .. _checkin-test.py:
4824
4825 Pre-push Testing using checkin-test.py
4826 --------------------------------------
4827
4828 CMake provides the integrated tool CTest (executable ``ctest``) which is used
4829 to define and run different tests. However, a lot more needs to be done to
4830 effectively test changes for a large project before pushing to the master
4831 branch(es) in the main repository(s). Things get especially complicated and
4832 tricky when multiple version-control (VC) repositories are involved. The
4833 TriBITS system provides the tool ``checkin-test.py`` for automating the
4834 process of:
4835
4836 1) Determining if the local VC repo(s) are ready to integrate with the remote
4837 master branch(es),
4838
4839 2) Pulling and integrating the most current changes from the remote VC repo(s),
4840
4841 3) Figuring out what TriBITS packages need to be enabled and testing (by
4842 examining VC file diffs),
4843
4844 4) Configuring only the necessary TriBITS packages and tests (and their
4845 downstream dependencies by default) and building, running tests, and
4846 reporting results (via email), and
4847
4848 5) Only if all specified builds and tests pass, amending the last commit
4849 message with the test results, then pushing local commits to the remove VC
4850 repo(s) and sending out summary emails.
4851
4852 There are several advantages to using a project's ``checkin-test.py`` tool for
4853 pushing changes to the main development branch which include:
4854
4855 a) provides a consistent definition for "okay to push" for all developers
4856
4857 b) protects other developers from pulling badly broken code
4858
4859 c) reduces the number of packages that need to be tested by automatically
4860 determining the set based on changed files and package dependencies
4861
4862 d) avoids developer mistakes in performing repetitive tasks and forgetting
4863 important steps in the process
4864
4865 e) marks a set of working commits that are safe to search with ``git bisect``
4866 to find problems (see `Using Git Bisect with checkin-test.py workflows`_)
4867
4868 When using the ``checkin-test.py`` tool, every TriBITS project defines one or
4869 more "default builds" (specified through the ``--default-builds`` argument)
4870 for pre-push CI testing that form the criteria for if it is okay to push code
4871 changes or not. The "default builds" select repositories, packages and
4872 code, and individual tests as described in `Pre-Push CI Testing`_. A TriBITS
4873 project defines its default pre-push builds using the file
4874 `<projectDir>/project-checkin-test-config.py`_. For an example, the file
4875 `TribitsExampleProject`_/``project-checkin-test-config.py`` is shown below:
4876
4877 .. include:: ../../examples/TribitsExampleProject/project-checkin-test-config.py
4878 :literal:
4879
4880 This gives ``--default-builds=MPI_DEBUG,SERIAL_RELEASE``. As shown, typically
4881 two default builds are defined so that various options can be toggled between
4882 the two builds. Typical options to toggle include enabling/disabling MPI and
4883 enabling/disabling run-time debug mode checking (i.e. toggle
4884 ``${PROJECT_NAME}_ENABLE_DEBUG``). Typically, other important options will
4885 also be toggled between these two builds.
4886
4887 Note that both of the default builds shown above, including the ``MPI_DEBUG``
4888 build, actually set optimized compiler flags with
4889 ``-DCMAKE_BUILD_TYPE:STRING=RELEASE``. What makes the ``MPI_DEBUG`` build a
4890 "debug" build is turning on optional run-time debug-mode checking, not
4891 disabling optimized code. This is important so that the defined tests run
4892 fast. For most projects, the default pre-push builds should **not** be used
4893 to debug-enabled code which is suitable to run through a debugger
4894 (e.g. ``gdb``). Instead, these "debug" builds are designed to test changes to
4895 the project's code efficiently before pushing changes. Typically, a
4896 development team should not have to test the chosen compiler's ability to
4897 generate non-optimized debug code and suffer slower test times before pushing.
4898
4899 Note that turning on ``-DTribitsExProj_ENABLE_CHECKED_STL=ON`` as shown above
4900 can only be used when the external packages/TPLs have no C++ code using the
4901 C++ STL or if that particular build points to C++ TPLs also compiled with
4902 checked STL enabled. The `TribitsExampleProject`_ default builds do not
4903 depend on any C++ TPLs that might use the C++ STL so enabling this option adds
4904 additional positive debug-mode checking for C++ code.
4905
4906 The ``checkin-test.py`` tool is a fairly sophisticated piece of software that
4907 is well tested and very robust. The level of testing of this tool is likely
4908 greater than any of the software that it will be used to test (unless the
4909 project is a real-time flight control system or nuclear reactor control system
4910 or something). This is needed so as to provide confidence in the developers
4911 that the tool will only push their changes if everything checks out as it
4912 should. There are a lot of details and boundary cases that one has to
4913 consider and a number of use cases that need to be supported by such a tool.
4914 For more detailed documentation, see `checkin-test.py --help`_.
4915
4916 Note that the ``checkin-test.py`` tool can also be used to implement
4917 "poor-man's" post-push testing processes as described in `Post-Push CI and
4918 Nightly Testing using checkin-test.py`_. However, most software projects will
4919 want to go with the more elaborate and more feature-full CTest/CDash system
4920 described in `TriBITS CTest/CDash Driver`_.
4921
4922 .. ToDo: Describe the standard workflow for using the checkin-test.py tool.
4923
4924 .. ToDo: Describe why the --default-builds must only include PT code and not
4925 .. ST due to changing the behavior of the PT software. As an example, discuss
4926 .. Boost usage in Teuchos and how that changes behavior of a few things.
4927 .. Another example is BinUtils.
4928
4929 .. ToDo: Describe the system for mapping changed files to changed packages.
4930 .. Discuss how important the directory structure is.
4931
4932 .. ToDo: Describe the local test, remote push process by pulling from the
4933 .. Trilinos developers webpage.
4934
4935 .. ToDo: Describe the role that checkin-test.py plays in multi-repo ACI
4936
4937
4938 TriBITS CTest/CDash Driver
4939 --------------------------
4940
4941 The TriBITS system uses a sophisticated and highly customized CTest -S driver
4942 script to test TriBITS projects and submit results to a CDash server. The
4943 primary code for driving this is contained in the CTest function
4944 `tribits_ctest_driver()`_ contained in the file
4945 ``TribitsCTestDriverCore.cmake``. This script loops through all of the
4946 specified TriBITS packages for a given TriBITS project and does a configure,
4947 built, and test and then submits results to the specified CDash server
4948 incrementally. If the configure or library build of any upstream TriBITS
4949 package fails, then that TriBITS package is disabled in all downstream TriBITS
4950 package builds so as not to propagate already known failures. Each TriBITS
4951 top-level package is assigned its own CDash regression email address (see
4952 `CDash regression email addresses`_) and each package configure/build/test is
4953 given its own row for the package build in the CDash server. A CTest script
4954 using `tribits_ctest_driver()`_ is run in one of three different modes.
4955 First, it can run standard once-a-day, from-scratch builds as described in
4956 `CTest/CDash Nightly Testing`_. Second, it can run as a CI server as
4957 described in `CTest/CDash CI Server`_. Third, it can run in experimental mode
4958 testing a local repository using the TriBITS-defined `make dashboard`_ target.
4959
4960 .. ToDo: Discuss the behavior of Continuous, Nightly, and Experimental
4961 .. CTest/CDash tracks and what that means in terms of sending out CDash error
4962 .. emails and allowing for multiple builds of the same name in a single test
4963 .. day.
4964
4965 .. ToDo: Document CTEST_TEST_TIMEOUT and DART_TESTING_TIMEOUT and how these
4966 .. interact.
4967
4968 .. ToDo: Import and format the contents of tribits/ctest/README related to
4969 .. this topic.
4970
4971
4972 CTest/CDash Nightly Testing
4973 +++++++++++++++++++++++++++
4974
4975 When a TriBITS CTest script using `tribits_ctest_driver()`_ is run in
4976 "Nightly" testing mode, it builds the project from scratch package-by-package
4977 and submits results to the TriBITS project's CDash project on the designated
4978 CDash server.
4979
4980 .. ToDo: Give example of the package-by-package build.
4981
4982 CTest/CDash CI Server
4983 +++++++++++++++++++++
4984
4985 When a TriBITS ctest driver script is used in continuous integration (CI)
4986 mode, it starts every day with a clean from-scratch build and then performs
4987 incremental rebuilds as new commits are pulled from the master branch in the
4988 main repository(s). In this mode, a continuous loop is performed after the
4989 initial baseline build constantly pulling commits from the master git
4990 repository(s). If any package changes are detected (looking at git file
4991 diffs), then the tests and examples for those packages and all downstream
4992 packages are enabled and run using a reconfigure/rebuild. Since all of the
4993 upstream package libraries are already built, this rebuild and retest can take
4994 place in a fraction of the time of a complete from-scratch build and test of
4995 the project.
4996
4997 .. ToDo Give some examples of the behavior of this process for the
4998 .. ReducedMockTrilins project. We could show example output from the script
4999 .. that shows how it behaves.
5000
5001 .. ToDo: Discuss the usage of ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE
5002 .. and ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE_EXCEPT to allow the CI
5003 .. server to directly processing and running tests for upstream packages which
5004 .. are not primary packages in the TriBITS meta-project.
5005
5006 .. ToDo: Mention that failing packages are always explicitly processed even if
5007 .. the current CI iteration does not pull any changes.
5008
5009 .. ToDo: Give some examples from ReducedMockTrilinos of how file changes
5010 .. result in package enables and rebuilds.
5011
5012
5013 TriBITS CDash Customizations
5014 ----------------------------
5015
5016 CDash is not currently designed to accommodate multi-package, multi-repository
5017 VC projects in the way they are supported by TriBITS. However, CDash provides
5018 some ability to customize a CDash project and submits to address missing
5019 features. Each TriBITS package is given a CTest/CDash "Label" with the name
5020 of the TriBITS package. CDash will then aggregate the different package
5021 configure/build/test runs for each package into aggregated "builds". The
5022 commits pulled for each of the extra VC repos listed in the
5023 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file are shown in an
5024 uploaded CDash "Notes" file for each TriBITS package configure/build/test
5025 submit. This uploaded "Notes" file also contains a cleaned up version of the
5026 CMakeCache.txt file as well as a copy of the ``ctest -s`` script that ran the
5027 case.
5028
5029 CDash offers numerous features such as the ability to construct a number of
5030 different types of queries and is extremely helpful in using past test data.
5031
5032 .. ToDo: Describe more on what CDash provides and how TriBITS uses it. This
5033 .. could be a *long* section with a lot of screen shots.
5034
5035
5036 CDash regression email addresses
5037 ++++++++++++++++++++++++++++++++
5038
5039 Every TriBITS Package has a regression email address associated with it that
5040 gets uploaded to a CDash project on a CDash server that is used to determine
5041 what email address to use when a package has configure, build, or test
5042 failures. Because of the complex organizational nature of different projects
5043 and different integration models, a single static email address for a given
5044 package in every project build is not practical.
5045
5046 The TriBITS system allows for a package's regression email address to be
5047 specified in the following order of precedence:
5048
5049 .. _${REPOSITORY_NAME}_REPOSITORY_OVERRIDE_PACKAGE_EMAIL_LIST:
5050
5051 1) **${REPOSITORY_NAME}_REPOSITORY_OVERRIDE_PACKAGE_EMAIL_LIST** (typically
5052 defined in `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_): Defines a
5053 single email address for all packages for the repository
5054 ``${REPOSITORY_NAME}`` and overrides all other package email regression
5055 specification variables. This is typically used by a meta-project to redefine
5056 the regression email addresses for all of the packages in an externally
5057 developed repository.
5058
5059 2) **REGRESSION_EMAIL_LIST** (defined in
5060 `<packageDir>/cmake/Dependencies.cmake`_): Package-specific email address
5061 specified in the package's ``Dependencies.cmake`` file using
5062 `tribits_package_define_dependencies()`_.
5063
5064 .. _${REPOSITORY_NAME}_REPOSITORY_EMAIL_URL_ADDRESS_BASE:
5065
5066 3) **${REPOSITORY_NAME}_REPOSITORY_EMAIL_URL_ADDRESS_BASE** (set in
5067 `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_): A base email address
5068 specified at the Repository level creating package-specific email addresses
5069 (e.g. ``<lower-case-package-name>[email protected]``, where
5070 ``${REPOSITORY_NAME}_REPOSITORY_EMAIL_URL_ADDRESS_BASE=some.repo.gov``).
5071 This variable is used, for example, by the Trilinos project to provide
5072 automatic regression email addresses for packages.
5073
5074 .. _${REPOSITORY_NAME}_REPOSITORY_MASTER_EMAIL_ADDRESS:
5075
5076 4) **${REPOSITORY_NAME}_REPOSITORY_MASTER_EMAIL_ADDRESS** (set in
5077 `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_): A single email address
5078 for all packages specified at the Repository level
5079 (e.g. ``[email protected]``). This variable is used for
5080 smaller repositories with smaller development groups who just want all
5081 regression emails for the repository's packages going to a single email
5082 address. This reduces the overhead of managing a bunch of individual package
5083 email addresses but at the expense of spamming too many people with CDash
5084 failure emails.
5085
5086 .. _${PROJECT_NAME}_PROJECT_EMAIL_URL_ADDRESS_BASE:
5087
5088 5) **${PROJECT_NAME}_PROJECT_EMAIL_URL_ADDRESS_BASE** (set in
5089 `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_): A base email address
5090 specified at the Project level creating package-specific email addresses
5091 (e.g. ``<lower-case-package-name>[email protected]``, where
5092 ``${PROJECT_NAME}_PROJECT_EMAIL_URL_ADDRESS_BASE=some.project.gov``). If not
5093 already set, this variable will be set to
5094 ``${REPOSITORY_NAME}_REPOSITORY_EMAIL_URL_ADDRESS_BASE`` for the first
5095 repository processed that has this set. This behavior is used, for example by
5096 the Trilinos project to automatically assign email addresses for add-on
5097 packages and was added to maintain backward compatibility.
5098
5099 .. _${PROJECT_NAME}_PROJECT_MASTER_EMAIL_ADDRESS:
5100
5101 6) **${PROJECT_NAME}_PROJECT_MASTER_EMAIL_ADDRESS** (set in
5102 `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_): A single default email
5103 address for all packages specified at the Project level
5104 (e.g. ``[email protected]``). If not already set, this
5105 variable will be set to
5106 ``${REPOSITORY_NAME}_REPOSITORY_MASTER_EMAIL_ADDRESS`` for the first
5107 repository processed that has this set. Every meta-project should set this
5108 variable so that it will be the default email address for any new package
5109 added.
5110
5111 WARNING: If any of the email lists or URL string variables listed above are
5112 set to ``"OFF"`` or ``"FALSE"`` (or some other value that CMake interprets as
5113 false, see `CMake Language Overview and Gotchas`_) then the variables are
5114 treated as empty and not set.
5115
5116 If a TriBITS project does not use CDash, then no email address needs to be
5117 assigned to packages at all (and therefore none of the above variables need be
5118 set).
5119
5120 As a general rule, repository-level settings override project-level settings
5121 and package-level settings override both. Also, a project can redefine a
5122 repository's regression email list settings by resetting the variables in the
5123 project's `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_ file.
5124
5125 All of the email dependency management logic must be accessible by just running
5126 the macro::
5127
5128 tribits_read_all_project_deps_files_create_deps_graph()
5129
5130 The above email address configuration variables are read from the Repository
5131 and Project files `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_ and
5132 `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_, respectively. The
5133 ``RepositoryDependenciesSetup.cmake`` files are read first in the specified
5134 repository order followed up by reading the ``ProjectDependenciesSetup.cmake``
5135 file. In this way, the project can override any of the repository settings.
5136
5137 In review, the precedence order for how regression email addresses are
5138 selected for a given package is:
5139
5140 1) Package-specific email list is selected if defined (unless an override is
5141 in place).
5142
5143 2) Repository-level option is selected over a project-level option.
5144
5145 3) Default email form with repository or project address base is selected over
5146 single repository or project email address.
5147
5148 4) If none of the above are selected, then no email address is assigned for a
5149 given package.
5150
5151 What the above setup does is it results in the TriBITS system (in the
5152 `tribits_ctest_driver()`_ function called in a ``ctest -S`` script) creating a
5153 file called ``CDashSubprojectDependencies.xml`` (which contains the list of
5154 TriBITS packages, which CDash calls "subprojects", and email address for each
5155 package to send regression emails to) and that file gets sent to the CDash
5156 server. CDash then takes this file and creates, or updates, a set of CDash
5157 users (same name and password as the email list address) and sets up a mapping
5158 of Labels (which are used for TriBITS package names) to CDash user emails
5159 addresses. CDash is automatically set up to process this XML file and create
5160 and update CDash users. There are several consequences of this implementation
5161 of which project maintainers need to be aware.
5162
5163 First, **one should not list the email address for a CDash user account
5164 already on CDash**. This is because labels will be added for the TriBITS
5165 packages that this email address is associated with and CDash emails will not
5166 be sent out for any other TriBITS packages, no matter what setting that CDash
5167 user account has. Therefore, one should only list email addresses as CDash
5168 regression email lists that are not already CDash user accounts and wish to be
5169 maintained separately. For example, if there is an email list that one wants
5170 to have CDash emails sent to but is already a CDash user account, then one can
5171 create another email list (e.g. using Mailman) which can then be registered
5172 with the TriBITS packages in the ``CDashSubprojectDependencies.xml`` file and
5173 then that new email list forward email to the target email list.
5174
5175 Second, the CDash implementation currently is not set up to remove labels from
5176 existing users when an email address is disassociated with a TriBITS package
5177 in the ``CDashSubprojectDependencies.xml`` file. Therefore, **if one changes
5178 a TriBITS package's CDash regression email address then one needs to manually
5179 remove the associated labels from the old email address**. CDash will not
5180 remove them automatically. Otherwise, email will continue to be sent to that
5181 email address for that package.
5182
5183 Therefore, **to change the mapping of CDash regression email addresses to
5184 TriBITS packages, one must perform the following actions**:
5185
5186 1) Change the TriBITS CMake files as described above that will result in the
5187 desired email addresses in the ``CDashSubprojectDependencies.xml``
5188 file. One can debug this by generating the file
5189 `<Project>PackageDependencies.xml`_ by using the cmake -P script
5190 `TribitsDumpDepsXmlScript.cmake`_.
5191
5192 2) Log onto the CDash server using an administrator account and then remove
5193 the auto-generated account for the CDash user email address for which
5194 labels are being removed (i.e. no longer associated with a TriBITS
5195 package). This is needed since CDash seems to be unable to remove labels
5196 from an existing CDash user (however this might be fixed in a current
5197 version of CDash).
5198
5199 3) The next time a CDash submit is performed by a CTest driver script calling
5200 `tribits_ctest_driver()`_, the CDash user associated with the mail list
5201 with labels being removed will get automatically recreated with the right
5202 list of labels (according to the current
5203 ``CDashSubprojectDependencies.xml`` file). Also, any new CDash users for
5204 new email addresses will be created.
5205
5206 Hopefully that should be enough information to manage the mapping of CDash
5207 regression email lists to TriBITS packages for single and multi-repository
5208 TriBITS projects.
5209
5210
5211 Multi-Repository Support
5212 ========================
5213
5214 TriBITS has built-in support for projects involving multiple `TriBITS
5215 Repositories`_ which contain multiple `TriBITS Packages`_ (see `How to set up
5216 multi-repository support`_). The basic configuration, build, and test of such
5217 projects requires only raw CMake/CTest, just like any other CMake project (see
5218 `TriBITS System Project Dependencies`_). Every TriBITS project automatically
5219 supports tacking on add-on TriBITS packages and external packages/TPLs through
5220 the `${PROJECT_NAME}_EXTRA_REPOSITORIES`_ cmake cache variable as described in
5221 `Enabling extra repositories with add-on packages`_. In addition, a TriBITS
5222 project can be set up to pull in other TriBITS Repositories using the
5223 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file. A special form of
5224 this type of project is a `TriBITS Meta-Project`_ that contains no native
5225 packages or TPLs of its own. The ability to create meta-projects out of
5226 individual TriBITS repositories allows TriBITS to be used to provide
5227 coordinated builds (or meta-builds) of large aggregations of software.
5228
5229 To help set up a full-featured development environment (i.e. not just the
5230 basic configure, build, test, and install) for TriBITS projects with multiple
5231 repositories, TriBITS provides some extra development tools implemented using
5232 Python which are provided in the "extended" parts of TriBITS (see
5233 `TriBITS/tribits/ Directory Contents`_). The primary tools supporting
5234 multi-repository projects are the Python tools `clone_extra_repos.py`_,
5235 `gitdist`_, and `checkin-test.py`_.
5236
5237 To demonstrate, consider the TriBITS meta-project with the following
5238 ``ExtraRepositoriesList.cmake`` file::
5239
5240 tribits_project_define_extra_repositories(
5241 ExtraRepo1 "" GIT [email protected]:ExtraRepo1 "" Continuous
5242 ExtraRepo2 "ExtraRepo1/ExtraRepos2" GIT [email protected]:ExtraRepo2
5243 NOPACKAGES Continuous
5244 ExtraRepo3 "" GIT [email protected]:ExtraRepo3 "" Nightly
5245 )
5246
5247 Once cloned, the directories would be laid out as::
5248
5249 MetaProject/
5250 .git/
5251 .gitignore
5252 ExtraRepo1/
5253 .git/
5254 ExrraRepo2/
5255 .git/
5256 ExtraRepo3/
5257 .git/
5258
5259 .. _clone_extra_repos.py:
5260
5261 The tool **clone_extra_repos.py** is used to clone the extra repositories for
5262 a multi-repositories TriBITS project. It reads the repository URLs and
5263 destination directories from the file
5264 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ and does the clones. For
5265 example, to clone all the repos for the ``MetaProject`` project, one would use
5266 the commands::
5267
5268 $ git clone [email protected]:MetaProject
5269 $ cd MetaProject/
5270 $ ./cmake/tribits/ci_support/clone_extra_repos.py
5271
5272 which produces the output like::
5273
5274 ...
5275
5276 ***
5277 *** Clone the selected extra repos:
5278 ***
5279
5280 Cloning repo ExtraRepo1 ...
5281
5282 Running: git clone [email protected]:ExtraRepo1 ExtraRepo1
5283
5284 Cloning repo ExtraRepo2 ...
5285
5286 Running: git clone [email protected]:ExtraRepo2 ExtraRepo1/ExtraRepo2
5287
5288 Cloning repo ExtraRepo3 ...
5289
5290 Running: git clone [email protected]:ExtraRepo3 ExtraRepo3
5291
5292 See `clone_extra_repos.py --help`_ for more details.
5293
5294 .. _gitdist:
5295
5296 Once cloned, one needs to work with the multiple repositories to perform basic
5297 VC operations. For this, TriBITS provides the tool **gitdist** which is a
5298 simple stand-alone Python script that distributes a git command across a set
5299 of git repos. This tool is not specific to TriBITS but it is very useful for
5300 dealing with TriBITS projects with multiple repositories. It only requires a
5301 local base git repo and a set of zero or more git repos cloned under it.
5302
5303 To use ``gitdist`` with this aggregate meta-project, one would first set up
5304 the file ``MetaProject/.gitdist`` (or a version-controlled
5305 ``MetaProject.gitdist.default`` file) which would contain the lines::
5306
5307 ExtraRepo1
5308 ExtraRepo1/ExtraRepo2
5309 ExtraRepo3
5310
5311 and one would set up the tracked ignore file ``MetaProject/.gitignore`` which
5312 contains the lines::
5313
5314 /ExtraRepo1/
5315 /ExtraRepo1/ExtraRepo2/
5316 /ExtraRepo3/
5317
5318 To use ``gitdist``, one would put ``gitdist`` into their path and also set up
5319 the command-line shell aliases ``gitdist-status`` and ``gitdist-mod`` (see
5320 `gitdist --dist-help=aliases`_).
5321
5322 Some of the aggregate commands that one would typically run under the base
5323 ``MetaProject/`` directory are::
5324
5325 # See status of all repos at once
5326 gitdist-status
5327
5328 # Pull updates to all
5329 gitdist pull
5330
5331 # Push local commits to tracking branches
5332 gitdist push
5333
5334 The tool ``gitdist`` is provided under TriBITS directory::
5335
5336 cmake/tribits/python_utils/gitidst
5337
5338 and can be installed by the `install_devtools.py`_ tool (see `TriBITS
5339 Development Toolset`_). See `gitdist documentation`_ for more details.
5340
5341 For projects with a standard set of extra repositories defined in the
5342 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file, the
5343 ``checkin-test.py`` tool only requires passing in the option
5344 ``--extra-repos-file=project`` and ``--extra-repos-type=Continuous`` (or
5345 ``Nightly``, see `Repository Test Classification`_) and it will automatically
5346 perform all of the various actions for all of the selected repositories. See
5347 `checkin-test.py`_ and `checkin-test.py --help`_ for more details.
5348
5349 To keep track of compatible versions of the git repos, TriBITS provides
5350 support for a ``<Project>RepoVersion.txt`` file. Any TriBITS project can
5351 generate this file automatically by setting the option
5352 `${PROJECT_NAME}_GENERATE_REPO_VERSION_FILE`_. For the above example
5353 ``MetaProject``, this file looks like::
5354
5355 *** Base Git Repo: MetaProject
5356 e102e27 [Mon Sep 23 11:34:59 2013 -0400] <[email protected]>
5357 First summary message
5358 *** Git Repo: ExtraRepo1
5359 b894b9c [Fri Aug 30 09:55:07 2013 -0400] <[email protected]>
5360 Second summary message
5361 *** Git Repo: ExtraRepo1/ExtraRepo2
5362 97cf1ac [Thu Dec 1 23:34:06 2012 -0500] <author2someurl.com>
5363 Third summary message
5364 *** Git Repo: ExtraRepo3
5365 cd4a3af [Mon Mar 9 19:39:06 2013 -0400] <author3someurl.com>
5366 Fourth summary message
5367
5368 This file gets created in the build directory, gets echoed in the configure
5369 output, gets installed into the install directory, and get added to the source
5370 distributions tarball. It also gets pushed up to CDash for all automated
5371 builds. The tool `gitdist`_ can then use this file to checkout and tag
5372 compatible versions, difference two versions of the meta-project, etc. (see
5373 `gitdist documentation`_ for more details on git operations).
5374
5375 The TriBITS approach to managing multiple VC repos described above works well
5376 for around 20 or 30 VC repos but is likely not a good solution for many more
5377 git repos. For larger numbers of VC repos, one should consider nested
5378 integration creating snapshot git repos (e.g. using the tool
5379 `snapshot-dir.py`_) that aggregate several related repositories into a single
5380 git repo. Another approach might be to use git submodules. (However, note
5381 that the TriBITS tools and processes described here are **not** currently set
5382 up to support aggregate VC repos that use git submodules.) The design
5383 decision with TriBITS was to explicitly handle the different git VC repos by
5384 listing them in the `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file and
5385 then using the simple, easy to understand, tools `clone_extra_repos.py`_ and
5386 `gitdist`_. There are advantages and disadvantages to explicitly handling the
5387 different git repos as is currently employed by the TriBITS software
5388 development tools verses using git submodules. It is possible that TriBITS
5389 will add support for aggregate git repos using git submodules in the future
5390 but only if there are important projects that choose to use them. The
5391 discussion of these various approaches and strategies to dealing with
5392 aggregate repos is beyond the scope of this document.
5393
5394
5395 Development Workflows
5396 ======================
5397
5398 In this section, the typical development workflows for a TriBITS project are
5399 described. First, the `Basic Development Workflow`_ for a single-repository
5400 TriBITS project is described. This is followed up with a slightly more
5401 complex `Multi-Repository Development Workflow`_.
5402
5403
5404 Basic Development Workflow
5405 --------------------------
5406
5407 The basic development workflow of a TriBITS project is not much different than
5408 with any other CMake project that uses CTest to define and run tests. One
5409 pulls updates from the master VC repo then configures with ``cmake``, and
5410 iteratively builds, runs tests, adds files, changes files, does a final test,
5411 then pushes updates. The major difference is that a well constructed
5412 development process will use the `checkin-test.py`_ tool to test and push all
5413 changes that affect the build or the tests. The basic steps in configuring,
5414 building, running tests, etc., are given in the project's
5415 `<Project>BuildReference`_. file (see `Project-Specific Build Reference`_).
5416
5417 Multi-Repository Development Workflow
5418 -------------------------------------
5419
5420 The development workflow for a project with multiple VC repos is very similar
5421 to a project with just a single VC repo if the project provides a standard
5422 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ file. The major difference
5423 is in making changes, creating commits, etc. The `gitdist`_ tool makes these
5424 steps easier and has been shown to work fairly well for up to 20 extra VC
5425 repos (as used in the CASL VERA project). The `checkin-test.py`_ tool
5426 automatically handles all of the details of pulling, diffing, pushing etc. to
5427 all the VC repos.
5428
5429
5430
5431 .. ToDo: Discuss usage of 'gitdist' and the repo clone script.
5432
5433
5434 Howtos
5435 ======
5436
5437 This section provides short, succinct lists of the steps to accomplish a few
5438 common tasks. Extra details are referenced.
5439
5440
5441 How to add a new TriBITS Package
5442 --------------------------------
5443
5444 To add a new TriBITS package, it is recommended to take the template from one
5445 of the `TribitsExampleProject`_ packages that most closely fits the needs of
5446 the new package and modify it for the new package. For example, the files for
5447 the ``SimpleCxx`` package can be copied one at a time and modified for the new
5448 package.
5449
5450 To add a new TriBITS package (with no subpackages), do the following:
5451
5452 1) Chose a name ``<packageName>`` for the new package and which TriBITS
5453 repository (``<repoDir>``) to put the package into. **WARNING!** The
5454 chosen name ``<packageName>`` must be unique across all TriBITS
5455 repositories (see `Globally unique TriBITS package names`_).
5456
5457 2) Create the directory ``<repoDir>/<packageDir>/`` for the new package and
5458 put in skeleton files for `<packageDir>/cmake/Dependencies.cmake`_ and
5459 `<packageDir>/CMakeLists.txt`_. Set the desired upstream TPL and package
5460 dependencies in the new ``Dependencies.cmake`` file but initially comment
5461 out everything in the ``CMakeLists.txt`` file except for the
5462 `tribits_package()`_ and `tribits_package_postprocess()`_ commands.
5463
5464 3) Add a row for the new package to the `<repoDir>/PackagesList.cmake`_ file
5465 after all of its upstream dependent packages. If a mistake is made and it
5466 is listed before one of its upstream dependent packages, the TriBITS CMake
5467 code will catch this and issue an error.
5468
5469 4) Configure the TriBITS project enabling the new empty package
5470 ``<packageName>``. This will enable the listed dependencies.
5471
5472 5) Incrementally fill in the package's ``CMakeLists.txt`` files defining
5473 libraries, executables, tests and examples. The project should be built
5474 and tests run as new pieces are added.
5475
5476 Once the new package is defined, downstream packages can define
5477 dependencies on this new package.
5478
5479 .. ToDo: Expand on the above bullets a lot!
5480
5481
5482 How to add a new TriBITS Package with Subpackages
5483 -------------------------------------------------
5484
5485 Adding a new package with subpackages is similar to adding a new regular
5486 package described in `How to add a new TriBITS Package`_. Again, it is
5487 recommended that one copies an example package from `TribitsExampleProject`_.
5488 For example, one could copy files and directories from the example package
5489 ``WithSubpackages``.
5490
5491 To add a new TriBITS package with packages, do the following:
5492
5493 1) Chose a name ``<packageName>`` for the new package and which TriBITS
5494 repository (``<repoDir>``) to put the package into. **WARNING!** The
5495 chosen name ``<packageName>`` must be unique across all TriBITS
5496 repositories (see `Globally unique TriBITS package names`_).
5497
5498 2) Create the directory ``<repoDir>/<packageDir>/`` for the new package and
5499 put in skeleton files for `<packageDir>/cmake/Dependencies.cmake`_ and
5500 `<packageDir>/CMakeLists.txt`_. Initially don't define any subpackages and
5501 comment out everything in the ``CMakeLists.txt`` file except for the
5502 `tribits_package()`_ and `tribits_package_postprocess()`_ commands.
5503
5504 3) Add a new row for the new package to the `<repoDir>/PackagesList.cmake`_
5505 file after all of the upstream dependencies of its to-be-defined
5506 subpackages.
5507
5508 4) Configure the TriBITS project enabling the new empty package
5509 ``<packageName>``.
5510
5511 5) Incrementally add the subpackages as described in `How to add a new TriBITS
5512 Subpackage`_, filling out the various ``CMakeLists.txt`` files defining
5513 libraries, executables, tests and examples.
5514
5515 Once the new packages are defined, downstream packages can define
5516 dependencies on these.
5517
5518
5519 How to add a new TriBITS Subpackage
5520 -----------------------------------
5521
5522 Given an existing top-level TriBITS package that is already broken down into
5523 subpackages (see `How to add a new TriBITS Package with Subpackages`_), adding
5524 a new subpackage does not require changing any project-level or
5525 repository-level files. One only needs to add the declaration for the new
5526 subpackages in its parent's `<packageDir>/cmake/Dependencies.cmake`_ file then
5527 fill out the pieces of the new subpackage defined in the section `TriBITS
5528 Subpackage Core Files`_. It is recommended to copy files from one of the
5529 `TribitsExampleProject`_ subpackages in the ``WithSubpackages``
5530 package.
5531
5532 To add a new TriBITS subpackage to a top-level package that already has
5533 subpackages, do the following:
5534
5535 1) Chose a name ``<spkgName>`` for the new subpackage which only has to be
5536 different than the other subpackages in the parent package. This name gets
5537 appended to the parent package's name ``<packageName>`` to form the package
5538 name ``<packageName><spkgName>``.
5539
5540 2) Create the directory ``<packageDir><spkgDir>/`` for the new package and put
5541 in skeleton files for `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_
5542 and `<packageDir>/<spkgDir>/CMakeLists.txt`_. Set the desired upstream TPL
5543 and package dependencies in the new ``Dependencies.cmake`` file but
5544 initially comment out everything in the ``CMakeLists.txt`` file except for
5545 the `tribits_subpackage()`_ and `tribits_subpackage_postprocess()`_
5546 commands.
5547
5548 3) Add a row for the new subpackage to the argument
5549 `SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS`_ in the macro call
5550 `tribits_package_define_dependencies()`_ in the parent package's
5551 `<packageDir>/cmake/Dependencies.cmake`_ file after all of its upstream
5552 dependent subpackages. If a mistake is made and it is listed before one of
5553 its upstream dependent subpackages, the TriBITS CMake code will catch this
5554 and issue an error.
5555
5556 4) Configure the TriBITS project enabling the new empty package
5557 ``<packageName><spkgName>``. This will enable the listed dependencies.
5558
5559 5) Incrementally fill in the subpackage's ``CMakeLists.txt`` files defining
5560 libraries, executables, tests and examples. The project should be built
5561 and tests run as new pieces are added.
5562
5563
5564 .. _How to add a new TriBITS TPL:
5565
5566 How to add a new TriBITS external package/TPL
5567 ---------------------------------------------
5568
5569 In order for a TriBITS package to define a dependency on a new `TriBITS
5570 External Package/TPL`_ (i.e. a TPL that has not already been declared in the
5571 current repo's or an upstream repo's `<repoDir>/TPLsList.cmake`_ file), one
5572 must add and modify a few repository-level files in addition to modifying
5573 files within the TriBITS packages that use the new external package/TPL.
5574
5575 To add a new TriBITS TPL, do the following:
5576
5577 1) **Chose a name <tplName>** for the new TPL (must be globally unique across
5578 all TriBITS repos. (See `Globally unique TriBITS TPL names`_.)
5579
5580 2) **Choose the subdirectory <tplDefsDir> for the new TPL files**. These
5581 files are usually placed under the TriBITS repository directory
5582 ``<repoDir>/`` (e.g. ``<repoDir>/cmake/tpls/``) where the first downstream
5583 dependent package is defined or can be under a TriBITS package directory
5584 ``<packageDir>/`` (e.g. ``<packageDir>/cmake/tpls/``) if only one package
5585 is using that TPL.
5586
5587 3) **Create the FindTPL<tplName>.cmake file** (or some other name, see
5588 `<tplName>_FINDMOD`_) under ``<tplDefsDir>/``. (See `Creating the
5589 FindTPL<tplName>.cmake file`_.) However, if the external package/TPL is a
5590 `TriBITS-compliant external package`_ this file is not needed and is
5591 ignored.
5592
5593 4) **[Optional] Create the FindTPL<tplName>Dependencies.cmake file** in the
5594 same directory as the ``FindTPL<tplName>.cmake`` file, ``<tplDefsDir>/``.
5595 (See `FindTPL<tplName>Dependencies.cmake`_.) NOTE: This file is need for a
5596 `TriBITS-compliant external package`_ if it has upstream dependent external
5597 packages/TPLs (where the file ``FindTPL<tplName>.cmake`` is not needed).
5598
5599 5) **Add a row to the <repoDir>/TPLsList.cmake file** for the new external
5600 package/TPL after any upstream TPLs that this new TPL may depend on. NOTE:
5601 For a `TriBITS-compliant external package`_, the special value
5602 ``TRIBITS_PKG`` is used for the TPL (See `<repoDir>/TPLsList.cmake`_.)
5603
5604 6) **Configure the TriBITS project enabling the new TPL with
5605 TPL_ENABLE_<tplName>=ON** and see that the TPL is found correctly at
5606 configure time.
5607
5608 7) **Add <tplName> to the package Dependencies.cmake files** of downstream
5609 dependent packages that will use this TPL (see
5610 `<packageDir>/cmake/Dependencies.cmake`_ or
5611 `<packageDir>/<spkgDir>/cmake/Dependencies.cmake`_ for a subpackage).
5612
5613 8) **[Optional] Add #cmakedefine for an optional package LIB TPL dependency in
5614 the package's <packageName>_config.h.in file** using::
5615
5616 #cmakedefine HAVE_<PACKAGE_NAME_UC>_<TPL_NAME_UC>
5617
5618 so that the package's LIB code build knows if the TPL is defined or not
5619 (see `<packageName>_config.h.in`_ and `tribits_configure_file()`_). (NOTE:
5620 Do **not** add this define for a optional test-only TPL dependency. We
5621 don't want all of the libraries in a package to have to be rebuild when we
5622 enable or disable tests for the package.)
5623
5624 9) **Use the TPL functionality in the packages** that define the dependency on
5625 the new TPL, configure, test, etc.
5626
5627
5628 .. _Creating the FindTPL<tplName>.cmake file:
5629
5630 **Creating the FindTPL<tplName>.cmake file**
5631
5632 The main axes of variation in defining ``FindTPL<tplName>.cmake`` modules are:
5633
5634 a) Use an inner ``find_package(<externalPkg>)`` call or just use a simple list
5635 of header file names and library names?
5636
5637 b) The call ``find_package(<externalPkg>)`` defines modern IMPORTED targets or
5638 only provides variables for the list of include directories and libraries?
5639
5640 c) Maintain backward compatibility with the legacy TriBITS TPL system that
5641 allows the user to set cache variables for the include directories and link
5642 libraries or force users to always call ``find_package(<externalPkg>)``?
5643
5644 See the different variations in the below sections:
5645
5646 * `Creating FindTPL<tplName>.cmake using find_package()`_
5647
5648 * `Creating FindTPL<tplName>.cmake using find_package() with IMPORTED targets`_
5649
5650 * `Creating FindTPL<tplName>.cmake using find_package() without IMPORTED targets`_
5651
5652 * `Creating a FindTPL<tplName>.cmake module without find_package()`_
5653
5654 * `Requirements for FindTPL<tplName>.cmake modules`_
5655
5656
5657 Creating FindTPL<tplName>.cmake using find_package()
5658 ++++++++++++++++++++++++++++++++++++++++++++++++++++
5659
5660 When defining a ``FindTPL<tplName>.cmake`` file, it encouraged to utilize
5661 ``find_package(<externalPkg> ...)`` to provide the default find operation (and
5662 also the definition of the IMPORTED targets needed to use the external package
5663 when supported). (Here, typically ``<tplName>`` and ``<externalPkg>`` are the
5664 same names, but there are cases where the names may be different or use
5665 different capitalization.) However, the state of the current ecosystem of
5666 ``Find<externalPkg>.cmake`` modules and ``<externalPkg>Config.cmake`` package
5667 config files is a bit uneven. While all find modules and package config files
5668 should be defining modern CMake IMPORTED targets which contains all the needed
5669 usage requirements (such as the target properties
5670 ``INTERFACE_INCLUDE_DIRECTORIES`` and ``INTERFACE_LINK_LIBRARIES``) and use
5671 ``find_dependency()`` to get all of their required external upstream package
5672 dependencies, many do not. Also, many of these don't provide a complete
5673 ``<tplName>::all_libs`` target which is required for a TriBITS-compliant
5674 external package/TPL.
5675
5676 In this case, the ``FindTPL<tplName>.cmake`` file provides a thin "glue" layer
5677 to adapt the information and objects provided by the
5678 ``find_package(<externalPkg> ...)`` call into a complete
5679 ``<tplName>::all_libs`` target and a wrapper ``<tplName>Config.cmake`` file
5680 for consumption by downstream TriBITS-compliant packages.
5681
5682 The following subsections will describe how to create these TriBITS-compliant
5683 ``FindTPL<tplName>.cmake`` modules for all of the various cases using an
5684 internal call to ``find_package(<externalPkg> ...)``:
5685
5686 * `Creating FindTPL<tplName>.cmake using find_package() with IMPORTED targets`_
5687 * `Creating FindTPL<tplName>.cmake using find_package() without IMPORTED targets`_
5688
5689
5690 Creating FindTPL<tplName>.cmake using find_package() with IMPORTED targets
5691 ..........................................................................
5692
5693 For cases where ``find_package(<externalPkg>)`` provides complete and proper
5694 modern (namespaced) IMPORTED targets (but is missing the
5695 ``<tplName>::all_libs`` target or the name ``<tplName>`` and ``<externalPkg>``
5696 name are different), these ``FindTPL<tplName>.cmake`` modules can call the
5697 function `tribits_extpkg_create_imported_all_libs_target_and_config_file()`_
5698 after calling ``find_package(<externalPkg>)`` to create a very thin find
5699 module file ``FindTPL<tplName>.cmake``. In these cases, such a
5700 ``FindTPL<tplName>.cmake`` module file is nothing more than::
5701
5702 find_package(<externalPkg> REQUIRED)
5703 tribits_extpkg_create_imported_all_libs_target_and_config_file(
5704 <tplName>
5705 INNER_FIND_PACKAGE_NAME <externalPkg>
5706 IMPORTED_TARGETS_FOR_ALL_LIBS <importedTarget0> <importedTarget1> ... )
5707
5708 The function
5709 `tribits_extpkg_create_imported_all_libs_target_and_config_file()`_ creates
5710 the target ``<tplName>::all_libs`` and the wrapper file
5711 ``<tplName>Config.cmake`` which is installed by TriBITS. The only unique
5712 information required to create this glue module is the name of the external
5713 package ``<externalPkg>`` and the exact full names of the IMPORTED targets
5714 ``<importedTarget0> <importedTarget1> ...`` provided by the
5715 ``find_package(<externalPkg> ...)`` call. The TriBITS function
5716 ``tribits_extpkg_create_imported_all_libs_target_and_config_file()`` takes
5717 care of all of the rest of the details.
5718
5719 Such simple ``FindTPL<tplName>.cmake`` modules do not follow the legacy
5720 TriBITS TPL convention of allowing users to specify a TPL by setting the cache
5721 variables ``<tplName>_INCLUDE_DIRS``, ``<tplName>_LIBRARY_DIRS``, and
5722 ``<tplName>_LIBRARY_NAMES`` or by setting ``TPL_<tplName>_INCLUDE_DIRS`` and
5723 ``<tplName>_LIBRARIES``. But as the ecosystem of CMake software transitions
5724 to modern CMake along with the proper usage of complete
5725 ``<Package>Config.cmake`` files, this is the reasonable thing to do.
5726
5727 However, to maintain backwards compatibility with the legacy TriBITS TPL
5728 system (such as when upgrading a existing ``FindTPL<tplName>.cmake`` file), a
5729 ``FindTPL<tplName>.cmake`` file can be extended to use the function
5730 `tribits_tpl_allow_pre_find_package()`_ in combination with the functions
5731 ``tribits_extpkg_create_imported_all_libs_target_and_config_file()`` and
5732 `tribits_tpl_find_include_dirs_and_libraries()`_ as follows::
5733
5734 set(REQUIRED_HEADERS <header0> <header1> ...)
5735 set(REQUIRED_LIBS_NAMES <libname0> <libname1> ...)
5736 set(IMPORTED_TARGETS_FOR_ALL_LIBS <importedTarget0> <importedTarget1> ...)
5737
5738 tribits_tpl_allow_pre_find_package(<tplName> <tplName>_ALLOW_PREFIND)
5739
5740 if (<tplName>_ALLOW_PREFIND)
5741 message("-- Using find_package(<externalPkg> ...) ...")
5742 find_package(<externalPkg>)
5743 if (<externalPkg>_FOUND)
5744 message("-- Found <externalPkg>_DIR='${<externalPkg>_DIR}'")
5745 message("-- Generating <tplName>::all_libs and <tplName>Config.cmake")
5746 tribits_extpkg_create_imported_all_libs_target_and_config_file(<tplName>
5747 INNER_FIND_PACKAGE_NAME <externalPkg>
5748 IMPORTED_TARGETS_FOR_ALL_LIBS ${IMPORTED_TARGETS_FOR_ALL_LIBS} )
5749 endif()
5750 endif()
5751
5752 if (NOT TARGET <tplName>::all_libs)
5753 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5754 REQUIRED_HEADERS ${REQUIRED_HEADERS}
5755 REQUIRED_LIBS_NAMES ${REQUIRED_LIBS_NAMES} )
5756 endif()
5757
5758 Above, if ``find_package(<externalPkg>)`` is called and returns
5759 ``<externalPkg>_FOUND=FALSE``, then as fallback it will call
5760 ``tribits_tpl_find_include_dirs_and_libraries()`` to find the components of
5761 the TPL. See the documentation for `tribits_tpl_allow_pre_find_package()`_
5762 for conditions where ``<tplName>_ALLOW_PREFIND`` is set to ``FALSE`` (and
5763 therefore ``find_package(<externalPkg>)`` is not called).
5764
5765 Note in the above ``FindTPL<tplName>.cmake`` file that
5766 ``find_package(<externalPkg>)`` will be called even on reconfigures. That is
5767 critical since ``find_package(<externalPkg>)`` defines IMPORTED targets that
5768 must be available each time configure is called. Also, if
5769 ``find_package(<externalPkg>)`` is called and ``<externalPkg>_FOUND=TRUE``,
5770 then the function
5771 `tribits_extpkg_create_imported_all_libs_target_and_config_file()`_
5772 is called which defines the targets ``<tplName>::all_libs`` which means that
5773 the function ``tribits_tpl_find_include_dirs_and_libraries()`` will **not** be
5774 called.
5775
5776 A concrete (and tested) example of the latter case can be found in the file
5777 ``TribitsExampleProject2/cmake/tpls/FindTPLTpl2.cmake``:
5778
5779 .. include:: ../../examples/TribitsExampleProject2/cmake/tpls/FindTPLTpl2.cmake
5780 :literal:
5781
5782
5783 Creating FindTPL<tplName>.cmake using find_package() without IMPORTED targets
5784 .............................................................................
5785
5786 There are cases where calling ``find_package(<externalPkg>)`` to find the
5787 parts of an external package does not create proper IMPORTED targets that can
5788 be directly used. For example, legacy ``Find<externalPkg>.cmake`` modules
5789 (even many standard ``Find*.cmake`` modules shipped with CMake as of CMake
5790 3.23) do not provide IMPORTED targets and instead only provide variables for
5791 the list of include directories ``<externalPkg>_INCLUDE_DIRS``, a list of
5792 library files ``<externalPkg>_LIBRARIES``, and other information. In cases
5793 such as this, one can implement the ``FindTPL<tplName>.cmake`` module by
5794 calling ``find_package(<externalPkg>)``, setting
5795 ``TPL_<tplName>_INCLUDE_DIRS`` and ``TPL_<tplName>_LIBRARIES``, and then
5796 calling `tribits_tpl_find_include_dirs_and_libraries()`_ to create the
5797 ``<tplName>::all_libs`` target and the ``<tplName>Config.cmake`` wrapper
5798 config file.
5799
5800 The simplest way to implement ``FindTPL<tplName>.cmake`` is to always call
5801 ``find_package(<externalPkg>)`` as::
5802
5803 message("-- Using find_package(<externalPkg> ...) ...")
5804 find_package(<externalPkg> REQUIRED)
5805
5806 # Tell TriBITS that we found <tplName> and there no need to look any further
5807 set(TPL_<tplName>_INCLUDE_DIRS ${<externalPkg>_INCLUDE_DIRS} CACHE PATH "...")
5808 set(TPL_<tplName>_LIBRARIES ${<externalPkg>_LIBRARIES} CACHE FILEPATH "...")
5809 set(TPL_<tplName>_LIBRARY_DIRS ${<externalPkg>_LIBRARY_DIRS} CACHE PATH "...")
5810
5811 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5812 REQUIRED_HEADERS neverFindThisHeader
5813 REQUIRED_LIBS_NAMES neverFindThisLib
5814 )
5815
5816 The above will always call ``find_package(<externalPkg> REQUIRED)``, and if it
5817 can't find the package, it will error out and stop the configure process.
5818
5819 While the above ``FindTPL<tplName>.cmake`` file is pretty simple, it does not
5820 allow for a fall-back using the legacy TriBITS TPL find system using
5821 `tribits_tpl_find_include_dirs_and_libraries()`_. One can allow for a
5822 fall-back find by passing a set of header files and library names for
5823 ``tribits_tpl_find_include_dirs_and_libraries()`` to find. This can be done
5824 using the ``FindTPL<tplName>.cmake`` module::
5825
5826 set(REQUIRED_HEADERS <header0> <header1> ...)
5827 set(REQUIRED_LIBS_NAMES <libname0> <libname1> ...)
5828
5829 message("-- Using find_package(<externalPkg> ...) ...")
5830 find_package(<externalPkg>)
5831 if (<externalPkg>_FOUND)
5832 # Tell TriBITS that we found <tplName> and there no need to look any further
5833 set(TPL_<tplName>_INCLUDE_DIRS ${<externalPkg>_INCLUDE_DIRS} CACHE PATH "...")
5834 set(TPL_<tplName>_LIBRARIES ${<externalPkg>_LIBRARIES} CACHE FILEPATH "...")
5835 set(TPL_<tplName>_LIBRARY_DIRS ${<externalPkg>_LIBRARY_DIRS} CACHE PATH "...")
5836 endif()
5837
5838 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5839 REQUIRED_HEADERS ${REQUIRED_HEADERS}
5840 REQUIRED_LIBS_NAMES ${REQUIRED_LIBS_NAMES}
5841 MUST_FIND_ALL_LIBS )
5842
5843 Above, if ``find_package(<externalPkg>)`` can't find ``<externalPkg>``, then
5844 the function ``tribits_tpl_find_include_dirs_and_libraries()`` will try to
5845 find the listed required header files and libraries, using the legacy TriBITS
5846 TPL find system. And if ``find_package(<externalPkg>)`` can find
5847 ``<externalPkg>``, then all the call to
5848 `tribits_tpl_find_include_dirs_and_libraries()`_ will not look for anything
5849 and will just take the information in the variables
5850 ``TPL_<tplName>_INCLUDE_DIRS`` and ``TPL_<tplName>_LIBRARIES`` and build
5851 IMPORTED targets and the ``<tplName>::all_libs`` target.
5852
5853 Finally, if one is upgrading an existing ``FindTPL<tplName>.cmake`` file to
5854 use ``find_package(<externalPkg>)`` but needs to maintain backward
5855 compatibility for existing configure scripts for the project that might be
5856 using the legacy TriBITS TPL system, one can use the function
5857 `tribits_tpl_allow_pre_find_package()`_ to determine if
5858 ``find_package(<externalPkg>)`` can be called or if the legacy TriBITS TPL
5859 find must be used (to maintain backward compatibility). This
5860 ``FindTPL<tplName>.cmake`` looks like::
5861
5862 set(REQUIRED_HEADERS <header0> <header1> ...)
5863 set(REQUIRED_LIBS_NAMES <libname0> <libname1> ...)
5864
5865 tribits_tpl_allow_pre_find_package(<tplName> <tplName>_ALLOW_PREFIND)
5866
5867 if (<tplName>_ALLOW_PREFIND)
5868 message("-- Using find_package<externalPkg> ...) ...")
5869 find_package(<externalPkg>)
5870 if (<externalPkg>_FOUND)
5871 # Tell TriBITS that we found <tplName> and there no need to look any further
5872 set(TPL_<tplName>_INCLUDE_DIRS ${<externalPkg>_INCLUDE_DIRS} CACHE PATH "...")
5873 set(TPL_<tplName>_LIBRARIES ${<externalPkg>_LIBRARIES} CACHE FILEPATH "...")
5874 set(TPL_<tplName>_LIBRARY_DIRS ${<externalPkg>_LIBRARY_DIRS} CACHE PATH "...")
5875 endif()
5876 endif()
5877
5878 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5879 REQUIRED_HEADERS ${REQUIRED_HEADERS}
5880 REQUIRED_LIBS_NAMES ${REQUIRED_LIBS_NAMES}
5881 MUST_FIND_ALL_LIBS )
5882
5883 The above will result in skipping the call of ``find_package(<externalPkg>)``
5884 if any of the legacy TPL find variables are set. But if the legacy TPL find
5885 variables are not set, then the default find will use
5886 ``find_package(<externalPkg>)`` and will only fall back on the legacy TriBITS
5887 TPL find operation if ``find_package(<externalPkg>)`` sets
5888 ``<externalPkg>_FOUND=FALSE``.
5889
5890 With this last approach, the ``FindTPL<tplName>.cmake`` module preserves all
5891 of the user behavior described in `Enabling support for an optional
5892 Third-Party Library (TPL)`_ for overriding what TPL components to look for,
5893 where to look, and finally to override what is actually used. That is, if the
5894 user sets the cache variables ``TPL_<tplName>_INCLUDE_DIRS``,
5895 ``TPL_<tplName>_LIBRARIES``, or ``TPL_<tplName>_LIBRARY_DIRS``, then they
5896 should be used without question (which is why the ``set( ... CACHE ...)``
5897 calls in the above example do not use ``FORCE``).
5898
5899 If one wants to skip and ignore the standard TriBITS TPL override variables
5900 ``<tplName>_INCLUDE_DIRS``, ``<tplName>_LIBRARY_NAMES``, or
5901 ``<tplName>_LIBRARY_DIRS``, then one can set::
5902
5903 set(<tplName>_FORCE_PRE_FIND_PACKAGE TRUE CACHE BOOL
5904 "Always first call find_package(<tplName> ...) unless explicit override")
5905
5906 at the top of the file ``FindTPL<tplName>.cmake`` and
5907 ``tribits_tpl_allow_pre_find_package()`` will ignore these variables and
5908 return ``TRUE``. This avoids name classes with the variables
5909 ``<externalPkg>_INCLUDE_DIRS`` and ``<externalPkg>_LIBRARY_DIRS`` with
5910 ``<tplName>_INCLUDE_DIRS`` and ``<tplName>_LIBRARY_DIRS`` (when ``<tplName> ==
5911 <externalPkg>``) which are often used in the concrete legacy CMake
5912 ``Find<tplName>.cmake`` module files themselves.
5913
5914 For a slightly more complex (but real-life) example, see ``FindTPLHDF5.cmake``
5915 which is:
5916
5917 .. include:: ../../common_tpls/FindTPLHDF5.cmake
5918 :literal:
5919
5920 Note that some specialized ``Find<externalPkg>.cmake`` modules do more than
5921 just return a list of include directories and libraries. Some, like
5922 ``FindQt4.cmake`` also return other variables that are used in downstream
5923 packages. therefore, in these cases, ``find_package(Qt4 ...)`` must be called
5924 on every configure. Such find modules cannot completely adhere to the
5925 standard legacy TriBITS behavior described in `Enabling support for an
5926 optional Third-Party Library (TPL)`_.
5927
5928
5929 Creating a FindTPL<tplName>.cmake module without find_package()
5930 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
5931
5932 For external packages that don't have a ``Find<externalPkg>.cmake`` module or
5933 ``<externalPkg>Config.cmake`` package config file, it may make sense to create
5934 a simple ``FindTpl<tplName>.cmake`` module that just calls
5935 `tribits_tpl_find_include_dirs_and_libraries()`_ with the set of required
5936 header files and libraries that must be found. A simple
5937 ``FindTPL<tplName>.cmake`` module of this form is::
5938
5939 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5940 REQUIRED_HEADERS <header0> <header1> ...
5941 REQUIRED_LIBS_NAMES <libname0> <libname1> ...
5942 MUST_FIND_ALL_LIBS
5943 )
5944
5945 Note that a set of alternate names for each library can be specified using
5946 quotes around the set of alternative library names using the syntax::
5947
5948 tribits_tpl_find_include_dirs_and_libraries( <tplName>
5949 ...
5950 REQUIRED_LIBS_NAMES "<libname0> <libname0alt0> <libname0alt1> ..." ...
5951 ...
5952 )
5953
5954 This is most commonly used for simple single-library TPLs like BLAS that has
5955 different potential implementations like::
5956
5957 tribits_tpl_find_include_dirs_and_libraries( BLAS
5958 REQUIRED_LIBS_NAMES "blas openblas atlas"
5959 ...
5960 )
5961
5962 Requirements for FindTPL<tplName>.cmake modules
5963 +++++++++++++++++++++++++++++++++++++++++++++++
5964
5965 It is possible to create a ``FindTPL<tplName>.cmake`` find module without
5966 using any TriBITS functions. The only firm requirements for a
5967 ``FindTPL<tplName>.cmake`` file are:
5968
5969 * The target ``<tplName>::all_libs`` must be created and it must contain all
5970 of the needed libraries, include directories, and other usage requirements
5971 (including for all upstream external packages/TPLs) to implement a
5972 `TriBITS-compliant package`_ to be consumed by downstream TriBITS packages.
5973
5974 * The file ``<buildDir>/external_packages/<tplName>/<tplName>Config.cmake``
5975 must be created in the build directory, and when included, it must define
5976 the equivalent IMPORTED target ``<tplName>::all_libs``, pull all of the
5977 ``<UpstreamTpl>Config.cmake`` files for upstream external packages/TPLs, and
5978 define the needed variables to provide a `TriBITS-compliant external
5979 package`_.
5980
5981 TriBITS will set the remaining variables to provide a complete
5982 `TriBITS-Compliant Package`_ for the current CMake project and will add the
5983 install target to install the file
5984 ``<buildDir>/external_packages/<tplName>/<tplName>Config.cmake`` to create a
5985 `TriBITS-compliant external package`_. TriBITS will also automatically create
5986 an appropriate package version file ``<tplName>ConfigVersion.cmake``.
5987
5988 Some of issues to consider in this case (and the role of the
5989 ``<tplName>ConfigVersion.cmake`` file) are described in the section `Tricky
5990 considerations for TriBITS-generated <tplName>Config.cmake files`_.
5991
5992
5993 How to add a new TriBITS Repository
5994 -----------------------------------
5995
5996 To add a new TriBITS and/ git VC repository to a TriBITS project that already
5997 contains other extra repositories, do the following:
5998
5999 1) Add a row for the new TriBITS and/or git repo to the file
6000 `<projectDir>/cmake/ExtraRepositoriesList.cmake`_. Commit this file.
6001
6002 2) Add an ignore for the extra repo name to the base project's git repo's
6003 ``.gitignore`` file (see `Multi-Repository Support`_). Commit this file.
6004
6005 3) Set up the new package dependencies for the new package in the new TriBITS
6006 repo or make other adjustments.
6007
6008 4) Consider the potential for missing upstream repos and packages by using
6009 `tribits_allow_missing_external_packages()`_.
6010
6011 See `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ for more details and
6012 links.
6013
6014 .. ToDo: Provide more detail!
6015
6016
6017 How to add a new TriBITS Package dependency
6018 -------------------------------------------
6019
6020 It is often the case where one will want to add a new dependency for an
6021 existing `downstream`_ package to an existing `upstream`_ (internal or
6022 external) `TriBITS Package`_. This can either be a required dependency or an
6023 optional dependency. Here, we will refer to the downstream package as
6024 ``<packageName>`` with base directory ``<packageDir>`` and will refer to the
6025 upstream (internal or external) package as ``<upstreamPackageName>``.
6026
6027 The process for adding a new dependency to an existing upstream package is
6028 as follows:
6029
6030 1) **Add the name of the upstream package to the downstream package's
6031 Dependencies.cmake file:** Add ``<upstreamPackagename>`` to the call of
6032 `tribits_package_define_dependencies()`_ in the downstream package's
6033 `<packageDir>/cmake/Dependencies.cmake`_ file. If this is to be a required
6034 library dependency, then ``<upstreamPackagename>`` is added to the
6035 ``LIB_REQUIRED_PACKAGES`` argument. Alternatively, if this is to be an
6036 optional library dependency, then ``<upstreamPackagename>`` is added to the
6037 ``LIB_OPTIONAL_PACKAGES`` argument. (For example, see the file
6038 ``packages/EpetaExt/cmake/Dependencies.cmake`` file in the
6039 `ReducedMockTrilinos`_ project.) If only the test and/or example sources,
6040 and not the package's core library sources, will have the required or
6041 optional dependency, then ``<upstreamPackagename>`` is added to the
6042 arguments ``TEST_REQUIRED_PACKAGES`` or ``TEST_OPTIONAL_PACKAGES``,
6043 respectively.
6044
6045 2) **For an optional dependency, add `HAVE_` preprocessor macro to the
6046 package's configured header file:** If this is an optional dependency,
6047 typically a C/C++ processor macro will be added to the package's configured
6048 `<packageDir>/cmake/<packageName>_config.h.in`_ file using the line::
6049
6050 #cmakedefine HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>
6051
6052 (see `HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>`_.)
6053
6054 .. _Warning, do not add optional defines for tests/examples to configured header files:
6055
6056 **Warning, do not add optional defines for tests/examples to configured
6057 header files:** If this is a test-only and/or example-only dependency then
6058 please do **not** add a ``#cmakedefine`` to the package's core
6059 `<packageDir>/cmake/<packageName>_config.h.in`_ file. Instead, add the
6060 ``#cmakedefine`` line to a configured header that is only included by
6061 sources for the tests/examples or just add a define on the compiler command
6062 line (see the ``DEFINES`` argument to `tribits_add_library()`_ and
6063 `tribits_add_executable()`_, but see the warning about problems with
6064 ``add_definitions()`` in `Miscellaneous Notes (tribits_add_library())`_).
6065 We don't want the package's header files to change or libraries to have to
6066 be rebuilt if tests/examples get enabled or disabled. Otherwise, the
6067 `TriBITS CTest/CDash Driver`_ process will result in unnecessary rebuilds
6068 of software over and over again.
6069
6070 3) **Use the features of the upstream package in the source files of the
6071 downstream package sources and/or tests/examples:** Usage of the features
6072 of the upstream package ``<upstreamPackageName>`` in the downstream package
6073 ``<packageName>`` will typically involve adding ``#include
6074 <upstreamPackageName>_<fileName>`` in the package's C/C++ source (or
6075 test/example) files (or the equivalent in Fortran). If it is an optional
6076 dependency, then these includes will typically be protected using
6077 preprocessor ifdefs, for example, as::
6078
6079 #include "<packageName>_config.h"
6080
6081 #if HAVE_<PACKAGE_NAME_UC>_<UPSTREAM_PACKAGE_NAME_UC>
6082 # include "<upstreamPackageName>_<fileName>"
6083 #endif
6084
6085 4) **For an optional dependency, use CMake if() statements based on
6086 ${PACKAGE_NAME}_ENABLE_<upstreamPackageName>:** When a package
6087 ``PACKAGE_NAME`` has an optional dependency on an upstream package
6088 ``<upstreamPackageName>`` and needs to put in optional logic in a
6089 ``CMakeLists.txt`` file, then the ``if()`` statements should use the
6090 variable ``${PACKAGE_NAME}_ENABLE_<upstreamPackageName>`` and **not** the
6091 variable ``${PROJECT_NAME}_ENABLE_<upstreamPackageName>`` or
6092 ``TPL_ENABLE_<upstreamPackageName>`` (if ``<upstreamPackageName>`` is an
6093 external package/TPL). For example, to optionally enable a test that
6094 depends on the enable of the optional upstream dependent package
6095 ``<upstreamPackageName>``, one would use::
6096
6097 tribits_add_test( ...
6098 EXCLUDE_IF_NOT_TRUE ${PACKAGE_NAME}_ENABLE_<upstreamPackageName>
6099 )
6100
6101 or::
6102
6103 if (${PACKAGE_NAME}_ENABLE_<upstreamPackageName>)
6104 tribits_add_test( ... )
6105 endif()
6106
6107 .. ToDo: Find an example to point to in TribitsExampleProject.
6108
6109 NOTE: TriBITS will automatically add the include directories for the upstream
6110 package to the compile lines for the downstream package source builds and will
6111 add the libraries for the upstream package to the link lines to the downstream
6112 package library and executable links. See documentation in the functions
6113 `tribits_add_library()`_ and `tribits_add_executable()`_, and the ``DEPLIBS``
6114 argument to these functions, for more details.
6115
6116
6117 How to tentatively enable an external package/TPL
6118 -------------------------------------------------
6119
6120 A TriBITS package can request the tentative enable of any of its optional
6121 external packagse/TPLs (see `How to add a new TriBITS Package dependency`_).
6122 This is done by calling `tribits_tpl_tentatively_enable()`_ in the package's
6123 `<packageDir>/cmake/Dependencies.cmake`_ file. For example::
6124
6125 tribits_package_define_dependencies(
6126 ...
6127 LIB_OPTIONAL_TPLS SomeTpl
6128 ...
6129 )
6130
6131 tribits_tpl_tentatively_enable(SomeTpl)
6132
6133 This will result is an attempt to find the components for the TPL ``SomeTpl``.
6134 But if that attempt fails, then the TPL will be disabled and
6135 ``${PACKAGE_NAME}_ENABLE_SomeTpl`` will be set to ``OFF``.
6136
6137
6138 How to insert a package into an upstream repo
6139 ---------------------------------------------
6140
6141 Sometimes it is desired to insert a package from a downstream VC repo into an
6142 upstream `TriBITS Repository`_ in order for one or more packages in the
6143 upstream repo to define a dependency on that package. The way this is
6144 supported in TriBITS is to just list the inserted package into the
6145 ``PackagesList.cmake`` file of the upstream TriBITS repo after the packages it
6146 depends on and before the packages that will use it then call the
6147 `tribits_allow_missing_external_packages()`_ function to allow the package to
6148 be missing. This is demonstrated in `TribitsExampleProject`_ with the package
6149 ``InsertedPkg`` which is **not** included in the default
6150 ``TribitsExampleProject`` source tree. The
6151 `TribitsExampleProject`_/``PackagesList.cmake`` file looks like:
6152
6153 .. include:: ../../examples/TribitsExampleProject/PackagesList.cmake
6154 :literal:
6155
6156 In this example, ``InsertedPkg`` has a required dependency on ``SimpleCxx``
6157 and the package ``WithSubpackagesB`` has an optional dependency on
6158 ``InsertedPkg``. Therefore, the inserted package ``InsertedPkg`` has upstream
6159 and downstream dependencies on packages in the ``TribitsExampleProject`` repo.
6160
6161 The function ``tribits_allow_missing_external_packages()`` tells TriBITS to
6162 treat ``InsertedPkg`` the same as any other package if the directory
6163 ``TribitsExampleProject/InsertedPkg`` exists or to completely ignore the
6164 package ``InsertedPkg`` otherwise. In addition, TriBITS will automatically
6165 disable of all downstream package dependencies for the missing package (and
6166 print a note about the disables). NOTE: By default TriBITS will silently
6167 ignore missing inserted packages and disable optional support for the missing
6168 package. To see what packages are missing and being ignored, configure with::
6169
6170 -D <Project>_WARN_ABOUT_MISSING_EXTERNAL_PACKAGES=TRUE
6171
6172 The way one would set up ``TribitsExampleProject`` to enable ``InsertedPkg``,
6173 if these were in separate VC (e.g. git) repos for example, would be to do::
6174
6175 $ git clone <some-url-base>/TribitsExampleProject
6176 $ cd TribitsExampleProject
6177 $ git clone <some-other-url-base>/ExteranlPkg
6178 $ echo /InsertedPkg/ >> .git/info/excludes
6179
6180 Then, when you configure ``TribitsExampleProject``, the package
6181 ``InsertedPkg`` would automatically appear and could then be enabled or
6182 disabled like any other TriBITS package. The TriBITS test
6183 ``Tribits_TribitsExampleProject_InsertedPkg`` demonstrates this.
6184
6185 Assuming that one would put the (new) external package in a separate VC repo,
6186 one would perform the following steps:
6187
6188 1) Pick a name for the new inserted external package ``<insertedPackageName>``
6189 (see `Globally unique TriBITS package names`_). (NOTE: The external
6190 package may already exist in which case the name would already be
6191 selected).
6192
6193 2) Create a new VC repo containing the new package ``<insertedPackageName>``
6194 or add it to an existing extra VC repo. (Or, add the new package to an
6195 existing downstream repo but don't add it to the ``PackagesList.cmake``
6196 file to downstream repo. That would define the package twice!)
6197
6198 3) Clone the downstream VC repo under the upstream TriBITS repository. (This
6199 is currently needed since TriBITS only allows package dirs to be contained
6200 under the repository directory.)
6201
6202 4) Insert the package into the `<repoDir>/PackagesList.cmake`_ file as
6203 described in `How to add a new TriBITS Package`_ except one must also call
6204 ``tribits_allow_missing_external_packages(<insertedPackageName>)`` as
6205 described above.
6206
6207 5) Flesh out the new package and use it in downstream packages just like it
6208 was any other package. But note that any downstream package that has a
6209 required dependency on ``<insertedPackageName>`` will always be hard
6210 disabled when the source for ``<insertedPackageName>`` is missing.
6211
6212 6) When configuring and building to get the package working, add
6213 ``-D<insertedPackageName>_ALLOW_MISSING_EXTERNAL_PACKAGE=FALSE`` so that
6214 TriBITS will catch mistakes in specifying the package directory.
6215 Otherwise, to see notes about ignoring missing inserted/external packages,
6216 set the variable ``-D<Project>_WARN_ABOUT_MISSING_EXTERNAL_PACKAGES=TRUE``
6217 and TriBITS will print warnings about missing external packages.
6218
6219
6220 How to put a TriBITS and raw CMake build system side-by-side
6221 ------------------------------------------------------------
6222
6223 There are cases where it is advantageous to have a raw CMake build system and
6224 a TriBITS CMake build system sit side-by-side in a CMake project. There are
6225 various ways to accomplish this but a very simple way that has minimal impact
6226 on the raw CMake build system is described here. An example of how to
6227 accomplish this is shown in the example project ``RawAndTribitsHelloWorld``.
6228 This CMake project is a copy of the `TribitsHelloWorld`_ project that puts a
6229 primary default raw CMake build system side-by-side with a secondary TriBITS
6230 CMake build system. The key aspects of this basic approach shown in this
6231 example are:
6232
6233 1) An ``if()`` statement must be added to the base project ``CMakeLists.txt``
6234 file to switch between the two build systems. (This is required since the
6235 raw CMake commands ``cmake_minimum_required()`` and ``project()`` must
6236 exist in the base ``CMakeLists.txt`` file and not in and included
6237 ``*.cmake`` file.) The switch trigger in the ``if()`` statement can be any
6238 logic desired, but a simple way is to look for the
6239 ``${PROJECT_NAME}_TRIBITS_DIR`` cache variable being set.
6240
6241 2) The TriBITS build system in every subdirectory is contained in files of the
6242 name ``CMakeLists.tribits.cmake`` beside the ``CMakeLists.txt`` files for
6243 the raw CMake build system. (The file ``CMakeLists.tribits.cmake`` ends
6244 with a ``*.cmake`` extension so that source editors pick it up as a CMake
6245 file.)
6246
6247 3) At the top of every raw CMake build system ``CMakeLists.txt`` file is a
6248 call to a simple macro ``include_tribits_build()`` which includes the
6249 ``CMakeLists.tribits.cmake`` file and then returns if doing a TriBITS build
6250 and otherwise does nothing for a raw CMake build.
6251
6252 The top file ``RawAndTribitsHelloWorld/CMakeLists.txt`` file demonstrates the
6253 basic approach:
6254
6255 .. include:: ../../examples/RawAndTribitsHelloWorld/CMakeLists.txt
6256 :literal:
6257
6258 Then every raw ``CMakeLists.txt`` file starts with the command
6259 ``include_tribits_build()`` at the very top as shown in the example file
6260 ``RawAndTribitsHelloWorld/hello_world/CMakeLists.txt``:
6261
6262 .. include:: ../../examples/RawAndTribitsHelloWorld/hello_world/CMakeLists.txt
6263 :literal:
6264
6265 To configure the project as a raw CMake project, just configure it as with any
6266 raw CMake project as::
6267
6268 cmake [options] <some_base_dir>/RawAndTribitsHelloWorld
6269
6270 To configure it as a TriBITS project, just set the cache var
6271 ``RawAndTribitsHelloWorld_TRIBITS_DIR`` to point to valid TriBITS source tree
6272 as::
6273
6274 cmake [options] \
6275 -DRawAndTribitsHelloWorld_TRIBITS_DIR=<tribits_dir> \
6276 <some_base_dir>/RawAndTribitsHelloWorld
6277
6278 A twist on this use case is for a package that only builds as a TriBITS
6279 package inside of some larger TriBITS project and not as its own TriBITS CMake
6280 project. In this case, some slight changes are needed to this example but the
6281 basic approach is nearly identical. One still needs an ``if()`` statement at
6282 the top of the first ``CMakeLists.txt`` file (this time for the package) and
6283 the macro ``include_tribits_build()`` needs to be defined at the top of that
6284 file as well. Then every ``CMakeLists.txt`` file in subdirectories just calls
6285 ``include_tribits_build()``. That is it.
6286
6287
6288 How to implement a TriBITS-compliant internal package using raw CMake
6289 ---------------------------------------------------------------------
6290
6291 As described in `TriBITS-Compliant Internal Packages`_, it is possible to
6292 create a raw CMake build system for a CMake package that can build under a
6293 parent TriBITS CMake project. The raw CMake code for such a package must
6294 provide the ``<Package>::all_libs`` target both in the current CMake build
6295 system and also in the generated ``<Package>Config.cmake`` file for the build
6296 directory and in the installed ``<Package>Config.cmake`` file. Every such
6297 TriBITS-compliant internal package therefore is **also capable of installing a
6298 TriBITS-compliant external package** ``<Package>Config.cmake`` file (see `How
6299 to implement a TriBITS-compliant external package using raw CMake`_).
6300
6301 .. ToDo: Consider listing out the key features of a raw CMake build system
6302 that is needed for a TriBITS-compliant internal package.
6303
6304 A raw CMake build system for a TriBITS-compliant internal package is
6305 demonstrated in the `TribitsExampleProject2`_ project ``Package1`` package.
6306 The base ``CMakeLists.txt`` file for building ``Package1`` with a raw CMake
6307 build system (called ``package1/CMakeLists.raw.cmake`` in that directory)
6308 looks like:
6309
6310 .. include:: TribitsExampleProject2_Package1_CMakeLists.raw.internal.cmake
6311 :literal:
6312
6313 As shown above, this simple CMake package contains the basic features of any
6314 CMake project/package including calling the ``cmake_minimum_required()`` and
6315 ``project()`` commands as well as including ``GNUInstallDirs``. In this
6316 example, the project/package being built ``Package1`` has a dependency on an
6317 external upstream package ``Tpl1`` pulled in with ``find_package(Tpl1)``.
6318 Also in this example, the package has native tests it defines with
6319 ``include(CTest)`` and ``add_subdirectory()`` (if ``Package1_ENABLE_TESTS`` is
6320 set to ``ON``).
6321
6322 The file ``package1/src/CMakeLists.raw.cmake`` (which gets included from
6323 ``package1/src/CMakeLists.txt``) creates a library and executable for the
6324 package and has the contents:
6325
6326 .. include:: ../../examples/TribitsExampleProject2/packages/package1/src/CMakeLists.raw.cmake
6327 :literal:
6328
6329 This creates a single installable library target ``Package1_package1`` which
6330 is aliased as ``Package1::package1`` in the current CMake project and sets up
6331 to create the IMPORTED target ``Package1::package1`` in the generated
6332 ``Package1ConfigTarget.cmake`` file, which gets included in the installed
6333 ``Package1Config.cmake`` (``<Package>Config.cmake``) file (as recommenced in
6334 the book "Professional CMake", see below). In addition, the above code
6335 creates the installable executable ``package1-prg``.
6336
6337 The ``Package1::all_libs`` (``<Package>::all_libs``) target is defined and set
6338 up inside of the included file
6339 ``package1/cmake/raw/DefineAllLibsTarget.cmake`` which contains the code:
6340
6341 .. include:: ../../examples/TribitsExampleProject2/packages/package1/cmake/raw/DefineAllLibsTarget.cmake
6342 :literal:
6343
6344 The above code contains the ALIAS library target ``Package1::all_libs``
6345 (``<Package>::all_libs``) for the current CMake project as well as sets up for
6346 the IMPORTED target ``Package1::all_libs`` (``<Package>::all_libs``) getting
6347 put in the generated ``Package1ConfigTargets.cmake`` file (see below).
6348
6349 The ``Package1Config.cmake`` (``<Package>Config.cmake``) file for the build
6350 directory is generated inside of the included file
6351 ``package1/cmake/raw/GeneratePackageConfigFileForBuildDir.cmake`` which has
6352 the contents:
6353
6354 .. include:: ../../examples/TribitsExampleProject2/packages/package1/cmake/raw/GeneratePackageConfigFileForBuildDir.cmake
6355 :literal:
6356
6357 The above code uses the ``export(EXPORT ...)`` command to generate the file
6358 ``Package1ConfigTargets.cmake`` for the build directory which provides the
6359 IMPORTED targets ``Package1::package1`` and ``Package1::all_libs``. The
6360 command ``configure_file(...)`` generates the ``Package1Config.cmake`` file
6361 that includes it for the build directory
6362 ``<buildDir>/cmake_packages/Package1/``. (NOTE: The above code only runs when
6363 the package is being built from inside of a TriBITS project which defines the
6364 command ``tribits_package``. So this code gets skipped when building
6365 ``Package1`` as a stand-alone raw CMake project.)
6366
6367 Finally, the code for generating and installing the ``Package1Config.cmake``
6368 file for the install directory ``CMAKE_PREFIX_PATH=<installDir>`` is specified
6369 in the included file
6370 ``package1/cmake/raw/GeneratePackageConfigFileForInstallDir.cmake`` with the
6371 contents:
6372
6373 .. include:: ../../examples/TribitsExampleProject2/packages/package1/cmake/raw/GeneratePackageConfigFileForInstallDir.cmake
6374 :literal:
6375
6376 The above uses the command ``install(EXPORT ...)`` to have CMake automatically
6377 generate and install the file ``Package1ConfigTargets.cmake`` in the install
6378 directory ``<installDir>/libs/cmake/Package1/`` which provides the IMPORTED
6379 targets ``Package1::package1`` and ``Package1::all_libs``. The command
6380 ``configure_file()`` is used to generate the file
6381 ``Package1Config.install.cmake`` in the build directory from the template file
6382 ``Package1Config.cmake.in``. Finally, the ``install()`` command is used in
6383 the file ``GeneratePackageConfigFileForInstallDir.cmake`` to set up the
6384 installation of the ``Package1Config.cmake`` file.
6385
6386 Note, the template file ``package1/cmake/raw/Package1Config.cmake.in`` (which
6387 is unique to ``Package1``) is:
6388
6389 .. include:: ../../examples/TribitsExampleProject2/packages/package1/cmake/raw/Package1Config.cmake.in
6390 :literal:
6391
6392 As shown in the all of the above code, there is a lot of boilerplate CMake
6393 code needed to correctly define the targets such that they get put into the
6394 installed ``Package1Config.cmake`` file using the correct namespace
6395 ``Package1::`` and care must be taken to ensure that a consistent "export set"
6396 is used for this purpose. (For more details, see the book "Professional
6397 CMake".)
6398
6399 **NOTE:** One should compare the above raw CMakeLists files to the more
6400 compact TriBITS versions for the base ``package1/CMakeLists.txt`` file (called
6401 ``package1/CMakeLists.tribits.cmake`` in the base directory ``pacakge1/``):
6402
6403 .. include:: ../../examples/TribitsExampleProject2/packages/package1/CMakeLists.tribits.cmake
6404 :literal:
6405
6406 and the TriBITS ``package1/src/CMakeLists.txt`` file (called
6407 ``package1/src/CMakeLists.tribits.cmake``):
6408
6409 .. include:: ../../examples/TribitsExampleProject2/packages/package1/src/CMakeLists.tribits.cmake
6410 :literal:
6411
6412 This shows the amount of boilerplate code that TriBITS addresses automatically
6413 (which reduces the overhead of finer-grained packages and avoids common
6414 mistakes with tedious low-level CMake code).
6415
6416
6417 How to implement a TriBITS-compliant external package using raw CMake
6418 ---------------------------------------------------------------------
6419
6420 As described in `TriBITS-Compliant External Packages`_, it is possible to
6421 create a raw CMake build system for a CMake package such that once it is
6422 installed, satisfies the requirements for a TriBITS-compliant external
6423 package. These installed packages provide a ``<Package>Config.cmake`` file
6424 that provides the required targets and behaviors as if it was produced by a
6425 TriBITS project. For most existing raw CMake projects that already produce a
6426 "Professional CMake" compliant ``<Package>Config.cmake`` file, that usually
6427 just means adding the IMPORTED target called ``<Package>::all_libs`` to the
6428 installed ``<Package>Config.cmake`` file.
6429
6430 A raw CMake build system for a TriBITS-compliant external package is
6431 demonstrated in the `TribitsExampleProject2`_ project ``Package1`` package.
6432 The base ``package1/CMakeLists.txt`` file for building ``Package1`` with a raw
6433 CMake build system (called ``package1/CMakeLists.raw.cmake``) for implementing
6434 a TriBITS-compliant internal package looks like:
6435
6436 .. include:: TribitsExampleProject2_Package1_CMakeLists.raw.external.cmake
6437 :literal:
6438
6439 Note that the raw build system this example is identical to the build system
6440 for the raw TriBITS-compliant internal package described in `How to implement
6441 a TriBITS-compliant internal package using raw CMake`_. The only differences
6442 are:
6443
6444 1) The ``Package1Config.cmake`` (``<Package>Config.cmake``) file does **not**
6445 need to be generated for the build directory and therefore the code in
6446 ``cmake/raw/GeneratePackageConfigFileForBuildDir.cmake`` does **not** need
6447 to be included.
6448
6449 2) The ALIAS library target ``Package1::all_libs`` (``<Package>::all_libs``)
6450 does **not** need to be generated (but should be to be "Professional CMake"
6451 compliant).
6452
6453 Other than that, see `How to implement a TriBITS-compliant internal package
6454 using raw CMake`_ for how to implement a TriBITS-compliant external package.
6455
6456
6457 How to use TriBITS testing support in non-TriBITS project
6458 ---------------------------------------------------------
6459
6460 The TriBITS test support functions `tribits_add_test()`_ and
6461 `tribits_add_advanced_test()`_ can be used from any raw (i.e. non-TriBITS)
6462 CMake project. To do so, one just needs to include the TriBITS modules:
6463
6464 * ``<tribitsDir>/core/test_support/TribitsAddTest.cmake``
6465 * ``<tribitsDir>/core/test_support/TribitsAddAdvancedTest.cmake``
6466
6467 and set the variable ``${PROJECT_NAME}_ENABLE_TESTS`` to ``ON``. For an
6468 MPI-enabled CMake project, the CMake variables ``MPI_EXEC``,
6469 ``MPI_EXEC_PRE_NUMPROCS_FLAGS``, ``MPI_EXEC_NUMPROCS_FLAG`` and
6470 ``MPI_EXEC_POST_NUMPROCS_FLAGS`` must also be set which define the MPI runtime
6471 program launcher command-line used in the TriBITS testing functions::
6472
6473 ${MPI_EXEC} ${MPI_EXEC_PRE_NUMPROCS_FLAGS}
6474 ${MPI_EXEC_NUMPROCS_FLAG} <NP>
6475 ${MPI_EXEC_POST_NUMPROCS_FLAGS}
6476 <TEST_EXECUTABLE_PATH> <TEST_ARGS>
6477
6478 (NOTE: These variables are defined automatically in a TriBITS project when
6479 ``TPL_ENABLE_MPI`` is set to ``ON``.)
6480
6481 This is demonstrated in the `TribitsExampleProject2`_ project ``Package1``
6482 package. The base ``pacakge1/CMakeLists.txt`` file for building ``Package1``
6483 with a raw CMake build system using TriBITS testing functions (called
6484 ``package1/CMakeLists.raw.cmake``) looks like:
6485
6486 .. include:: TribitsExampleProject2_Package1_CMakeLists.raw.tribits_test.cmake
6487 :literal:
6488
6489 The only difference between this base ``package1/CMakeLists.txt`` file and one
6490 for a raw CMake project is the inclusion of the file
6491 ``package1/cmake/raw/EnableTribitsTestSupport.cmake`` which has the contents:
6492
6493 .. include:: ../../examples/TribitsExampleProject2/packages/package1/cmake/raw/EnableTribitsTestSupport.cmake
6494 :literal:
6495
6496 The key lines are::
6497
6498 include("${Package1_TRIBITS_DIR}/core/test_support/TribitsAddTest.cmake")
6499 include("${Package1_TRIBITS_DIR}/core/test_support/TribitsAddAdvancedTest.cmake")
6500
6501 This defines the CMake functions `tribits_add_test()`_ and
6502 `tribits_add_advanced_test()`_, respectively.
6503
6504 The above code demonstrates that ``CMAKE_MODULE_PATH`` does **not** need to be
6505 updated to use these TriBITS ``test_support`` modules. However, one is free
6506 to update ``CMAKE_MODULE_PATH`` and then include the modules by name only
6507 like::
6508
6509 list(PREPEND CMAKE_MODULE_PATH "${Package1_TRIBITS_DIR}/core/test_support")
6510 include(TribitsAddTest)
6511 include(TribitsAddAdvancedTest)
6512
6513 Once these TriBITS modules are included, one can use the TriBITS functions as
6514 demonstrated in the file ``package1/test/CMakeLists.tribits.cmake`` (which is
6515 included from the file ``package1/test/CMakeLists.txt``) and has the contents:
6516
6517 .. include:: ../../examples/TribitsExampleProject2/packages/package1/test/CMakeLists.tribits.cmake
6518 :literal:
6519
6520 Note that in this example, the executable ``package1-prg`` was already
6521 created. If new test libraries and executables need to be created, then the
6522 raw CMake commands to create those will need to be added as well.
6523
6524
6525 How to check for and tweak TriBITS "ENABLE" cache variables
6526 -----------------------------------------------------------
6527
6528 TriBITS defines a number of special ``<XXX>_ENABLE_<YYY>`` variables for
6529 enabling/disabling various entities that allow for a default "undefined" empty
6530 ``""`` enable status. Examples of these special variables include:
6531
6532 * ``${PROJECT_NAME}_ENABLE_<TRIBITS_PACKAGE>`` (Packages)
6533 * ``TPL_ENABLE_<tplName>`` (External Packages/TPLs)
6534 * ``<TRIBITS_PACKAGE>_ENABLE_<TRIBITS_DEP_PACKAGE>`` (Support for a
6535 package ``<TRIBITS_DEP_PACKAGE>`` in a downstream package
6536 ``<TRIBITS_PACKAGE>``)
6537 * ``<TRIBITS_PACKAGE>_ENABLE_TESTS`` (Package tests)
6538 * ``<TRIBITS_PACKAGE>_ENABLE_EXAMPLES`` (Package examples)
6539 * ``${PROJECT_NAME}_ENABLE_TESTS`` (Tests for explicitly enabled packages)
6540 * ``${PROJECT_NAME}_ENABLE_EXAMPLES`` (Examples for explicitly enabled
6541 packages)
6542
6543 (see `TriBITS Dependency Handling Behaviors`_).
6544
6545 To check for and tweak these special "ENABLE" variables, perform the
6546 following:
6547
6548 1) To check to see if an ``ENABLE`` variable has been enabled or disabled
6549 (either explicitly or through auto enable/disable logic), use::
6550
6551 if ("${<XXX>_ENABLE_<YYY>}" STREQUAL "")
6552 # Variable has not been set to 'ON' or 'OFF' yet
6553 ...
6554 endif()
6555
6556 This will work correctly independent of if the cache variable has been
6557 default defined or not.
6558
6559 2) To tweak the enable/disable of one or more of these variables after user
6560 input but **before** the step "Adjust package and TPLs enables and
6561 disables" in `Full Processing of TriBITS Project Files`_:
6562
6563 a) To tweak the enables/disables for a TriBITS Repository (i.e. affecting
6564 all TriBITS projects) add enable/disable code to the file
6565 `<repoDir>/cmake/RepositoryDependenciesSetup.cmake`_.
6566
6567 b) To tweak the enables/disables for a specific TriBITS Project
6568 (i.e. affecting only that TriBITS project) add enable/disable code to the
6569 file `<projectDir>/cmake/ProjectDependenciesSetup.cmake`_.
6570
6571 For example, one might default disable a package if it has not been
6572 explicitly enabled (or disabled) in one of these files using logic like::
6573
6574 if (NOT ${PROJECT_NAME}_ENABLE_Fortran)
6575 if ("${${PROJECT_NAME}_ENABLE_<SomeFortranPackage>}" STREQUAL "")
6576 message("-- " "NOTE: Setting ${PROJECT_NAME}_ENABLE_<SomeFortranPackage>=OFF because"
6577 "${PROJECT_NAME}_ENABLE_Fortran = '${${PROJECT_NAME}_ENABLE_Fortran}'")
6578 set(${PROJECT_NAME}_ENABLE_<SomeFortranPackage> OFF)
6579 endif()
6580 endif()
6581
6582 In order to understand the above steps for properly querying and tweaking
6583 these ``ENABLE`` variables, one must understand how TriBITS CMake code defines
6584 and interprets variables of this type.
6585
6586 First, note that most of these particular ``ENABLE`` variables are not
6587 ``BOOL`` cache variables but are actually ``STRING`` variables with the
6588 possible values of ``ON``, ``OFF`` and empty ``""`` (see the macro
6589 `set_cache_on_off_empty()`_). Therefore, just because the value of a
6590 ``<XXX>_ENABLE_<YYY>`` variable is defined (e.g. ``if (DEFINED
6591 <XXX>_ENABLE_<YYY>) ... endif()``) does not mean that it has been set to
6592 ``ON`` or ``OFF`` yet (or any non-empty values that evaluates to true or false
6593 in CMake). To see if an ``ENABLE`` variable is one of these types, look in
6594 the ``CMakeCache.txt``. If the type of the variable ``<XXX>_ENABLE_<YYY>`` is
6595 ``STRING`` and you see another variable set with the name
6596 ``<XXX>_ENABLE_<YYY>_-STRINGS``, then it is most likely this special type of
6597 ``ENABLE`` variable with a typical default value of empty ``""``. However, if
6598 the cache variable is of type ``BOOL``, then it is likely a standard bool
6599 variable that is not allowed to have a value of empty ``""``.
6600
6601 Second, note that the value of empty ``""`` evaluates to ``FALSE`` in CMake
6602 ``if()`` statements. Therefore, if one just wants to know if one of these
6603 variables evaluates to true, then just use ``if (<XXX>_ENABLE_<YYY>)
6604 ... endif()``.
6605
6606 Third, note that TriBITS will not define cache variables for these ``ENABLE``
6607 variables until TriBITS processes the ``Dependencies.cmake`` files on the
6608 first configure (see `Full TriBITS Project Configuration`_). On future
6609 reconfigures, these variables are all defined (but most will have a default
6610 value of empty ``""`` stored in the cache).
6611
6612 The reason the files ``RepositoryDependenciesSetup.cmake`` and
6613 ``ProjectDependenciesSetup.cmake`` are the best places to put in these tweaks
6614 is because, as shown in `Full Processing of TriBITS Project Files`_, they get
6615 processed after all of the user input has been read (in CMake cache variables
6616 set with ``-D<variable>=<value>`` and read in from
6617 `${PROJECT_NAME}_CONFIGURE_OPTIONS_FILE`_ files) but before TriBITS adjusts
6618 the package enables and disables (see `Package Dependencies and Enable/Disable
6619 Logic`_). Also, these files get processed in `Reduced Package Dependency
6620 Processing`_ as well so they get processed in all contexts where
6621 enable/disable logic is applied.
6622
6623 However, if one wants to tweak these variables once packages are starting to
6624 be processed (in step "For each ``<packageDir>`` in all enabled top-level
6625 packages" in `Full TriBITS Project Configuration`_), there are fewer
6626 situations where that can be done correctly as described in the next section.
6627
6628
6629 How to tweak downstream TriBITS "ENABLE" variables during package configuration
6630 -------------------------------------------------------------------------------
6631
6632 There are cases where one may need to enable or disable some feature that
6633 TriBITS may have enabled by default (such as in "Adjust package and TPLs
6634 enables and disables" in `Full Processing of TriBITS Project Files`_) and that
6635 decision can only be made while processing a package's
6636 `<packageDir>/CMakeLists.txt`_ file and not before. (And therefore the logic
6637 for this disable cannot be performed in the ``ProjectDependenciesSetup.cmake``
6638 or ``RepositoryDependenciesSetup.cmake`` files as described in `How to check
6639 for and tweak TriBITS "ENABLE" cache variables`_.) Also, there are cases
6640 where it is necessary to make this change visible to downstream packages. The
6641 main example is when optional support of an upstream package in a downstream
6642 package ``<DownstreamPackage>_ENABLE_<UpstreamPackage>`` must be changed in
6643 the package's `<packageDir>/CMakeLists.txt`_ file. But there are other
6644 examples such as support for a given data-type that may impact multiple
6645 downstream packages.
6646
6647 When the internal configuration of a package (i.e. while processing its
6648 ``<packageDir>/CMakeLists.txt`` file) determines that an optional feature
6649 ``<Package>_ENABLE_<YYY>`` must change the value previously set (e.g. that was
6650 set automatically by TriBITS during the "Adjust package and TPLs enables and
6651 disables" stage in `Full Processing of TriBITS Project Files`_), one cannot
6652 use a simple ``set()`` statement. Changing the value of a
6653 ``<Package>_ENABLE_<YYY>`` variable inside a package's
6654 ``<packageDir>/CMakeLists.txt`` file using a raw ``set(<Package>_ENABLE_<YYY>
6655 <newValue>)`` statement only changes the variable's value inside the package's
6656 scope, but all other packages will see the old value of
6657 ``<Package>_ENABLE_<YYY>``. To correctly change the value of one of these
6658 variables, instead use `dual_scope_set()`_ from the top-level
6659 ``<packageDir>/CMakeLists.txt`` file. To perform this disable more robustly
6660 than calling ``dual_scope_set()`` directly, use the provided macro
6661 `tribits_disable_optional_dependency()`_. For example, to disable optional
6662 support for ``<UpstreamPackage>`` in ``<DownstreamPackage>`` in
6663 ``<DownstreamPackage>`` package's ``<packageDir>/CMakeLists.txt`` file based
6664 on some criteria, add the CMake code::
6665
6666 if (<some-condition>)
6667 tribits_disable_optional_dependency( <UpstreamPackage>
6668 "NOTE: ${PACKAGE_NAME}_ENABLE_<UpstreamPackage> being set to OFF because of <reason>" )
6669 endif()
6670
6671 Calling ``dual_scope_set()`` in the package's top-level
6672 ``<packageDir>/CMakeLists.txt`` file sets the value in both the local scope of
6673 ``<packageDir>/CMakeLists.txt`` (and therefore propagated to all other
6674 ``CMakeLists.txt`` files in that package) and in base-level (global) project
6675 scope. (But this does **not** change the value of a cache variable
6676 ``<Package>_ENABLE_<YYY>`` that may have been set by the user or some other
6677 means which is the desired behavior; see `TriBITS auto-enables/disables done
6678 using non-cache local variables`_.) In this way, any downstream package
6679 (configured after processing ``<packageDir>/CMakeLists.txt``) will see the new
6680 value for ``<Package>_ENABLE_<YYY>``.
6681
6682 It is also strongly recommended that a message be printed to CMake STDOUT
6683 using ``message("-- " "NOTE: <message>")`` when changing the value of one of
6684 these ``<Package>_ENABLE_<YYY>`` variables. The user may have set it
6685 explicitly or TriBITS may have printed automatic logic for setting it by
6686 default, and user needs to know why and where the value is being overridden.
6687
6688 **NOTE:** However, it is **not** allowed to try to change the value of a
6689 global enable of a upstream or downstream package by trying to change the
6690 value of ``<Project>_ENABLE_<Package>`` or ``TPL_ENABLE_<Package>`` in a
6691 ``<packageDir>/CMakeLists.txt`` file. Changing the value of these variables
6692 after the "Adjust package and TPLs enables and disables" stage in `Full
6693 Processing of TriBITS Project Files`_ will result in undefined behavior.
6694
6695
6696 How to set up multi-repository support
6697 --------------------------------------
6698
6699 The following steps describe how to set up support for TriBITS project
6700 managing multiple version control and TriBITS repositories by default (see
6701 `Multi-Repository Support`_).
6702
6703 1) Add file `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ and list out
6704 extra repos
6705
6706 For example, this file would contain something like::
6707
6708 tribits_project_define_extra_repositories(
6709 ExtraRepo1 "" GIT [email protected]:ExtraRepo1 "" Continuous
6710 ExtraRepo2 "ExtraRepo1/ExtraRepos2" GIT [email protected]:ExtraRepo2
6711 NOPACKAGES Continuous
6712 ExtraRepo3 "" GIT [email protected]:ExtraRepo3 "" Nightly
6713 )
6714
6715 NOTE: If one will not be using the `checkin-test.py`_ tool, or
6716 `clone_extra_repos.py`_ tool, or the `TriBITS CTest/CDash Driver`_ system,
6717 then one can leave the **REPO_VCTYPE** and **REPO_URL** fields empty (see
6718 `tribits_project_define_extra_repositories()`_ for details). (TriBITS Core
6719 does not have any dependencies on any specific VC tool. These fields are
6720 listed here to avoid duplicating the list of repos in another file when
6721 using these additional TriBITS tools.)
6722
6723 2) Set default values for `${PROJECT_NAME}_EXTRAREPOS_FILE`_ and
6724 ``${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE`` (and possibly
6725 ``${PROJECT_NAME}_IGNORE_MISSING_EXTRA_REPOSITORIES``) in the file
6726 `<projectDir>/ProjectName.cmake`_
6727
6728 For example, add::
6729
6730 set(${PROJECT_NAME}_EXTRAREPOS_FILE cmake/ExtraRepositoriesList.cmake
6731 CACHE FILEPATH "Set in ProjectName.cmake")
6732 set(${PROJECT_NAME}_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE Continuous
6733 CACHE STRING "Set in ProjectName.cmake")
6734
6735 to the `<projectDir>/ProjectName.cmake`_ file. Otherwise, no extra repos
6736 will be defined or processed by default when configuring the project.
6737
6738 And if the project can operate without all of its extra repos, the project
6739 can set the following default in this file as well with::
6740
6741 set(${PROJECT_NAME}_IGNORE_MISSING_EXTRA_REPOSITORIES TRUE
6742 CACHE STRING "Set in ProjectName.cmake")
6743
6744 Otherwise, all of the extra repos need to be present or the project
6745 configure will fail.
6746
6747 3) If using git as the VC tool, then set the variable
6748 ``${PROJECT_NAME}_GENERATE_REPO_VERSION_FILE_DEFAULT`` in the
6749 `<projectDir>/ProjectName.cmake`_ file
6750
6751 For example::
6752
6753 set(${PROJECT_NAME}_GENERATE_REPO_VERSION_FILE_DEFAULT TRUE)
6754
6755 4) If wanting a clone tool with git repos, set up a link to the
6756 `clone_extra_repos.py`_ tool in the base ``<projectDir>/`` directory
6757
6758 Create a symlink to the script `clone_extra_repos.py`_ in the base project
6759 repo, for example with::
6760
6761 cd <projecDir>/
6762 ln -s cmake/tribits/ci_support/clone_extra_repos.py .
6763 git add clone_extra_repos.py
6764 git commit
6765
6766
6767 How to submit testing results to a CDash site
6768 ---------------------------------------------
6769
6770 The following steps describe how to submit results to a CDash site using the
6771 `TriBITS CTest/CDash Driver`_ support.
6772
6773 1) Create a CDash project ``<ProjectName>`` on the targeted CDash site.
6774
6775 To do this, one must have an account on the CDash site and the permissions
6776 to create a new CDash project. The name of the project on CDash should
6777 generally match the name of the TriBITS project ``PROJECT_NAME`` but it does
6778 not have to. In fact, one can generally submit to any CDash project with
6779 any name so creating a new CDash project is actually optional.
6780
6781 NOTE: For open-source projects, Kitware provides the free CDash site
6782 my.cdash.org that allows a limited number of submits and data per day. But
6783 it should be enough to test out submitting to CDash.
6784
6785 2) Create ``CTestConfig.cmake`` and ``CTestCustom.cmake.in`` files for the
6786 project.
6787
6788 * The file `<projectDir>/CTestConfig.cmake`_ can be copied and pasted from
6789 TribitsExampleProject/CTestConfig.cmake. To customize for your project,
6790 you generally just need to update the variables ``CTEST_DROP_SITE``,
6791 ``CTEST_PROJECT_NAME``, and ``CTEST_DROP_LOCATION``.
6792
6793 * The file `<projectDir>/cmake/ctest/CTestCustom.cmake.in`_ can be copied
6794 and pasted from ``TribitsExampleProject/cmake/ctest/CTestCustom.cmake.in``
6795 and them modified as desired.
6796
6797 3) Test experimental submits with ``make dashboard``.
6798
6799 Once the CDash project and the `<projectDir>/CTestConfig.cmake`_ and
6800 `<projectDir>/cmake/ctest/CTestCustom.cmake.in`_ files are created, one
6801 perform an experimental submission by just configuring the project as normal
6802 (except configuring additionally with ``-DCTEST_BUILD_FLAGS=-j8`` and
6803 ``-DCTEST_PARALLEL_LEVEL=8`` to use parallelism in the build and testing in
6804 the ``ctest -S`` script) and then running the build target::
6805
6806 make dashboard
6807
6808 That will configure, build, test, and submit results to the CDash site to
6809 the ``Experimental`` track of the target CDash project (see `Dashboard
6810 Submissions`_).
6811
6812 To work out problems locally without spamming the CDash site, one can run
6813 with::
6814
6815 env CTEST_DO_SUBMIT=OFF make dashboard
6816
6817 To submit to a different CDash site and project, change the cache vars
6818 ``CTEST_DROP_SITE``, ``CTEST_PROJECT_NAME``, and ``CTEST_DROP_LOCATION`` at
6819 configure time. For example, to submit TribitsExampleProject results to a
6820 different CDash site, configure with::
6821
6822 cmake \
6823 -DCTEST_DROP_SITE=testing.sandia.gov/cdash \
6824 -DCTEST_PROJECT_NAME=TribitsExProj \
6825 -DCTEST_DROP_LOCATION="/submit.php?project=TribitsExProj" \
6826 [other cmake options] \
6827 <baseDir>/TribitsExamplProject
6828
6829 and then run ``make dashboard``.
6830
6831 4) Add custom CTest -S driver scripts.
6832
6833 For driving different builds and tests, one needs to set up one or more
6834 CTest -S driver scripts. There are various ways to do this but a simple
6835 approach that avoids duplication is to first create a file like
6836 ``TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake``:
6837
6838 .. include:: ../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
6839 :literal:
6840
6841 and then create a set of CTest -S driver scripts that uses that file. One
6842 example is the file
6843 ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:
6844
6845 .. include:: ../../examples/TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake
6846 :literal:
6847
6848 5) Test CTest -S driver scripts
6849
6850 Once a CTest -S driver script (like the ``ctest_serial_debug.cmake`` example
6851 shown above) is created, one can test it locally and then test a submit to
6852 CDash. To test the exact state of the repository locally, one can create a
6853 temporary base directory, symbolically link in the local project source
6854 directory, and then run the CTest -S script by setting
6855 ``CTEST_DO_SUBMIT=OFF``. For example, the TribitsExampleProject CTest -S
6856 script can be run and tested locally by doing::
6857
6858 $ mkdir MOCK_TRIBITSEXPROJ_SERIAL_DEBUG
6859 $ cd MOCK_TRIBITSEXPROJ_SERIAL_DEBUG/
6860 $ ln -s $TRIBITS_DIR/examples/TribitsExampleProject .
6861 $ env CTEST_DASHBOARD_ROOT=$PWD \
6862 CTEST_DROP_SITE=testing.sandia.gov/cdash \
6863 CTEST_PROJECT_NAME=TribitsExProj \
6864 CTEST_DROP_LOCATION="/submit.php?project=TribitsExProj" \
6865 CTEST_TEST_TYPE=Experimental \
6866 CTEST_DO_SUBMIT=OFF \
6867 CTEST_DO_UPDATES=OFF \
6868 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \
6869 ctest -V -S \
6870 $TRIBITS_DIR/examples/TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake \
6871 &> console.out
6872
6873 where ``TRIBITS_DIR`` is an env var that points to the location of the
6874 ``TriBITS/tribits`` directory on the local machine (and the location of the
6875 CDash site and project is changed, since the free my.cdash.org site can only
6876 accept as small number of builds each day).
6877
6878 Once that CTest -S driver script is working correctly without submitting to
6879 CDash, the above ``ctest -S`` command can be run with ``CTEST_DO_SUBMIT=ON``
6880 which submit the CDash and then print the location on CDash for the
6881 submitted configure, build, and test results.
6882
6883 6) Set up automated runs of CTest -S driver scripts
6884
6885 The custom CTest -S driver scripts created above can be run and used to
6886 submit to CDash in a variety of ways:
6887
6888 * Cron jobs can be set up to run them at the same time every day.
6889
6890 * Jenkins jobs can be set up to run them based on various criteria.
6891
6892 * GitHub Actions can run them to respond to GitHub pushes or to test pull
6893 requests.
6894
6895 * GitLab CI can run them to respond to GitLab pushes or the test merge
6896 requests.
6897
6898 * Use the legacy `TriBITS Dashboard Driver`_ system (not recommended).
6899
6900 The setup of Jenkins, GitHub Actions, GitLab CI and other more sophisticated
6901 automated testing systems will not be described here. What will be briefly
6902 outlined below is the setup using cron jobs on a Linux machine. That is
6903 sufficient for most smaller projects and provides tremendous value.
6904
6905 To set up an automated build using a cron job, one will typically create a
6906 shell driver script that sets the env and then calls the ``ctest -S
6907 <script>`` command. Then one just adds a call to that shell driver script
6908 using ``crontab -e``. That is about all there is to it.
6909
6910 .. ToDo: Add some more details and examples on how to do this, using
6911 .. TribitsExampleProject implementations. In particular, you need two
6912 .. clones of your base git repo. One to provide the CTest -S script, and
6913 .. the other cloned and updated by the CTest driver script.
6914
6915
6916 How to submit testing results to a custom CDash Group
6917 -----------------------------------------------------
6918
6919 Following up on `How to submit testing results to a CDash site`_, to submit
6920 build and test results to a custom "Group" on CDash (instead of just
6921 "Nightly", "Continuous" or "Experimental"), one just has to create the new
6922 group on CDash using the CDash GUI interface and then tell the ctest -S local
6923 driver to submit results to that CDash group. The steps for doing this are
6924 given below.
6925
6926 1. Create the new CDash group ``<special_group>`` for CDash project
6927 ``<ProjectName>`` on the targeted CDash site.
6928
6929 If the CDash group ``<special_group>`` is not already created, then one can
6930 create it by first logging into CDash with an account that can modify the
6931 CDash project ``<ProjectName>``. Once logged in, go to the project edit
6932 page and select "Settings" and "Groups". From there, create the new group
6933 ``<special_group>``. Set the "Build Type" to either "Daily" or "Latest".
6934
6935 2. Set ``${PROJECT_NAME}_TRACK=<special_group>`` with the CTest -S driver
6936 script.
6937
6938 One can either do that by setting ``set(${PROJECT_NAME}_TRACK
6939 <special_group>)`` in the CTest -S ``*.cmake`` driver script itself or can
6940 set it in the environment when running the ctest -S driver script. For
6941 example::
6942
6943 $ env <Project>_TRACK=<special_group> ... \
6944 ctest -V -S <ctest_driver>.cmake
6945
6946 If the "build type" for the CDash group ``<special_group>`` was set to
6947 "Daily", then set `CTEST_TEST_TYPE`_ to ``Nightly``. Otherwise,
6948 ``CTEST_TEST_TYPE`` can be set to ``Continuous`` or ``Experimental``.
6949
6950
6951 Miscellaneous Topics
6952 ====================
6953
6954 In this section, a number of miscellaneous topics and TriBITS features are
6955 discussed. These features and topics are either not considered primary
6956 features of TriBITS or don't neatly fit into one of the other sections.
6957
6958
6959 TriBITS Repository Contents
6960 ---------------------------
6961
6962 The TriBITS git repository is organized as a `TriBITS Project`_, `TriBITS
6963 Repository`_, and `TriBITS Package`_ all in the same base directory. The base
6964 contents are described in the file::
6965
6966 TriBITS/README.DIRECTORY_CONTENTS.rst
6967
6968 The part of TriBITS that is installed or snapshotted in contained in the
6969 subdirectory `TriBITS/tribits/`_ and is described in the following section.
6970
6971 .. include:: TriBITS.README.DIRECTORY_CONTENTS.rst
6972
6973 .. _TriBITS/tribits/:
6974
6975 .. include:: ../../README.DIRECTORY_CONTENTS.rst
6976
6977
6978 TriBITS System Project Dependencies
6979 -----------------------------------
6980
6981 The core TriBITS system itself (see ``tribts/core/`` in `TriBITS/tribits/`_)
6982 which is used to configure, built, test, create tarballs, and install software
6983 has no dependencies other than a basic installation of CMake (which typically
6984 includes the executables ``cmake``, ``ctest``, and ``cpack``). Great effort
6985 has been expended to implement all of this core functionality of TriBITS just
6986 using raw CMake. That means that anyone who needs to configure, build, and
6987 install software that uses TriBITS just needs a compatible CMake
6988 implementation. CMake is becoming iniquitous enough that many machines will
6989 already have a current-enough version of CMake installed by default on their
6990 systems and therefore no one will need to download or install any extra
6991 software when building and installing a project that uses TriBITS (assuming
6992 the necessary compilers etc. required by the project are also installed). If
6993 a current-enough version of CMake is not installed on a given system, it is
6994 easy to download the source code and all it needs is a basic C++ compiler to
6995 build and install.
6996
6997 However, note that a specific TriBITS project is free to use any newer CMake
6998 features it wants and therefore these projects will require newer versions of
6999 CMake than what is required by TriBITS (see discussion of
7000 ``cmake_minimum_required()`` in `<projectDir>/CMakeLists.txt`_). But also
7001 note that specific TriBITS projects and packages will also require additional
7002 tools like compilers, Python (see `Python Support`_), Perl, or many other such
7003 dependencies. It is just that TriBITS itself does not require any of these in
7004 order to perform the basic configure, build, test, and install of software.
7005 The goal of TriBITS is not to make the portability of software that uses it
7006 any worse than it already is but instead to make it easier in most cases (that
7007 after all is the whole goal of CMake).
7008
7009 While the TriBITS Core functionality to configure, build, test, and install
7010 software is written using only raw CMake, the more sophisticated development
7011 tools needed to implement the full TriBITS development environment require
7012 Python 2.7 (or higher including Python 3.x) (see `Python Support`_). Python
7013 is needed for tools like `checkin-test.py`_ and `gitdist`_. In addition,
7014 these python tools are used in `tribits_ctest_driver()`_ to drive automated
7015 testing and submits to CDash. Also note that ``git`` is the chosen version
7016 control tool for the TriBITS software development tools and all the VC-related
7017 functionality in TriBITS. But none of this is required for doing the most
7018 basic building, testing, or installation of a project using TriBITS Core.
7019
7020
7021 Python Support
7022 --------------
7023
7024 TriBITS Core does not require anything other than raw CMake. However, Python
7025 Utils, TriBITS CI Support, and other extended TriBITS components require
7026 Python. These extra TriBITS tools only require Python 3.6+. By default, when
7027 a TriBITS project starts to configure using CMake, it will try to find Python
7028 3.6+ on the system (see `Full Processing of TriBITS Project Files`_). If
7029 Python is found, it will set the global cache variable ``Python3_EXECUTABLE``.
7030 If it is not found, then it will print a warning and ``Python3_EXECUTABLE``
7031 will be empty. With this default behavior, if Python is found, then the
7032 TriBITS project can use it. Otherwise, it can do without it.
7033
7034 While the default behavior for finding Python described above is useful for
7035 many TriBITS projects, some TriBITS projects need different behavior such as:
7036
7037 1. The TriBITS project may not ever use Python so there is no need to look for
7038 it at all. In this case, the TriBITS project would set
7039 `${PROJECT_NAME}_USES_PYTHON`_ to ``FALSE``.
7040
7041 2. Some TriBITS projects require Python and should not even configure if it
7042 can't be found. In this case, the TriBITS project would set
7043 `${PROJECT_NAME}_REQUIRES_PYTHON`_ to ``TRUE``.
7044
7045 3. Some TriBITS projects may require a version of Python more recent than 3.6.
7046 In this case, the TriBITS project would set
7047 `${PROJECT_NAME}_Python3_FIND_VERSION`_ to some value higher than ``3.6``.
7048 For example, may newer systems have higher versions of Python installed by
7049 default, and projects developed on such a system typically require this
7050 version or higher.
7051
7052
7053 Project-Specific Build Reference
7054 --------------------------------
7055
7056 If a project that uses TriBITS is going to have a significant user base that
7057 will configure, build, and test the project, then having some documentation
7058 that explains how to do this would be useful. For this purpose, TriBITS
7059 provides a mechanism to quickly create a project-specific build reference
7060 document in restructured text (RST) format and with HTML and LaTeX/PDF
7061 outputs. This document are generally created in the base project source tree
7062 and given then name ``<Project>BuildReference.[rst,html,pdf]``. This document
7063 consists of two parts. One part is a generic template document::
7064
7065 tribits/doc/TribitsBuildReferenceBody.rst
7066
7067 provided in the TriBITS source tree that uses the place-holder ``<Project>``
7068 for the for the real project name. The second part is a project-specific
7069 template file::
7070
7071 <projectDir>/cmake/<Project>BuildReferenceTemplate.rst
7072
7073 which provides the outer RST document (with title, authors, abstract,
7074 introduction, other introductory sections). From these two files, the
7075 script::
7076
7077 tribits/doc/build_ref/create-project-build-quickref.py
7078
7079 is used to replace ``<Project>`` in the ``TribitsBuildReferenceBody.rst`` file
7080 with the real project name (read from the project's ``ProjectName.cmake`` file
7081 by default) and then generates the read-only files::
7082
7083 <projectDir>/
7084 <Project>BuildReference.rst
7085 <Project>BuildReference.html
7086 <Project>BuildReference.pdf
7087
7088 For a simple example of this, see::
7089
7090 tribits/doc/build_ref/create-build-ref.sh
7091
7092 A project-independent version of this file is provided in the
7093 `TribitsBuildReference`_.[rst,html,pdf] which is referred to many times in this
7094 developers guide.
7095
7096
7097 Project and Repository Versioning and Release Mode
7098 ----------------------------------------------------
7099
7100 TriBITS has built-in support for project and repository versioning and
7101 release-mode control. When the project contains the file
7102 `<projectDir>/Version.cmake`_, it is used to define the project's official
7103 version. The idea is that when it is time to branch for a release, the only
7104 file that needs to be changed is the file `<projectDir>/Version.cmake`_.
7105
7106 Each TriBITS repository can also contain a `<repoDir>/Version.cmake`_ file
7107 that sets version-related variables which TriBITS packages in that repository
7108 can use to derive development and release version information. If the TriBITS
7109 repository also contains a `<repoDir>/Copyright.txt`_ file, then the
7110 information in ``<repoDir>/Version.cmake`` and ``<repoDir>/Copyright.txt`` are
7111 used to configure a repository version header file::
7112
7113 ${${REPOSITORY_NAME}_BINARY_DIR}/${REPOSITORY_NAME}_version.h
7114
7115 The configured header file ``${REPOSITORY_NAME}_version.h`` defines C
7116 pre-processor macros that give the repository version number in several
7117 formats, which allows C/C++ code (or any software that uses the C
7118 preprocessor) to write conditional code like::
7119
7120 #if Trilinos_MAJOR_MINOR_VERSION > 100200
7121 /* Contains feature X */
7122 ...
7123 #else
7124 /* Does not contain feature X */
7125 ...
7126 #endif
7127
7128 Of course when the TriBITS project and the TriBITS repository are the same
7129 directory, the ``<projectDir>/Version.cmake`` and ``<repoDir>/Version.cmake``
7130 files are the same file, which works just fine.
7131
7132
7133 TriBITS Environment Probing and Setup
7134 -------------------------------------
7135
7136 Part of the TriBITS Framework is to probe the environment, set up the
7137 compilers, and get ready to compile code. This was mentioned in `Full
7138 Processing of TriBITS Project Files`_. This is executed by the TriBITS macro
7139 ``tribits_setup_env()``. Some of the things this macro does are:
7140
7141 .. _Probe and set up the environment:
7142
7143 **Probe and set up the environment:**
7144
7145 * Set ``CMAKE_BUILD_TYPE`` default (if not already set)
7146 * Set up for MPI (MPI compilers, etc.)
7147 * Set up C, C++, and Fortran compilers using ``enable_language(<LANG>)``
7148 * ``include(`` `<projectDir>/cmake/ProjectCompilerPostConfig.cmake`_ ``)``
7149 * Find Perl (sets ``PERL_EXECUTABLE``)
7150 * Determine mixed language C/Fortran linking
7151 * Set up OpenMP (with ``find_package(OpenMP)``)
7152 * Set up optional Windows support
7153 * Find Doxygen (sets ``DOXYGEN_EXECUTABLE``)
7154 * Perform some other configure-time tests (see ``cmake`` configure output)
7155
7156 At the completion of this part of the processing, the TriBITS CMake project is
7157 ready to compile code. All of the major variables set as part of this process
7158 are printed to the ``cmake`` stdout when the project is configured.
7159
7160
7161 Tricky considerations for TriBITS-generated <tplName>Config.cmake files
7162 -----------------------------------------------------------------------
7163
7164 An issue that comes up with external packages/TPLs like HDF5 that needs to be
7165 discussed here is the fact that ``FindTPL<tplName>.cmake`` module files create
7166 (See `How to add a new TriBITS TPL`_) and TriBITS installs package config
7167 files of the name ``<tplName>Config.cmake``. These TriBITS-generated package
7168 config files ``<tplName>Config.cmake`` could potentially be found by calls to
7169 ``find_package(<externalPkg>)`` (i.e. when ``<tplName> == <externalPkg>`` like
7170 with HDF5). These TriBITS-generated ``<tplName>Config.cmake`` files are
7171 primarily meant to provide a `TriBITS-compliant external package`_ for
7172 downstream TriBITS-compliant ``<Package>Config.cmake`` files. These
7173 TriBITS-generated ``<tplName>Config.cmake`` files will usually not behave the
7174 same way existing ``Find<tplName>.config`` find modules or native
7175 ``<tplName>Config.cmake`` package config files would behave as expected by
7176 downstream projects when found by ``find_package(<tplName>)`` commands called
7177 in some arbitrary downstream raw CMake project. Therefore, to avoid having an
7178 installed TriBITS-generated ``HDF5Config.cmake`` file, for example, being
7179 found by the inner call to ``find_package(HDF5 ...)`` in the file
7180 ``FindTPLHDF5.cmake`` (which could be disastrous), TriBITS employs two
7181 safeguards.
7182
7183 First, TriBITS-generated ``<tplName>Config.cmake`` package config files are
7184 placed into the build directory under::
7185
7186 <buildDir>/external_packages/<tplName>/<tplName>Config.cmake
7187
7188 and installed into the installation directory under::
7189
7190 <installDir>/lib/external_packages/<tplName>/<tplName>Config.cmake
7191
7192 so they will not be found by ``find_package(<tplName>)`` by default when
7193 ``<buildDir>/cmake_packages`` and/or ``<installDir>``, respectively, are added
7194 to ``CMAKE_PREFIX_PATH``.
7195
7196 Second, even if the directories ``<installDir>/lib/external_packages`` or
7197 ``<buildDir>/external_packages`` do get added to the search path somehow
7198 (e.g. by appending those to ``CMAKE_INSTALL_PREFIX``), the companion
7199 TriBITS-generated ``<tplName>ConfigVersion.cmake`` files will set
7200 ``PACKAGE_VERSION_COMPATIBLE=OFF`` and result in ``find_package(<tplName>)``
7201 not selecting the TriBITS-generated ``<tplName>Config.cmake`` file. (It turns
7202 out that CMake's ``find_package(<Package>)`` command always includes the file
7203 ``<Package>ConfigVersion.cmake``, even if no version information is passed to
7204 the command ``find_package(<Package>)``. This allows special logic to be
7205 placed in the file ``<Package>ConfigVersion.cmake`` to determine if
7206 ``find_package(<Package>)`` will select a given ``<Package>Config.cmake`` file
7207 that is in the search path based on a number of different criteria such as in
7208 this case.)
7209
7210
7211 Installation considerations
7212 ---------------------------
7213
7214 For the most part, installation is pretty straightforward with a TriBITS-based
7215 CMake project. TriBITS automatically puts in appropriate default
7216 ``install()`` commands to install header files, libraries, executables, and
7217 other commonly installed artifacts (such as TriBITS-autogenerated
7218 ``<Package>Config.cmake`` files). And packages can add their own custom
7219 ``install()`` commands to install items under ``CMAKE_INSTALL_PREFIX`` (or the
7220 subdirs under ``CMAKE_INSTALL_PREFIX`` mentioned in `Setting the install
7221 prefix`_). However, there are some special situations that need to be
7222 addressed and some tweaks to built-in CMake support that need to be made.
7223
7224 One issue that can occur is that there are cases where a Unix/Linux system is
7225 set up not to honor the group sticky bit and therefore one cannot control what
7226 group owns the created installed files and directories (i.e. the default group
7227 will be used). Also, there are cases were one cannot easily control the
7228 default file or directory creation permissions using ``umask``. And there are
7229 cases where one would like to recursively install a set of directories and
7230 files where some of these files may be scripts that need to have the execute
7231 permission set on them for them to work. The only to flexible accomplish
7232 that with CMake (if one does not know the exist list of those files or
7233 extensions of those files) is to pass in the ``SOURCE_PERMISSIONS`` option to
7234 the ``install(DIRECTORY ...)`` command. An example of this is shown in:
7235
7236 * ``TribitsExampleProject/packages/with_subpackages/b/CMakeLists.txt``
7237
7238 that has::
7239
7240 install( DIRECTORY "${CMAKE_CURRENT_LIST_DIR}/stuff"
7241 DESTINATION "${CMAKE_INSTALL_PREFIX}/share/${PACKAGE_NAME}"
7242 USE_SOURCE_PERMISSIONS PATTERN "*~" EXCLUDE )
7243
7244 In this case, CMake will preserve the execute permission on any of the scripts
7245 contained under the ``stuff/`` subdirectory but ``group`` and ``other``
7246 permissions will not be set based on ``umask`` or the default CMake install
7247 permissions. Instead, these permissions are set based on the source directory
7248 permissions (which is often set to ``700`` or ``rwx------``).
7249
7250 To address cases like this, TriBITS can automatically run ``chgrp`` and
7251 ``chmod`` on the created files and directories that are created during the
7252 ``install`` target as described in `Setting install ownership and
7253 permissions`_. This is completely automatic and requires nothing for the
7254 TriBITS Project developers to do to enable support for this (other than to
7255 note the below warning).
7256
7257 **WARNING**: Do not add any ``install()`` commands after the
7258 `tribits_project()`_ command completes. Otherwise, any extra files or
7259 directories will not have their group and permissions fixed by these special
7260 TriBITS-added ``chgrp`` and ``chmod`` commands run at install time. Instead,
7261 try to put all ``install()`` commands inside of a package's
7262 `<packageDir>/CMakeLists.txt`_ file. Currently, there really is no good place
7263 to add repo-level or project-level ``install()`` commands. But if one had to
7264 sneak them in, they could add various ``install()`` commands to files like
7265 `<projectDir>/CMakeLists.txt`_ (before the ``tribits_project_()`` command),
7266 `<repoDir>/cmake/CallbackSetupExtraOptions.cmake`_,
7267 `<projectDir>/cmake/CallbackDefineProjectPackaging.cmake`_ and/or
7268 `<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake`_. (Note that
7269 install commands from the former two files are run before install commands for
7270 the enabled packages while install commands from the latter two files are run
7271 after.)
7272
7273 One can also change what compilers are written into the generated
7274 ``<Project>Config.cmake`` and ``<Package>Config.cmake`` files for the build
7275 and the install trees. By default, the compilers pointed to in these
7276 ``Config.cmake`` files will be ``CMAKE_<LANG>_COMPILER`` where ``<LANG>`` =
7277 ``CXX``, ``C``, and ``Fortran``. But one can change this by setting any of
7278 the following::
7279
7280 set(CMAKE_CXX_COMPILER_FOR_CONFIG_FILE_BUILD_DIR <path>)
7281 set(CMAKE_C_COMPILER_FOR_CONFIG_FILE_BUILD_DIR <path>)
7282 set(CMAKE_Fortran_COMPILER_FOR_CONFIG_FILE_BUILD_DIR <path>)
7283 set(CMAKE_CXX_COMPILER_FOR_CONFIG_FILE_INSTALL_DIR <path>)
7284 set(CMAKE_C_COMPILER_FOR_CONFIG_FILE_INSTALL_DIR <path>)
7285 set(CMAKE_Fortran_COMPILER_FOR_CONFIG_FILE_INSTALL_DIR <path>)
7286
7287 before the ``Config.cmake`` files are generated. These can also be set in the
7288 CMake cache using, for example,
7289 ``-DCMAKE_CXX_COMPILER_FOR_CONFIG_FILE_INSTALL_DIR:FILEPATH=<path>``.
7290
7291 This is used, for example, when compiler wrappers are used for the build tree
7292 and are set to ``CMAKE_<LANG>_COMPILER`` but when one wants to point to the
7293 original underlying compilers for the installed ``Config.cmake`` files.
7294
7295
7296 RPATH Handling
7297 --------------
7298
7299 As explained in `Setting install RPATH`_, TriBITS changes the CMake defaults
7300 to write in the RPATH for shared libraries and executables so that they run
7301 right out of the install directory without needing to set paths in the
7302 environment (e.g. ``LD_LIBRARY_PATH``). However, these defaults can be
7303 changed by changing setting different project defaults for the variables
7304 `${PROJECT_NAME}_SET_INSTALL_RPATH`_ and `CMAKE_INSTALL_RPATH_USE_LINK_PATH`_.
7305 But most projects should likely keep these defaults in place since they make
7306 it so that doing builds and installations on a single machine work correctly
7307 by default out of the box. For other installation/distribution use cases, the
7308 user is told how to manipulate CMake variables for those cases in `Setting
7309 install RPATH`_.
7310
7311
7312 Configure-time System Tests
7313 ---------------------------
7314
7315 CMake has good support for defining configure-time checks of the system to
7316 help in configuring the project. One can check for whether a header file
7317 exists, if the compiler supports a given data-type or language feature, or
7318 perform almost any other type of check that one can imagine that can be done
7319 using the configured compilers, libraries, system tools, etc. An example was
7320 given in `TribitsExampleProject`_. Just follow that example, look at some of
7321 the built-in CMake configure-time test modules, and consult provided on-line
7322 CMake documentation in order to learn how to create a configure-time test for
7323 almost anything.
7324
7325 .. ToDo: Provide more detail how to integrate a configure-time test into a
7326 .. TriBITS project and TriBITS package.
7327
7328
7329 Creating Source Distributions
7330 -----------------------------
7331
7332 The TriBITS system uses CMake's built-in CPack support to create source
7333 distributions in a variety of zipped and tarred formats. (Note that the term
7334 "source tarball" or just "tarball" may be used below but should be interpreted
7335 as "source distribution".) TriBITS will automatically add support for CPack
7336 when the variable `${PROJECT_NAME}_ENABLE_CPACK_PACKAGING`_ is set to ``ON``.
7337 The commands for creating a source distribution are described in `Creating a
7338 tarball of the source tree`_ using the built-in ``package_source`` build
7339 target. The value added by TriBITS is that TriBITS will automatically exclude
7340 the source for any defined packages that are not enabled and TriBITS provides
7341 a framework for systematically excluding files and directories from individual
7342 repositories and packages. In addition, the source for non-enabled
7343 subpackages can also be excluded depending on the value of
7344 `${PROJECT_NAME}_EXCLUDE_DISABLED_SUBPACKAGES_FROM_DISTRIBUTION`_. All of
7345 this allows one to create distributions which only includes subsets of a
7346 larger project (even a single package in some cases).
7347
7348 Unlike other build systems (like autotools), CMake will put **EVERYTHING**
7349 into the source distribution (e.g. tarball) that is sitting in the source tree
7350 by default. Therefore, setting up for a source distribution usually means
7351 deciding what extra files and directories should be excluded. Beyond the
7352 directories for non-enabled packages, further files can be selected to be
7353 excluded on a package-by-package based and at the repository level (see
7354 below).
7355
7356 Individual packages can list additional files/directories under the package's
7357 source tree to be excluded from the project's source distribution using a call
7358 to `tribits_exclude_files()`_ in their `<packageDir>/CMakeLists.txt`_ file.
7359 Note that if the package is not enabled, these excludes will never get added!
7360 That is no problem if these excludes only apply to the given package since
7361 TriBITS will add an exclude for the entire package but is a problem if a
7362 package lists excludes for files outside of the package's source tree.
7363
7364 Additional files and entire directories can also be excluded at the repository
7365 level by listing them in the
7366 `<repoDir>/cmake/CallbackDefineRepositoryPackaging.cmake`_ file which
7367 typically just appends the built-in CMake variable
7368 ``CPACK_SOURCE_IGNORE_FILES``. However, if a repository is not processed,
7369 this file is never processed and therefore no files will be excluded from a
7370 repository sitting in the source tree that is not processed (see below).
7371
7372 There are a number of project-level settings that need to be defined and these
7373 are specified in the file
7374 `<projectDir>/cmake/CallbackDefineProjectPackaging.cmake`_.
7375
7376 The `TribitsExampleProject`_ is set up for creating source distributions and
7377 this is demonstrated in one of the tests defined in::
7378
7379 TriBITS/test/core/ExamplesUnitTests/CMakeLists.txt
7380
7381 There are a few points of caution to note about creating source distributions.
7382
7383 **NOTE:** It is worth stressing again that **EVERY** file that is in the
7384 source tree will be included in the source distribution (tarball) unless there
7385 is an exclude regex matching it appended to the variable
7386 ``CPACK_SOURCE_IGNORE_FILES``. TriBITS can only add excludes for defined
7387 non-enabled packaged. Every other file listed in the source tree will be
7388 included in the tarball.
7389
7390 **NOTE:** The entries in ``CPACK_SOURCE_IGNORE_FILES`` are interpreted as
7391 **REGULAR EXPRESSIONS** not globs so if you add ``"someFile.*"`` as an
7392 exclude, it will exclude every file in the entire source tree that has
7393 ``"someFile"`` in the name! This is because, in regex terminology, the
7394 trailing ``".*"`` means "match any character zero or more times" and
7395 ``"someFile"`` can match anywhere in the file name path. Also, note that if
7396 you add in an exclude like ``"*.pyc"`` (i.e. trying to exclude all of the
7397 generated Python byte code files) that it will exclude every file that has
7398 ``"pyc"`` in the name and **not** just those with the file extension
7399 ``"pyc"``. For example, the exclude ``".pyc"`` would exclude the files
7400 ``"puppyc"``, ``"lpycso"``, etc. If you want to exclude all files with
7401 extension ``"pyc"``, you have to add the exclude regex ``".*[.]pyc$"``!
7402 One's lack of understanding of this fact will cost someone hours of lost time
7403 debugging what happens when random files are missing when one tries to
7404 configure what is left. Sometimes, what is left will actually configure and
7405 might almost build!
7406
7407 **NOTE:** As warned in `TriBITS Package Core Files`_ and `TriBITS Subpackage
7408 Core Files`_, Packages must have directories that are strictly independent
7409 of the directories of other packages. If they don't, then the source
7410 directory for an enabled package will get excluded from the source
7411 distribution if its directory is under the directory of a package that is not
7412 enabled. For example, if ``PackageA`` is enabled but its package directory
7413 ``packageb/packagea/`` is under the package directory ``packageb/`` for the
7414 disabled package ``PackageB``, then every file and directory under
7415 ``packageb/`` will be excluded from the source distribution (tarball),
7416 including everything under ``packageb/packagea/``! It would be too expensive
7417 to put in an automated check for cases like this so package developers should
7418 just take care not to nest the directories of packages inside of each other to
7419 avoid problems like this.
7420
7421 **NOTE:** Extra repositories that are sitting in the source tree but not
7422 processed by TriBITS for some reason (e.g. due to explicitly listing in the
7423 variable `${PROJECT_NAME}_EXTRA_REPOSITORIES`_ only a subset of the repositories
7424 listed in `<projectDir>/cmake/ExtraRepositoriesList.cmake`_ ) will get added
7425 to the source distribution in full by default even though there are no enabled
7426 packages from these repos. These non-processed repo dirs are like any other
7427 random directory sitting in the source tree, they will get copied over into
7428 the source distribution!.
7429
7430 **NOTE:** When debugging tarball creation problems, always configure with the
7431 variable ``<Project>_DUMP_CPACK_SOURCE_IGNORE_FILES=ON``. If you don't see a
7432 regex listed for the file or directory you expect to be excluded, then that
7433 file/directory it will be included in the source distribution!
7434
7435
7436 Using Git Bisect with checkin-test.py workflows
7437 -----------------------------------------------
7438
7439 There are cases where a customer will do an update of an upstream project from
7440 a git repo and then find out that some feature or behavior is broken with
7441 respect to their usage. This can happen even if the upstream project's own
7442 test suite passes all of its tests. Depending on the situation, there may be
7443 hundreds to thousands of commits between the last known "good" working version
7444 of the code and pulled "bad" version. To help customers find the first commit
7445 that contains the changes which are causing the breakage, git supplies ``git
7446 bisect``. This set of commands does a binary search of the commits in the
7447 range ``<good-sha>..<bad_sha>`` and finds the first commit that is "bad" (or a
7448 range of commits which contains the first "bad" commit if commits are skipped,
7449 as described below).
7450
7451 But the ``git bisect`` commands require that all of the commits in the range
7452 ``<good-sha>..<bad_sha>`` be complete commits that provide full working
7453 versions of the upstream project. However, many beginning git developers and
7454 even many experienced developers don't always create individual git commits
7455 that build and pass all of the upstream project's tests and therefore can
7456 create false "bad" commits during the binary search. This can happen when
7457 developers create intermediate "check-point" commits during the development
7458 process but did not squash the intermediate commits together to create
7459 cohesive working commits. This can also happen when experienced developers
7460 have a set of what they believe are all working commits but do not actually
7461 test all of the commits to know that they pass *all* of the upstream project's
7462 tests before pushing these commits to the main development branch. This lack
7463 of detailed testing of each and every individual commit can give rise to false
7464 "bad" commits which will result in ``git bisect`` reporting the wrong first
7465 "bad" commit.
7466
7467 Projects that use the `checkin-test.py`_ tool to push sets of commits to the
7468 main development branch have an advantage in the usage of ``git bisect``.
7469 This is because the default mode of the ``checkin-test.py`` script is to amend
7470 the top commit message with a summary of what was tested and therefore marks a
7471 set of commits that are known to have more complete testing. For example, the
7472 ``checkin-test.py`` tool amends the top commit (after the final pull and
7473 rebase by default) as shown in the following Trilinos commit::
7474
7475 commit 71ce56bd2d268922fda7b8eca74fad0ffbd7d807
7476 Author: Roscoe A. Bartlett <[email protected]>
7477 Date: Thu Feb 19 12:04:11 2015 -0500
7478
7479 Define HAVE_TEUCHOSCORE_CXX11 in TeuchosCore_config.h
7480
7481 This makes TeuchosCore a good example for how Trilinos (or any TriBITS)
7482 subpackages should put in an optional dependency on C++11.
7483
7484 Build/Test Cases Summary
7485 Enabled Packages: TeuchosCore
7486 Disabled Packages: [...]
7487 0) MPI_DEBUG => passed: passed=44,notpassed=0 (2.61 min)
7488 1) SERIAL_RELEASE => passed: passed=43,notpassed=0 (1.08 min)
7489
7490 Therefore, these special known-tested commits can be flagged by grepping the
7491 ``git log -1 HEAD`` output for the string ``"Build/Test Cases Summary"``. By
7492 bisecting on these commits, one has a lower chance of encountering false "bad"
7493 commits and has a higher chance of finding a smaller range of commits where
7494 the first true "bad" commit might be found. To aid in performing ``git
7495 bisect`` and only checking ``checkin-test.py``-tested commits, the tool
7496 `is_checkin_tested_commit.py`_ is provided.
7497
7498 To demonstrate how the ``is_checkin_tested_commit.py`` tool can be used with
7499 ``git bisect``, suppose that someone writes a customized script
7500 ``build_and_test_customer_code.sh`` that will build the upstream project and
7501 the downstream customer's code and then run a set of tests to see if the "bad"
7502 behavior seen by the customer code is the current ``HEAD`` version of the
7503 upstream project. Assume this script is copied into the upstream project's
7504 local git repo using::
7505
7506 $ cd <upstream-repo>/
7507 $ cp ~/build_and_test_customer_code.sh .
7508 $ cat /build_and_test_customer_code.sh >> .git/info/exclude
7509
7510 Now, one could use ``build_and_test_customer_code.sh`` directly with::
7511
7512 $ git bisect run ./build_and_test_customer_code.sh
7513
7514 but that would result in testing *all* the commits which may have a high
7515 chance of producing false "bad" commits as described above and fail to
7516 correctly bracket the location of the true first "bad" commit.
7517
7518 So instead, one can write a filtered testing script
7519 ``safe_build_and_test_customer_code.sh`` which calls
7520 ``is_checkin_tested_commit.py`` and ``build_and_test_customer_code.sh`` as
7521 follows::
7522
7523 #!/bin/bash
7524 #
7525 # Script: safe_build_and_test_customer_code.sh
7526
7527 $TRIBITS_DIR/ci_support/is_checkin_tested_commit.py
7528 IS_CHECKIN_TESTED_COMMIT_RTN=$?
7529 if [ "$IS_CHECKIN_TESTED_COMMIT_RTN" != "0" ] ; then
7530 exit 125 # Skip the commit because HEAD is not known to be tested!
7531 fi
7532
7533 ./build_and_test_customer_code.sh # Rtn 0 "good", or [1, 124] if "bad"
7534
7535 The above test script ``safe_build_and_test_customer_code.sh`` will skip the
7536 testing of commits that are not marked by the ``checkin-test.py`` tool.
7537
7538 To demonstrate how to use the ``is_checkin_tested_commit.py`` script with
7539 ``git bisect``, an example from Trilinos is used below. (Note that the
7540 current Trilinos public repository may have been filtered so the commit SHA1s
7541 shown below may not match what is in the current Trilinos repository. But one
7542 can use the commit summary message, author, and author date to find the
7543 updated SHA1s and then to update this example for the current repository.)
7544
7545 Consider a scenario where a customer application updates Trilinos from the
7546 commit::
7547
7548 d44c17d "Merge branch 'master' of software.sandia.gov:/space/git/Trilinos"
7549 Author: Roscoe A. Bartlett <[email protected]>
7550 Date: Tue May 26 12:43:25 2015 -0400
7551
7552 to the commit::
7553
7554 605b91b "Merge branch 'master' of software.sandia.gov:/git/Trilinos"
7555 Author: Vitus Leung <[email protected]>
7556 Date: Tue Sep 29 20:18:54 2015 -0600
7557
7558 and it is found that some critical feature broke or is not behaving acceptably
7559 for the customer code (but all of the tests for Trilinos pass just fine).
7560 This range of commits ``d44c17d..605b91b`` gives 2257 commits to search as
7561 shown by::
7562
7563 $ cd Trilinos/
7564 $ git log-oneline d44c17d..605b91b | wc -l
7565 2257
7566
7567 However, as described above, it is likely that doing ``git bisect`` on that
7568 full set of 2257 commits may result in hitting false "bad" commits and
7569 therefore result in a false bracketing of the first "bad" commit. This is
7570 where the usage of the ``checkin-test.py`` tool helps which is used by many
7571 (but not currently all) Trilinos developers to push changes to the Trilinos
7572 'master' branch in the current single-branch workflow. The commits marked
7573 with the ``checkin-test.py`` tool are known (with some caveats mentioned
7574 below) to be working commits and for this the range of commits
7575 ``d44c17d..605b91b`` yields 166 commits as shown by::
7576
7577 $ git log-oneline --grep="Build/Test Cases Summary" d44c17d..605b91b | wc -l
7578 166
7579
7580 That is an average of 2257/166 = 13.6 commits between commits pushed with the
7581 ``checkin-test.py`` tool. So bisecting on just the commits marked by
7582 ``checkin-test.py`` should bound the "bad" commit in a set of 13.6 commits on
7583 average. Bisecting on this set of 166 commits should likely give no false
7584 “bad” commits, and therefore result in the correct bracketing of the first
7585 "bad" commit.
7586
7587 Using the ``safe_build_and_test_customer_code.sh`` shown above, one would
7588 search for the first bad commit over this range using::
7589
7590 $ git bisect start 605b91b d44c17d
7591 $ env DUMMY_TEST_COMMIT_BAD_SHA=83f05e8 \
7592 time git bisect run ./safe_build_and_test_customer_code.sh
7593
7594 and this would return the range of commits that contains the first "bad"
7595 commit (listed at the end of ``git bisect log`` output, see example below).
7596
7597 To provide a concrete example, suppose the commit that first introduced the
7598 problem in the range of commits ``d44c17d..605b91b`` was::
7599
7600 83f05e8 "MueLu: stop semi-coarsening if no z-layers are left."
7601 Author: Tobias Wiesner <[email protected]>
7602 Date: Wed Jul 1 14:54:20 2015 -0600
7603
7604 And instead of using the script ``safe_build_and_test_customer_code.sh``, we
7605 use a dummy driver script ``dummy_test_commit.sh`` to simulate this which is
7606 provided in the set of TriBITS documentation::
7607
7608 $TRIBITS_DIR/doc/developers_guide/scripts/dummy_test_commit.sh
7609
7610 as:
7611
7612 .. include:: scripts/dummy_test_commit.sh
7613 :literal:
7614
7615 This driver script allows one to simulate the usage of ``git bisect`` to
7616 understand how it works without having to actually build and test code. It is
7617 a useful training and experimentation tool.
7618
7619 Using ``git bisect`` (with git version 2.1.0) over the range of commits
7620 ``d44c17d..605b91b`` searching for the first "bad" commit is done by running
7621 the commands::
7622
7623 $ git bisect start 605b91b d44c17d
7624 $ env DUMMY_TEST_COMMIT_BAD_SHA=83f05e8 \
7625 time git bisect run \
7626 $TRIBITS_DIR/doc/developers_guide/scripts/dummy_test_commit.sh \
7627 &> ../git_bisect_run.log
7628 $ git2 bisect log &> ../git_bisect_log.log
7629 $ cat ../git_bisect_log.log | grep "possible first bad commit" | \
7630 sed "s/possible first bad commit://g" | sed "s/[a-z0-9]\{30\}\]/]/g"
7631 $ git bisect reset
7632
7633 This set of commands yield the output::
7634
7635 Bisecting: 1128 revisions left to test after this (roughly 10 steps)
7636 [9634d462dba77704b598e89ba69ba3ffa5a71471] Revert "Trilinos: remove _def.hpp [...]"
7637
7638 real 1m22.961s
7639 user 0m57.157s
7640 sys 3m40.376s
7641
7642 # [165067ce53] MueLu: SemiCoarsenPFactory. Use strided maps to properly transfer [...]
7643 # [ada21a95a9] MueLu: refurbish LineDetectionFactory
7644 # [83f05e8970] MueLu: stop semi-coarsening if no z-layers are left.
7645
7646 Previous HEAD position was 83f05e8... MueLu: stop semi-coarsening if no z-layers are left.
7647 Switched to branch 'master'
7648
7649 This output shows the dummy bad commit 83f05e8 in a set of just 3 commits,
7650 bounded in the set of commits ``8b79832..165067c``::
7651
7652 165067c "MueLu: SemiCoarsenPFactory. Use strided maps to properly [...]."
7653 Author: Tobias Wiesner <[email protected]>
7654 Date: Thu Jul 2 12:11:24 2015 -0600
7655
7656 8b79832 "Ifpack2: RBILUK: adding additional ETI types"
7657 Author: Jonathan Hu <[email protected]>
7658 Date: Thu Jul 2 14:17:40 2015 -0700
7659
7660 The set of commits that were actually tested by ``git bisect run <script>`` is
7661 shown by::
7662
7663 $ cat ../git_bisect_log.log | grep "\(good:\|bad:\)" | sed "s/[a-z0-9]\{30\}\]/]/g"
7664 # bad: [605b91b012] Merge branch 'master' of software.sandia.gov:/git/Trilinos
7665 # good: [d44c17d5d2] Merge branch 'master' of software.sandia.gov:/space/git/Trilinos
7666 # good: [7e13a95774] Ifpack2: If the user does not provide the bandwidth of the banded [...]
7667 # bad: [7335d8bc92] MueLu: fix documentation
7668 # bad: [9997ecf0ba] Belos::LSQRSolMgr: Fixed bug in setParameters.
7669 # bad: [b6e0453224] MueLu: add a nightly test for the combination of semicoarsening [...]
7670 # bad: [165067ce53] MueLu: SemiCoarsenPFactory. Use strided maps to properly [...]
7671 # good: [3b5453962e] Ifpack2: Nuking the old ETI system
7672 # good: [8b79832f1d] Ifpack2: RBILUK: adding additional ETI types
7673
7674 This is only 9 commits out of the possible set of 166 ``checkin-test.py``
7675 marked commits which is out of the total set of 2257 possible commits. With
7676 just 9 build/test cycles, it bounded the first "bad" commit in a set of 3
7677 commits in this case. And it does not matter how sloppy or broken the
7678 intermediate commits are in Trilinos. All that matters is the usage of the
7679 ``checkin-test.py`` tool (another motivation for the usage of the
7680 ``checkin-test.py`` tool, see `Pre-push Testing using checkin-test.py`_ for
7681 others as well).
7682
7683 Note that above, we grep the output from ``git bisect log`` for the set of
7684 possible "bad" commits instead of just looking at the output from the ``git
7685 bisect run <script>`` command (which also lists the set of possible "bad"
7686 commits). This is because the direct output from the ``git bisect run
7687 <script>`` command (shown in the log file ``git_bisect_run.log``) shows the
7688 set of possible bad commits at the end of the output but they are unsorted and
7689 give no other git commit information::
7690
7691 There are only 'skip'ped commits left to test.
7692 The first bad commit could be any of:
7693 83f05e89706590c4b384dd191f51ef4ab00ce9bb
7694 ada21a95a991cd238581e5a6a96800d209a57924
7695 165067ce538af2cd0bd403e2664171726ec86f3f
7696 We cannot bisect more!
7697 bisect run cannot continue any more
7698
7699 The problem with unsorted commits is that it is difficult to use an unsorted
7700 set to do further bisection. However, the output of the set of commits from
7701 ``git bisect log`` is sorted and also shows the commit summary message and
7702 therefore is much more useful. (But note that older versions of git don’t
7703 show this set of commits at the end of ``git bisect log`` so make sure and use
7704 an updated version of git, preferably >= 2.1.0.)
7705
7706 Now that one has the range of possible "bad" commits (just 3 in this example)
7707 doing a further manual bisection or even manual inspection of these commits
7708 may be enough to find the change that is causing the problem for the
7709 downstream customer application.
7710
7711 Without the usage of the ``checkin-test.py`` tool, one would not have an
7712 automated way to ensure that ``git bisect`` avoids false "bad" commits. This
7713 allows for less experienced developers to create commits and push to the main
7714 development branch but still ensure effective usage of ``git bisect``. (This
7715 is another example where automated tools in TriBITS help to overcome lacking
7716 developer experience and discipline.)
7717
7718
7719 Multi-Repository Almost Continuous Integration
7720 ----------------------------------------------
7721
7722 The `checkin-test.py`_ tool can be used to the implement staged integration of
7723 the various repositories in a multi-repo TriBITS project (see
7724 `Multi-Repository Support`_) . This is referred to here as Almost Continuous
7725 Integration (ACI). The basic concept of Almost Continuous Integration (ACI)
7726 is defined and described in the paper [`Integration Strategies for CSE,
7727 2009`_].
7728
7729 This topic is broken down into the following subsections:
7730
7731 * `ACI Multi-Git/TriBITS Repo Integration Example`_
7732 * `ACI Local Sync Git Repo Setup`_
7733 * `ACI Integration Build Directory Setup`_
7734 * `ACI Sync Driver Script`_
7735 * `ACI Cron Job Setup`_
7736 * `Addressing ACI Failures and Summary`_
7737
7738
7739 ACI Introduction
7740 ++++++++++++++++
7741
7742 The TriBITS system allows for setting up composite meta-builds of large
7743 collections of software pulled in from many different git/TriBITS code
7744 repositories as described in the section `Multi-Repository Support`_. The
7745 `checkin-test.py`_ tool is a key tool to enable the testing of a set of
7746 packages in different git/TriBITS repos before pushing to remote tracking
7747 branches for the set of git repos; all in one robust command invocation.
7748
7749 While the ``checkin-test.py`` tool was originally designed and its default
7750 behavior is to test a set of local commits created by a developer before
7751 pushing changes to one or more (public) git repos, it can also be used to set
7752 up an Almost Continuous Integration (ACI) process to keep these various
7753 git/TriBITS repos in sync thereby integrating the work of various disconnected
7754 development teams and projects. To use the ``checkin-test.py`` tool for ACI
7755 requires some setup and changing what the tool does a little by passing in
7756 additional options that a regular developer typically never uses.
7757
7758 The following subsections describe how to use the `checkin-test.py`_ tool to
7759 implement an ACI process for a given set of git/TriBITS repositories and also
7760 provides a little background and context behind ACI.
7761
7762
7763 ACI Multi-Git/TriBITS Repo Integration Example
7764 ++++++++++++++++++++++++++++++++++++++++++++++
7765
7766 In order to set up the context for the ACI process, consider the following
7767 simple TriBITS project with two extra repositories::
7768
7769 BaseProj/
7770 ExtraRepo1
7771 ExtraRepo2
7772
7773 Here, ``BaseProj`` is the base TriBITS project/repository and ``ExtraRepo1``
7774 and ``ExtraRepo2`` are extra repositories that supply additional TriBITS
7775 packages that are appended to the TriBITS packages defined in ``BaseProj``
7776 (see `<projectDir>/cmake/ExtraRepositoriesList.cmake`_). Also, assume that
7777 ``BaseProj``, ``ExtraRepo1``, and ``ExtraRepo2`` are developed by three
7778 different development teams that all have different funding sources and
7779 different priorities so they tend not to work closely together or consider the
7780 other efforts too much when developing their software. However, in this
7781 example, there is great value in combining all of this software into a single
7782 integrated TriBITS meta-project. This combined meta-build is driven by a 4th
7783 integration/development team. In this case, the core developers for each of
7784 these three different git/TriBITS repos do not test compatibility with the
7785 other git/TriBITS repos when pushing commits to their own git/TriBITS repos.
7786 This gives three different git repos on three different machines:
7787
7788 * ``BaseProj`` main repo: Pushed to by the core BaseProj team::
7789
7790 url1.gov:/git/BaseProj
7791
7792 * ``ExtraRepo1`` main repo: Pushed to by the core ExtraRepo1 team::
7793
7794 url2.gov:/git/ExtraRepo1
7795
7796 * ``ExtraRepo2`` main repo: Pushed to by the core ExtraRepo2 team::
7797
7798 url3.gov:/git/ExtraRepo2
7799
7800 Because of the independent development processes of these three teams, unless
7801 these development teams maintain 100% backward compatibility w.r.t. the
7802 interfaces and behavior of the combined software, one cannot at any time pull
7803 the code from these three different git repos and expect to be able to
7804 successfully build all of the code and have all of the tests pass. Therefore,
7805 how does the 4th integration team expect to be able to build, test, and
7806 possibly extend the combined software? In this case, the integration team
7807 would set up their own clones of all three git/TriBITS repos on their own
7808 machine such as:
7809
7810 **Integration project mirrored git repos**::
7811
7812 url4.gov:/git/BaseProj
7813 url4.gov:/git/ExtraRepo1
7814 url4.gov:/git/ExtraRepo2
7815
7816 Once an initial collaboration effort between the integration team and the
7817 three other development teams is able to get a version of all three
7818 git/TriBITS repos to work correctly in the combined meta-project, these
7819 versions (assume the ``master`` branches) would be pushed to the git repos on
7820 the git integration server ``url4.gov``. The state where the TriBITS packages
7821 in the three different git/TriBITS repos in the ``master`` branch on
7822 ``url4.gov`` all work together correctly constitutes the initial condition for
7823 the ACI process described below. From that initial condition, the ACI
7824 processes ensures that updates the ``master`` branches for the git/TriBITS
7825 repos on ``url4.gov`` do not break any builds or tests of the integrated
7826 software.
7827
7828 In order to describe how to set up an ACI process using the
7829 ``checkin-test.py`` tool, the following subsections will focus on the update
7830 of the git/TriBITS ``ExtraRepo1`` repo keeping the other two git/TriBITS repos
7831 ``BaseProj`` and ``ExtraRepo2`` constant as the ACI use case.
7832
7833
7834 ACI Local Sync Git Repo Setup
7835 +++++++++++++++++++++++++++++
7836
7837 In order to set up an ACI process for the multi-git/TriBITS repo example
7838 outlined above, first local repos are created by cloning the repos on the
7839 integration server url4.gov as follows (all of which become 'origin')::
7840
7841 $ cd $SYNC_BASE_DIR
7842 $ git clone url4.gov:/git/BaseProj
7843 $ cd BaseProj
7844 $ git clone url4.gov:/git/ExtraRepo1
7845 $ git clone url4.gov:/git/ExtraRepo2
7846
7847 where, ``SYNC_BASE_DIR=~/sync_base_dir`` for example, which must already be
7848 created.
7849
7850 Next, one defines a remote to pull changes for the ``ExtraRepo1`` from the
7851 main development repo:
7852
7853 $ cd $SYNC_BASE_DIR/BaseProj/ExtraRepo1
7854 $ git remote add public url2.gov:/git/ExtraRepo1
7855
7856 Here, one should pick a name for the remote repo for ``ExtraRepo1`` that is
7857 most descriptive for that particular situation. In this case, the name
7858 ``public`` is chosen to signify the main public development repo.
7859
7860 This gives the remotes::
7861
7862 $ cd $SYNC_BASE_DIR/BaseProj
7863 $ gitdist remote -v | grep -v push | grep -v "^$"
7864 *** Base Git Repo: BaseProj
7865 origin url4.gov:/git/BaseProj (fetch)
7866 *** Git Repo: ExtraRepo1
7867 origin url4.gov:/git/ExtraRepo1 (fetch)
7868 public url2.gov:/git/ExtraRepo1 (fetch)
7869 *** Git Repo: ExtraRepo2
7870 origin url4.gov:/git/ExtraRepo2 (fetch)
7871
7872 The remote ``public`` is used by the ``checkin-test.py`` wrapper script (see
7873 `ACI Sync Driver Script`_ below) to pull and merge in additional changes that
7874 will be tested and pushed to the 'origin' repos on ``url4.gov``. In this
7875 case, the ``ExtraRepo1`` remote ``public`` will result in updates being pulled
7876 from the main development repo on ``url2.gov``, thereby facilitating the
7877 update of ``ExtraRepo1`` in the integrated meta-project.
7878
7879
7880 ACI Integration Build Directory Setup
7881 +++++++++++++++++++++++++++++++++++++
7882
7883 After the git repos are cloned and the remotes are set up as described above,
7884 a build base directory is set up as::
7885
7886 $ cd $SYNC_BASE_DIR
7887 $ mkdir BUILDS
7888 $ mkdir BUILDS/CHECKIN
7889
7890 An ACI wrapper script for ``checkin-test.py`` is created to drive the syncing
7891 process. It is assumed that this script would be called only once a day and
7892 not continuously in a loop (but that is possible as well but is not documented
7893 here).
7894
7895 NOTE: Other build directory structures are possible, it all depends how one
7896 writes the ``checkin-test.py`` wrapper scripts but the above directory
7897 structure is fairly standard in the usage of the ``checkin-test.py`` script.
7898
7899
7900 ACI Sync Driver Script
7901 ++++++++++++++++++++++
7902
7903 The sync driver script for this example should be called something like
7904 ``sync_ExtraRepo1.sh``, placed under version control, and would look something
7905 like::
7906
7907 #!/bin/bash -e
7908
7909 # Set up the environment (i.e. PATH; needed for cron jobs)
7910 ...
7911
7912 SYNC_BASE_DIR=~/sync_base_dir
7913 CHECKIN_TEST_WRAPPER=$SYNC_BASE_DIR/BaseProj/sampleScripts/checkin-test-foo.sh
7914
7915 cd $SYNC_BASE_DIR/BUILDS/CHECKIN
7916
7917 $CHECKIN_TEST_WRAPPER \
7918 --extra-pull-from=ExtraRepo1:public:master \
7919 --abort-gracefully-if-no-changes-to-push \
7920 --enable-extra-packages=Package1A \
7921 --send-build-case-email=only-on-failure \
7922 [email protected] \
7923 [email protected] \
7924 --no-append-test-results --no-rebase \
7925 --do-all --push \
7926 -j16 \
7927 --wipe-clean \
7928 "$@"
7929
7930 NOTE, in the above example ``sync_ExtraRepo1.sh`` script, the variable
7931 ``CHECKIN_TEST_WRAPPER`` is set to a wrapper script::
7932
7933 BaseProj/sampleScripts/checkin-test-foo.sh
7934
7935 which would be set up to call the project's `checkin-test.py`_ tool with
7936 configure options for the specific machine. The location and the nature of
7937 the wrapper script will vary from project to project and machine to machine.
7938 In some simple cases, ``CHECKIN_TEST_WRAPPER`` might just be set to be the raw
7939 machine-independent ``checkin-test.py`` tool for the project.
7940
7941 A description of each option passed into this invocation of the
7942 `checkin-test.py`_ tool is given below (see `checkin-test.py --help`_ for more
7943 details):
7944
7945 ``--extra-pull-from=ExtraRepo1:public:master``
7946
7947 This option instructs the ``checkin-test.py`` tool to pull and merge in
7948 commits that define the integration. One could do the pull(s) manually of
7949 doing so has the disadvantage that if they fail for some reason, they will
7950 not be seen by the ``checkin-test.py`` tool and no notification email
7951 would go out.
7952
7953 ``--abort-gracefully-if-no-changes-to-push``
7954
7955 The option ``--abort-gracefully-if-no-changes-to-push`` makes the
7956 ``checkin-test.py`` tool gracefully terminate without sending out any
7957 emails if after all the pulls, there are no local changes to push to the
7958 'origin' repos. This can happen, for example, if no commits were pushed
7959 to the main development git repo for ``ExtraRepo1`` at
7960 ``url2.gov:/git/ExtraRepo1`` since the last time this sync process was
7961 run. This avoids getting confusing and annoying emails like ``"PUSH
7962 FAILED"``. The reason this option is not generally needed for local
7963 developer usage of the ``checkin-test.py`` tool is that in general a
7964 developer will not run the ``checkin-test.py`` tool with ``--push`` unless
7965 they have made local changes; it just does not make any sense at all to do
7966 that and if they do by accident, they should get an error email. However,
7967 for an automated ACI sync process, there is no easy way to know a-priori
7968 if changes need to be synced so the script supports this option to deal
7969 with that case gracefully.
7970
7971 ``--enable-extra-packages=Package1A``
7972
7973 This option should be set if one wants to ensure that all commits get
7974 synced, even when these changes don't impact the build or the tests of the
7975 project. If not setting ``--enable-extra-packages=<some-package>`` , then
7976 the ``checkin-test.py`` tool will only decide on its own what packages to
7977 test just based on what packages have changed files in the ``ExtraRepo1``
7978 repo and if no modified files map to a package, then no packages will be
7979 auto-enabled and therefore no packages will be enabled at all. For
7980 example, if a top-level README file in the base ``ExtraRepo1`` repo gets
7981 modified that does not sit under a package directory, then the automatic
7982 logic in the checkin-test.py tool will not trigger a package enable. In
7983 that case, no configure, build, testing, or push will take place (must run
7984 at least some tests in order to assume it is safe to push) and therefore
7985 the sync will not occur. Therefore, if one wants to ensure that every
7986 commit gets safely synced over on every invocation, then the safest way to
7987 that is to always enable at least one or more packages by specify
7988 ``--enable-extra-packages=<pkg0>,<pkg1>``. **WARNING:** it is not
7989 advisable to manually set ``--enable-packages=<package-list>`` since it
7990 turns off the auto-enable logic for changed files. This is because if
7991 there are changes to other packages, then these packages will not get
7992 enabled and not get tested, which could break the global build and tests.
7993 Also, this is fragile if new packages are added to ``ExtraRepo1`` later
7994 that are not listed in ``--enable-packages=<pkg0>,<pkg1>,...`` as they
7995 will not be included in the testing. Also, if someone makes local commits
7996 in other local git repos before running the sync script again, then these
7997 packages will not get enabled and tested. Therefore, in general, don't
7998 set ``--enable-packages=<pkg0>,<pkg1>,...`` in a sync script, only set
7999 ``--enable-extra-packages=<pkg0>,<pkg1>,...`` to be robust and safe.
8000
8001 ``--send-build-case-email=only-on-failure``
8002
8003 This makes the checkin-test.py tool skip sending email about a build case
8004 (e.g. ``MPI_DEBUG``) unless it fails. That way, if everything passes,
8005 then only a final ``DID PUSH`` email will go out. But if a build case
8006 does fail (i.e. configure, build, or tests fail), then an "early warning"
8007 email will still go out. However, if one wants to never get the early
8008 build-case emails, one can turn this off by setting
8009 ``--send-build-case-email=never``.
8010
8011 ``[email protected]``
8012
8013 The results of the builds will be sent this email address. If you only
8014 want an email sent when a push actually happens, you can set
8015 ``--send-email-to=''`` and rely on ``--send-email-to-on-push``.
8016
8017 ``[email protected]``
8018
8019 A confirmation and summary email will be sent to this address if the push
8020 happens. This can be a different email address than set by the
8021 ``--send-email-to`` option. It is highly recommended that a mail list be
8022 used for this email address since this will be the only more permanent
8023 logging of the ACI process.
8024
8025 ``--no-append-test-results --no-rebase``
8026
8027 These options are needed to stop the ``checkin-test.py`` tool from
8028 modifying the commits being tested and pushed from one public git repo to
8029 another. The option ``--no-append-test-results`` is needed to instruct
8030 the ``checkin-test.py`` tool to **NOT** amend the last commit with the
8031 test results. The option ``--no-rebase`` is needed to avoid rebasing the
8032 new commits pulled. While the default behavior of the ``checkin-test.py``
8033 tool is to amend the last commit message and rebase the local commits
8034 (which is considered best practice when testing local commits), this is a
8035 very bad thing to do when a ACI sync server is only testing and moving
8036 commits between public repos. Amending the last commit would change the
8037 SHA1 of the commit (just as a rebase would) and would fork the history and
8038 mess up a number of workflows that otherwise should work smoothly. Since
8039 an email logging what was tested will go out if a push happens due to the
8040 ``--send-email-to-on-push`` argument, there is no value in appending the
8041 test results to the last commit pulled and merged (which will generally
8042 not be a merge commit but just a fast-forward). There are cases, however,
8043 where appending the test results in an ACI process might be acceptable but
8044 they are not discussed here.
8045
8046 ``--do-all --push -j16``
8047
8048 These are standard options that always get used when invoking the
8049 ``checkin-test.py`` tool and need no further explanation.
8050
8051 ``--wipe-clean``
8052
8053 This option is added if you want to make the sync server more robust to
8054 changes that might require a clean configure from script. If you care
8055 more about using less computer resources and testing that rebuilds work
8056 smoothly, remove this option.
8057
8058 The sync script can be created and tested locally to ensure that it works
8059 correctly first, before setting it as a cron job as described next. Also, the
8060 sync script should be version controlled in one of the project's git repos.
8061 This ensures that changes to script pushed to the repos will get invoked
8062 automatically when they are pulled (but only on the second invocation of the
8063 script). If a change to script is critical in order to do the pull, then one
8064 must manually pull the updated commit to the local sync repo.
8065
8066 Note, if using this in a continuous sync server that runs many times in a day
8067 in a loop, you also want to set the option
8068 ``--abort-gracefully-if-no-changes-pulled`` in addition to the option
8069 ``--abort-gracefully-if-no-changes-to-push``. That is because if the updated
8070 repos are in a broken state such that there are always local changes at every
8071 CI iteration (because they have not been pushed to origin), you don't want to
8072 do a new CI build unless something has changed that would otherwise perhaps
8073 make the error go away. That allows the CI server to sit ready to try out any
8074 change that gets pulled that might allow the integrated build to work and then
8075 push the updates.
8076
8077
8078 ACI Cron Job Setup
8079 ++++++++++++++++++
8080
8081 Once the sync script ``sync_ExtraRepo1.sh`` has been locally tested, then it
8082 should be committed to a version control git repo and then run automatically
8083 as a cron job. For example, the cron script shown below would fire off the
8084 daily ACI process at 8pm local time every night::
8085
8086 # ----------------- minute (0 - 59)
8087 # | -------------- hour (0 - 23)
8088 # | | ----------- day of month (1 - 31)
8089 # | | | -------- month (1 - 12)
8090 # | | | | ----- day of week (0 - 7) (Sunday=0 or 7)
8091 # | | | | |
8092 # * * * * * command to be executed
8093 00 20 * * * ~/sync_base_dir/sync_ExtraRepo1.sh &> ~/sync_base_dir/sync_ExtraRepo1.out
8094
8095 In the above crontab file (set with ``'crontab -e'`` or ``'crontab
8096 my-crontab-file.txt'``), the script::
8097
8098 ~/sync_base_dir/sync_ExtraRepo1.sh
8099
8100 is assumed to be a soft symbolic link to some version controlled copy of the
8101 ACI sync script. For example, it might make sense for this script to be
8102 version controlled in the ``BaseProj`` repo and therefore the symbolic link
8103 would be created with something like::
8104
8105 $ cd ~/sync_base_dir/
8106 $ ln -s BaseProj/sampleScripts/sync_ExtraRepo1.sh .
8107
8108 Such a setup would ensure that sync scripts would always be up-to-date due to
8109 the git pulls part of the ACI process.
8110
8111
8112 Addressing ACI Failures and Summary
8113 +++++++++++++++++++++++++++++++++++
8114
8115 After the above cron job starts running (setup described above), the
8116 ``checkin-test.py`` tool will send out emails to the email addresses passed
8117 into the underlying ``checkin-test.py`` tool. If the emails report an update,
8118 configure, build, or test failure, then someone will need to log onto the
8119 machine where the ACI sync server is running and investigate what went wrong,
8120 just like they would if they were running the ``checkin-test.py`` tool for
8121 testing locally modified changes before pushing.
8122
8123 In the above example, only a single git/TriBITS repo is integrated in this ACI
8124 sync scripts. For a complete system, other ACI sync scripts would be written
8125 to sync the two other git/TriBITS repos in order to maintain some
8126 independence. Or, a single ACI sync script that tries to update all three
8127 git/TriBITS repos at could would be written and used. The pattern of
8128 integrations chosen will depend many different factors and these patterns can
8129 change over time according to current needs and circumstances.
8130
8131 In summary, the `checkin-test.py`_ tool can be used to set up robust and
8132 effective Almost Continuous Integration (ACI) sync servers that can be used to
8133 integrate large collections of software in logical configurations at any
8134 logical frequency. Such an approach, together with the practice of `Regulated
8135 Backward Compatibility and Deprecated Code`_, can allow very large collections
8136 of software to be kept integrated in short time intervals using ACI.
8137
8138 Post-Push CI and Nightly Testing using checkin-test.py
8139 ------------------------------------------------------
8140
8141 While the post-push CI and Nightly testing processes using ``ctest -S``
8142 scripts using `tribits_ctest_driver()`_ (see `TriBITS CTest/CDash Driver`_)
8143 which posts results to a CDash server (see `TriBITS CDash Customizations`_) is
8144 a very attractive testing system with many advantages, setting up a CDash
8145 server can be a bit difficult and a CDash server can require non-trivial
8146 storage and CPU resources (due to the MySQL DB of test results) and requires
8147 some amount of extra maintenance. As an intermediate approach, one can
8148 consider just using the project's `checkin-test.py`_ tool to implement basic
8149 post-push CI and/or Nightly testing servers using simple cron jobs and some
8150 other helper scripts. The ``checkin-test.py`` tool will robustly pull new
8151 commits, configure the project, build, run tests, and send out emails with
8152 results and pass/fail. A bunch of builds can be run at once using multiple
8153 builds specified in the ``--default-builds``, ``--st-extra-builds``, and
8154 ``--extra-build`` arguments, or different invocations of the
8155 ``checkin-test.py`` tool can be run in parallel for better machine
8156 utilization.
8157
8158 What one gives up with this approach over the full-blow CTest/CDash
8159 implementation is:
8160
8161 1) Test results over multiple builds/days are not archived in a database for
8162 later retrieval and query. Only the latest build's detailed results are
8163 accessible.
8164
8165 2) Detailed test results (pass or fail) cannot be easily looked up on a
8166 website. Only detailed results from the last build can be found and one
8167 has to go to the machine where the build was run from to see the results.
8168
8169 The process for setting up a basic nightly build using ``checkin-test.py`` as
8170 a cron job is a subset of the steps needed to set up a driver and cron job for
8171 the more complex ACI process described in `ACI Sync Driver Script`_ and `ACI
8172 Cron Job Setup`_. Setting up a CI server using ``checkin-test.py`` is a
8173 little more involved than setting up a nightly build (and less desirable
8174 because it blows away old CI iteration results pretty fast) but can be done
8175 using existing tools.
8176
8177 .. ToDo: Show how to set up a cron job using checkin-test.py.
8178
8179 For smaller, private, less-critical projects with few developers, setting up
8180 CI and Nightly testing using ``checkin-test.py`` may be quite adequate. In
8181 fact, post-push testing processes implemented with ``checkin-test.py`` are
8182 much more solid and feature-full than have been employed in many software
8183 projects that we have seen over the years that were larger, more public, had
8184 many developers, and were quite important to many users and development
8185 teams.
8186
8187
8188 TriBITS Dashboard Driver
8189 ------------------------
8190
8191 TriBITS also includes a system based on CMake/CTest/CDash to drive the builds
8192 of a TriBITS project and post results to another CDash project. This system
8193 is contained under the directory::
8194
8195 tribits/ctest/tdd/
8196
8197 If the TriBITS project name is ``<projectName>``, the TDD driver CDash project
8198 is typically called ``<projectName>Driver``. Using CTest to drive the Nightly
8199 builds of a TriBITS project makes sense because CTest can run different builds
8200 in parallel, can time-out builds that are taking too long, and will report
8201 results to a dashboard and submit notification emails when things fail.
8202 However, this is the most confusing and immature part of the TriBITS system.
8203 The `TriBITS CTest/CDash Driver`_ system using the `tribits_ctest_driver()`_
8204 can be used without this TriBITS Dashboard Driver (TDD) system.
8205
8206 However, this TriBITS subsystem is not well tested with automated tests, is
8207 difficult to extend and test manually, and has other problems. Therefore, it
8208 is not recommended that projects adopt the usage of this subsystem. A simple
8209 set of cron jobs or a tool like Jenkins is likely a better option (if done
8210 carefully).
8211
8212 .. ToDo: Import and format the contents of tribits/ctest/README related to
8213 .. this topic into this section.
8214
8215
8216 Regulated Backward Compatibility and Deprecated Code
8217 ----------------------------------------------------
8218
8219 The motivation and ideas behind `Regulated Backward Compatibility` and
8220 deprecated code are described in the `TriBITS Lifecycle Model`_ document.
8221 Here, the details of the implementation in TriBITS are given and how
8222 transitions between non-backward compatible major versions is accomplished.
8223
8224 This section describes the process for deprecating code within a major
8225 backward-compatible version sequence and finally removing deprecated code when
8226 transitioning to a new major version X to X+1 for the semantic versioning
8227 numbering scheme X.Y.Z. This information is given as a process with different
8228 phases.
8229
8230 The processing for managing deprecated code is as follows and more details are
8231 given below:
8232
8233 * `Setting up support for deprecated code handling`_
8234
8235 * `Deprecating code but maintaining backward compatibility`_
8236
8237 * `Deprecating C/C++ classes, structs, functions, typdefs, etc.`_
8238 * `Deprecating preprocessor macros`_
8239 * `Deprecating an entire header and/or source file`_
8240
8241 * `Hiding deprecated code to certify and facilitate later removal`_
8242
8243 * `Hiding C/C++ entities`_
8244 * `Hiding entire deprecated header and source files`_
8245
8246 * `Physically removing deprecated code`_
8247
8248 * `Removing entire deprecated header and source files`_
8249 * `Removing deprecated code from remaining files`_
8250
8251
8252 Setting up support for deprecated code handling
8253 +++++++++++++++++++++++++++++++++++++++++++++++
8254
8255 Setting up support for managing deprecated code in a TriBITS package requires
8256 just two simple changes to the TriBITS-related files in a package. First, the
8257 top-level package `<packageDir>/CMakeLists.txt`_ file needs to have a call
8258 to::
8259
8260 tribits_add_show_deprecated_warnings_option()
8261
8262 Second, the package's configure header::
8263
8264 <packageDir>/cmake/<PACKAGE_UCNAME>_config.h.in
8265
8266 file needs to have::
8267
8268 @<PACKAGE_UCNAME>_DEPRECATED_DECLARATIONS@
8269
8270 where ``<PACKAGE_UCNAME>`` is the upper-case name of the TriBITS package.
8271
8272 That is all that is needed to provide support for TriBITS deprecated code.
8273
8274 When a TriBITS project is configured with::
8275
8276 -D<Project>_SHOW_DEPRECATED_WARNINGS=ON
8277
8278 by default, all packages will show deprecated warnings. These deprecated
8279 warnings can be turned on and off on a package-by-package basis by setting::
8280
8281 -D<PackageName>_SHOW_DEPRECATED_WARNINGS=[ON|OFF]
8282
8283 This gives developers and users a little extra control over what deprecated
8284 warnings are shown.
8285
8286 In addition, deprecated code can be hidden from the build to help certify that
8287 downstream code is clean by configuring with::
8288
8289 -D<Project>_HIDE_DEPRECATED_CODE=ON
8290
8291 This will remove the deprecated code from the build and install (see details
8292 below) so that other software in the TriBITS project can be shown to build
8293 clean without deprecated code and so that outside client code can be shown to
8294 be clean of deprecated code.
8295
8296 As with deprecated warnings, showing or hiding deprecated code can be
8297 controlled on a package-by-package basis by setting::
8298
8299 -D<PackageName>_HIDE_DEPRECATED_CODE=[ON|OFF]
8300
8301 In this case, hiding deprecated code on a package-by-package basis may not
8302 work because deprecated code in a downstream package might rely on deprecated
8303 code in an upstream package (which might have its deprecated code hidden).
8304
8305
8306 Deprecating code but maintaining backward compatibility
8307 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
8308
8309 One of the most important aspects of the `TriBITS Lifecycle Model`_ for
8310 later-stage Production Growth and Production Maintenance code is to provide
8311 backward compatibility between a continuous stream of versions of the software
8312 within a major version number X in the version numbering scheme X.Y.Z. In all
8313 cases, if a piece of client code builds and runs correctly against version
8314 X0.Y0.Z0, it should also build, without modification, against versions
8315 X0.Y1,Z1 for all Y1 >= Y0 and all Z1 and up to (but not including) X0+1.0.0.
8316 There are many types of constructs that one will want to deprecate and
8317 therefore later remove. When deprecating code, one wants to give users
8318 compile-time warnings of the usage of deprecated features so that they know
8319 what they need to remove. One also wants to allow them to certify that their
8320 code is free of deprecated warnings by hiding deprecated code. Below, the
8321 different types of entities that one wants to deprecate and how to support
8322 hiding code (which also facilitates removing it later) are described.
8323
8324
8325 Deprecating C/C++ classes, structs, functions, typdefs, etc.
8326 ............................................................
8327
8328 To deprecate standard C/C++ constructs, one can just use the standard TriBITS
8329 compile-time macro ``<PACKAGE_UCNAME>_DEPRECATED`` which is properly ifdefed
8330 by the TriBITS system to add a GCC/Intel deprecated attribute or not. For
8331 example, one would deprecate standard C/C++ constructs for the package
8332 ``SomePackage`` with::
8333
8334 // Deprecate a class (or struct)
8335 class SOMEPACKAGE_DEPRECATED SomeClass { ... };
8336
8337 // Deprecate a function
8338 SOMEPACKAGE_DEPRECATED int somefunc(...);
8339
8340 // Deprecate a typedef
8341 SOMEPACKAGE_DEPRECATED typedef someTypeDef int;
8342
8343 The GCC (version 3.1 and newer) and Intel C and C++ compilers both support
8344 adding extra attributes including the ``__deprecated__`` attribute. When this
8345 attribute is applied to a given C/C++ entity, it produces a compiler warning
8346 that can be searched for in the compiler output and elevated to an error (when
8347 ``-Werror`` is also passed to the compiler).
8348
8349 In addition to the basic deprecated warning, one can also add an optional
8350 deprecated warning message using the macro
8351 ``<PACKAGE_UCNAME>_deprecated_msg()``. For example, if a new function is
8352 replacing an old function, one might use::
8353
8354 // Deprecated old unsafe function taking raw pointers
8355 somepackage_deprecated_msg(
8356 "Please use the safe somefunc(const Ptr<const std::string>&) instead!")
8357 void somefunc(const std::string *str);
8358
8359 // New version take does not take raw pointers (and is therefore safer)
8360 void somefunc(const Teuchos::Ptr<const std::string> &str);
8361
8362 Then, if user code calls the version ``somefunc(const std::string*)`` they
8363 will get the string::
8364
8365 "Please use the safe somefunc(const Ptr<const std::string>&) instead!"
8366
8367 printed as well by the compiler. Note that the custom message will only be
8368 printed for GCC versions 4.5 and newer. If an older version of GCC is used,
8369 then the message string is ignored.
8370
8371
8372 Deprecating preprocessor macros
8373 ...............................
8374
8375 A C/C++ preprocessor macro is not an entity seen by the C/C++ compiler and
8376 therefore cannot directly take advantage of a feature such as the
8377 ``__deprecated__`` attribute of the GCC/Intel compilers. However, in some
8378 cases, for function-like macros can such as::
8379
8380 // The definition of the macro
8381 #define some_old_func_macro(ARG1, ARG2, ...) ...
8382 ...
8383 // A use of the macro
8384 some_old_func_macro(a, b, ...)
8385
8386 there is a strategy where one can define the macro to call a dummy deprecated
8387 function such as with::
8388
8389 SOMEPACKAGE_DEPRECATED
8390 inline void some_old_func_macro_is_deprecated()
8391 {}
8392
8393 // The definition of the macro
8394 #define some_old_func_macro(ARG1, ARG2, ...) \
8395 { \
8396 some_old_func_macro_is_deprecated(); \
8397 ... \
8398 }
8399
8400 ...
8401
8402 // A use of the macro
8403 some_old_func_macro(a, b, ...)
8404
8405 In the above example, client code calling ``some_old_func_macro()`` will now
8406 result in a deprecated compiler warning which should make it clear what is
8407 being deprecated.
8408
8409 Note that this approach will not work in cases were a macro is only providing
8410 a single value such as a constant (but one should not use macros for providing
8411 constants anyway).
8412
8413
8414 Deprecating an entire header and/or source file
8415 ...............................................
8416
8417 There are times when one wants to deprecate an entire set of files and all of
8418 the contents in those files. In addition to deprecating the contents of the
8419 files one will want to deprecate the entire file as well. There are a few
8420 steps to this. First, one wants to put a warning in the file such as::
8421
8422 #ifdef __GNUC__
8423 # warning "This header <THIS_HEADER> is deprecated! Use <NEW_HEADER> instead!"
8424 #endif
8425
8426 The above ``ifdef`` on ``__GNUC__`` is needed in order avoid the non-standard
8427 ``#warning`` preprocessor directive on non-compliant compilers (but should
8428 work on all later version GCC and Intel compilers).
8429
8430
8431 Hiding deprecated code to certify and facilitate later removal
8432 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
8433
8434 In addition to adding deprecation warnings at preprocessing or compile-time,
8435 it is also highly desirable to allow the deprecated code to be removed from
8436 the build to help certify that client code indeed no longer needs the
8437 deprecated code. The following subsections describe how to hide deprecated
8438 code from existing files and how to hide deprecated files entirely.
8439
8440
8441 Hiding C/C++ entities
8442 .....................
8443
8444 In the case when various C/C++ entities will be removed from an existing file,
8445 but the file will remain, then the deprecated code can be ifdefed out, for the
8446 package ``SomePackage`` for example, using::
8447
8448 #ifndef SOMEPACKAGE_HIDE_DEPRECATED_CODE
8449 // Deprecate a class (or struct)
8450 class SOMEPACKAGE_DEPRECATED SomeClass { ... };
8451 ...
8452 #endif /* SOMEPACKAGE_HIDE_DEPRECATED_CODE */
8453
8454 In this way, when the CMake variable ``SomePackae_HIDE_DEPRECATED_CODE=ON``,
8455 then the deprecated code will be completely removed resulting in compile
8456 errors for any downstream code still using them.
8457
8458
8459 Hiding entire deprecated header and source files
8460 ................................................
8461
8462 In order to hide entire deprecated header and source files when the CMake
8463 variable ``<PackageName>_HIDE_DEPRECATED_CODE=ON`` is set, one needs to move
8464 the headers and sources to another directory and provide for conditional
8465 inclusion in the TriBITS build of the library. For example, suppose one wants
8466 to remove support for the deprecated files ``SomeOldStuff.hpp`` and
8467 ``SomeOldStuff.cpp``. In this case, one would move the files onto a new
8468 ``deprecated/`` sub-directory and then write the ``CMakeLists.txt`` file
8469 like::
8470
8471 set(HEADERS "")
8472 set(SOURCES "")
8473
8474 set_and_inc_dirs(DIR ${CMAKE_CURRENT_SOURCE_DIR})
8475 append_glob(HEADERS ${DIR}/*.hpp)
8476 append_glob(SOURCES ${DIR}/*.cpp)
8477
8478 if (NOT ${PACKAGE_NAME}_HIDE_DEPRECATED_CODE)
8479 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/deprecated)
8480 append_set(HEADERS
8481 SomeOldStuff.hpp
8482 )
8483 append_set(SOURCES
8484 SomeOldStuff.cpp
8485 )
8486 endif()
8487
8488 ...
8489
8490 tribits_add_library(
8491 <LIBRARY_NAME>
8492 HEADERS ${HEADERS}
8493 SOURCES ${SOURCES}
8494 )
8495
8496 In this way, when ``${PACKAGE_NAME}_HIDE_DEPRECATED_CODE=TRUE``, then the
8497 directory for the deprecated headers will not be in the include path (so
8498 downstream clients will not even be able to see them) and they will not be
8499 installed so external clients will not be able to see them either. However,
8500 when ``${PACKAGE_NAME}_HIDE_DEPRECATED_CODE=FALSE``, then these files will be
8501 included in the build and include path and downstream clients can use them.
8502
8503 Once these files need to be permanently removed, one just then needs to remove
8504 them from the version control repository (i.e. ``git rm <files_to_remove>``)
8505 and then remove them from the above ``CMakeLists.txt`` code.
8506
8507
8508 Physically removing deprecated code
8509 +++++++++++++++++++++++++++++++++++
8510
8511 The final step in the code deprecation cycle is to actually remove the
8512 deprecated code. This is necessary to clean house, remove clutter and finally
8513 get the payoff in the reduction of technical debt that occurs when removing
8514 what is no longer needed or wanted.
8515
8516 It is recommended to remove deprecated files first, then remove deprecated
8517 file fragments from remaining files. Also, it is recommended to create git
8518 commits after each step.
8519
8520 It is also recommended that some time before deprecated code is actually
8521 removed, that a TriBITS repo change the default of
8522 ``<Project>_HIDE_DEPRECATED_CODE`` from ``OFF`` to ``ON`` so that downstream
8523 clients will see the effects of hiding the deprecated code before the code is
8524 actually removed. In fact, this default should be changed several days to a
8525 week or more before the code is actually removed. This way, downstream code
8526 developers will get a "shock" about removal of the deprecated code but can
8527 manually configure with ``-D<Project>_HIDE_DEPRECATED_CODE=OFF`` to keep
8528 building in the short-term until they can remove their usage of deprecated
8529 code.
8530
8531
8532 Removing entire deprecated header and source files
8533 ..................................................
8534
8535 To remove entire deprecated header and source files one just needs to first
8536 remove them from the version control repository and local directories
8537 (e.g. ``git rm deprecated/*``) and then remove any traces of them from the
8538 ``CMakeLists.txt`` file. For the example in `Hiding entire deprecated header
8539 and source files`_, one would just remove the files ``SomeOldStuff.hpp`` and
8540 ``SomeOldStuff.cpp`` from the ``CMakeLists.txt`` file leaving::
8541
8542 if (NOT ${PACKAGE_NAME}_HIDE_DEPRECATED_CODE)
8543 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/deprecated)
8544 append_set(HEADERS
8545 )
8546 append_set(SOURCES
8547 )
8548 endif()
8549
8550 Since more files may be deprecated later, it may be a good idea to leave the
8551 machinery for conditionally including deprecated files by leaving the above
8552 empty CMake code or just commenting it out.
8553
8554 To find which ``CMakeLists.txt`` files need to be modified, do a search like::
8555
8556 $ find . -name CMakeLists.txt -exec grep -nH HIDE_DEPRECATED_CODE {} \;
8557
8558 After removing the files, create a local commit of the removed files and the
8559 updated ``CMakeLists.txt`` files before removing deprecated fragments from the
8560 source files. In other words, do::
8561
8562 $ emacs -nw CMakeLists.txt # Remove the references to the deprecated files
8563 $ git rm SomeOldStuff.hpp SomeOldStuff.cpp
8564 $ git commit -m "Removing deprecated files"
8565
8566
8567 Removing deprecated code from remaining files
8568 .............................................
8569
8570 The deprecated ifdefed blocks described in `Hiding C/C++ entities`_ can be
8571 removed manually but it is generally preferred to use a tool. One simple tool
8572 that can do this is called ``unifdef``, that can be downloaded and it is
8573 documented at:
8574
8575 http://dotat.at/prog/unifdef/
8576
8577 Just download, build, and install the program ``unifdef`` (see
8578 ``unifdef/INSTALL`` in untarred source) and then run it as described below.
8579 In the example below, assume the program is installed in the user's home
8580 directory under::
8581
8582 ~/install/bin/unifdef
8583
8584 For a given TriBITS package, the program is then run as::
8585
8586 $ find . -name "*pp" -exec ~/install/bin/unifdef \
8587 -DSomePackage_HIDE_DEPRECATED_CODE {} -o {} \;
8588
8589 After the above command runs, look at the diffs to make sure the ifdef
8590 deprecated code blocks were removed correctly. For example, run::
8591
8592 $ git diff -- .
8593
8594 If the diffs look correct, commit the changes::
8595
8596 $ git commit -m "Removing deprecated code blocks" -- .
8597
8598 Then test everything and push using the `checkin-test.py`_ tool.
8599
8600 After that, all deprecated code is removed and the next period of incremental
8601 change and deprecation begins.
8602
8603
8604 Installation and Backward Compatibility Testing
8605 -----------------------------------------------
8606
8607 TriBITS has some built-in support for installation testing and backward
8608 compatibility testing. The way it works is that one can install the headers,
8609 libraries, and executables for a TriBITS project and then configure the tests
8610 and examples in the TriBITS project against the installed
8611 headers/libraries/executables. In this mode, the TriBITS project's libraries
8612 and executables are **not** build and the header file locations to local
8613 source are not included.
8614
8615 When the same version of the project sources are used to build the
8616 tests/examples against the installed headers/libraries/executables, then this
8617 constitutes *installation testing*. When an older version of the project is
8618 used to build and run tests and examples against the
8619 headers/libraries/executables for a version of the project, then this
8620 constitutes *backward compatibility testing* which also includes installation
8621 testing of course.
8622
8623 .. ToDo: Describe how the installation testing and backward compatibility
8624 .. testing process works with some examples.
8625
8626 .. ToDo: Discuss how this fits into the TriBITS lifecycle model.
8627
8628
8629 Wrapping Externally Configured/Built Software
8630 ---------------------------------------------
8631
8632 It is possible to take an external piece of software that uses any arbitrary
8633 build system and wrap it as a TriBITS package and have it integrate in with
8634 the package dependency infrastructure. The `TribitsExampleProject`_ package
8635 ``WrapExternal`` shows how this can be done. Support for this in TriBITS is
8636 slowly evolving but some key TriBITS features that have been added to support
8637 the arbitrary case include:
8638
8639 * `tribits_determine_if_current_package_needs_rebuilt()`_: Uses brute-force
8640 searches for recently modified files in upstream packages to determine if
8641 the external piece of software needs to be rebuilt.
8642
8643 * `tribits_write_flexible_package_client_export_files()`_: Write an export
8644 makefile or a ``XXXConfig.cmake`` file for usage by the wrapped externally
8645 configured and built software.
8646
8647 .. ToDo: Show how this is done in detail once TriBITS has nice support for
8648 .. external software and the WrapExternal example looks cleaner.
8649
8650 While it is possible to wrap nearly any externally configured and built piece
8651 of software as a TriBITS package, in most cases, it is usually better to just
8652 create a TriBITS build system for the software. For projects that use a raw
8653 CMake build system, a TriBITS build system can be created side-by-side with
8654 the existing raw CMake build using a number of approaches. The common approach
8655 that is not too invasive is to create a ``CMakeLists.tribits.txt`` file along
8656 side every native ``CMakeLists.txt`` file in the external software project and
8657 have the native ``CMakeLists.txt`` file defined like::
8658
8659 if (DOING_A_TRIBITS_BUILD)
8660 include("${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.tribits.txt")
8661 return()
8662 endif()
8663
8664 # Rest of native CMakeLists.txt file ...
8665
8666 Experience from the CASL VERA project has shown that, overall, there is less
8667 hassle, less work, and better portability when creating a native TriBITS
8668 build, even if it is a secondary build system for a given piece of software.
8669
8670 The decision whether to just wrap the build system for the existing software
8671 or to create a (secondary) TriBITS build system for it depends on a number of
8672 factors.
8673
8674 Note that while it is generally recommended to create a TriBITS build for an
8675 existing piece of software, it is generally not recommended with switch over
8676 all of the tests to use CTest (unless the existing software is going to ditch
8677 their current test driver and reporting system). Instead, the external
8678 software's native test system can just be called by the wrapped TriBITS
8679 package in one or more CTest tests.
8680
8681
8682 TriBITS directory snapshotting
8683 ------------------------------
8684
8685 Some TriBITS projects choose to snapshot the `TriBITS/tribits/`_ directory
8686 source tree into their project's source tree, typically under
8687 `<projectDir>/cmake/tribits/`_. The independent ``TriBITS/tribts/`` source
8688 tree contains the tool ``snapshot_tribits.py`` (calls `snapshot-dir.py`_) that
8689 allows one to update the snapshot of the TriBITS source tree as simply as::
8690
8691 $ cd <projectDir>/cmake/tribits/
8692 $ <some-base-dir>/TriBITS/tribits/snapshot_tribits.py
8693
8694 This will create a git commit in the local ``<projectDir>/`` git repo that
8695 looks like::
8696
8697 Automatic snapshot commit from tribits at f8c1682
8698
8699 Origin repo remote tracking branch: 'casl-dev-collab/tribits_reorg_26'
8700 Origin repo remote repo URL: 'casl-dev-collab = git@casl-dev:collaboration/TriBITS'
8701
8702 At commit:
8703
8704 f8c1682 Assert TriBITS min CMake version in TriBITS itself
8705 Author: Roscoe A. Bartlett <[email protected]>
8706 Date: Fri Dec 5 05:40:49 2014 -0500
8707
8708 This, of course, assumes that ``<projectDir>/`` is a local git repo (or is in
8709 local git repo). If that is not the case, then one cannot use the script
8710 ``snapshot_tribits.py`` or must use it with the ``--skip-commit`` option.
8711
8712 See `snapshot-dir.py --help`_ for more details. Note the guidance on using a
8713 different branch for the snapshot sync followed by a merge. This allows for
8714 one to maintain local changes to TriBITS and use git to manage the merges.
8715 However, this will increase the changes of merge conflicts so one should
8716 consider just directly snapshotting into the master branch to avoid merge
8717 conflicts.
8718
8719
8720 TriBITS Development Toolset
8721 ---------------------------
8722
8723 Most TriBITS projects need git, a compiler (e.g. GCC), MPI, and a number of
8724 other standard TPLs and other tools in order to develop on and test the
8725 project code. To this end, TriBITS contains some helper scripts for
8726 downloading, configuring, building, and installing packages like git, cmake,
8727 GCC, MPICH, and others needed to set up a development environment for a
8728 typical computational science software project. These tools are used to set
8729 up development environments on new machines for projects like Trilinos and
8730 CASL VERA. Scripts with names like ``install-gcc.py`` are defined which pull
8731 sources from public git repos then configure, build, and install into
8732 specified installation directories.
8733
8734 .. _install_devtools.py:
8735
8736 The script **install_devtools.py** is provided in the directory::
8737
8738 tribits/devtools_install/
8739
8740 To use this script, one just needs to create some scratch directory like::
8741
8742 $ mkdir scratch
8743 $ cd scratch/
8744
8745 then install the tools using, for example::
8746
8747 $ install_devtools.py --install-dir=~/install/tribits_devtools \
8748 --parallel=16 --do-all
8749
8750 Then to access installed development environment, one just needs to source the
8751 script::
8752
8753 ~/install/tribits_devtools/load_dev_env.sh
8754
8755 and then the installed versions of GCC, MPICH, CMake, and `gitdist`_ are
8756 placed in one's path.
8757
8758 See `install_devtools.py --help`_ for more details.
8759
8760
8761 .. Words specific to this documentation collection:
8762
8763 .. LocalWords: projectDir packageDir
8764
8765 .. TriBITS words:
8766
8767 .. LocalWords: TRIBITS TriBITS tribits TPL TPLs
8768 .. LocalWords: Subpackages subpackages Subpackage subpackage
8769 .. LocalWords: PackagesList TPLsList
8770 .. LocalWords: NativeRepositoriesList ExtraRepositoriesList
8771 .. LocalWords: TribitsCTestDriverCore
8772 .. LocalWords: TribitsExampleProject TribitsExProj DTribitsExProj SimpleCXX MixedLang
8773 .. LocalWords: MockTrilinos
8774 .. LocalWords: WithSubpackages WithSubpackagesA WithSubpackagesB WithSubpackagesC
8775
8776 .. General CMake words:
8777
8778 .. LocalWords: CMAKE CMake cmake CTEST CTest ctest CDash
8779 .. LocalWords: CMakeLists CMakeCache CTestConfig
8780 .. LocalWords: Kitware
8781 .. LocalWords: endif foreach endforeach endmacro subdirectory
8782 .. LocalWords: Fortran
8783
8784
8785 .. Other general words:
8786
8787 .. LocalWords: Trilinos executables Versioning versioning
8788 .. LocalWords: Namespaced namespaced symlinks initializations