Debian Codegraph @

cmake/tribits/ctest_driver/README

raw
1 ======================================
2 Submitting a TribitsDriver dashboard:
3 ======================================
4
5
6 A) Prerequisites:
7 =================
8
9 First, manually ensure that the following prerequisites are satisfied:
10
11 (*) python version 2.6 or later
12
13 (*) eg or git version 1.6.x or later
14
15
16 B) Create a base directory:
17 ===========================
18
19 Create a directory as a base and clone the sources
20
21 $ cd <somebasedir>
22 $ mkdir Dashboards
23 $ cd Dashboards
24 $ git clone something.blah.gov/<PROJECT>
25 # Also, clone any other sources as well needed.
26
27
28 C) Create machine specific content:
29 ===================================
30
31 Add a subdirectory (optionally) named after your machine in
32 <PROJECT>/cmake/ctest/drivers. If you name the directory exactly as your
33 machine is named by `uname -n` then you simply need to add some files to that
34 directory. You can drop ".domain.gov" stuff from the end of the machine name,
35 as long as the name does not conflict with any other directory names already
36 in the drivers directory.
37
38 Add some ctest -S scripts to run as tests. You can base the *.cmake
39 files in this directory on those used on an existing machine that is
40 substantially similar to yours. If there isn't one, just copy the
41 files from the s903186 directory and work from there.
42
43 Add a CMakeLists.txt file to the new directory for your machine. It
44 should contain content something like this (from the flagship example,
45 s903186):
46
47 # Initially:
48 #
49 tribits_driver_setup()
50
51 # Make one of these calls for each script
52 #
53 tribits_driver_add_dashboard(
54 MPI_OPT
55 ctest_mac_nightly_mpi_opt_s903186.cmake
56 )
57
58 tribits_driver_add_dashboard(
59 SERIAL_DEBUG
60 ctest_mac_nightly_serial_debug_s903186.cmake
61 )
62
63 # Finally:
64 #
65 tribits_add_required_cmake_installs()
66
67 The first argument to TRIBITS_DRIVER_ADD_DASHBOARD calls is the test
68 name as it will appear on the <Project>Driver dashboard. The second
69 argument is the name of the script in the same directory that will be
70 run as a ctest -S script as a test on the driver dashboard.
71
72 If you have a dashboard that must run all by itself, without any other
73 dashboards being run in parallel, then you should pass the RUN_SERIAL
74 argument.
75
76 For more details on other optional arguments, see the comments at the
77 function definition for TRIBITS_DRIVER_ADD_DASHBOARD in
78 TribitsDriverSupport.cmake.
79
80 WARNING: The name of the build (the first argument, e.g. MPI_OPT)
81 should be the same as the name of the build set in the *.cmake driver
82 file (the second argument,
83 e.g. ctest_mac_nightly_mpi_opt_s903186.cmake). There are no checks in
84 place to make sure that this is true. You have been warned!
85
86 Next, add a cron_driver.sh for your machine. (Or cron_driver.bat on
87 Windows.)
88
89 You can set the environment variable TDD_PARALLEL_LEVEL to control how
90 many dashboards are run in parallel (as tests) by the <Project>Driver
91 dashboard.
92
93 By default, the TDD system will download the version of CMake/CTest using the
94 download-cmake.py script. This allows one to use specific versions of
95 CMake/CTest for the build and also to have the versions upgraded automatically
96 according to how download-cmake.py is set up (see download-cmake.py --help for
97 details). However, if TRIBITS_TDD_USE_SYSTEM_CTEST=1 is set in the env, then
98 CTest/CMake as found in the default env at the time the TDD driver
99 tdd_driver.py is run will be used instead of downloading and installing
100 CMake/CTest.
101
102 If TRIBITS_TDD_USE_SYSTEM_CTEST != 1, then the download of CMake/CTest will
103 occur. In this case, you may also need to set the environment variable
104 TDD_HTTP_PROXY for your http proxy which is needed if the script
105 download-cmake.py can't figure it out the proxy automatically. For Sandia
106 machines, this is http://wwwproxy.sandia.gov:80/.
107
108 Typical cron_driver.sh contents:
109
110 export TDD_PARALLEL_LEVEL=2
111 export TDD_HTTP_PROXY="http://wwwproxy.sandia.gov:80/"
112 ../cron_driver.py
113
114 Typical cron_driver.bat contents:
115
116 set TDD_PARALLEL_LEVEL=2
117 set TDD_HTTP_PROXY="http://wwwproxy.sandia.gov:80/"
118 python "%~dp0..\cron_driver.py"
119
120
121 D) Run the scripts manually to test them:
122 =========================================
123
124 Before setting up the final cron job, you should test the individual
125 CTest driver scripts and the outer <Project>Driver script.
126
127
128 D.1) Test out individual CTest dashboard builds first:
129 ------------------------------------------------------
130
131 Before running your outer cron driver script, you would be wise to first test
132 out your individual CTest driver scripts (e.g. MPI_OPT, SERIAL_DEBUG, etc.).
133 You can test individual basic ctest driver scripts to clone the project and
134 create the BUILD directory by creating your own temp base directory and then
135 overriding CTEST_DASHBOARD_ROOT for example as:
136
137 $ mkdir MOCK_MPI_OPT
138 $ cd MOCK_MPI_OPT
139 $ env CTEST_DASHBOARD_ROOT=$PWD \
140 <Project>_PACKAGES=Teuchos \
141 <Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \
142 CTEST_TEST_TYPE=Experimental \
143 CTEST_DO_SUBMIT=OFF \
144 CTEST_DO_UPDATES=TRUE \
145 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \
146 CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE \
147 CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE \
148 ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake
149
150 Running the above will clone <Project> into MOCK_MPI_OPT/<Project> and will
151 create the build directory in MOCK_MPI_OPT/BUILD. After the initial clone is
152 performed above, you can speed up subsequent testing by additionally setting
153 CTEST_DO_UPDATES=FALSE and CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE on
154 the environment. You can keep running this 'env ... ctest -S ...' command
155 until you get everything working. All of the info needed to debug problems
156 should be present in the output from 'ctest -VV -S' (which you should pipe to
157 a file and then examine). The option
158 CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE is set to avoid the generation
159 of the file CDashSubprojectDependencies.xml. There is no reason to generate
160 or submit that file when you are just doing local testing.
161
162 If you want to test the exact state of the local changes in <somedir>, you can
163 avoid clonging new repos or pulling commits by creating a symbolic link and
164 running as:
165
166 $ mkdir MOCK_MPI_OPT
167 $ cd MOCK_MPI_OPT
168 $ ln -s <somedir> <Project>
169 $ env CTEST_DASHBOARD_ROOT=$PWD \
170 <Project>_PACKAGES=Teuchos \
171 <Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \
172 <Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=ON \
173 CTEST_TEST_TYPE=Experimental \
174 CTEST_DO_SUBMIT=OFF \
175 CTEST_DO_UPDATES=OFF \
176 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \
177 CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE \
178 CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE \
179 ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake
180
181 Note that above it is critical that you set CTEST_DO_UPDATES=OFF so that it
182 will not try to reset your repos and wipe out your changes or have merge
183 conflicts for extra repos. The option
184 <Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=ON allows you to be missing some
185 extra repos so that you can test with just the extra repos that you have
186 cloned. Note that this will not test the exact state of the project that
187 would be in automated testing. If you want to assert that the exact list of
188 repos is cloned already, then set
189 <Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=OFF or leave off that argument.
190
191 After the local CTest processes are working, just to make sure the
192 submit works, you can run a final trial build with:
193
194 $ env CTEST_DASHBOARD_ROOT=$PWD <Project>_PACKAGES=Teuchos \
195 CTEST_TEST_TYPE=Experimental CTEST_DO_UPDATES=FALSE \
196 <Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \
197 CTEST_DO_SUBMIT=ON CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \
198 ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake
199
200 If this submits to the 'Experimental' CDash dashboard, you should be
201 in good shape for the nightly test. If you can't find you
202 experimental build on the CDash dashboard, then something is wrong.
203
204 NOTES:
205
206 a) Different Trilinos git repos: When CTEST_TEST_TYPE=Experimental or
207 CTEST_TEST_TYPE=Continuous then Trilinos is initially cloned from form the
208 main development repo. However, when CTEST_TEST_TYPE=Nightly then Trilinos
209 will be initially cloned from a specially maintained 'nightly' repo. (The
210 purpose of the 'nightly' repo is to ensure that all Nightly builds use exactly
211 the same version of the sources and report consistent results in case someone
212 pushes in between the first Nightly pull and later Nightly pulls.) Therefore,
213 if you use CTEST_TEST_TYPE=Experimental when first testing in the actually
214 final directory location for the Nightly dashboard (such as will occur when
215 testing the cron_driver.sh script described below) then subsequent pulls will
216 occur from the main development repo *NOT* the special 'nightly' repo. To get
217 back to the 'nightly' repo, you must delete the locally cloned Trilinos repo
218 so that when the script runs with CTEST_TEST_TYPE=Nightly, it will clone from
219 the 'nightly' repo and do subsequent updates form the 'nightly' repo. If you
220 want to totally override which repo gets pulled form, you can set that with
221 the variable <Project>_REPOSITORY_LOCATION.
222
223
224 D.2) Testing the outer cron driver script:
225 ------------------------------------------
226
227 To test this script manually, you must move into the driver scripts directory
228 and run from there:
229
230 $ cd <DRIVERS_BASE_DIR>
231 $ env \
232 <PROJECT_NAME>_PACKAGES=Teuchos \
233 TDD_IN_TESTING_MODE=ON \
234 TDD_CTEST_TEST_TYPE=Experimental \
235 CTEST_TEST_TYPE=Experimental \
236 CTEST_DO_UPDATES=TRUE \
237 TDD_DO_SUBMIT=FALSE \
238 CTEST_DO_SUBMIT=FALSE \
239 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \
240 ./cron_driver_somemachine.sh
241
242 Once you are done with the initial run, you can test it out
243 interactively in a rapid, iterative process by setting up the
244 following additionial environment variables:
245
246 $ cd <DRIVERS_BASE_DIR>
247 $ env \
248 <Project>_PACKAGES=Teuchos \
249 CTEST_DO_UPDATES=FALSE \
250 CTEST_DO_SUBMIT=FALSE \
251 TDD_DO_SUBMIT=FALSE \
252 TDD_IN_TESTING_MODE=ON \
253 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \
254 TDD_CTEST_TEST_TYPE=Experimental \
255 CTEST_TEST_TYPE=Experimental \
256 TDD_FORCE_CMAKE_INSTALL=0 \
257 TDD_FORCE_INNER_CMAKE_INSTALL=0 \
258 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \
259 ./cron_driver_somemachine.sh
260
261 (of course you must use set commands on Windows).
262
263 NOTES:
264
265 (*) Set <Project>_PACKAGES=Teuchos to whatever your project name is and
266 some upstream package. For Trilinos, this would be something like
267 Trilinos_PACKAGES=Teuchos. For another project you would use its
268 project name and one of its upstream packages.
269 (*) Set TDD_FORCE_CMAKE_INSTALL=0 to skip global CMake reinstall.
270 (*) Set TDD_FORCE_INNER_CMAKE_INSTALL=0 to skip tests that install
271 the inner CMake/CTest used by individual builds.
272 (*) Set CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE after a first round of
273 builds to speed up subsequent builds.
274
275 Work out any kinks you may have in your environment and your scripts
276 by running this over and over again.
277
278 Once you are confident that your driver script is working correctly,
279 you should try to run this as a cron job using the command (see the
280 entire command below in the "cron driver" section):
281
282 $ env <Project>_PACKAGES=Teuchos CTEST_DO_SUBMIT=ON TDD_DO_SUBMIT=ON \
283 CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \
284 CTEST_TEST_TYPE=Experimental \
285 TDD_FORCE_CMAKE_INSTALL=1 # If you want to force outer CMake upgrade \
286 <yourdirverssroucedir>/cron_driver_somemachine.sh
287
288 Note the submits variables are turned on which means that the inner and outer
289 submits will be performed to the <Project> and <Project>Driver dashboards,
290 respectively. If this submits to the CDash dashboards, you should be ready to
291 set up your final cron job driver (see below).
292
293 WARNING: Blow away all of the base dashboard directories that get created
294 during this process before running the final cron job! You must do this so
295 that the correct git nightly repository will get cloned in the inner CTest
296 dashboards. You would do this with:
297
298 $ cd ~/T
299 $ rm -rf MPI_DEBUG SERIAL_RELEASE ...
300
301
302 E) Debugging problems:
303 ======================
304
305 Debugging the inner CTest dashboards should be more straightforward as
306 described in D.1 above.
307
308 When first getting the outer <Project>Driver script to run, there can be
309 problems that you will have to debug. The output to the console from the
310 cron_driver.py script is very sparse but will be of some help. Almost all of
311 the output from the various stages are contained under the base build/test
312 directory. The base build directory for the outer CTest <Project>Driver
313 script is in:
314
315 ~/T/TDD_BUILD
316
317 You can find the output from the various configure, build, tests in
318 the date-specific subdirectory under:
319
320 ~/T/TDD_BUILD/Testing/Temporary/
321
322 Note that you will also find the output from the
323 TribitsCTestDriverCore.cmake script as the test output in:
324
325 T/TDD_BUILD/Testing/Temporary/LastTest_XXX.log
326
327 Also, of course all of the output for each build case (BUILD_CASE) is
328 in:
329
330 T/BUILD_CASE/Testing/Temporary/
331
332 where BUILD_CASE = MPI_OPT, SERIAL_DEBUG etc.
333
334 Good luck in debugging!
335
336
337 F) Set up a crontab entry (or a scheduled task on Windows):
338 ===========================================================
339
340 After you have successfully manually testing the inner and outer
341 driver scripts, you will be ready to create the final crontab entry.
342 You can set this with 'crontab -e' or install the cron driver script
343 in whole.
344
345 The contents of your cron driver should look like:
346
347 [email protected]
348 10 00 * * * cd ~/T/<Project>/cmake/ctest/drivers/MACHINE_NAME && ./cron_driver.sh &> ~/T/cron_driver.log
349
350 Typical values are MM=10 and HH=00 which starts things off at 10 minutes after
351 midnight. The testing day time may move earlier (e.g. 8 PM MST) but it will
352 not start after Midnight so that value is safe. You can start your cron
353 driver at any time after the beginning of a new testing day (as defined by
354 CTest/CDash configuration).
355
356
357 G) Commit changes to the driver scripts:
358 ========================================
359
360 You must commit and push changes in <Project>/cmake/ctest/drivers into the
361 main Trilinos git repository. To do this, do:
362
363 $ cd ~/T/<Project>
364 $ eg remote add trilinos software:/space/git/Trilinos
365 $ eg pull trilinos
366 $ eg commit
367 $ eg rebase --against trilinos/origin
368 $ eg push
369
370 Hopefully, that should push your changes back and you should be ready
371 to let the cron job run on its own every day. If that does not work,
372 cherry-pick your commits into a different local Trilinos repo cloned
373 directly from software:/space/git/Trilinos and do the push from there
374 (this is what I have had to do in the past). Do this as:
375
376 $ cd $SOME_OTHER_BASE/Trilinos
377 $ eg remote add nighly_cdash ~/T/Trilinos
378 $ eg pull nighly_cdash master
379 $ eg pull
380 $ eg rebase --against origin/master
381 $ eg push
382
383 Of course, if you change anything other than what is in your
384 driver/MACHINE_NAME directory, you should use the checkin-test.py
385 script to test and push your commits.