| 1 |
====================================== |
| 2 |
Submitting a TribitsDriver dashboard: |
| 3 |
====================================== |
| 4 |
|
| 5 |
|
| 6 |
A) Prerequisites: |
| 7 |
================= |
| 8 |
|
| 9 |
First, manually ensure that the following prerequisites are satisfied: |
| 10 |
|
| 11 |
(*) python version 2.6 or later |
| 12 |
|
| 13 |
(*) eg or git version 1.6.x or later |
| 14 |
|
| 15 |
|
| 16 |
B) Create a base directory: |
| 17 |
=========================== |
| 18 |
|
| 19 |
Create a directory as a base and clone the sources |
| 20 |
|
| 21 |
$ cd <somebasedir> |
| 22 |
$ mkdir Dashboards |
| 23 |
$ cd Dashboards |
| 24 |
$ git clone something.blah.gov/<PROJECT> |
| 25 |
# Also, clone any other sources as well needed. |
| 26 |
|
| 27 |
|
| 28 |
C) Create machine specific content: |
| 29 |
=================================== |
| 30 |
|
| 31 |
Add a subdirectory (optionally) named after your machine in |
| 32 |
<PROJECT>/cmake/ctest/drivers. If you name the directory exactly as your |
| 33 |
machine is named by `uname -n` then you simply need to add some files to that |
| 34 |
directory. You can drop ".domain.gov" stuff from the end of the machine name, |
| 35 |
as long as the name does not conflict with any other directory names already |
| 36 |
in the drivers directory. |
| 37 |
|
| 38 |
Add some ctest -S scripts to run as tests. You can base the *.cmake |
| 39 |
files in this directory on those used on an existing machine that is |
| 40 |
substantially similar to yours. If there isn't one, just copy the |
| 41 |
files from the s903186 directory and work from there. |
| 42 |
|
| 43 |
Add a CMakeLists.txt file to the new directory for your machine. It |
| 44 |
should contain content something like this (from the flagship example, |
| 45 |
s903186): |
| 46 |
|
| 47 |
# Initially: |
| 48 |
# |
| 49 |
tribits_driver_setup() |
| 50 |
|
| 51 |
# Make one of these calls for each script |
| 52 |
# |
| 53 |
tribits_driver_add_dashboard( |
| 54 |
MPI_OPT |
| 55 |
ctest_mac_nightly_mpi_opt_s903186.cmake |
| 56 |
) |
| 57 |
|
| 58 |
tribits_driver_add_dashboard( |
| 59 |
SERIAL_DEBUG |
| 60 |
ctest_mac_nightly_serial_debug_s903186.cmake |
| 61 |
) |
| 62 |
|
| 63 |
# Finally: |
| 64 |
# |
| 65 |
tribits_add_required_cmake_installs() |
| 66 |
|
| 67 |
The first argument to TRIBITS_DRIVER_ADD_DASHBOARD calls is the test |
| 68 |
name as it will appear on the <Project>Driver dashboard. The second |
| 69 |
argument is the name of the script in the same directory that will be |
| 70 |
run as a ctest -S script as a test on the driver dashboard. |
| 71 |
|
| 72 |
If you have a dashboard that must run all by itself, without any other |
| 73 |
dashboards being run in parallel, then you should pass the RUN_SERIAL |
| 74 |
argument. |
| 75 |
|
| 76 |
For more details on other optional arguments, see the comments at the |
| 77 |
function definition for TRIBITS_DRIVER_ADD_DASHBOARD in |
| 78 |
TribitsDriverSupport.cmake. |
| 79 |
|
| 80 |
WARNING: The name of the build (the first argument, e.g. MPI_OPT) |
| 81 |
should be the same as the name of the build set in the *.cmake driver |
| 82 |
file (the second argument, |
| 83 |
e.g. ctest_mac_nightly_mpi_opt_s903186.cmake). There are no checks in |
| 84 |
place to make sure that this is true. You have been warned! |
| 85 |
|
| 86 |
Next, add a cron_driver.sh for your machine. (Or cron_driver.bat on |
| 87 |
Windows.) |
| 88 |
|
| 89 |
You can set the environment variable TDD_PARALLEL_LEVEL to control how |
| 90 |
many dashboards are run in parallel (as tests) by the <Project>Driver |
| 91 |
dashboard. |
| 92 |
|
| 93 |
By default, the TDD system will download the version of CMake/CTest using the |
| 94 |
download-cmake.py script. This allows one to use specific versions of |
| 95 |
CMake/CTest for the build and also to have the versions upgraded automatically |
| 96 |
according to how download-cmake.py is set up (see download-cmake.py --help for |
| 97 |
details). However, if TRIBITS_TDD_USE_SYSTEM_CTEST=1 is set in the env, then |
| 98 |
CTest/CMake as found in the default env at the time the TDD driver |
| 99 |
tdd_driver.py is run will be used instead of downloading and installing |
| 100 |
CMake/CTest. |
| 101 |
|
| 102 |
If TRIBITS_TDD_USE_SYSTEM_CTEST != 1, then the download of CMake/CTest will |
| 103 |
occur. In this case, you may also need to set the environment variable |
| 104 |
TDD_HTTP_PROXY for your http proxy which is needed if the script |
| 105 |
download-cmake.py can't figure it out the proxy automatically. For Sandia |
| 106 |
machines, this is http://wwwproxy.sandia.gov:80/. |
| 107 |
|
| 108 |
Typical cron_driver.sh contents: |
| 109 |
|
| 110 |
export TDD_PARALLEL_LEVEL=2 |
| 111 |
export TDD_HTTP_PROXY="http://wwwproxy.sandia.gov:80/" |
| 112 |
../cron_driver.py |
| 113 |
|
| 114 |
Typical cron_driver.bat contents: |
| 115 |
|
| 116 |
set TDD_PARALLEL_LEVEL=2 |
| 117 |
set TDD_HTTP_PROXY="http://wwwproxy.sandia.gov:80/" |
| 118 |
python "%~dp0..\cron_driver.py" |
| 119 |
|
| 120 |
|
| 121 |
D) Run the scripts manually to test them: |
| 122 |
========================================= |
| 123 |
|
| 124 |
Before setting up the final cron job, you should test the individual |
| 125 |
CTest driver scripts and the outer <Project>Driver script. |
| 126 |
|
| 127 |
|
| 128 |
D.1) Test out individual CTest dashboard builds first: |
| 129 |
------------------------------------------------------ |
| 130 |
|
| 131 |
Before running your outer cron driver script, you would be wise to first test |
| 132 |
out your individual CTest driver scripts (e.g. MPI_OPT, SERIAL_DEBUG, etc.). |
| 133 |
You can test individual basic ctest driver scripts to clone the project and |
| 134 |
create the BUILD directory by creating your own temp base directory and then |
| 135 |
overriding CTEST_DASHBOARD_ROOT for example as: |
| 136 |
|
| 137 |
$ mkdir MOCK_MPI_OPT |
| 138 |
$ cd MOCK_MPI_OPT |
| 139 |
$ env CTEST_DASHBOARD_ROOT=$PWD \ |
| 140 |
<Project>_PACKAGES=Teuchos \ |
| 141 |
<Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \ |
| 142 |
CTEST_TEST_TYPE=Experimental \ |
| 143 |
CTEST_DO_SUBMIT=OFF \ |
| 144 |
CTEST_DO_UPDATES=TRUE \ |
| 145 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \ |
| 146 |
CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE \ |
| 147 |
CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE \ |
| 148 |
ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake |
| 149 |
|
| 150 |
Running the above will clone <Project> into MOCK_MPI_OPT/<Project> and will |
| 151 |
create the build directory in MOCK_MPI_OPT/BUILD. After the initial clone is |
| 152 |
performed above, you can speed up subsequent testing by additionally setting |
| 153 |
CTEST_DO_UPDATES=FALSE and CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE on |
| 154 |
the environment. You can keep running this 'env ... ctest -S ...' command |
| 155 |
until you get everything working. All of the info needed to debug problems |
| 156 |
should be present in the output from 'ctest -VV -S' (which you should pipe to |
| 157 |
a file and then examine). The option |
| 158 |
CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE is set to avoid the generation |
| 159 |
of the file CDashSubprojectDependencies.xml. There is no reason to generate |
| 160 |
or submit that file when you are just doing local testing. |
| 161 |
|
| 162 |
If you want to test the exact state of the local changes in <somedir>, you can |
| 163 |
avoid clonging new repos or pulling commits by creating a symbolic link and |
| 164 |
running as: |
| 165 |
|
| 166 |
$ mkdir MOCK_MPI_OPT |
| 167 |
$ cd MOCK_MPI_OPT |
| 168 |
$ ln -s <somedir> <Project> |
| 169 |
$ env CTEST_DASHBOARD_ROOT=$PWD \ |
| 170 |
<Project>_PACKAGES=Teuchos \ |
| 171 |
<Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \ |
| 172 |
<Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=ON \ |
| 173 |
CTEST_TEST_TYPE=Experimental \ |
| 174 |
CTEST_DO_SUBMIT=OFF \ |
| 175 |
CTEST_DO_UPDATES=OFF \ |
| 176 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \ |
| 177 |
CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE \ |
| 178 |
CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE \ |
| 179 |
ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake |
| 180 |
|
| 181 |
Note that above it is critical that you set CTEST_DO_UPDATES=OFF so that it |
| 182 |
will not try to reset your repos and wipe out your changes or have merge |
| 183 |
conflicts for extra repos. The option |
| 184 |
<Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=ON allows you to be missing some |
| 185 |
extra repos so that you can test with just the extra repos that you have |
| 186 |
cloned. Note that this will not test the exact state of the project that |
| 187 |
would be in automated testing. If you want to assert that the exact list of |
| 188 |
repos is cloned already, then set |
| 189 |
<Project>_IGNORE_MISSING_EXTRA_REPOSITORIES=OFF or leave off that argument. |
| 190 |
|
| 191 |
After the local CTest processes are working, just to make sure the |
| 192 |
submit works, you can run a final trial build with: |
| 193 |
|
| 194 |
$ env CTEST_DASHBOARD_ROOT=$PWD <Project>_PACKAGES=Teuchos \ |
| 195 |
CTEST_TEST_TYPE=Experimental CTEST_DO_UPDATES=FALSE \ |
| 196 |
<Project>_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Nightly \ |
| 197 |
CTEST_DO_SUBMIT=ON CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \ |
| 198 |
ctest -V -S <somedir>/cmake/ctest/drivers/s903186/ctest_mac_nightly_mpi_opt_s903186.cmake |
| 199 |
|
| 200 |
If this submits to the 'Experimental' CDash dashboard, you should be |
| 201 |
in good shape for the nightly test. If you can't find you |
| 202 |
experimental build on the CDash dashboard, then something is wrong. |
| 203 |
|
| 204 |
NOTES: |
| 205 |
|
| 206 |
a) Different Trilinos git repos: When CTEST_TEST_TYPE=Experimental or |
| 207 |
CTEST_TEST_TYPE=Continuous then Trilinos is initially cloned from form the |
| 208 |
main development repo. However, when CTEST_TEST_TYPE=Nightly then Trilinos |
| 209 |
will be initially cloned from a specially maintained 'nightly' repo. (The |
| 210 |
purpose of the 'nightly' repo is to ensure that all Nightly builds use exactly |
| 211 |
the same version of the sources and report consistent results in case someone |
| 212 |
pushes in between the first Nightly pull and later Nightly pulls.) Therefore, |
| 213 |
if you use CTEST_TEST_TYPE=Experimental when first testing in the actually |
| 214 |
final directory location for the Nightly dashboard (such as will occur when |
| 215 |
testing the cron_driver.sh script described below) then subsequent pulls will |
| 216 |
occur from the main development repo *NOT* the special 'nightly' repo. To get |
| 217 |
back to the 'nightly' repo, you must delete the locally cloned Trilinos repo |
| 218 |
so that when the script runs with CTEST_TEST_TYPE=Nightly, it will clone from |
| 219 |
the 'nightly' repo and do subsequent updates form the 'nightly' repo. If you |
| 220 |
want to totally override which repo gets pulled form, you can set that with |
| 221 |
the variable <Project>_REPOSITORY_LOCATION. |
| 222 |
|
| 223 |
|
| 224 |
D.2) Testing the outer cron driver script: |
| 225 |
------------------------------------------ |
| 226 |
|
| 227 |
To test this script manually, you must move into the driver scripts directory |
| 228 |
and run from there: |
| 229 |
|
| 230 |
$ cd <DRIVERS_BASE_DIR> |
| 231 |
$ env \ |
| 232 |
<PROJECT_NAME>_PACKAGES=Teuchos \ |
| 233 |
TDD_IN_TESTING_MODE=ON \ |
| 234 |
TDD_CTEST_TEST_TYPE=Experimental \ |
| 235 |
CTEST_TEST_TYPE=Experimental \ |
| 236 |
CTEST_DO_UPDATES=TRUE \ |
| 237 |
TDD_DO_SUBMIT=FALSE \ |
| 238 |
CTEST_DO_SUBMIT=FALSE \ |
| 239 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \ |
| 240 |
./cron_driver_somemachine.sh |
| 241 |
|
| 242 |
Once you are done with the initial run, you can test it out |
| 243 |
interactively in a rapid, iterative process by setting up the |
| 244 |
following additionial environment variables: |
| 245 |
|
| 246 |
$ cd <DRIVERS_BASE_DIR> |
| 247 |
$ env \ |
| 248 |
<Project>_PACKAGES=Teuchos \ |
| 249 |
CTEST_DO_UPDATES=FALSE \ |
| 250 |
CTEST_DO_SUBMIT=FALSE \ |
| 251 |
TDD_DO_SUBMIT=FALSE \ |
| 252 |
TDD_IN_TESTING_MODE=ON \ |
| 253 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \ |
| 254 |
TDD_CTEST_TEST_TYPE=Experimental \ |
| 255 |
CTEST_TEST_TYPE=Experimental \ |
| 256 |
TDD_FORCE_CMAKE_INSTALL=0 \ |
| 257 |
TDD_FORCE_INNER_CMAKE_INSTALL=0 \ |
| 258 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \ |
| 259 |
./cron_driver_somemachine.sh |
| 260 |
|
| 261 |
(of course you must use set commands on Windows). |
| 262 |
|
| 263 |
NOTES: |
| 264 |
|
| 265 |
(*) Set <Project>_PACKAGES=Teuchos to whatever your project name is and |
| 266 |
some upstream package. For Trilinos, this would be something like |
| 267 |
Trilinos_PACKAGES=Teuchos. For another project you would use its |
| 268 |
project name and one of its upstream packages. |
| 269 |
(*) Set TDD_FORCE_CMAKE_INSTALL=0 to skip global CMake reinstall. |
| 270 |
(*) Set TDD_FORCE_INNER_CMAKE_INSTALL=0 to skip tests that install |
| 271 |
the inner CMake/CTest used by individual builds. |
| 272 |
(*) Set CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE after a first round of |
| 273 |
builds to speed up subsequent builds. |
| 274 |
|
| 275 |
Work out any kinks you may have in your environment and your scripts |
| 276 |
by running this over and over again. |
| 277 |
|
| 278 |
Once you are confident that your driver script is working correctly, |
| 279 |
you should try to run this as a cron job using the command (see the |
| 280 |
entire command below in the "cron driver" section): |
| 281 |
|
| 282 |
$ env <Project>_PACKAGES=Teuchos CTEST_DO_SUBMIT=ON TDD_DO_SUBMIT=ON \ |
| 283 |
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=FALSE \ |
| 284 |
CTEST_TEST_TYPE=Experimental \ |
| 285 |
TDD_FORCE_CMAKE_INSTALL=1 # If you want to force outer CMake upgrade \ |
| 286 |
<yourdirverssroucedir>/cron_driver_somemachine.sh |
| 287 |
|
| 288 |
Note the submits variables are turned on which means that the inner and outer |
| 289 |
submits will be performed to the <Project> and <Project>Driver dashboards, |
| 290 |
respectively. If this submits to the CDash dashboards, you should be ready to |
| 291 |
set up your final cron job driver (see below). |
| 292 |
|
| 293 |
WARNING: Blow away all of the base dashboard directories that get created |
| 294 |
during this process before running the final cron job! You must do this so |
| 295 |
that the correct git nightly repository will get cloned in the inner CTest |
| 296 |
dashboards. You would do this with: |
| 297 |
|
| 298 |
$ cd ~/T |
| 299 |
$ rm -rf MPI_DEBUG SERIAL_RELEASE ... |
| 300 |
|
| 301 |
|
| 302 |
E) Debugging problems: |
| 303 |
====================== |
| 304 |
|
| 305 |
Debugging the inner CTest dashboards should be more straightforward as |
| 306 |
described in D.1 above. |
| 307 |
|
| 308 |
When first getting the outer <Project>Driver script to run, there can be |
| 309 |
problems that you will have to debug. The output to the console from the |
| 310 |
cron_driver.py script is very sparse but will be of some help. Almost all of |
| 311 |
the output from the various stages are contained under the base build/test |
| 312 |
directory. The base build directory for the outer CTest <Project>Driver |
| 313 |
script is in: |
| 314 |
|
| 315 |
~/T/TDD_BUILD |
| 316 |
|
| 317 |
You can find the output from the various configure, build, tests in |
| 318 |
the date-specific subdirectory under: |
| 319 |
|
| 320 |
~/T/TDD_BUILD/Testing/Temporary/ |
| 321 |
|
| 322 |
Note that you will also find the output from the |
| 323 |
TribitsCTestDriverCore.cmake script as the test output in: |
| 324 |
|
| 325 |
T/TDD_BUILD/Testing/Temporary/LastTest_XXX.log |
| 326 |
|
| 327 |
Also, of course all of the output for each build case (BUILD_CASE) is |
| 328 |
in: |
| 329 |
|
| 330 |
T/BUILD_CASE/Testing/Temporary/ |
| 331 |
|
| 332 |
where BUILD_CASE = MPI_OPT, SERIAL_DEBUG etc. |
| 333 |
|
| 334 |
Good luck in debugging! |
| 335 |
|
| 336 |
|
| 337 |
F) Set up a crontab entry (or a scheduled task on Windows): |
| 338 |
=========================================================== |
| 339 |
|
| 340 |
After you have successfully manually testing the inner and outer |
| 341 |
driver scripts, you will be ready to create the final crontab entry. |
| 342 |
You can set this with 'crontab -e' or install the cron driver script |
| 343 |
in whole. |
| 344 |
|
| 345 |
The contents of your cron driver should look like: |
| 346 |
|
| 347 |
[email protected] |
| 348 |
10 00 * * * cd ~/T/<Project>/cmake/ctest/drivers/MACHINE_NAME && ./cron_driver.sh &> ~/T/cron_driver.log |
| 349 |
|
| 350 |
Typical values are MM=10 and HH=00 which starts things off at 10 minutes after |
| 351 |
midnight. The testing day time may move earlier (e.g. 8 PM MST) but it will |
| 352 |
not start after Midnight so that value is safe. You can start your cron |
| 353 |
driver at any time after the beginning of a new testing day (as defined by |
| 354 |
CTest/CDash configuration). |
| 355 |
|
| 356 |
|
| 357 |
G) Commit changes to the driver scripts: |
| 358 |
======================================== |
| 359 |
|
| 360 |
You must commit and push changes in <Project>/cmake/ctest/drivers into the |
| 361 |
main Trilinos git repository. To do this, do: |
| 362 |
|
| 363 |
$ cd ~/T/<Project> |
| 364 |
$ eg remote add trilinos software:/space/git/Trilinos |
| 365 |
$ eg pull trilinos |
| 366 |
$ eg commit |
| 367 |
$ eg rebase --against trilinos/origin |
| 368 |
$ eg push |
| 369 |
|
| 370 |
Hopefully, that should push your changes back and you should be ready |
| 371 |
to let the cron job run on its own every day. If that does not work, |
| 372 |
cherry-pick your commits into a different local Trilinos repo cloned |
| 373 |
directly from software:/space/git/Trilinos and do the push from there |
| 374 |
(this is what I have had to do in the past). Do this as: |
| 375 |
|
| 376 |
$ cd $SOME_OTHER_BASE/Trilinos |
| 377 |
$ eg remote add nighly_cdash ~/T/Trilinos |
| 378 |
$ eg pull nighly_cdash master |
| 379 |
$ eg pull |
| 380 |
$ eg rebase --against origin/master |
| 381 |
$ eg push |
| 382 |
|
| 383 |
Of course, if you change anything other than what is in your |
| 384 |
driver/MACHINE_NAME directory, you should use the checkin-test.py |
| 385 |
script to test and push your commits. |