| 1 |
file: florist/tests/README [$Revision$] |
| 2 |
|
| 3 |
THIS SOFTWARE AND THE ACCOMPANYING DOCUMENTATION ARE DISTRIBUTED |
| 4 |
IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY; |
| 5 |
WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS |
| 6 |
FOR A PARTICULAR PURPOSE -- NOTWITHSTANDING ANY STATEMENTS MADE |
| 7 |
BELOW IN THIS FILE. |
| 8 |
|
| 9 |
POSIX.5B VALIDATION TESTS: |
| 10 |
|
| 11 |
This directory contains working drafts of validation tests for |
| 12 |
IEEE Standard 1003.5b. These test programs, which are derived |
| 13 |
from programs originally written as tests of Florist, have been |
| 14 |
developed under contract to Logicon Eagle Technologies for the |
| 15 |
U.S. Department of Defense, Defense Information Systems Agency, |
| 16 |
Center for Standards. |
| 17 |
|
| 18 |
PREREQUISITES: |
| 19 |
|
| 20 |
The tests are supposed to be 100% portable, but the procedures |
| 21 |
for compiling them and running them will depend on the environment |
| 22 |
in which you are working. |
| 23 |
|
| 24 |
The Makefile here has been used successfully in the following |
| 25 |
environment: |
| 26 |
|
| 27 |
make utility : GNU Make version 3.74 |
| 28 |
Ada compiler : GNATMAKE 3.13a |
| 29 |
POSIX.5b implementation : Florist 1.4, 31 Dec 1998 |
| 30 |
OS : Solaris 2.7 |
| 31 |
architecture : SPARCstation 20 HS14 |
| 32 |
|
| 33 |
The Makefile depends on the "make" utility, how to call the Ada |
| 34 |
compiler, and where the sources and library files associated with |
| 35 |
the implementation of POSIX.5b are installed. |
| 36 |
|
| 37 |
The Makefile in this directory is designed to work with the Gnu |
| 38 |
"make" utility. (It will probably work with other "make" |
| 39 |
utilities, except for the pattern-matching rules, which use the |
| 40 |
extended Gnu syntax.) |
| 41 |
|
| 42 |
The Makefile is set up to work with the Florist implementation of |
| 43 |
POSIX.5/.5b, and the GNAT Ada compiler. It assumes you have |
| 44 |
compiled Florist, that the sources and object library are in the |
| 45 |
directory ./floristlib. |
| 46 |
|
| 47 |
With this Makefile you can compile all the tests using the command |
| 48 |
"make tests". You can run all the tests by typing "make |
| 49 |
run_tests". The tests will be run in a subdirectory called |
| 50 |
"test.dir", and the results will be logged to the file |
| 51 |
"test.dir/run_tests_1.log". It will take quite a long time to run |
| 52 |
all the tests, even when they all pass. In particular, the "p99*" |
| 53 |
series of tests normally take a long time. There is also high |
| 54 |
probability that one of the tests may crash the test run, or |
| 55 |
"hang". In the latter case, you will need to kill the test run, |
| 56 |
and run the remaining tests in smaller groups. Try commenting the |
| 57 |
bad tests out of the script "run_tests_1", and running them |
| 58 |
individually. |
| 59 |
|
| 60 |
VERBOSITY CONTROLS: |
| 61 |
|
| 62 |
You can modify how much output the test produces, using the |
| 63 |
following command line options. |
| 64 |
|
| 65 |
-t = Terse output. This eliminates all but the first and last |
| 66 |
lines of output |
| 67 |
|
| 68 |
-v = Verbose output. This provides additional output, that |
| 69 |
may be helpful in narrowing down the nature of a test |
| 70 |
failure. |
| 71 |
|
| 72 |
The output of the default verbosity level is described below. |
| 73 |
|
| 74 |
NORMAL TEST OUTPUT: |
| 75 |
|
| 76 |
Each test will produce an initial line giving the name of the test |
| 77 |
and some version information, e.g. |
| 78 |
|
| 79 |
,.,. p020400 POSIX Ada Validation Tests, Version 1.2 |
| 80 |
|
| 81 |
By default, the test will then produce a series of lines to |
| 82 |
indicate its progress through various internal subdivisions, like |
| 83 |
the following: |
| 84 |
|
| 85 |
---- *-Subtest: package POSIX |
| 86 |
---- *-Subtest: version query functions |
| 87 |
---- *-Subtest: optional facility subtypes |
| 88 |
---- *-Subtest: bytes and I/O counts |
| 89 |
---- *-Subtest: Blocking_Behavior type |
| 90 |
---- *-Subtest: Signal_Masking type |
| 91 |
---- *-Subtest: POSIX_Character |
| 92 |
---- *-Subtest: POSIX_String type |
| 93 |
---- *-Subtest: POSIX_String_List type |
| 94 |
---- *-Subtest: Option_Set type |
| 95 |
---- *-Subtest: Error_Code type |
| 96 |
---- *-Subtest: uname-derived functions |
| 97 |
---- *-Subtest: Timespec type |
| 98 |
|
| 99 |
If the test passes it will then terminate and produce the message: |
| 100 |
|
| 101 |
==== Test Completed Successfully. |
| 102 |
|
| 103 |
ORDINARY FAILURES: |
| 104 |
|
| 105 |
Any messages of the following form indicate a failure at some |
| 106 |
point within the test: |
| 107 |
|
| 108 |
!!TEST FAILED: ... |
| 109 |
|
| 110 |
If the test fails and successfully terminates, it will produce a |
| 111 |
message of the form: |
| 112 |
|
| 113 |
==== Failed ...some number... tests. |
| 114 |
|
| 115 |
EXTREME FAILURES: |
| 116 |
|
| 117 |
Some tests may hang, or terminate without completing. We have |
| 118 |
tried to design the tests so that they can tolerate some |
| 119 |
anticipated kinds of failures and still run to completion, but the |
| 120 |
range of possible failures is practically infinite. In these |
| 121 |
cases, it may help to use the "-v" option to see more precisely |
| 122 |
how far the test got before the problem occurred. |
| 123 |
|
| 124 |
UNSUPPORTED FEATURES: |
| 125 |
|
| 126 |
Some tests will produce "Nonsupport of ......" message after the |
| 127 |
"Completed successfully" message. This indicates that the feature |
| 128 |
you are testing is not provided by the POSIX Ada binding |
| 129 |
implementation. This should only happen if it the corresponding |
| 130 |
capablity is not provided by the underlying operating system. |
| 131 |
This is not a failure, unless the implementation actually claims |
| 132 |
to support the feature, or there are other tests where the feature |
| 133 |
appears to be supported. |
| 134 |
|
| 135 |
You can use the "-v" command-line argument option when you run the |
| 136 |
test to see messages indicating each of the points where the |
| 137 |
test found a feature unsupported. |
| 138 |
|
| 139 |
PRIVILEGE FAILURES: |
| 140 |
|
| 141 |
Some tests attempt to perform operations that may require special |
| 142 |
privilege. These tests passed when we ran them on Solaris 2.6 |
| 143 |
with "root" privilege, but running strange programs with root |
| 144 |
permission is always a risky thing to do. You will need to review |
| 145 |
these tests and determine what needs to be done on your system to |
| 146 |
give them the appropriate privilege, and arrange to test them in |
| 147 |
an environment where they will do no serious harm if the run amok. |
| 148 |
|
| 149 |
TAILORABLE FEATURES: |
| 150 |
|
| 151 |
The POSIX standards provide considerable leeway for variation |
| 152 |
among implementations. For example, an implementation may impose |
| 153 |
arbitrary restrictions on the form of the name of a message queue. |
| 154 |
In order to accomodate these variations, we have attempted to |
| 155 |
parameterize the tests, and to isolate the implementation dependent |
| 156 |
constants (such as message queue names) in a single package, named |
| 157 |
Test_Parameters. You may need to edit this package to fit the |
| 158 |
implementation being tested. |
| 159 |
|
| 160 |
OTHER SETUP: |
| 161 |
|
| 162 |
Some tests require other external setup, before the test program |
| 163 |
is executed. For example, some require setting an environment |
| 164 |
variable, and some require putting an executable file in a known |
| 165 |
location. See the script "run_tests_1" for examples. |
| 166 |
|
| 167 |
STATUS: |
| 168 |
|
| 169 |
These are the final versions of the tests delivered to the |
| 170 |
Government under the POSIX.5b Test Suite project. |