Overview ================================================== This directory contains a random test generator for the gmp compatibility layer of imath. The tests cases are randomly generated and run with both gmp and imath. The results are compared and any mismatching results are flagged as failures. You should not see any failures when running these tests. Requirements ================================================== These tests use the python ffi to run the imath and gmp functions. To run these tests you will need the following items * libimath.so * python 3 * gmp library and header files Running ================================================== All the tests cases will be generated and run automatically by the makefile. First, make sure you have built libimath.so in the top level imath directory. Use the following command to generate and run the tests $ make TESTS=random.tests This should generate all the needed wrapper files along with the randomized unit tests. By default the unit tests are output to the random.tests file. The tests can be run by hand using the following command line $ ./runtest random.tests You can also write your own unit tests and run them by passing the file name to the runtest script. Testing Methodology ================================================== The goal of our testing is to ensure that we are compatible with the gmp api. To test our compatibility layer we generate inputs for each gmp api that we wrap and then call the gmp version of the api and compare the results to the imath version. The results should be identical. Any output mismatch is considered an error. Testcase Generation -------------------- The test data generation is inspired by the QuickCheck testing methodolgy. We want to test both random data and important values such as 0,1,-1, etc. We generate input data based on the type of the input parameter. Most apis take either an mpz_t or a signed/unsigned long. For each parameter of the given type we generate the following values: mpz_t: 0,1,-1, all min/max values, all min/max values +/- 1 small numbers: 1-4 digits medium numbers: 5-20 digits large numbers: 20-100 digits long : 0,1,-1 short,int,long min/max values random long values unsigned long: 0,1,-1, unsigned short, unsigned int, unsigned long min/max values random unsigned long values The generated data for each paramater is combined to produce a series of inputs to the function. The input data format looks like: mpz_add|0,1,2 Which represents the call mpz_add(0, 1, 2). Additional test cases can be written by hand. Test Structure -------------------- The tests are run using the python ffi (the ctypes module). We have a single description of each api that is used to generate the following: * intput data * c function wrapper to call the libgmp or libimath function * python wrapper to call both libgmp and libimath wrappers and compare results The test_gmp.so and test_imath.so libraries are loaded at runtime by the python runner scrip and used to run each test input and compare the results. The code generation pattern looks something like this: genctest.py ~~generates~~> gmp_test.c <--includes-- gmp_custom_test.c gmp_test.c ~~generates~~> gmp_tests.so <--links-- libgmp.so genpytest.py ~~generates~~> wrappers.py <--calls-- gmp_test.so gmp_test.so --loads--> runtest.py <--includes-- wrappers.py ^ | reads | gendata.py ~~generates~~> random.tests Adding tests for new APIs ================================================== New apis can be tested by modifying the gmpapy.py file. This file contains a list of all apis that will be tested. Adding an api to the apis list will cause code and input data to be generated for that api. Rerunning make will generate the test data and code to run the test.