| 1 |
""" |
| 2 |
This is a top level package, hosting the entire CSB test framework. It is divided |
| 3 |
into several major parts: |
| 4 |
|
| 5 |
- test cases, located under csb.test.cases |
| 6 |
- test data, in C{/csb/test/data} (not a package) |
| 7 |
- test console, in C{/csb/test/app.py} |
| 8 |
|
| 9 |
This module, csb.test, contains all the glue-code functions, classes and |
| 10 |
decorators you would need in order to write tests for CSB. |
| 11 |
|
| 12 |
1. Configuration and Tree |
| 13 |
|
| 14 |
L{Config<csb.test.Config>} is a common config object shared between CSB |
| 15 |
tests. Each config instance contains properties like: |
| 16 |
|
| 17 |
- data: the data folder, automatically discovered and loaded in |
| 18 |
csb.test.Config.DATA at module import time |
| 19 |
- temp: a default temp folder, which test cases can use |
| 20 |
|
| 21 |
Each L{Config<csb.test.Config>} provides a convenient way to retrieve |
| 22 |
files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile} |
| 23 |
and L{Config.getPickle}. In case you need a temp file, use |
| 24 |
L{Config.getTempStream} or have a look at L{csb.io.TempFile} and |
| 25 |
L{csb.io.TempFolder}. |
| 26 |
|
| 27 |
All test data files should be placed in the C{data} folder. All test |
| 28 |
modules must be placed in the root package: csb.test.cases. There is |
| 29 |
a strict naming convention for test modules: the name of a test module |
| 30 |
should be the same as the name of the CSB API package it tests. For |
| 31 |
example, if you are writing tests for C{csb/bio/io/__init__.py}, the |
| 32 |
test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases} |
| 33 |
is the root package of all test modules in CSB. |
| 34 |
|
| 35 |
2. Writing Tests |
| 36 |
|
| 37 |
Writing a test is easy. All you need is to import csb.test and then |
| 38 |
create your own test cases, derived from L{csb.test.Case}: |
| 39 |
|
| 40 |
>>> import csb.test |
| 41 |
>>> @csb.test.unit |
| 42 |
class TestSomeClass(csb.test.Case): |
| 43 |
def setUp(self): |
| 44 |
super(TestSomeClass, self).setUp() |
| 45 |
# do something with self.config here... |
| 46 |
|
| 47 |
In this way your test case instance is automatically equipped with a |
| 48 |
reference to the test config, so your test method can be: |
| 49 |
|
| 50 |
>>> @csb.test.unit |
| 51 |
class TestSomeClass(csb.test.Case): |
| 52 |
def testSomeMethod(self): |
| 53 |
myDataFile = self.config.getTestFile('some.file') |
| 54 |
self.assert... |
| 55 |
|
| 56 |
The "unit" decorator marks a test case as a collection of unit tests. |
| 57 |
All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom}, |
| 58 |
and L{csb.test.regression}. |
| 59 |
|
| 60 |
Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit |
| 61 |
more work. Custom tests must be functions, not classes. Basically a |
| 62 |
custom test is a function, which builds a unittest.TestSuite instance |
| 63 |
and then returns it when called without arguments. |
| 64 |
|
| 65 |
Regression tests are usually created in response to reported bugs. Therefore, |
| 66 |
the best practice is to mark each test method with its relevant bug ID: |
| 67 |
|
| 68 |
>>> @csb.test.regression |
| 69 |
class SomeClassRegressions(csb.test.Case) |
| 70 |
def testSomeFeature(self) |
| 71 |
\""" |
| 72 |
@see: [CSB 000XXXX] |
| 73 |
\""" |
| 74 |
# regression test body... |
| 75 |
|
| 76 |
3. Style Guide: |
| 77 |
|
| 78 |
- name test case packages as already described |
| 79 |
- group tests in csb.test.Case-s and name them properly |
| 80 |
- prefix test methods with "test", like "testParser" - very important |
| 81 |
- use camelCase for methods and variables. This applies to all the |
| 82 |
code under csb.test (including test) and does not apply to the rest |
| 83 |
of the library! |
| 84 |
- for functional tests it's okay to define just one test method: runTest |
| 85 |
- for unit tests you should create more specific test names, for example: |
| 86 |
"testParseFile" - a unit test for some method called "parse_file" |
| 87 |
- use csb.test decorators to mark tests as unit, functional, regression, etc. |
| 88 |
- make every test module executable:: |
| 89 |
|
| 90 |
if __name__ == '__main__': |
| 91 |
csb.test.Console() # Discovers and runs all test cases in the module |
| 92 |
|
| 93 |
4. Test Execution |
| 94 |
|
| 95 |
Test discovery is handled by C{test builders} and a test runner |
| 96 |
C{app}. Test builders are subclasses of L{AbstractTestBuilder}. |
| 97 |
For every test type (unit, functional, regression, custom) there is a |
| 98 |
corresponding test builder. L{AnyTestBuilder} is a special builder which |
| 99 |
scans for unit, regression and functional tests at the same time. |
| 100 |
|
| 101 |
Test builder classes inherit the following test discovery methods: |
| 102 |
|
| 103 |
- C{loadTests} - load tests from a test namespace. Wildcard |
| 104 |
namespaces are handled by C{loadAllTests} |
| 105 |
- C{loadAllTests} - load tests from the given namespace, and |
| 106 |
from all sub-packages (recursive) |
| 107 |
- C{loadFromFile} - load tests from an absolute file name |
| 108 |
- C{loadMultipleTests} - calls C{loadTests} for a list of |
| 109 |
namespaces and combines all loaded tests in a single suite |
| 110 |
|
| 111 |
Each of those return test suite objects, which can be directly executed |
| 112 |
with python's unittest runner. |
| 113 |
|
| 114 |
Much simpler way to execute a test suite is to use our test app |
| 115 |
(C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}:: |
| 116 |
|
| 117 |
$ python csb/test/app.py --help |
| 118 |
|
| 119 |
The app has two main arguments: |
| 120 |
|
| 121 |
- test type - tells the app which TestBuilder to use for test dicsovery |
| 122 |
("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.) |
| 123 |
- test namespaces - a list of "dotted" test modules, for example:: |
| 124 |
|
| 125 |
csb.test.cases.bio.io.* # io and sub-packages |
| 126 |
csb.test.cases.bio.utils # only utils |
| 127 |
. # current module |
| 128 |
|
| 129 |
In addition to running the app from the command line, you can run it |
| 130 |
also programmatically by instantiating L{csb.test.Console}. You can |
| 131 |
construct a test console object by passing a list of test namespace(s) |
| 132 |
and a test builder class to the Console's constructor. |
| 133 |
|
| 134 |
|
| 135 |
5. Commit Policies |
| 136 |
|
| 137 |
Follow these guidelines when making changes to the repository: |
| 138 |
|
| 139 |
- B{no bugs in "trunk"}: after fixing a bug or implementing a new |
| 140 |
feature, make sure at least the default test set passes by running |
| 141 |
the test console without any arguments. This is equivalent to: |
| 142 |
app.py -t any "csb.test.cases.*". (If no test case from this set covers |
| 143 |
the affected code, create a test case first, as described in the other |
| 144 |
policies) |
| 145 |
|
| 146 |
- B{no recurrent issues}: when a bug is found, first write a regression |
| 147 |
test with a proper "@see: BugID" tag in the docstring. Run the test |
| 148 |
to make sure it fails. After fixing the bug, run the test again before |
| 149 |
you commit, as required by the previous policy |
| 150 |
|
| 151 |
- B{test all new features}: there should be a test case for every new feature |
| 152 |
we implement. One possible approach is to write a test case first and |
| 153 |
make sure it fails; when the new feature is ready, run the test again |
| 154 |
to make sure it passes |
| 155 |
|
| 156 |
@warning: for compatibility reasons do NOT import and use the unittest module |
| 157 |
directly. Always import unittest from csb.test, which is guaranteed |
| 158 |
to be python 2.7+ compatible. |
| 159 |
""" |
| 160 |
import os |
| 161 |
import sys |
| 162 |
import imp |
| 163 |
import types |
| 164 |
import time |
| 165 |
import tempfile |
| 166 |
import traceback |
| 167 |
import argparse |
| 168 |
|
| 169 |
import csb.io |
| 170 |
import csb.core |
| 171 |
|
| 172 |
try: |
| 173 |
from unittest import skip, skipIf |
| 174 |
import unittest |
| 175 |
except ImportError: |
| 176 |
import unittest2 as unittest |
| 177 |
|
| 178 |
from abc import ABCMeta, abstractproperty |
| 179 |
|
| 180 |
|
| 181 |
class Attributes(object): |
| 182 |
|
| 183 |
UNIT = '__CSBUnitTest__' |
| 184 |
CUSTOM = '__CSBCustomTest__' |
| 185 |
FUNCTIONAL = '__CSBFunctionalTest__' |
| 186 |
REGRESSION = '__CSBRegressionTest__' |
| 187 |
|
| 188 |
class Config(object): |
| 189 |
""" |
| 190 |
General CSB Test Config. Config instances contain the following properties: |
| 191 |
|
| 192 |
- data - path to the CSB Test Data directory. Default is L{Config.DATA} |
| 193 |
- temp - path to the system's temp directory. Default is L{Config.TEMP} |
| 194 |
- config - the L{Config} class |
| 195 |
""" |
| 196 |
|
| 197 |
DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data') |
| 198 |
""" |
| 199 |
@cvar: path to the default test data directory: <install dir>/csb/test/data |
| 200 |
""" |
| 201 |
GENERATED_DATA = DATA |
| 202 |
""" |
| 203 |
@cvar: path to the default data directory for generated test files |
| 204 |
""" |
| 205 |
TEMP = os.path.abspath(tempfile.gettempdir()) |
| 206 |
""" |
| 207 |
@cvar: path to the default system's temp directory |
| 208 |
""" |
| 209 |
|
| 210 |
@staticmethod |
| 211 |
def setDefaultDataRoot(path): |
| 212 |
""" |
| 213 |
Override the default L{Config.DATA} with a new data root directory. |
| 214 |
|
| 215 |
@param path: full directory path |
| 216 |
@type path: str |
| 217 |
""" |
| 218 |
if not os.path.isdir(path): |
| 219 |
raise IOError('Path not found: {0}'.format(path)) |
| 220 |
|
| 221 |
Config.DATA = os.path.abspath(path) |
| 222 |
|
| 223 |
@staticmethod |
| 224 |
def setDefaultGeneratedDataRoot(path): |
| 225 |
""" |
| 226 |
Override the default L{Config.GENERATED_DATA} with a new data root directory. |
| 227 |
|
| 228 |
@param path: full directory path |
| 229 |
@type path: str |
| 230 |
""" |
| 231 |
if not os.path.isdir(path): |
| 232 |
raise IOError('Path not found: {0}'.format(path)) |
| 233 |
|
| 234 |
Config.GENERATED_DATA = os.path.abspath(path) |
| 235 |
|
| 236 |
@property |
| 237 |
def data(self): |
| 238 |
""" |
| 239 |
Test data directory |
| 240 |
@rtype: str |
| 241 |
""" |
| 242 |
return Config.DATA |
| 243 |
|
| 244 |
@property |
| 245 |
def generatedData(self): |
| 246 |
""" |
| 247 |
Test data directory for generated files |
| 248 |
@rtype: str |
| 249 |
""" |
| 250 |
return Config.GENERATED_DATA |
| 251 |
|
| 252 |
@property |
| 253 |
def temp(self): |
| 254 |
""" |
| 255 |
Test temp directory |
| 256 |
@rtype: str |
| 257 |
""" |
| 258 |
return Config.TEMP |
| 259 |
|
| 260 |
def getTestFile(self, fileName, subDir=''): |
| 261 |
""" |
| 262 |
Search for C{fileName} in the L{Config.DATA} directory. If not found, |
| 263 |
try also L{Config.GENERATED_DATA} (if different). |
| 264 |
|
| 265 |
@param fileName: the name of a test file to retrieve |
| 266 |
@type fileName: str |
| 267 |
@param subDir: scan a sub-directory of L{Config.DATA} |
| 268 |
@type subDir: str |
| 269 |
|
| 270 |
@return: full path to C{fileName} |
| 271 |
@rtype: str |
| 272 |
|
| 273 |
@raise IOError: if no such file is found |
| 274 |
""" |
| 275 |
for data in [self.data, self.generatedData]: |
| 276 |
file = os.path.join(data, subDir, fileName) |
| 277 |
|
| 278 |
if os.path.isfile(file): |
| 279 |
return file |
| 280 |
|
| 281 |
raise IOError('Test file not found: {0}'.format(fileName)) |
| 282 |
|
| 283 |
def getPickle(self, fileName, subDir=''): |
| 284 |
""" |
| 285 |
Same as C{self.getTestFile}, but try to unpickle the the file |
| 286 |
and return the unpickled object. Pickles are usually stored in |
| 287 |
L{Config.GENERATED_DATA}. |
| 288 |
|
| 289 |
@param fileName: the name of a test file to retrieve |
| 290 |
@type fileName: str |
| 291 |
@param subDir: scan a sub-directory of L{Config.DATA} |
| 292 |
@type subDir: str |
| 293 |
|
| 294 |
@rtype: object |
| 295 |
""" |
| 296 |
file = self.getTestFile(fileName, subDir) |
| 297 |
return csb.io.Pickle.load(open(file, 'rb')) |
| 298 |
|
| 299 |
def getContent(self, fileName, subDir=''): |
| 300 |
""" |
| 301 |
Same as C{self.getTestFile}, but also read and return the contents of |
| 302 |
the file. |
| 303 |
|
| 304 |
@param fileName: the name of a test file to retrieve |
| 305 |
@type fileName: str |
| 306 |
@param subDir: scan a sub-directory of L{Config.DATA} |
| 307 |
@type subDir: str |
| 308 |
|
| 309 |
@rtype: str |
| 310 |
""" |
| 311 |
with open(self.getTestFile(fileName, subDir)) as f: |
| 312 |
return f.read() |
| 313 |
|
| 314 |
def getTempStream(self, mode='t'): |
| 315 |
""" |
| 316 |
Return a temporary file stream:: |
| 317 |
|
| 318 |
with self.getTempStream() as tmp: |
| 319 |
tmp.write(something) |
| 320 |
tmp.flush() |
| 321 |
file_name = tmp.name |
| 322 |
|
| 323 |
@param mode: file open mode (text, binary), default=t |
| 324 |
@type mode: str |
| 325 |
@rtype: file stream |
| 326 |
""" |
| 327 |
return csb.io.TempFile(mode=mode) |
| 328 |
|
| 329 |
def ensureDataConsistency(self): |
| 330 |
""" |
| 331 |
Try to deserialize some pickled data files. Call L{Config.updateDataFiles} |
| 332 |
if the pickles appeared incompatible with the current interpreter. |
| 333 |
""" |
| 334 |
try: |
| 335 |
self.getPickle('1nz9.model1.pickle') |
| 336 |
except: |
| 337 |
self.updateDataFiles() |
| 338 |
|
| 339 |
def updateDataFiles(self): |
| 340 |
""" |
| 341 |
Refresh the pickled structures in csb/test/data. This might be needed when |
| 342 |
the internal representation of some classes has changed. |
| 343 |
""" |
| 344 |
from csb.io import Pickle |
| 345 |
from csb.bio.io.wwpdb import RegularStructureParser |
| 346 |
from csb.bio.structure import Ensemble, ChemElements |
| 347 |
|
| 348 |
parser = RegularStructureParser(self.getTestFile('1nz9.pdb')) |
| 349 |
model1 = parser.parse_structure(model=1) |
| 350 |
model2 = parser.parse_structure(model=2) |
| 351 |
|
| 352 |
ensemble = Ensemble() |
| 353 |
ensemble.models.append(model1) |
| 354 |
ensemble.models.append(model2) |
| 355 |
Pickle.dump(ensemble, open(os.path.join(self.generatedData, '1nz9.full.pickle'), 'wb')) |
| 356 |
|
| 357 |
mse = model1.chains['A'].find(164) |
| 358 |
mse.label = 'MSE' |
| 359 |
mse.atoms['SD']._element = ChemElements.Se |
| 360 |
mse.atoms['SD']._full_name = 'SE ' |
| 361 |
Pickle.dump(model1, open(os.path.join(self.generatedData, '1nz9.model1.pickle'), 'wb')) |
| 362 |
|
| 363 |
class Case(unittest.TestCase): |
| 364 |
""" |
| 365 |
Base class, defining a CSB Test Case. Provides a default implementation |
| 366 |
of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}. |
| 367 |
""" |
| 368 |
|
| 369 |
@property |
| 370 |
def config(self): |
| 371 |
""" |
| 372 |
Test config instance |
| 373 |
@rtype: L{Config} |
| 374 |
""" |
| 375 |
return self.__config |
| 376 |
|
| 377 |
def setUp(self): |
| 378 |
""" |
| 379 |
Provide a reference to the CSB Test Config in the C{self.config} property. |
| 380 |
""" |
| 381 |
self.__config = Config() |
| 382 |
assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory' |
| 383 |
assert self.config.data, 'The CSB Test Config must contain the data directory' |
| 384 |
|
| 385 |
def reRaise(self, addArgs=()): |
| 386 |
""" |
| 387 |
Re-raise the last exception with its full traceback, but modify the |
| 388 |
argument list with C{addArgs} and the original stack trace. |
| 389 |
|
| 390 |
@param addArgs: additional arguments to append to the exception |
| 391 |
@type addArgs: tuple |
| 392 |
""" |
| 393 |
klass, ex, _tb = sys.exc_info() |
| 394 |
ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())] |
| 395 |
|
| 396 |
raise klass(ex.args) |
| 397 |
|
| 398 |
def assertAlmostEqual(self, first, second, places=None, msg=None, delta=None): |
| 399 |
|
| 400 |
if first == second: |
| 401 |
return |
| 402 |
if delta is not None and places is not None: |
| 403 |
raise TypeError("specify delta or places not both") |
| 404 |
|
| 405 |
if delta is not None: |
| 406 |
|
| 407 |
if abs(first - second) <= delta: |
| 408 |
return |
| 409 |
|
| 410 |
m = '{0} != {1} within {2} delta'.format(first, second, delta) |
| 411 |
msg = self._formatMessage(msg, m) |
| 412 |
|
| 413 |
raise self.failureException(msg) |
| 414 |
|
| 415 |
else: |
| 416 |
if places is None: |
| 417 |
places = 7 |
| 418 |
|
| 419 |
return super(Case, self).assertAlmostEqual(first, second, places=places, msg=msg) |
| 420 |
|
| 421 |
def assertFasterThan(self, duration, callable, *args, **kargs): |
| 422 |
""" |
| 423 |
Fail if it took more than C{duration} seconds to invoke C{callable}. |
| 424 |
|
| 425 |
@param duration: maximum amount of seconds allowed |
| 426 |
@type duration: float |
| 427 |
""" |
| 428 |
|
| 429 |
start = time.time() |
| 430 |
callable(*args, **kargs) |
| 431 |
execution = time.time() - start |
| 432 |
|
| 433 |
if execution > duration: |
| 434 |
self.fail('{0}s is slower than {1}s)'.format(execution, duration)) |
| 435 |
|
| 436 |
@classmethod |
| 437 |
def execute(cls): |
| 438 |
""" |
| 439 |
Run this test case. |
| 440 |
""" |
| 441 |
suite = unittest.TestLoader().loadTestsFromTestCase(cls) |
| 442 |
runner = unittest.TextTestRunner() |
| 443 |
|
| 444 |
return runner.run(suite) |
| 445 |
|
| 446 |
class InvalidNamespaceError(NameError, ImportError): |
| 447 |
pass |
| 448 |
|
| 449 |
class AbstractTestBuilder(object): |
| 450 |
""" |
| 451 |
This is a base class, defining a test loader which exposes the C{loadTests} |
| 452 |
method. |
| 453 |
|
| 454 |
Subclasses must override the C{labels} abstract property, which controls |
| 455 |
what kind of test cases are loaded by the test builder. |
| 456 |
""" |
| 457 |
|
| 458 |
__metaclass__ = ABCMeta |
| 459 |
|
| 460 |
@abstractproperty |
| 461 |
def labels(self): |
| 462 |
pass |
| 463 |
|
| 464 |
def loadFromFile(self, file): |
| 465 |
""" |
| 466 |
Load L{csb.test.Case}s from a module file. |
| 467 |
|
| 468 |
@param file: test module file name |
| 469 |
@type file: str |
| 470 |
|
| 471 |
@return: a C{unittest.TestSuite} ready for the test runner |
| 472 |
@rtype: C{unittest.TestSuite} |
| 473 |
""" |
| 474 |
mod = self._loadSource(file) |
| 475 |
suite = unittest.TestLoader().loadTestsFromModule(mod) |
| 476 |
return unittest.TestSuite(self._filter(suite)) |
| 477 |
|
| 478 |
def loadTests(self, namespace): |
| 479 |
""" |
| 480 |
Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace |
| 481 |
ends with a wildcard, tests from sub-packages will be loaded as well. |
| 482 |
If the namespace is '__main__' or '.', tests are loaded from __main__. |
| 483 |
|
| 484 |
@param namespace: test module namespace, e.g. 'csb.test.cases.bio' will |
| 485 |
load tests from '/csb/test/cases/bio/__init__.py' |
| 486 |
@type namespace: str |
| 487 |
|
| 488 |
@return: a C{unittest.TestSuite} ready for the test runner |
| 489 |
@rtype: C{unittest.TestSuite} |
| 490 |
""" |
| 491 |
if namespace.strip() == '.*': |
| 492 |
namespace = '__main__.*' |
| 493 |
elif namespace.strip() == '.': |
| 494 |
namespace = '__main__' |
| 495 |
|
| 496 |
if namespace.endswith('.*'): |
| 497 |
return self.loadAllTests(namespace[:-2]) |
| 498 |
else: |
| 499 |
loader = unittest.TestLoader() |
| 500 |
tests = loader.loadTestsFromName(namespace) |
| 501 |
return unittest.TestSuite(self._filter(tests)) |
| 502 |
|
| 503 |
def loadMultipleTests(self, namespaces): |
| 504 |
""" |
| 505 |
Load L{csb.test.Case}s from a list of given CSB C{namespaces}. |
| 506 |
|
| 507 |
@param namespaces: a list of test module namespaces, e.g. |
| 508 |
('csb.test.cases.bio', 'csb.test.cases.bio.io') will |
| 509 |
load tests from '/csb/test/cases/bio.py' and |
| 510 |
'/csb/test/cases/bio/io.py' |
| 511 |
@type namespaces: tuple of str |
| 512 |
|
| 513 |
@return: a C{unittest.TestSuite} ready for the test runner |
| 514 |
@rtype: C{unittest.TestSuite} |
| 515 |
""" |
| 516 |
if not csb.core.iterable(namespaces): |
| 517 |
raise TypeError(namespaces) |
| 518 |
|
| 519 |
return unittest.TestSuite(self.loadTests(n) for n in namespaces) |
| 520 |
|
| 521 |
def loadAllTests(self, namespace, extension='.py'): |
| 522 |
""" |
| 523 |
Load L{csb.test.Case}s recursively from the given CSB C{namespace} and |
| 524 |
all of its sub-packages. Same as:: |
| 525 |
|
| 526 |
builder.loadTests('namespace.*') |
| 527 |
|
| 528 |
@param namespace: test module namespace, e.g. 'csb.test.cases.bio' will |
| 529 |
load tests from /csb/test/cases/bio/*' |
| 530 |
@type namespace: str |
| 531 |
|
| 532 |
@return: a C{unittest.TestSuite} ready for the test runner |
| 533 |
@rtype: C{unittest.TestSuite} |
| 534 |
""" |
| 535 |
suites = [] |
| 536 |
|
| 537 |
try: |
| 538 |
base = __import__(namespace, level=0, fromlist=['']).__file__ |
| 539 |
except ImportError: |
| 540 |
raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace)) |
| 541 |
|
| 542 |
if os.path.splitext(os.path.basename(base))[0] != '__init__': |
| 543 |
suites.append(self.loadTests(namespace)) |
| 544 |
|
| 545 |
else: |
| 546 |
|
| 547 |
for entry in os.walk(os.path.dirname(base)): |
| 548 |
|
| 549 |
for item in entry[2]: |
| 550 |
file = os.path.join(entry[0], item) |
| 551 |
if extension and item.endswith(extension): |
| 552 |
suites.append(self.loadFromFile(file)) |
| 553 |
|
| 554 |
return unittest.TestSuite(suites) |
| 555 |
|
| 556 |
def _loadSource(self, path): |
| 557 |
""" |
| 558 |
Import and return the Python module identified by C{path}. |
| 559 |
|
| 560 |
@note: Module objects behave as singletons. If you import two different |
| 561 |
modules and give them the same name in imp.load_source(mn), this |
| 562 |
counts for a redefinition of the module originally named mn, which |
| 563 |
is basically the same as reload(mn). Therefore, you need to ensure |
| 564 |
that for every call to imp.load_source(mn, src.py) the mn parameter |
| 565 |
is a string that uniquely identifies the source file src.py. |
| 566 |
""" |
| 567 |
name = os.path.splitext(os.path.abspath(path))[0] |
| 568 |
name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep) |
| 569 |
|
| 570 |
return imp.load_source(name, path) |
| 571 |
|
| 572 |
def _recurse(self, obj): |
| 573 |
""" |
| 574 |
Extract test cases recursively from a test C{obj} container. |
| 575 |
""" |
| 576 |
cases = [] |
| 577 |
if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj): |
| 578 |
for item in obj: |
| 579 |
cases.extend(self._recurse(item)) |
| 580 |
else: |
| 581 |
cases.append(obj) |
| 582 |
return cases |
| 583 |
|
| 584 |
def _filter(self, tests): |
| 585 |
""" |
| 586 |
Filter a list of objects using C{self.labels}. |
| 587 |
""" |
| 588 |
filtered = [] |
| 589 |
|
| 590 |
for test in self._recurse(tests): |
| 591 |
for label in self.labels: |
| 592 |
if hasattr(test, label) and getattr(test, label) is True: |
| 593 |
filtered.append(test) |
| 594 |
|
| 595 |
return filtered |
| 596 |
|
| 597 |
class AnyTestBuilder(AbstractTestBuilder): |
| 598 |
""" |
| 599 |
Build a test suite of cases, marked as either unit, functional or regression |
| 600 |
tests. For detailed documentation see L{AbstractTestBuilder}. |
| 601 |
""" |
| 602 |
@property |
| 603 |
def labels(self): |
| 604 |
return [Attributes.UNIT, Attributes.FUNCTIONAL, Attributes.REGRESSION] |
| 605 |
|
| 606 |
class UnitTestBuilder(AbstractTestBuilder): |
| 607 |
""" |
| 608 |
Build a test suite of cases, marked as unit tests. |
| 609 |
For detailed documentation see L{AbstractTestBuilder}. |
| 610 |
""" |
| 611 |
@property |
| 612 |
def labels(self): |
| 613 |
return [Attributes.UNIT] |
| 614 |
|
| 615 |
class FunctionalTestBuilder(AbstractTestBuilder): |
| 616 |
""" |
| 617 |
Build a test suite of cases, marked as functional tests. |
| 618 |
For detailed documentation see L{AbstractTestBuilder}. |
| 619 |
""" |
| 620 |
@property |
| 621 |
def labels(self): |
| 622 |
return [Attributes.FUNCTIONAL] |
| 623 |
|
| 624 |
class RegressionTestBuilder(AbstractTestBuilder): |
| 625 |
""" |
| 626 |
Build a test suite of cases, marked as regression tests. |
| 627 |
For detailed documentation see L{AbstractTestBuilder}. |
| 628 |
""" |
| 629 |
@property |
| 630 |
def labels(self): |
| 631 |
return [Attributes.REGRESSION] |
| 632 |
|
| 633 |
class CustomTestBuilder(AbstractTestBuilder): |
| 634 |
""" |
| 635 |
Build a test suite of cases, marked as custom tests. CustomTestBuilder will |
| 636 |
search for functions, marked with the 'custom' test decorator, which return |
| 637 |
a dynamically built C{unittest.TestSuite} object when called without |
| 638 |
parameters. This is convenient when doing data-related tests, e.g. |
| 639 |
instantiating a single type of a test case many times iteratively, for |
| 640 |
each entry in a database. |
| 641 |
|
| 642 |
For detailed documentation see L{AbstractTestBuilder}. |
| 643 |
""" |
| 644 |
@property |
| 645 |
def labels(self): |
| 646 |
return [Attributes.CUSTOM] |
| 647 |
|
| 648 |
def loadFromFile(self, file): |
| 649 |
|
| 650 |
mod = self._loadSource(file) |
| 651 |
suites = self._inspect(mod) |
| 652 |
|
| 653 |
return unittest.TestSuite(suites) |
| 654 |
|
| 655 |
def loadTests(self, namespace): |
| 656 |
|
| 657 |
if namespace.strip() == '.*': |
| 658 |
namespace = '__main__.*' |
| 659 |
elif namespace.strip() == '.': |
| 660 |
namespace = '__main__' |
| 661 |
|
| 662 |
if namespace.endswith('.*'): |
| 663 |
return self.loadAllTests(namespace[:-2]) |
| 664 |
else: |
| 665 |
try: |
| 666 |
mod = __import__(namespace, fromlist=['']) |
| 667 |
except ImportError: |
| 668 |
raise InvalidNamespaceError('Namespace {0} is not importable'.format(namespace)) |
| 669 |
suites = self._inspect(mod) |
| 670 |
return unittest.TestSuite(suites) |
| 671 |
|
| 672 |
def _inspect(self, module): |
| 673 |
|
| 674 |
objects = map(lambda n: getattr(module, n), dir(module)) |
| 675 |
return self._filter(objects) |
| 676 |
|
| 677 |
def _filter(self, factories): |
| 678 |
""" |
| 679 |
Filter a list of objects using C{self.labels}. |
| 680 |
""" |
| 681 |
filtered = [] |
| 682 |
|
| 683 |
for obj in factories: |
| 684 |
for label in self.labels: |
| 685 |
if hasattr(obj, label) and getattr(obj, label) is True: |
| 686 |
suite = obj() |
| 687 |
if not isinstance(suite, unittest.TestSuite): |
| 688 |
raise ValueError('Custom test function {0} must return a ' |
| 689 |
'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite))) |
| 690 |
filtered.append(suite) |
| 691 |
|
| 692 |
return filtered |
| 693 |
|
| 694 |
def unit(klass): |
| 695 |
""" |
| 696 |
A class decorator, used to label unit test cases. |
| 697 |
|
| 698 |
@param klass: a C{unittest.TestCase} class type |
| 699 |
@type klass: type |
| 700 |
""" |
| 701 |
if not isinstance(klass, type): |
| 702 |
raise TypeError("Can't apply class decorator on {0}".format(type(klass))) |
| 703 |
|
| 704 |
setattr(klass, Attributes.UNIT, True) |
| 705 |
return klass |
| 706 |
|
| 707 |
def functional(klass): |
| 708 |
""" |
| 709 |
A class decorator, used to label functional test cases. |
| 710 |
|
| 711 |
@param klass: a C{unittest.TestCase} class type |
| 712 |
@type klass: type |
| 713 |
""" |
| 714 |
if not isinstance(klass, type): |
| 715 |
raise TypeError("Can't apply class decorator on {0}".format(type(klass))) |
| 716 |
|
| 717 |
setattr(klass, Attributes.FUNCTIONAL, True) |
| 718 |
return klass |
| 719 |
|
| 720 |
def regression(klass): |
| 721 |
""" |
| 722 |
A class decorator, used to label regression test cases. |
| 723 |
|
| 724 |
@param klass: a C{unittest.TestCase} class type |
| 725 |
@type klass: type |
| 726 |
""" |
| 727 |
if not isinstance(klass, type): |
| 728 |
raise TypeError("Can't apply class decorator on {0}".format(type(klass))) |
| 729 |
|
| 730 |
setattr(klass, Attributes.REGRESSION, True) |
| 731 |
return klass |
| 732 |
|
| 733 |
def custom(function): |
| 734 |
""" |
| 735 |
A function decorator, used to mark functions which build custom (dynamic) |
| 736 |
test suites when called. |
| 737 |
|
| 738 |
@param function: a callable object, which returns a dynamically compiled |
| 739 |
C{unittest.TestSuite} |
| 740 |
@type function: callable |
| 741 |
""" |
| 742 |
if isinstance(function, type): |
| 743 |
raise TypeError("Can't apply function decorator on a class") |
| 744 |
elif not hasattr(function, '__call__'): |
| 745 |
raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function))) |
| 746 |
|
| 747 |
setattr(function, Attributes.CUSTOM, True) |
| 748 |
return function |
| 749 |
|
| 750 |
def skip(reason, condition=None): |
| 751 |
""" |
| 752 |
Mark a test case or method for skipping. |
| 753 |
|
| 754 |
@param reason: message |
| 755 |
@type reason: str |
| 756 |
@param condition: skip only if the specified condition is True |
| 757 |
@type condition: bool/expression |
| 758 |
""" |
| 759 |
if isinstance(reason, types.FunctionType): |
| 760 |
raise TypeError('skip: no reason specified') |
| 761 |
|
| 762 |
if condition is None: |
| 763 |
return unittest.skip(reason) |
| 764 |
else: |
| 765 |
return unittest.skipIf(condition, reason) |
| 766 |
|
| 767 |
class Console(object): |
| 768 |
""" |
| 769 |
Build and run all tests of the specified namespace and kind. |
| 770 |
|
| 771 |
@param namespace: a dotted name, which specifies the test module |
| 772 |
(see L{csb.test.AbstractTestBuilder.loadTests}) |
| 773 |
@type namespace: str |
| 774 |
@param builder: test builder to use |
| 775 |
@type builder: any L{csb.test.AbstractTestBuilder} subclass |
| 776 |
@param verbosity: verbosity level for C{unittest.TestRunner} |
| 777 |
@type verbosity: int |
| 778 |
@param update: if True, refresh all pickles in csb/test/data |
| 779 |
@type update: bool |
| 780 |
@param generated_data: where to cache generated test files (directory) |
| 781 |
@type generated_data: str |
| 782 |
""" |
| 783 |
|
| 784 |
BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder, |
| 785 |
'custom': CustomTestBuilder, 'any': AnyTestBuilder, |
| 786 |
'regression': RegressionTestBuilder} |
| 787 |
|
| 788 |
|
| 789 |
def __init__(self, namespace=('__main__',), builder=AnyTestBuilder, verbosity=1, |
| 790 |
update=False, generated_data=Config.GENERATED_DATA, argv=None): |
| 791 |
|
| 792 |
if not argv: |
| 793 |
argv = sys.argv |
| 794 |
|
| 795 |
self._namespace = None |
| 796 |
self._builder = None |
| 797 |
self._verbosity = 1 |
| 798 |
self._update = False |
| 799 |
self._gendata = str(generated_data) |
| 800 |
self._program = os.path.basename(argv[0]) |
| 801 |
|
| 802 |
self.namespace = namespace |
| 803 |
self.builder = builder |
| 804 |
self.verbosity = verbosity |
| 805 |
self.update = update |
| 806 |
self.generated_data = generated_data |
| 807 |
|
| 808 |
self.parseArguments(argv[1:]) |
| 809 |
self.run() |
| 810 |
|
| 811 |
@property |
| 812 |
def namespace(self): |
| 813 |
return self._namespace |
| 814 |
@namespace.setter |
| 815 |
def namespace(self, value): |
| 816 |
if csb.core.iterable(value): |
| 817 |
self._namespace = list(value) |
| 818 |
else: |
| 819 |
self._namespace = [value] |
| 820 |
|
| 821 |
@property |
| 822 |
def builder(self): |
| 823 |
return self._builder |
| 824 |
@builder.setter |
| 825 |
def builder(self, value): |
| 826 |
self._builder = value |
| 827 |
|
| 828 |
@property |
| 829 |
def verbosity(self): |
| 830 |
return self._verbosity |
| 831 |
@verbosity.setter |
| 832 |
def verbosity(self, value): |
| 833 |
self._verbosity = value |
| 834 |
|
| 835 |
@property |
| 836 |
def builders(self): |
| 837 |
return ', '.join(Console.BUILDERS) |
| 838 |
|
| 839 |
@property |
| 840 |
def program(self): |
| 841 |
return self._program |
| 842 |
|
| 843 |
@property |
| 844 |
def update(self): |
| 845 |
return self._update |
| 846 |
@update.setter |
| 847 |
def update(self, value): |
| 848 |
self._update = bool(value) |
| 849 |
|
| 850 |
@property |
| 851 |
def generated_data(self): |
| 852 |
return self._gendata |
| 853 |
@generated_data.setter |
| 854 |
def generated_data(self, value): |
| 855 |
self._gendata = os.path.abspath(value) |
| 856 |
|
| 857 |
def run(self): |
| 858 |
|
| 859 |
Config.setDefaultGeneratedDataRoot(self.generated_data) |
| 860 |
|
| 861 |
if self.update: |
| 862 |
Config().updateDataFiles() |
| 863 |
else: |
| 864 |
Config().ensureDataConsistency() |
| 865 |
|
| 866 |
builder = self.builder() |
| 867 |
suite = builder.loadMultipleTests(self.namespace) |
| 868 |
|
| 869 |
runner = unittest.TextTestRunner(verbosity=self.verbosity) |
| 870 |
runner.run(suite) |
| 871 |
|
| 872 |
def parseArguments(self, argv): |
| 873 |
|
| 874 |
parser = argparse.ArgumentParser(prog=self.program, description="CSB Test Runner Console.") |
| 875 |
|
| 876 |
parser.add_argument("-t", "--type", type=str, default="any", choices=list(Console.BUILDERS), |
| 877 |
help="Type of tests to load from each namespace (default=any)") |
| 878 |
parser.add_argument("-v", "--verbosity", type=int, default=1, |
| 879 |
help="Verbosity level passed to unittest.TextTestRunner (default=1).") |
| 880 |
parser.add_argument("-u", "--update-files", default=False, action="store_true", |
| 881 |
help="Force update of the test pickles in " + Config.GENERATED_DATA) |
| 882 |
parser.add_argument("-g", "--generated-resources", type=str, default=Config.GENERATED_DATA, |
| 883 |
help="Generate, store and load additional test resources in this directory" |
| 884 |
" (default=" + Config.GENERATED_DATA + ")") |
| 885 |
|
| 886 |
parser.add_argument("namespaces", nargs='*', |
| 887 |
help="""An optional list of CSB test dotted namespaces, from which to |
| 888 |
load tests. '__main__' and '.' are interpreted as the |
| 889 |
current module. If a namespace ends with an asterisk |
| 890 |
'.*', all sub-packages will be scanned as well. |
| 891 |
|
| 892 |
Examples: |
| 893 |
"csb.test.cases.bio.*" |
| 894 |
"csb.test.cases.bio.io" "csb.test.cases.bio.utils" |
| 895 |
".")""") |
| 896 |
|
| 897 |
args = parser.parse_args(argv) |
| 898 |
|
| 899 |
self.builder = Console.BUILDERS[args.type] |
| 900 |
self.verbosity = args.verbosity |
| 901 |
self.update = args.update_files |
| 902 |
self.generated_data = args.generated_resources |
| 903 |
|
| 904 |
if args.namespaces: |
| 905 |
self.namespace = args.namespaces |
| 906 |
|
| 907 |
|
| 908 |
if __name__ == '__main__': |
| 909 |
|
| 910 |
Console() |