| 1 |
README FOR IDLE TESTS IN IDLELIB.IDLE_TEST |
| 2 |
|
| 3 |
0. Quick Start |
| 4 |
|
| 5 |
Automated unit tests were added in 3.3 for Python 3.x. |
| 6 |
To run the tests from a command line: |
| 7 |
|
| 8 |
python -m test.test_idle |
| 9 |
|
| 10 |
Human-mediated tests were added later in 3.4. |
| 11 |
|
| 12 |
python -m idlelib.idle_test.htest |
| 13 |
|
| 14 |
|
| 15 |
1. Test Files |
| 16 |
|
| 17 |
The idle directory, idlelib, has over 60 xyz.py files. The idle_test |
| 18 |
subdirectory contains test_xyz.py for each implementation file xyz.py. |
| 19 |
To add a test for abc.py, open idle_test/template.py and immediately |
| 20 |
Save As test_abc.py. Insert 'abc' on the first line, and replace |
| 21 |
'zzdummy' with 'abc. |
| 22 |
|
| 23 |
Remove the imports of requires and tkinter if not needed. Otherwise, |
| 24 |
add to the tkinter imports as needed. |
| 25 |
|
| 26 |
Add a prefix to 'Test' for the initial test class. The template class |
| 27 |
contains code needed or possibly needed for gui tests. See the next |
| 28 |
section if doing gui tests. If not, and not needed for further classes, |
| 29 |
this code can be removed. |
| 30 |
|
| 31 |
Add the following at the end of abc.py. If an htest was added first, |
| 32 |
insert the import and main lines before the htest lines. |
| 33 |
|
| 34 |
if __name__ == "__main__": |
| 35 |
from unittest import main |
| 36 |
main('idlelib.idle_test.test_abc', verbosity=2, exit=False) |
| 37 |
|
| 38 |
The ', exit=False' is only needed if an htest follows. |
| 39 |
|
| 40 |
|
| 41 |
|
| 42 |
2. GUI Tests |
| 43 |
|
| 44 |
When run as part of the Python test suite, Idle GUI tests need to run |
| 45 |
test.support.requires('gui'). A test is a GUI test if it creates a |
| 46 |
tkinter.Tk root or master object either directly or indirectly by |
| 47 |
instantiating a tkinter or idle class. GUI tests cannot run in test |
| 48 |
processes that either have no graphical environment available or are not |
| 49 |
allowed to use it. |
| 50 |
|
| 51 |
To guard a module consisting entirely of GUI tests, start with |
| 52 |
|
| 53 |
from test.support import requires |
| 54 |
requires('gui') |
| 55 |
|
| 56 |
To guard a test class, put "requires('gui')" in its setUpClass function. |
| 57 |
The template.py file does this. |
| 58 |
|
| 59 |
To avoid interfering with other GUI tests, all GUI objects must be |
| 60 |
destroyed and deleted by the end of the test. The Tk root created in a |
| 61 |
setUpX function should be destroyed in the corresponding tearDownX and |
| 62 |
the module or class attribute deleted. Others widgets should descend |
| 63 |
from the single root and the attributes deleted BEFORE root is |
| 64 |
destroyed. See https://bugs.python.org/issue20567. |
| 65 |
|
| 66 |
@classmethod |
| 67 |
def setUpClass(cls): |
| 68 |
requires('gui') |
| 69 |
cls.root = tk.Tk() |
| 70 |
cls.text = tk.Text(root) |
| 71 |
|
| 72 |
@classmethod |
| 73 |
def tearDownClass(cls): |
| 74 |
del cls.text |
| 75 |
cls.root.update_idletasks() |
| 76 |
cls.root.destroy() |
| 77 |
del cls.root |
| 78 |
|
| 79 |
The update_idletasks call is sometimes needed to prevent the following |
| 80 |
warning either when running a test alone or as part of the test suite |
| 81 |
(#27196). It should not hurt if not needed. |
| 82 |
|
| 83 |
can't invoke "event" command: application has been destroyed |
| 84 |
... |
| 85 |
"ttk::ThemeChanged" |
| 86 |
|
| 87 |
If a test creates instance 'e' of EditorWindow, call 'e._close()' before |
| 88 |
or as the first part of teardown. The effect of omitting this depends |
| 89 |
on the later shutdown. Then enable the after_cancel loop in the |
| 90 |
template. This prevents messages like the following. |
| 91 |
|
| 92 |
bgerror failed to handle background error. |
| 93 |
Original error: invalid command name "106096696timer_event" |
| 94 |
Error in bgerror: can't invoke "tk" command: application has been destroyed |
| 95 |
|
| 96 |
Requires('gui') causes the test(s) it guards to be skipped if any of |
| 97 |
these conditions are met: |
| 98 |
|
| 99 |
- The tests are being run by regrtest.py, and it was started without |
| 100 |
enabling the "gui" resource with the "-u" command line option. |
| 101 |
|
| 102 |
- The tests are being run on Windows by a service that is not allowed |
| 103 |
to interact with the graphical environment. |
| 104 |
|
| 105 |
- The tests are being run on Linux and X Windows is not available. |
| 106 |
|
| 107 |
- The tests are being run on Mac OSX in a process that cannot make a |
| 108 |
window manager connection. |
| 109 |
|
| 110 |
- tkinter.Tk cannot be successfully instantiated for some reason. |
| 111 |
|
| 112 |
- test.support.use_resources has been set by something other than |
| 113 |
regrtest.py and does not contain "gui". |
| 114 |
|
| 115 |
Tests of non-GUI operations should avoid creating tk widgets. Incidental |
| 116 |
uses of tk variables and messageboxes can be replaced by the mock |
| 117 |
classes in idle_test/mock_tk.py. The mock text handles some uses of the |
| 118 |
tk Text widget. |
| 119 |
|
| 120 |
|
| 121 |
3. Running Unit Tests |
| 122 |
|
| 123 |
Assume that xyz.py and test_xyz.py both end with a unittest.main() call. |
| 124 |
Running either from an Idle editor runs all tests in the test_xyz file |
| 125 |
with the version of Python running Idle. Test output appears in the |
| 126 |
Shell window. The 'verbosity=2' option lists all test methods in the |
| 127 |
file, which is appropriate when developing tests. The 'exit=False' |
| 128 |
option is needed in xyx.py files when an htest follows. |
| 129 |
|
| 130 |
The following command lines also run all test methods, including |
| 131 |
GUI tests, in test_xyz.py. (Both '-m idlelib' and '-m idlelib.idle' |
| 132 |
start Idle and so cannot run tests.) |
| 133 |
|
| 134 |
python -m idlelib.xyz |
| 135 |
python -m idlelib.idle_test.test_xyz |
| 136 |
|
| 137 |
The following runs all idle_test/test_*.py tests interactively. |
| 138 |
|
| 139 |
>>> import unittest |
| 140 |
>>> unittest.main('idlelib.idle_test', verbosity=2) |
| 141 |
|
| 142 |
The following run all Idle tests at a command line. Option '-v' is the |
| 143 |
same as 'verbosity=2'. |
| 144 |
|
| 145 |
python -m unittest -v idlelib.idle_test |
| 146 |
python -m test -v -ugui test_idle |
| 147 |
python -m test.test_idle |
| 148 |
|
| 149 |
IDLE tests are 'discovered' by idlelib.idle_test.__init__.load_tests |
| 150 |
when this is imported into test.test_idle. Normally, neither file |
| 151 |
should be changed when working on individual test modules. The third |
| 152 |
command runs unittest indirectly through regrtest. The same happens when |
| 153 |
the entire test suite is run with 'python -m test'. So that command must |
| 154 |
work for buildbots to stay green. IDLE tests must not disturb the |
| 155 |
environment in a way that makes other tests fail (GH-62281). |
| 156 |
|
| 157 |
To test subsets of modules, see idlelib.idle_test.__init__. This |
| 158 |
can be used to find refleaks or possible sources of "Theme changed" |
| 159 |
tcl messages (GH-71383). |
| 160 |
|
| 161 |
To run an individual Testcase or test method, extend the dotted name |
| 162 |
given to unittest on the command line or use the test -m option. The |
| 163 |
latter allows use of other regrtest options. When using the latter, |
| 164 |
all components of the pattern must be present, but any can be replaced |
| 165 |
by '*'. |
| 166 |
|
| 167 |
python -m unittest -v idlelib.idle_test.test_xyz.Test_case.test_meth |
| 168 |
python -m test -m idlelib.idle_test.text_xyz.Test_case.test_meth test_idle |
| 169 |
|
| 170 |
The test suite can be run in an IDLE user process from Shell. |
| 171 |
>>> import test.autotest # Issue 25588, 2017/10/13, 3.6.4, 3.7.0a2. |
| 172 |
There are currently failures not usually present, and this does not |
| 173 |
work when run from the editor. |
| 174 |
|
| 175 |
|
| 176 |
4. Human-mediated Tests |
| 177 |
|
| 178 |
Human-mediated tests are widget tests that cannot be automated but need |
| 179 |
human verification. They are contained in idlelib/idle_test/htest.py, |
| 180 |
which has instructions. (Some modules need an auxiliary function, |
| 181 |
identified with "# htest # on the header line.) The set is about |
| 182 |
complete, though some tests need improvement. To run all htests, run the |
| 183 |
htest file from an editor or from the command line with: |
| 184 |
|
| 185 |
python -m idlelib.idle_test.htest |
| 186 |
|
| 187 |
|
| 188 |
5. Test Coverage |
| 189 |
|
| 190 |
Install the coverage package into your Python 3.6 site-packages |
| 191 |
directory. (Its exact location depends on the OS). |
| 192 |
> python3 -m pip install coverage |
| 193 |
(On Windows, replace 'python3 with 'py -3.6' or perhaps just 'python'.) |
| 194 |
|
| 195 |
The problem with running coverage with repository python is that |
| 196 |
coverage uses absolute imports for its submodules, hence it needs to be |
| 197 |
in a directory in sys.path. One solution: copy the package to the |
| 198 |
directory containing the cpython repository. Call it 'dev'. Then run |
| 199 |
coverage either directly or from a script in that directory so that |
| 200 |
'dev' is prepended to sys.path. |
| 201 |
|
| 202 |
Either edit or add dev/.coveragerc so it looks something like this. |
| 203 |
--- |
| 204 |
# .coveragerc sets coverage options. |
| 205 |
[run] |
| 206 |
branch = True |
| 207 |
|
| 208 |
[report] |
| 209 |
# Regexes for lines to exclude from consideration |
| 210 |
exclude_lines = |
| 211 |
# Don't complain if non-runnable code isn't run: |
| 212 |
if 0: |
| 213 |
if __name__ == .__main__.: |
| 214 |
|
| 215 |
.*# htest # |
| 216 |
if not _utest: |
| 217 |
if _htest: |
| 218 |
--- |
| 219 |
The additions for IDLE are 'branch = True', to test coverage both ways, |
| 220 |
and the last three exclude lines, to exclude things peculiar to IDLE |
| 221 |
that are not executed during tests. |
| 222 |
|
| 223 |
A script like the following cover.bat (for Windows) is very handy. |
| 224 |
--- |
| 225 |
@echo off |
| 226 |
rem Usage: cover filename [test_ suffix] # proper case required by coverage |
| 227 |
rem filename without .py, 2nd parameter if test is not test_filename |
| 228 |
setlocal |
| 229 |
set py=f:\dev\3x\pcbuild\win32\python_d.exe |
| 230 |
set src=idlelib.%1 |
| 231 |
if "%2" EQU "" set tst=f:/dev/3x/Lib/idlelib/idle_test/test_%1.py |
| 232 |
if "%2" NEQ "" set tst=f:/dev/ex/Lib/idlelib/idle_test/test_%2.py |
| 233 |
|
| 234 |
%py% -m coverage run --pylib --source=%src% %tst% |
| 235 |
%py% -m coverage report --show-missing |
| 236 |
%py% -m coverage html |
| 237 |
start htmlcov\3x_Lib_idlelib_%1_py.html |
| 238 |
rem Above opens new report; htmlcov\index.html displays report index |
| 239 |
--- |
| 240 |
The second parameter was added for tests of module x not named test_x. |
| 241 |
(There were several before modules were renamed, now only one is left.) |