Commit graph

123 commits

Author SHA1 Message Date
Jouni Malinen
cb8cb33e55 tests: Allow run-tests.py to be run from the vm directory
Some of the paths in run-tests.py are hardcoded and could not handle
tests/hwsim/vm as the working directory. Modify the design enough to
allow ../run-tests.py -L to be used from the vm directory.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
5e3c0b02a7 tests: Make run-tests.py process test case selection arguments
This allows a list of matching test cases to be produced without having
to run the test cases. Previously, -L output included all defined test
cases regardless of what else was included on the command line.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:52 +02:00
Ilan Peer
3bfa5f10a9 tests: Add support for reading modules list from a file
The new run-tests.py argument "-l <file>" can now be used to specify the
test modules using a text file.

Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2014-10-26 23:27:26 +02:00
Jouni Malinen
ac69c33297 tests: Print failure exeception if logger output goes to file
This makes it more convenient to determine failure reasons during test
case development and VM testing.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-26 10:39:32 +02:00
Jouni Malinen
eb92d389a8 tests: Remove wlan5-related interfaces based on list
Instead of hardcoding reset_devs() to remove wlan5, remove all wlan*
interfaces renaming in the wpa_supplicant process to support the case of
dynamically added hwsim phy.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-20 12:52:25 +03:00
Jouni Malinen
5f797376f2 tests: Make RELOG command use global control interface
This allows RELOG to be used even if a wpa_supplicant process does not
have any interfaces configured.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-20 12:52:25 +03:00
Jouni Malinen
521b7e7925 tests: Verify global control interface before starting each test
This allows control interface issues to be caught in a bit more readable
way in the debug logs. In addition, dump pending monitor socket
information more frequently and within each test case in the log files
to make the output clearer and less likely to go over the socket buffer
limit.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-04-29 15:20:23 +03:00
Jouni Malinen
91f3cf6996 tests: Add option for running test cases that take a long time
run-tests.py now takes an optional --long parameter that can be used to
enable running of test cases that take a long time (multiple minutes).
By default, such test cases are skipped to avoid making the normal test
run take excessive amounts of time.

As an initial long test case, verify WPS PBC walk time expiration (two
minutes).

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-04-06 13:11:32 +03:00
Jouni Malinen
09e38c2fce tests: Add scripts to allow parallel execution of tests in VMs
"parallel-vm.sh <number of VMs> [arguments..]" can now be used to run
multiple VMs in parallel to speed up full test cycle significantly. In
addition, the "--split srv/total" argument used in this design would
also make it possible to split this to multiple servers to speed up
testing.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-03-24 23:37:42 +02:00
Jouni Malinen
9007391650 tests: Allow test cases to receive additional parameters
The optional third argument to the test case functions can now be used
to receive additional parameters from run-tests.py. As the initial
parameter, logdir value is provided so that test cases can use it to
review the debug logs from the test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-03-23 12:01:10 +02:00
Eduardo Abinader
b5008c2c2d tests: Fix trace parameter invocation
If trace-cmd command does not exist, run-tests.py could end up hanging
in a loop waiting for input. Fix this simply by checking whether the
trace-cmd command can be executed sucessfully and exiting the script if
not.

Signed-off-by: Eduardo Abinader <eduardo.abinader@openbossa.org>
2014-03-11 19:09:40 +02:00
Jouni Malinen
2ca47da4f7 tests: Fix trace-cmd output for host case
Both the output file path and the current working directory included the
log directory and this failed if log directory was not absolute (e.g.,
when using the default logs/current in the case a VM is not used).

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-03-11 17:04:33 +02:00
Jouni Malinen
8107d33361 tests: Fix the file comment to describe what this script does
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-02-21 20:25:25 +02:00
Roger Zanoni
040dc68e8c tests: Change the python interpreter directive
This makes the script check the environment for the current python
interpreter in use instead of assuming that the python executable points
to a python 2 interpreter.

Signed-off-by: Roger Zanoni <roger.zanoni@openbossa.org>
2014-02-21 20:23:56 +02:00
Jouni Malinen
f8949f5f8f tests: Reset hostapd state with FLUSH between test cases
This is similar to the FLUSH command for wpa_supplicant.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2014-01-27 22:08:13 +02:00
Jouni Malinen
aa5fcc558b tests: Show missing descriptions in more helpful format
Signed-hostap: Jouni Malinen <j@w1.fi>
2014-01-07 10:45:10 +02:00
Jouni Malinen
1d646f5ee1 tests: Write debug logs into the database for failed test cases
This makes it easier to build a web page for analyzing failures without
having to fetch the log files themselves from the test server.

Signed-hostap: Jouni Malinen <j@w1.fi>
2014-01-07 10:45:10 +02:00
Jouni Malinen
9489637b42 tests: Verify dynamic wpa_supplicant interface addition/removal
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-31 15:45:18 +02:00
Jouni Malinen
b1448311c2 tests: Show test counter on the START line
This makes it more convenient to figure out how far a test cycle is from
live logs.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-29 17:18:17 +02:00
Jouni Malinen
bcbd74155e tests: Mark kernel issues more clearly in the log file
It was previously not obvious from the <test case>.log file that a test
case was marked failed based on kernel issues. Make this very clear to
avoid wasting time on figuring out what caused the failure.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 18:11:07 +02:00
Jouni Malinen
848bb8dea9 tests: Use per-test case log/capture file from wlantest
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-17 21:47:06 +02:00
Johannes Berg
a572f0b960 hwsim tests: Print failed tests nicer
There's no reason to format the failed tests as a python
list, just print a (space-separated) list of test names.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-17 16:46:41 +02:00
Jouni Malinen
66f0bdb465 tests: Handle hostapd dying more gracefully
Catch exceptions from operations that try to remove hostapd interface
and rename the log file. If these operations fail due to socket
connection issues, hostapd has likely died or gotten stuck somewhere.
Report the test case as a failure and stop test run cleanly.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-05 13:21:58 +02:00
Johannes Berg
5061a301e5 hwsim tests: Check kernel messages for warnings/bugs
When a test passes but the kernel printed warnings, consider
the test to have failed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-05 10:17:35 +02:00
Jouni Malinen
b5476d940e tests: Add run-tests.py --no-reset
This can be used to request the previously used default behavior where
the devices are not stopped at the end of a test case if a single test
case is run.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 11:31:48 +02:00
Jouni Malinen
f5f59e4f57 tests: Clean up output formatting with str.format()
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 17:07:13 +02:00
Jouni Malinen
57661377df tests: Use a helper function to rename wpa_supplicant/hostapd logs
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 13:04:06 +02:00
Jouni Malinen
8f47f31dd8 tests: Always write run-tests.py debug logs a file
Remove the -l command like option from run-tests.py and always enable
writing of debug level logs to files. The stdout debug verbosity is
controlled independently of the debug log files.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 12:20:59 +02:00
Jouni Malinen
75428961d9 tests: Split hostapd debug log into per test case files
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 11:53:38 +02:00
Jouni Malinen
de684a2b83 tests: Add run-tests.py --shuffle-test
This optional argument can be used to randomize the order in which the
test cases are run. This can provide more coverage on testing
interactions of common use cases in various different sequences. Such
issues have already been found even with the fixed order of test cases,
but being able to reorder the tests makes this more efficient.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 10:30:37 +02:00
Johannes Berg
a5d7da3fd3 hwsim tests: Prefill database in VM tests
In some cases, e.g., with the VM tests if the VM crashes, it
can be useful to know which tests should have run but didn't
(or didn't finish). In order to catch these more easily, add
an option to prefill the database with all tests at the very
beginning of the testing (in a new NOTRUN state) and use the
option in the VM tests.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:17:44 +02:00
Johannes Berg
63f83fac0d hwsim tests: Create results database in VM tests
Create a results.db in the output directory when running
the tests in a VM. To make that easier, create the tables
in the python script if they don't exist.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:16:28 +02:00
Johannes Berg
4a1b8901ab hwsim tests: Refactor test results reporting
Refactor the test reporting to treat the different results
(success/skip/failure) identically. This makes the timing
seem a bit longer, but cleans up the code which will allow
for adding more checks (e.g., on the captured data files)
later.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:03:14 +02:00
Jouni Malinen
a633397756 tests: Test dynamic BSS addition/removal
test_ap_bss_add_remove verifies hostapd behavior when BSSes are
added/removed in multi-BSS configuration.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 23:04:50 +02:00
Jouni Malinen
a1ddc79adf tests: Remove separate error/failed text file
The new sqlite database for results deprecated this.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:39:09 +02:00
Jouni Malinen
904b6e7e75 tests: Remove separate results text file
The new sqlite database for results deprecated this.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:36:26 +02:00
Jouni Malinen
b72c300267 tests: Use a symlink for default LOGDIR between scripts
This allows run-tests.py to use the same logs/<date> default logdir as
start.sh which is quite convenient for manual test runs.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:27:57 +02:00
Jouni Malinen
353e5067ff tests: Split wpa_supplicant debug logs into per test case files
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 13:19:38 +02:00
Jouni Malinen
c9aa430896 tests: Split run-tests.py logger info into per test case files
The run-tests.py -l argument does not take an argument value anymore.
Instead, debug output is directed to a separate file <test>.log for each
test case.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 12:51:08 +02:00
Jouni Malinen
e955226208 tests: Remove the 'test_' prefix from test module names
This is unnecessary extra complexity for user, so use the 'test_' prefix
only internally within the python scripts and file names.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:49:58 +02:00
Jouni Malinen
4cd8343f4d tests: Remove the 'test_' prefix from test names
This is unnecessary extra complexity for user and reports, so use the
'test_' prefix only internally within the python scripts.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:43:45 +02:00
Jouni Malinen
6a5b4b1a66 tests: Fix sqlite run column to be an integer
Commit 781b65cfbb ended up accidentally
changing this from an integer to a string. Fix this by not converting
the variable into a string.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
8f946511c9 hwsim tests: Reset at the end of a test
Resetting at the beginning causes the reset logging/tracing
data to leak from the previous test into the next, and the
last one being missed at all - reset at the end of each run
instead. Also reset before all tests just in case running a
test actually crashed the python script.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
0d5a54048b hwsim tests: Allow collecting dmesg
In addition to tracing, allow collecting dmesg. There's no
provision for actually looking at it and finding problems
in it yet though.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
0141fa5244 hwsim tests: Pass --logdir to run-tests.py
Instead of passing the log directory for each option
(-l, -r, -e, and -T) pass it once and make the other
options just take the filename (optionally, even).

This will also make it easier to extend later.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:07:52 +02:00
Johannes Berg
435e37df7c hwsim tests: Allow run-tests.py to start tracing
In order to get tracing per test, allow run-tests.py to start
and stop tracing per test case. This is implemented using a
python 'with' context so it starts/stops automatically at the
right spots.

Instead of starting global tracing, also use it from run-all.sh
and put the trace files into the log dir.

Note that this only works right if you use a separate log dir
for all test runs as the trace files aren't timestamped.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:47:44 +02:00
Johannes Berg
aee9446d0c hwsim tests: Pass commit as an argument
Rather than saving the current commit to a file, pass it
as an argument to run-tests.py.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:29:00 +02:00
Johannes Berg
f72434b43e tests: Refactor run-all.sh
Reuse the code rather than duplicating the implementation
of starting the tests. To make that easier, allow passing
multiple modules with -f to run-tests.py.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:58:23 +02:00
Johannes Berg
0b1faa8bf1 tests: Use argparse module with hwsim
Instead of re-implementing a command-line parser, use the
argparse module.

The only real change (I hope) is that the test module must
now be given to the -f option without the .py suffix.

Also, --help now works, and if a test module/test name is
given that doesn't exist, the valid list is printed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:58:23 +02:00
Johannes Berg
2f3a3edd5d tests: Import sqlite3 only if used
Might help some systems that don't have/need it.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:20:07 +02:00
Johannes Berg
781b65cfbb tests: Use proper SQL construction in hwsim test reporting
Don't construct SQL strings on the fly but instead use the
argument placeholders.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:19:07 +02:00
Johannes Berg
b3e2ce3bae tests: Do not require wpaspy installation
Since the scripts must be run from the source directory to
find the tests to run, they can use a relative path to the
wpaspy module instead of requiring it to be installed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-27 18:28:28 +02:00
Jouni Malinen
f3be6eea9b tests: Allow test case descriptions to be written into database
"run-tests.py -S <db file> -L" can now be used to update a database
table with the current set of test cases and their descriptions.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-27 12:41:10 +02:00
Jouni Malinen
b74b7e87bb tests: Add support for sqlite results database
This is more convenient to use directly than going through the
text-based results file.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-27 10:04:47 +02:00
Jouni Malinen
d93a240731 tests: Stop test run on NOTE command failure
There is no point trying to go through a test case if the NOTE command
to write TEST-START entry does not succeed. This avoids some excessive
waits on buildbot trying to forcefully kill the programs on its timeout
if wpa_supplicant gets stuck waiting for something (like the current
issue with libnl events and commands having a chance of hitting a
blocking wait on netlink messages).

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 21:38:02 +03:00
Jouni Malinen
f9174c3b4e tests: Flush stdout after each status line print
This makes it more convenient to follow live test execution status.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 20:42:37 +03:00
Jouni Malinen
0d84c400f9 tests: Add mechanism for test cases to indicate a skipped run
This can be used by test cases that depend on some external component
that may not always be available to indicate clearly that a test case
was skipped rather than passed or failed.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:21:42 +03:00
Jouni Malinen
9b00f24a76 tests: run-tests.py -L to print out list of test cases
This can be used to get a summary of all the available test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:11:48 +03:00
Jouni Malinen
48d8a43a28 tests: Adjust debug verbosity levels to get rid of extra info
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:07:01 +03:00
Jouni Malinen
d85bb75d47 tests: Make run-tests.py command argument parser more flexible
This allow -e, -r, and -f arguments to be given in any order.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-28 18:35:17 +03:00
Jouni Malinen
3b7475d365 tests: Write test case result summary into a file
This can be used to build statistics from test execution without having
to go through number of log files.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-28 18:31:00 +03:00
Jouni Malinen
1fc114a16a tests: Add test duration and time to PASS/FAIL reports
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-28 18:20:32 +03:00
Jouni Malinen
c548fb27c8 tests: Save debug log to a file and clean up stdout status
This makes it easier to figure out what failed and allows builbot to
integrate multiple logs and state information about the test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-08-25 20:15:54 +03:00
Jouni Malinen
789959c493 tests: Continue even if device reset fails
This may provide better testing coverage by allowing temporary issues
during the test cycle to be ignored.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-08-24 21:16:04 +03:00
Jouni Malinen
3ed2814a1e tests: Use logger.info() instead of print to get one stream
print and logger.info() were directing output to different locations
(stdout and stderr, respectively) which resulted in buildbot showing
reordered entries. Use logger consistently to avoid that.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-08-24 19:48:04 +03:00
Jouni Malinen
4896ddaea1 tests: Check for exceptions during TEST-START/STOP
It looks like the NOTE commands can time out in some cases. Avoid
stopping the test run in such a case to get more coverage if this is a
temporary issue.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-08-24 19:41:08 +03:00
Jouni Malinen
0fa28afe35 tests: Use wpa_supplicant global control interface for P2P
This allows more consistent interface to be used regardless of which
P2P driver design is used (especially for P2P management operations
over netdev vs. dedicated P2P_DEVICE).

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-07-01 00:36:33 +03:00
Jouni Malinen
f59a10d405 tests: Generate a tarball of test results if a test fails
This makes it easier to collect failure data from test servers for
further analysis.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-04-26 13:48:48 +03:00
Jouni Malinen
6b67117a34 tests: Stop wpa_supplicant before hostapd
Stopping the AP first was not ideal for the test cases since it could
result in wpa_supplicant trying to connect back and start a scan at the
end of a test case and cause problems for the following test case that
tried to scan in the beginning while the previously started scan was
still in progress.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-04-01 00:59:50 +03:00
Jouni Malinen
62d58f7ae3 tests: Use FLUSH command to clear wpa_supplicant state
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-03-31 20:32:33 +03:00
Jouni Malinen
1a8e4761da tests: Use a more compact summary of test results
Do not print the potentially long list of passed test cases. In case of
failure(s), make sure the failed test list is the last item in the
report.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-03-31 18:10:17 +03:00
Jouni Malinen
7585f349c7 tests: Add command line argument -f for specifying a test file
This can be used to run all test cases from a single test file.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-03-31 18:05:42 +03:00
Jouni Malinen
63a020ead8 tests: Use a single set of scripts for running both P2P and AP tests
This removes the unnecessary separation of P2P (no hostapd) and AP
tests. The same scripts can be used to prepare for these tests and to
execute the tests.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-03-31 18:05:31 +03:00
Renamed from tests/hwsim/run-ap-tests.py (Browse further)