Commit graph

102 commits

Author SHA1 Message Date
Jouni Malinen
c55799e54c tests: Check per-wiphy specific country code between test cases
This allows more accurate logging of failures related to the cfg80211
country=98 (intersection) case. This version is trying to give some more
time to allow the country code to clear, but that does not seem to be
sufficient with the current cfg80211 implementation for country=98 (but
might be for other cases).

The additional check for country=98 at the beginning of each test case
is an attempt to force cfg80211 to restore world roaming state with a
new association and disconnection at the station side detected after the
AP side has already stopped. This is needed after the Linux kernel
commit 113f3aaa81bd ("cfg80211: Prevent regulatory restore during STA
disconnect in concurrent interfaces").

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-01-01 11:46:50 +02:00
Vasyl Vavrychuk
2494bcef24 tests: Add option --dbus for logging D-Bus messages
Signed-off-by: Vasyl Vavrychuk <vvavrychuk@gmail.com>
2018-02-17 18:21:47 +02:00
Jouni Malinen
8171d70406 tests: Write TEST-START and TEST-STOP into kernel logs with timestamps
This makes it easier to synchronize log entries in the kernel log
(seconds from boot) and wpa_supplicant/hostapd (UNIX timestamp).

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-05-27 11:08:16 +03:00
Masashi Honma
d760db19b1 tests: Save the log of wmediumd
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2017-03-01 00:35:40 +02:00
Jouni Malinen
1edc31904a tests: Beacon request - active scan mode and many BSSs
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2017-01-03 19:53:03 +02:00
Jouni Malinen
bb5d761c2e tests: Do not attach wpa_supplicant monitor interface unnecessarily
There is no need to attach the monitor interface was events when issuing
only a single INTERFACES command.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2016-12-14 13:56:15 +02:00
Jouni Malinen
78a37200f8 tests: Reset wlan5 device before RELOG command
This moves the wpa_supplicant debug entries from the end of a test case
using a dynamically added wlan5 interface to the correct test case,
i.e., the test case that added the interface instead of whatever test
case happens to follow this.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2016-09-23 16:03:59 +03:00
Jonathan Afek
8efc83d4e7 tests: Add support for wlantest for remote hwsim tests
Use a monitor interface given in the command line that is not also a
station or an AP as a monitor running wlantest on the channel used by
the test. This makes all the tests that use wlantest available for
execution on real hardware on remote hosts.

Signed-off-by: Jonathan Afek <jonathanx.afek@intel.com>
2016-05-28 16:34:09 +03:00
Jouni Malinen
0663ae22ff tests: Do not use tabs for indentation
Be more consistent with indentation (always uses spaces in Python
files).

Signed-off-by: Jouni Malinen <j@w1.fi>
2016-04-25 00:19:40 +03:00
Janusz Dziedzic
2147b3a7f8 tests: Print traceback if test fails
This is useful in case we hit a problem in test code.

Signed-off-by: Janusz Dziedzic <janusz.dziedzic@tieto.com>
2016-04-24 20:16:30 +03:00
Johannes Berg
1a4f18d8f4 tests: run-tests: Print more details about NameError
If encountering a NameError, print the entire traceback so that
it's actually debuggable.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-28 00:21:20 +02:00
Johannes Berg
f2a438f51d tests: Print test name to kernel log buffer
Print the test name to the kernel log buffer so debugging
cross-test issues becomes easier.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-24 16:28:24 +02:00
Jouni Malinen
a302ee342c tests: Use sqlite3.Binary() with the log files
This is needed to avoid issues in some cases where 8-bit bytestrings may
be present in the otherwise text debug log.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-09-07 16:53:23 +03:00
Jouni Malinen
1f9427c9cd tests: Explicitly close hostapd global control socket on RELOG
Without this, the run-tests.py socket could have been left in attached
to receive all hostapd global events during a test case. This could hit
the limit of pending messages on the socket since there is nothing
clearing this socket during the execution of a test case. Fix this by
explicitly closing the socket after having completed the RELOG command.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-07-26 18:12:09 +03:00
Anton Nayshtut
41a256ecd9 tests: Add FST module tests
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-07-16 18:26:16 +03:00
Johannes Berg
d2002f8334 tests: Allow running with arbitrary working directory
It's somewhat annoying that you can only run parallel-vm.py as
./parallel-vm.py, not from elsewhere by giving the full path,
so fix that by resolving the paths correctly in the scripts where
needed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:07 +02:00
Jouni Malinen
cfb5576d93 tests: Rename dmesg files to a unique name
This allows multiple dmesg files to be saved if a test case is executed
multiple times similarly to the other logfiles.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-02-21 16:07:53 +02:00
Jouni Malinen
c4668009d0 tests: Remove unnecessary use of sudo from test cases
run-tests.py is running as root, so sudo does not need to be used
anymore from within each test case.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-02-07 15:37:13 +02:00
Eliad Peller
a918e48af1 tests: Add printk tracer to trace-cmd
Signed-off-by: Eliad Peller <eliadx.peller@intel.com>
2015-02-05 13:48:40 +02:00
Jouni Malinen
a66d2248a0 tests: Close wlan5 control interface monitor more explicitly
There were couple of common cases where the control interface for the
dynamic wpa_supplicant instance could have been left in attached state
until Python ends up cleaning up the instance. This could result in
issues if many monitor interface events were queued for that attached
socket. Make this less likely to cause issues by explicitly detaching
and closing control interfaces before moving to the next test case.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-18 17:13:55 +02:00
Jouni Malinen
81e787b750 tests: Convert test skipping to use exception
Instead of returning "skip" from the test function, raise the new
HwsimSkip exception to indicate a test case was skipped.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-08 22:43:47 +02:00
Jouni Malinen
51c5aeb42a tests: Convert "hwsim-SKIP" exception to use a custom class
This makes the design a bit cleaner for catching the exceptions.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-07 16:31:22 +02:00
Jouni Malinen
2ec82e67c1 tests: wpa_supplicant D-Bus interface
This adds new files with test cases to verify both the old and new
wpa_supplicant D-Bus interface.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-02 22:50:27 +02:00
Jouni Malinen
1942b68172 tests: Disable terminal echo for stdin control (-i)
This makes vm-run.sh output cleaner to parse for parallel-vm.py.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
6ac593ec47 tests: Optimize run-tests.py test case dict iteration
Use a single iteration through the module dictionary rather than
iterating over the keys and separately fetching values to get the
function.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 20:08:11 +02:00
Jouni Malinen
9363f5e065 tests: Optimize run-tests.py --prefill-tests startup time
It took significant part of the startup latency to prefill the database
with test cases due to the SQL COMMIT operation between each added row.
Move COMMIT to outside the loop to speed startup significantly.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-11-27 20:08:15 +02:00
Jouni Malinen
499c5e9dc6 tests: Follow test sequence from run-tests.py command line
It can be useful to specify an exact order of test cases and also to
allow the same test case to be run multiple times when the list of tests
is provided on the command line.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-11-27 19:48:41 +02:00
Jouni Malinen
f6d1c5d9f8 tests: Fix regression in START line total count
The previous changes to enable stdin control broke the previous case of
showing the total number of test cases in the START lines. Fix that by
using a separate variable for the total number of test casess instead of
using length of the list of remaining test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 22:02:08 +02:00
Jouni Malinen
0d7456e9fd tests: Add run-tests.py -i to control execution from stdin
The new -i command line argument can be used to control test case
execution from stdin to run-tests.py and vm-run.sh. This can be used,
e.g., to run multiple repeated test sequences in a virtual machine
without havign to restart the VM between each iteration.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
cb8cb33e55 tests: Allow run-tests.py to be run from the vm directory
Some of the paths in run-tests.py are hardcoded and could not handle
tests/hwsim/vm as the working directory. Modify the design enough to
allow ../run-tests.py -L to be used from the vm directory.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
5e3c0b02a7 tests: Make run-tests.py process test case selection arguments
This allows a list of matching test cases to be produced without having
to run the test cases. Previously, -L output included all defined test
cases regardless of what else was included on the command line.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:52 +02:00
Ilan Peer
3bfa5f10a9 tests: Add support for reading modules list from a file
The new run-tests.py argument "-l <file>" can now be used to specify the
test modules using a text file.

Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2014-10-26 23:27:26 +02:00
Jouni Malinen
ac69c33297 tests: Print failure exeception if logger output goes to file
This makes it more convenient to determine failure reasons during test
case development and VM testing.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-26 10:39:32 +02:00
Jouni Malinen
eb92d389a8 tests: Remove wlan5-related interfaces based on list
Instead of hardcoding reset_devs() to remove wlan5, remove all wlan*
interfaces renaming in the wpa_supplicant process to support the case of
dynamically added hwsim phy.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-20 12:52:25 +03:00
Jouni Malinen
5f797376f2 tests: Make RELOG command use global control interface
This allows RELOG to be used even if a wpa_supplicant process does not
have any interfaces configured.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-20 12:52:25 +03:00
Jouni Malinen
521b7e7925 tests: Verify global control interface before starting each test
This allows control interface issues to be caught in a bit more readable
way in the debug logs. In addition, dump pending monitor socket
information more frequently and within each test case in the log files
to make the output clearer and less likely to go over the socket buffer
limit.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-04-29 15:20:23 +03:00
Jouni Malinen
91f3cf6996 tests: Add option for running test cases that take a long time
run-tests.py now takes an optional --long parameter that can be used to
enable running of test cases that take a long time (multiple minutes).
By default, such test cases are skipped to avoid making the normal test
run take excessive amounts of time.

As an initial long test case, verify WPS PBC walk time expiration (two
minutes).

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-04-06 13:11:32 +03:00
Jouni Malinen
09e38c2fce tests: Add scripts to allow parallel execution of tests in VMs
"parallel-vm.sh <number of VMs> [arguments..]" can now be used to run
multiple VMs in parallel to speed up full test cycle significantly. In
addition, the "--split srv/total" argument used in this design would
also make it possible to split this to multiple servers to speed up
testing.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-03-24 23:37:42 +02:00
Jouni Malinen
9007391650 tests: Allow test cases to receive additional parameters
The optional third argument to the test case functions can now be used
to receive additional parameters from run-tests.py. As the initial
parameter, logdir value is provided so that test cases can use it to
review the debug logs from the test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-03-23 12:01:10 +02:00
Eduardo Abinader
b5008c2c2d tests: Fix trace parameter invocation
If trace-cmd command does not exist, run-tests.py could end up hanging
in a loop waiting for input. Fix this simply by checking whether the
trace-cmd command can be executed sucessfully and exiting the script if
not.

Signed-off-by: Eduardo Abinader <eduardo.abinader@openbossa.org>
2014-03-11 19:09:40 +02:00
Jouni Malinen
2ca47da4f7 tests: Fix trace-cmd output for host case
Both the output file path and the current working directory included the
log directory and this failed if log directory was not absolute (e.g.,
when using the default logs/current in the case a VM is not used).

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-03-11 17:04:33 +02:00
Jouni Malinen
8107d33361 tests: Fix the file comment to describe what this script does
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-02-21 20:25:25 +02:00
Roger Zanoni
040dc68e8c tests: Change the python interpreter directive
This makes the script check the environment for the current python
interpreter in use instead of assuming that the python executable points
to a python 2 interpreter.

Signed-off-by: Roger Zanoni <roger.zanoni@openbossa.org>
2014-02-21 20:23:56 +02:00
Jouni Malinen
f8949f5f8f tests: Reset hostapd state with FLUSH between test cases
This is similar to the FLUSH command for wpa_supplicant.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2014-01-27 22:08:13 +02:00
Jouni Malinen
aa5fcc558b tests: Show missing descriptions in more helpful format
Signed-hostap: Jouni Malinen <j@w1.fi>
2014-01-07 10:45:10 +02:00
Jouni Malinen
1d646f5ee1 tests: Write debug logs into the database for failed test cases
This makes it easier to build a web page for analyzing failures without
having to fetch the log files themselves from the test server.

Signed-hostap: Jouni Malinen <j@w1.fi>
2014-01-07 10:45:10 +02:00
Jouni Malinen
9489637b42 tests: Verify dynamic wpa_supplicant interface addition/removal
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-31 15:45:18 +02:00
Jouni Malinen
b1448311c2 tests: Show test counter on the START line
This makes it more convenient to figure out how far a test cycle is from
live logs.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-29 17:18:17 +02:00
Jouni Malinen
bcbd74155e tests: Mark kernel issues more clearly in the log file
It was previously not obvious from the <test case>.log file that a test
case was marked failed based on kernel issues. Make this very clear to
avoid wasting time on figuring out what caused the failure.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 18:11:07 +02:00
Jouni Malinen
848bb8dea9 tests: Use per-test case log/capture file from wlantest
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-17 21:47:06 +02:00