Commit graph

56 commits

Author SHA1 Message Date
Jouni Malinen
31a3d4c234 tests: Recognize more test SKIP reasons
Signed-off-by: Jouni Malinen <j@w1.fi>
2020-01-26 16:03:31 +02:00
Johannes Berg
f124367afb tests: parallel-vm: allow running without curses
Allow running without curses, in which case the log is simply written to
stdout instead of a file. This is useful for automated (but parallel)
testing. Note that in most cases, you'd want to specify --debug, and so
I added a .rstrip() there on the lines to clean that up a bit.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2020-01-20 21:17:24 +02:00
Jouni Malinen
a6ba5cca45 tests: Allow more VMs to be started in parallel
Check the number of CPUs to determine how many VMs can be started in
parallel.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 20:31:33 +02:00
Jouni Malinen
c64b6f62cd tests: Use python selector in the parallel-vm.py main loop
This gets rid of the loop that was polling for things to do every 0.25
seconds and instead, reacts to any data from VMs as soon as it becomes
available. This avoids unnecessary operations when no new data is
available and avoids unnecessary waits when new data becomes available
more quickly.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 17:12:34 +02:00
Jouni Malinen
fd0465b85e tests: Print a summary of SKIP reasons
This makes it easier to determine reasons for test cases being skipped.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 16:03:50 +02:00
Jouni Malinen
179279eb34 tests: Move VM starting delay control into parallel-vm.py
This is more efficient since we can now start only the necessary number
of VMs instead of always forcing all VMs to start with one second delay.
This can also control the starting delay by keeping at most two VMs
starting at a time instead of using the hardcoded one second wait for
each consecutive VM.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 16:03:50 +02:00
Jouni Malinen
9dc07a1d55 tests: Fix 'unexpected exit' detection in parallel-vm.py
Now that parallel-vm.py is actually stopping VMs as soon as they are not
needed for retries, it is not really an unexpected exit to see a VM exit
while test cases remain in the queue as long as at least that many VMs
remain running. Get rid of confusing 'unexpected exit' status from the
UI in such cases.

Fixes: 4aaddecdd8 ("tests: Handle test retries through the same queue")
Signed-off-by: Jouni Malinen <j@w1.fi>
2019-11-23 17:09:56 +02:00
Pradeep Kumar Chitrapu
0a809529d0 tests: HE AP Functionality
Signed-off-by: Pradeep Kumar Chitrapu <pradeepc@codeaurora.org>
2019-10-25 19:29:53 +03:00
Jouni Malinen
4aaddecdd8 tests: Handle test retries through the same queue
This removes the separate rerun step from the parallel-vm.py processing
and instead, simply requeues the failed test cases into the same queue
that is used for the initial run. This is simpler and more efficient
since reruns start as soon as any VM is ready for processing them
instead of having to wait for all VMs to complete the first round.
Furthermore, this allows VMs to be stopped sooner when no more test
cases remain and that is helpful especially with the time travel patches
that make the wait-for-next-test step in the VM use all available CPU.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-07-27 20:19:28 +03:00
Jouni Malinen
fab49f6145 tests: Python coding style cleanup (pylint3 bad-whitespace)
Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-03-16 18:52:09 +02:00
Jouni Malinen
a2d300765b tests: Make parallel-vm.py log easier to use for debugging failures
Print a list of full paths to log files from failed test cases both the
parallel.log and stdout so that they can be easily opened for analysis.
In addition, renumber the VM lines in the <timestamp>-parallel.log to
match the i+1 numbering used in the log directories and UI that is
tracking test execution.

Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-03-15 21:10:00 +02:00
Masashi Honma
432496cdd8 tests: Switch default python version to 3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-17 17:26:01 +02:00
Masashi Honma
51c83edfca tests: Catch only specific exceptions
This allows unexpected cases to terminate parallel-vm.py without being
hidden by the exception handler.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-11 12:42:52 +02:00
Masashi Honma
45d3e2edbd tests: Explicitly flush stdin for python3
Without this flush(), test does not run.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
3069be8fe3 tests: Encode VM input for python3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
689a956049 tests: Decode VM output for python3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
0da6d93dbf tests: Change handling of reading non blocked empty stream for python3
The result of reading non blocked empty stream is different between
python2 and 3. The python2 sends "[Errno 11] Resource temporarily
unavailable" exception. The python3 could read "None" without
exception, so handle this "None" case as well.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Johannes Berg
e339511007 tests: Optionally start telnet server inside VMs
If telnetd is installed and --telnet <port> is passed on the
vm-run.sh command line, start a telnet server (directly connected
to bash, no login) inside the VM(s) to be able to look into them
when something is wrong. Use a user network in qemu with a single
host forward from the specified port for this, listening only on
'localhost'.

Please note that this provides unauthenticated access to the guest
system from anything that can open a TCP connection on the host system.
The guess system does have access to reading all files on the host that
the user account running kvm has access to (and even write access if the
default ROTAG ,readonly parameter is cleared). In other words, this
option should not be used on any multiuser systems where kvm is run
under user accounts that are not dedicated for testing purposes (i.e.,
do not have access to any files that should not be readable to
everyone).

This needs CONFIG_VIRTIO_NET=y in the guest kernel.

For parallel-vm.py, the --telnet argument specifies the base port
and each VM index (0, 1, ...) is added to it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-02-05 14:26:36 +02:00
Masashi Honma
89896c0008 tests: Use python3 compatible print statement
This patch is made by using 2to3 command.

$ find . -name *.py | xargs 2to3 -f print -w -n

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-01-26 12:37:25 +02:00
Jouni Malinen
43d174b8e2 tests: Automatic channel selection, HT scan, and DFS
Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-23 17:25:11 +02:00
Jouni Malinen
e4ce86f98b tests: Add dfs_etsi to the long_tests list
Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-08 15:33:31 +02:00
Jouni Malinen
326720c1db tests: Split proxyarp test cases into IPv4 and IPv6 parts
This is useful for now since the IPv6 support for proxyarp is not yet
included in the upstream kernel. This allows the IPv4 test cases to pass
with the current upstream kernel while allowing the IPv6 test cases to
report SKIP instead of FAIL.

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-01-29 14:32:17 +02:00
Jouni Malinen
803d01901b tests: DFS CAC functionality on channel 104 HT40-
Signed-off-by: Jouni Malinen <j@w1.fi>
2016-03-27 20:30:40 +03:00
Jouni Malinen
45e8a45bb5 tests: Show vm-run.sh command line for re-running failure sequences
Now that vm-run.sh supports a long list of test cases without crashing
the VM kernel, there is no need to use the "parallel-vm.py -1 1 <tests>"
workaround. Print the re-run example commands with vm-run.sh instead. In
addition, add the --long argument if it was specified for the test run
to avoid skipping test cases in the re-run case.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-11-30 19:42:56 +02:00
Johannes Berg
680ce356c0 tests: Honor HWSIM_TEST_LOG_DIR variable in VM runs
If /tmp has a relatively small size limit, or multiple people run the
tests on the same machine, using the same output directory can easily
cause problems.

Make the test framework honor the new HWSIM_TEST_LOG_DIR environment
variable to make it easier to avoid those problems.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-27 21:11:53 +02:00
Jouni Malinen
24b7f2822e tests: More WPS/HTTP test coverage
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-08-31 17:25:20 +03:00
Ilan Peer
ab653ed2f6 tests: Add P2P GO frequency change policy tests
Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2015-08-04 19:15:44 +03:00
Jouni Malinen
9f622398e4 tests: Print a command line to re-run failed test sequence
This makes it easier to test whether a specific test sequence results in
a reproducible test failure.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-06-18 21:39:24 +03:00
Jouni Malinen
2272f5aada tests: WPS AP iteration on no Selected Registrar and error case
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-04-24 17:55:33 +03:00
Jouni Malinen
ce591c74bf tests: Make parallel-vm.py FAIL parser more robust
It looks like it was possible to receive an incomplete FAIL line and
break out from test execution due to a parsing error. Handle this more
robustly and log the error.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-26 22:20:35 +02:00
Jouni Malinen
de52a2e259 tests: Fix parallel-vm.py test case name parsing
The 'params' argument was not used at all. Use it as an alternative
means for setting the list of test cases to execute.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:12:01 +02:00
Jouni Malinen
4f76eb81e9 tests: Add --valgrind argument to parallel-vm.py
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:09:23 +02:00
Jouni Malinen
68baa82cd2 tests: Add --short option for parallel-vm.py
This can be used to filter out test cases that take significantly longer
time to execute (15 seconds or longer). While this reduces testing
coverage, this can be useful to get a pretty quick coverage in
significantly faster time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-07 12:23:39 +02:00
Johannes Berg
d2002f8334 tests: Allow running with arbitrary working directory
It's somewhat annoying that you can only run parallel-vm.py as
./parallel-vm.py, not from elsewhere by giving the full path,
so fix that by resolving the paths correctly in the scripts where
needed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:07 +02:00
Johannes Berg
4f06261bd9 tests: parallel-vm.py: Use argparse module
Instead of hand-writing a (positional) parser, use the argparse module.
This also gets us nice help output.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:04 +02:00
Jouni Malinen
50c3825c6d tests: DFS with wpa_supplicant AP mode
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-05 17:25:16 +02:00
Jouni Malinen
00c35673fd tests: Allow --shuffle-tests to be used with parallel-vm.py
This can be used to get more random sequence of test case execution
into use when running multiple VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-03 09:47:25 +02:00
Jouni Malinen
f77d6d4bd8 tests: Mesh authentication failure events
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-02-08 13:25:35 +02:00
Jouni Malinen
915135e055 tests: hostapd OOM loop tests for WPA2-PSK and WPA2-EAP connection
These verify number of previously uncovered error paths.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-01-29 22:33:25 +02:00
Jouni Malinen
802bf82482 tests: Add optional -1 argument to parallel-vm.py
This can be used to skip rerunning of failed test cases
(e.g., with "./parallel-vm.py 1 -1 <test case>").

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-17 11:28:46 +02:00
Jouni Malinen
ed8a1fa496 tests: Limit number of failed test cases in the curses output
It was possible for the scr.addstr() operations to fail and terminate
parallel-vm.py if the number of failed test cases increased beyond what
fits on the screen.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-26 13:21:00 +02:00
Jouni Malinen
1f4de34ec1 tests: Maintain test result counters through the run
This avoids having to go through the full output buffer from each
VM for each update during a test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
8041102874 tests: Process VM stdout output in full lines
Merge partial lines together before processing them in parallel-vm.py.
This avoids issues in cases where the stdout read gets split into pieces
that do not include the full READY/PASS/FAIL/SKIP information. In
addition, strip unnecessary whitespace (mainly, '\r') from the log
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
d3fb9c1403 tests: Add debug logging for parallel-vm.py
parallel-vm.log is now written with details of test execution steps and
results. This makes it easier to debug if something goes wrong in VM
monitoring. The --debug option can be used to enable verbose debugging.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:36:51 +02:00
Jouni Malinen
3eb1db0377 tests: Retry failed cases automatically in parallel VM run
parallel-vm.py is now retrying failed cases once at the end of the run.
If all the failed test cases passed on the second attempt, that is noted
in the summary output. Results are also indicated as the exit value from
the run: 0 = all cases passed on first run, 1 = some cases failed once,
but everything passed after one retry, 2 = some cases failed did not
succeed at all.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 22:25:29 +02:00
Jouni Malinen
a628eeb42b tests: Add more long duration test cases to parallel-vm.py list
This adds the remaining test cases that took more than 15 seconds to run
into the list of test cases to run at the beginning of the execution to
avoid these being left at the end when only some of the VMs may be
running.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 20:34:13 +02:00
Jouni Malinen
a69d0cdc72 tests: Merge grpform_cred_ready_timeout test cases into a single one
These test cases had a long 120 seconds wait for the GO Negotiation
initiator to time out. This can be done using two devices in parallel to
save two minutes from total test execution time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 15:54:30 +02:00
Jouni Malinen
5554fbef46 tests: Make parallel VM start a bit more resource friendly
Wait one second between each kvm start to avoid hitting large number of
processes trying to start in parallel. This allows the VMs to be started
more efficiently for parallel-vm.py runs with large number of VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 22:04:54 +02:00
Jouni Malinen
2ffbf34db0 tests: Optimize test case execution order for multiple VMs
Move test cases with long duration to the beginning as an optimization
to avoid last part of the test execution running a long duration test
case on a single VM while all other VMs have already completed their
work.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 18:20:15 +02:00
Jouni Malinen
e0cccf26a4 tests: Add --codecov support to parallel-vm.py
This allows code coverage report to be generated must faster with the
help of parallel VMs executing test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 01:24:28 +02:00