Commit graph

28 commits

Author SHA1 Message Date
Jouni Malinen
2272f5aada tests: WPS AP iteration on no Selected Registrar and error case
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-04-24 17:55:33 +03:00
Jouni Malinen
ce591c74bf tests: Make parallel-vm.py FAIL parser more robust
It looks like it was possible to receive an incomplete FAIL line and
break out from test execution due to a parsing error. Handle this more
robustly and log the error.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-26 22:20:35 +02:00
Jouni Malinen
de52a2e259 tests: Fix parallel-vm.py test case name parsing
The 'params' argument was not used at all. Use it as an alternative
means for setting the list of test cases to execute.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:12:01 +02:00
Jouni Malinen
4f76eb81e9 tests: Add --valgrind argument to parallel-vm.py
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:09:23 +02:00
Jouni Malinen
68baa82cd2 tests: Add --short option for parallel-vm.py
This can be used to filter out test cases that take significantly longer
time to execute (15 seconds or longer). While this reduces testing
coverage, this can be useful to get a pretty quick coverage in
significantly faster time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-07 12:23:39 +02:00
Johannes Berg
d2002f8334 tests: Allow running with arbitrary working directory
It's somewhat annoying that you can only run parallel-vm.py as
./parallel-vm.py, not from elsewhere by giving the full path,
so fix that by resolving the paths correctly in the scripts where
needed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:07 +02:00
Johannes Berg
4f06261bd9 tests: parallel-vm.py: Use argparse module
Instead of hand-writing a (positional) parser, use the argparse module.
This also gets us nice help output.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:04 +02:00
Jouni Malinen
50c3825c6d tests: DFS with wpa_supplicant AP mode
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-05 17:25:16 +02:00
Jouni Malinen
00c35673fd tests: Allow --shuffle-tests to be used with parallel-vm.py
This can be used to get more random sequence of test case execution
into use when running multiple VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-03 09:47:25 +02:00
Jouni Malinen
f77d6d4bd8 tests: Mesh authentication failure events
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-02-08 13:25:35 +02:00
Jouni Malinen
915135e055 tests: hostapd OOM loop tests for WPA2-PSK and WPA2-EAP connection
These verify number of previously uncovered error paths.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-01-29 22:33:25 +02:00
Jouni Malinen
802bf82482 tests: Add optional -1 argument to parallel-vm.py
This can be used to skip rerunning of failed test cases
(e.g., with "./parallel-vm.py 1 -1 <test case>").

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-17 11:28:46 +02:00
Jouni Malinen
ed8a1fa496 tests: Limit number of failed test cases in the curses output
It was possible for the scr.addstr() operations to fail and terminate
parallel-vm.py if the number of failed test cases increased beyond what
fits on the screen.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-26 13:21:00 +02:00
Jouni Malinen
1f4de34ec1 tests: Maintain test result counters through the run
This avoids having to go through the full output buffer from each
VM for each update during a test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
8041102874 tests: Process VM stdout output in full lines
Merge partial lines together before processing them in parallel-vm.py.
This avoids issues in cases where the stdout read gets split into pieces
that do not include the full READY/PASS/FAIL/SKIP information. In
addition, strip unnecessary whitespace (mainly, '\r') from the log
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
d3fb9c1403 tests: Add debug logging for parallel-vm.py
parallel-vm.log is now written with details of test execution steps and
results. This makes it easier to debug if something goes wrong in VM
monitoring. The --debug option can be used to enable verbose debugging.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:36:51 +02:00
Jouni Malinen
3eb1db0377 tests: Retry failed cases automatically in parallel VM run
parallel-vm.py is now retrying failed cases once at the end of the run.
If all the failed test cases passed on the second attempt, that is noted
in the summary output. Results are also indicated as the exit value from
the run: 0 = all cases passed on first run, 1 = some cases failed once,
but everything passed after one retry, 2 = some cases failed did not
succeed at all.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 22:25:29 +02:00
Jouni Malinen
a628eeb42b tests: Add more long duration test cases to parallel-vm.py list
This adds the remaining test cases that took more than 15 seconds to run
into the list of test cases to run at the beginning of the execution to
avoid these being left at the end when only some of the VMs may be
running.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 20:34:13 +02:00
Jouni Malinen
a69d0cdc72 tests: Merge grpform_cred_ready_timeout test cases into a single one
These test cases had a long 120 seconds wait for the GO Negotiation
initiator to time out. This can be done using two devices in parallel to
save two minutes from total test execution time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 15:54:30 +02:00
Jouni Malinen
5554fbef46 tests: Make parallel VM start a bit more resource friendly
Wait one second between each kvm start to avoid hitting large number of
processes trying to start in parallel. This allows the VMs to be started
more efficiently for parallel-vm.py runs with large number of VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 22:04:54 +02:00
Jouni Malinen
2ffbf34db0 tests: Optimize test case execution order for multiple VMs
Move test cases with long duration to the beginning as an optimization
to avoid last part of the test execution running a long duration test
case on a single VM while all other VMs have already completed their
work.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 18:20:15 +02:00
Jouni Malinen
e0cccf26a4 tests: Add --codecov support to parallel-vm.py
This allows code coverage report to be generated must faster with the
help of parallel VMs executing test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 01:24:28 +02:00
Jouni Malinen
b54f697019 tests: Remove test names from VM command with parallel-vm.py
There is no need to pass the test case names to the VMs when using
parallel-vm.py. Removing those from the command line helps in avoiding
kernel panic if maximum number of kernel parameters limit is hit.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-07 15:33:22 +02:00
Jouni Malinen
077f13c3e1 tests: Move parallel-vm.py test control to the host
This allows all VMs to be used at the end of a test sequence by
assigning test cases to VMs based on which VM is available for a new
test case rather than splitting the full task at the beginning and
potentially getting stuck with the last VM running long test cases for
significantly longer than another VM that gets shorter duration tests
assigned to it.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
44ac019c96 tests: Report kernel panic clearly in parallel-vm.py
Previously, it was possible for a kernel panic to be missed since the
only sign of it in stdout was reduced number of passed test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:34:54 +02:00
Jouni Malinen
7e6942253c tests: Use a single timestamp in all vm-run.sh instances
This avoids possible mismatches in directory and log file timestamps if
the UNIX timestamp (seconds) changes during the startup sequence.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:24:18 +02:00
Jouni Malinen
71c41d45af tests: Provide more details of parallel testing with curses UI
This extends parallel-vm.py to show more details about testing progress
from each VM.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-19 11:19:49 +03:00
Jouni Malinen
a87e123a96 tests: Add Python-version of parallel-vm.sh
This is a more advanced version of the simple parallel-vm.sh script.
Status of each VM is printed out during the test and results are
provided in more convenient format in the end.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-12 21:49:36 +03:00