Commit graph

16 commits

Author SHA1 Message Date
Jouni Malinen
ed8a1fa496 tests: Limit number of failed test cases in the curses output
It was possible for the scr.addstr() operations to fail and terminate
parallel-vm.py if the number of failed test cases increased beyond what
fits on the screen.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-26 13:21:00 +02:00
Jouni Malinen
1f4de34ec1 tests: Maintain test result counters through the run
This avoids having to go through the full output buffer from each
VM for each update during a test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
8041102874 tests: Process VM stdout output in full lines
Merge partial lines together before processing them in parallel-vm.py.
This avoids issues in cases where the stdout read gets split into pieces
that do not include the full READY/PASS/FAIL/SKIP information. In
addition, strip unnecessary whitespace (mainly, '\r') from the log
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
d3fb9c1403 tests: Add debug logging for parallel-vm.py
parallel-vm.log is now written with details of test execution steps and
results. This makes it easier to debug if something goes wrong in VM
monitoring. The --debug option can be used to enable verbose debugging.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:36:51 +02:00
Jouni Malinen
3eb1db0377 tests: Retry failed cases automatically in parallel VM run
parallel-vm.py is now retrying failed cases once at the end of the run.
If all the failed test cases passed on the second attempt, that is noted
in the summary output. Results are also indicated as the exit value from
the run: 0 = all cases passed on first run, 1 = some cases failed once,
but everything passed after one retry, 2 = some cases failed did not
succeed at all.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 22:25:29 +02:00
Jouni Malinen
a628eeb42b tests: Add more long duration test cases to parallel-vm.py list
This adds the remaining test cases that took more than 15 seconds to run
into the list of test cases to run at the beginning of the execution to
avoid these being left at the end when only some of the VMs may be
running.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 20:34:13 +02:00
Jouni Malinen
a69d0cdc72 tests: Merge grpform_cred_ready_timeout test cases into a single one
These test cases had a long 120 seconds wait for the GO Negotiation
initiator to time out. This can be done using two devices in parallel to
save two minutes from total test execution time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 15:54:30 +02:00
Jouni Malinen
5554fbef46 tests: Make parallel VM start a bit more resource friendly
Wait one second between each kvm start to avoid hitting large number of
processes trying to start in parallel. This allows the VMs to be started
more efficiently for parallel-vm.py runs with large number of VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 22:04:54 +02:00
Jouni Malinen
2ffbf34db0 tests: Optimize test case execution order for multiple VMs
Move test cases with long duration to the beginning as an optimization
to avoid last part of the test execution running a long duration test
case on a single VM while all other VMs have already completed their
work.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 18:20:15 +02:00
Jouni Malinen
e0cccf26a4 tests: Add --codecov support to parallel-vm.py
This allows code coverage report to be generated must faster with the
help of parallel VMs executing test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 01:24:28 +02:00
Jouni Malinen
b54f697019 tests: Remove test names from VM command with parallel-vm.py
There is no need to pass the test case names to the VMs when using
parallel-vm.py. Removing those from the command line helps in avoiding
kernel panic if maximum number of kernel parameters limit is hit.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-07 15:33:22 +02:00
Jouni Malinen
077f13c3e1 tests: Move parallel-vm.py test control to the host
This allows all VMs to be used at the end of a test sequence by
assigning test cases to VMs based on which VM is available for a new
test case rather than splitting the full task at the beginning and
potentially getting stuck with the last VM running long test cases for
significantly longer than another VM that gets shorter duration tests
assigned to it.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
44ac019c96 tests: Report kernel panic clearly in parallel-vm.py
Previously, it was possible for a kernel panic to be missed since the
only sign of it in stdout was reduced number of passed test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:34:54 +02:00
Jouni Malinen
7e6942253c tests: Use a single timestamp in all vm-run.sh instances
This avoids possible mismatches in directory and log file timestamps if
the UNIX timestamp (seconds) changes during the startup sequence.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:24:18 +02:00
Jouni Malinen
71c41d45af tests: Provide more details of parallel testing with curses UI
This extends parallel-vm.py to show more details about testing progress
from each VM.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-19 11:19:49 +03:00
Jouni Malinen
a87e123a96 tests: Add Python-version of parallel-vm.sh
This is a more advanced version of the simple parallel-vm.sh script.
Status of each VM is printed out during the test and results are
provided in more convenient format in the end.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-10-12 21:49:36 +03:00