Commit graph

68 commits

Author SHA1 Message Date
Jouni Malinen
56dfb604a4 Revert "tests: vm: Output everything on console"
This reverts commit be9fe3d8af. While I
did manage to complete multiple test runs without failures, it looks
like this change increases full test run duration by about 30 seconds
when using seven VMs. The most visible reason for that seems to be in
"breaking" active scanning quite frequently with the Probe Response
frame coming out about 40 ms (or more) after the Probe Request frame
which is long enough for the station to already have left the channel.

Since this logging change is not critical, it is simplest to revert it
for now rather than make changes to huge number of test cases to allow
more scan attempts to be performed before timing out.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-12-30 21:51:52 +02:00
Johannes Berg
be9fe3d8af tests: vm: Output everything on console
When running tests, make printk put all messages, including debug
messages, onto the serial console to go into the console file.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-12-24 21:27:24 +02:00
Johannes Berg
05736da8c6 tests: vm: Honor EPATH in uevent.sh
This is useful for testing CRDA since it means you can use EPATH to
redirect the test scripts to a different crda binary.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-12-18 00:24:51 +02:00
Jouni Malinen
45e8a45bb5 tests: Show vm-run.sh command line for re-running failure sequences
Now that vm-run.sh supports a long list of test cases without crashing
the VM kernel, there is no need to use the "parallel-vm.py -1 1 <tests>"
workaround. Print the re-run example commands with vm-run.sh instead. In
addition, add the --long argument if it was specified for the test run
to avoid skipping test cases in the re-run case.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-11-30 19:42:56 +02:00
Johannes Berg
23fcfd6004 tests: Allow passing more arguments to vm-run.sh
The script is currently limited by the maximum kernel command line
length and if that's exceeded the kernel panics at boot. Fix this by
writing the arguments to a file and reading it in the VM.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-30 14:03:28 +02:00
Johannes Berg
680ce356c0 tests: Honor HWSIM_TEST_LOG_DIR variable in VM runs
If /tmp has a relatively small size limit, or multiple people run the
tests on the same machine, using the same output directory can easily
cause problems.

Make the test framework honor the new HWSIM_TEST_LOG_DIR environment
variable to make it easier to avoid those problems.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-27 21:11:53 +02:00
Johannes Berg
f2a438f51d tests: Print test name to kernel log buffer
Print the test name to the kernel log buffer so debugging
cross-test issues becomes easier.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-24 16:28:24 +02:00
Dan Williams
d7273180c6 tests: Fix VM tests for Fedora
Binaries aren't always in /usr/bin/, plus kvm is sometimes 'qemu-kvm'.

Signed-off-by: Dan Williams <dcbw@redhat.com>
2015-10-14 19:23:00 +03:00
Jouni Malinen
24b7f2822e tests: More WPS/HTTP test coverage
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-08-31 17:25:20 +03:00
Ilan Peer
ab653ed2f6 tests: Add P2P GO frequency change policy tests
Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2015-08-04 19:15:44 +03:00
Anton Nayshtut
41a256ecd9 tests: Add FST module tests
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-07-16 18:26:16 +03:00
Jouni Malinen
9f622398e4 tests: Print a command line to re-run failed test sequence
This makes it easier to test whether a specific test sequence results in
a reproducible test failure.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-06-18 21:39:24 +03:00
Michael Braun
54cf411f4c tests: Verify correct VLAN operation in multi-BSS multi-VLAN case
This adds hwsim test ap_vlan_iface_cleanup_multibss. It connects two
stations in different BSS but the same hostapd process. First both
stations are in VLAN 1, then they get reauthenticated into VLAN 2. Due
to the ordering of the stations moving around, this test checks that
bridge and tagged interface referencing counting is done globally, such
that the tagged interface is not removed too early and no bridge is
left over.

Signed-off-by: Michael Braun <michael-dev@fami-braun.de>
2015-06-14 13:40:50 +03:00
Jouni Malinen
2272f5aada tests: WPS AP iteration on no Selected Registrar and error case
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-04-24 17:55:33 +03:00
Jouni Malinen
ce591c74bf tests: Make parallel-vm.py FAIL parser more robust
It looks like it was possible to receive an incomplete FAIL line and
break out from test execution due to a parsing error. Handle this more
robustly and log the error.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-26 22:20:35 +02:00
Jouni Malinen
de52a2e259 tests: Fix parallel-vm.py test case name parsing
The 'params' argument was not used at all. Use it as an alternative
means for setting the list of test cases to execute.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:12:01 +02:00
Jouni Malinen
4f76eb81e9 tests: Add --valgrind argument to parallel-vm.py
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-14 12:09:23 +02:00
Jouni Malinen
68baa82cd2 tests: Add --short option for parallel-vm.py
This can be used to filter out test cases that take significantly longer
time to execute (15 seconds or longer). While this reduces testing
coverage, this can be useful to get a pretty quick coverage in
significantly faster time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-07 12:23:39 +02:00
Johannes Berg
d2002f8334 tests: Allow running with arbitrary working directory
It's somewhat annoying that you can only run parallel-vm.py as
./parallel-vm.py, not from elsewhere by giving the full path,
so fix that by resolving the paths correctly in the scripts where
needed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:07 +02:00
Johannes Berg
4f06261bd9 tests: parallel-vm.py: Use argparse module
Instead of hand-writing a (positional) parser, use the argparse module.
This also gets us nice help output.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-03-07 12:23:04 +02:00
Jouni Malinen
50c3825c6d tests: DFS with wpa_supplicant AP mode
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-03-05 17:25:16 +02:00
Jouni Malinen
00c35673fd tests: Allow --shuffle-tests to be used with parallel-vm.py
This can be used to get more random sequence of test case execution
into use when running multiple VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-03-03 09:47:25 +02:00
Jouni Malinen
8412dd52d1 tests: Increase default VM memory from 128M to 192M
It looks like the 128M default memory size for the hwsim test setup was
not large enough to cover all the needs anymore. Some of the test cases
using tshark could hit OOM with that size. Increase the default
allocation to 192M to avoid this type of issues.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-02-25 19:02:43 +02:00
Jouni Malinen
f77d6d4bd8 tests: Mesh authentication failure events
Signed-off-by: Jouni Malinen <j@w1.fi>
2015-02-08 13:25:35 +02:00
Jouni Malinen
915135e055 tests: hostapd OOM loop tests for WPA2-PSK and WPA2-EAP connection
These verify number of previously uncovered error paths.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-01-29 22:33:25 +02:00
Jouni Malinen
802bf82482 tests: Add optional -1 argument to parallel-vm.py
This can be used to skip rerunning of failed test cases
(e.g., with "./parallel-vm.py 1 -1 <test case>").

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-17 11:28:46 +02:00
Jouni Malinen
b0839232fa tests: Enable wpa_supplicant D-Bus support for hwsim tests
This allows automated testing of the wpa_supplicant D-Bus interface. The
instance controlling wlan0 registers with D-Bus if dbus-daemon was
started successfully. This is only used in VM testing, i.e., not when
run-tests.sh is used on the host system with D-Bus running for normal
system purposes.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-01-02 22:50:27 +02:00
Jouni Malinen
13113d127c tests: Filter out extra files from codecov reports
wpa_cli and hostapd_cli are not currently tested for code coverage, so
filter the files specific to those components away from the code
coverage reports. *_module_tests.c are not included in normal builds, so
drop them as well. In addition, drop the system header file (byteswap.h)
that gets somehow unnecessarily included in the reports for couple of
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-29 18:27:23 +02:00
Jouni Malinen
6532a84ed3 tests: Fix --codecov cases to find correct wpa_cli/hostapd_cli
It was possible for the separate builds to not include
wpa_cli/hostapd_cli in the default location. Make sure hostapd_cli gets
built for --codecov cases and update both WPACLI and HAPDCLI paths to
match the alternative location.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-29 15:49:04 +02:00
Jouni Malinen
ed8a1fa496 tests: Limit number of failed test cases in the curses output
It was possible for the scr.addstr() operations to fail and terminate
parallel-vm.py if the number of failed test cases increased beyond what
fits on the screen.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-26 13:21:00 +02:00
Jouni Malinen
1f4de34ec1 tests: Maintain test result counters through the run
This avoids having to go through the full output buffer from each
VM for each update during a test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
8041102874 tests: Process VM stdout output in full lines
Merge partial lines together before processing them in parallel-vm.py.
This avoids issues in cases where the stdout read gets split into pieces
that do not include the full READY/PASS/FAIL/SKIP information. In
addition, strip unnecessary whitespace (mainly, '\r') from the log
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
d3fb9c1403 tests: Add debug logging for parallel-vm.py
parallel-vm.log is now written with details of test execution steps and
results. This makes it easier to debug if something goes wrong in VM
monitoring. The --debug option can be used to enable verbose debugging.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:36:51 +02:00
Jouni Malinen
3eb1db0377 tests: Retry failed cases automatically in parallel VM run
parallel-vm.py is now retrying failed cases once at the end of the run.
If all the failed test cases passed on the second attempt, that is noted
in the summary output. Results are also indicated as the exit value from
the run: 0 = all cases passed on first run, 1 = some cases failed once,
but everything passed after one retry, 2 = some cases failed did not
succeed at all.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 22:25:29 +02:00
Jouni Malinen
a628eeb42b tests: Add more long duration test cases to parallel-vm.py list
This adds the remaining test cases that took more than 15 seconds to run
into the list of test cases to run at the beginning of the execution to
avoid these being left at the end when only some of the VMs may be
running.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 20:34:13 +02:00
Jouni Malinen
a69d0cdc72 tests: Merge grpform_cred_ready_timeout test cases into a single one
These test cases had a long 120 seconds wait for the GO Negotiation
initiator to time out. This can be done using two devices in parallel to
save two minutes from total test execution time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 15:54:30 +02:00
Jouni Malinen
5554fbef46 tests: Make parallel VM start a bit more resource friendly
Wait one second between each kvm start to avoid hitting large number of
processes trying to start in parallel. This allows the VMs to be started
more efficiently for parallel-vm.py runs with large number of VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 22:04:54 +02:00
Jouni Malinen
2ffbf34db0 tests: Optimize test case execution order for multiple VMs
Move test cases with long duration to the beginning as an optimization
to avoid last part of the test execution running a long duration test
case on a single VM while all other VMs have already completed their
work.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 18:20:15 +02:00
Jouni Malinen
f24489d94f tests: Use new scripts for vm-run.sh codecov
Now there is only one set of commands to maintain. The separate reports
for individual components have not been of much use in the past, so they
are dropped for now.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 16:41:59 +02:00
Jouni Malinen
e0cccf26a4 tests: Add --codecov support to parallel-vm.py
This allows code coverage report to be generated must faster with the
help of parallel VMs executing test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 01:24:28 +02:00
Jouni Malinen
19cbe06238 tests: Add ebtables rules for ProxyARP
Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2014-12-18 16:41:40 +02:00
Jouni Malinen
b54f697019 tests: Remove test names from VM command with parallel-vm.py
There is no need to pass the test case names to the VMs when using
parallel-vm.py. Removing those from the command line helps in avoiding
kernel panic if maximum number of kernel parameters limit is hit.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-07 15:33:22 +02:00
Johannes Berg
b67cdbc162 tests: Allow wext in configuration
Both the wpa_supplicant and kernel configuration need wext to
run the wext testcase, enable those in the default/example
configurations.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2014-11-28 23:02:29 +02:00
Javier Lopez
68157c0674 tests: Add test_wpas_mesh test cases
This wpa_supplicant tests include basic tests for:

- Mesh scan
- Mesh group add/remove
- Mesh peer connected/disconnected
- Add/Set/Remove to test mesh mode network
- Open mesh connectivity test
- Secure mesh connectivity test
- no_auto_peer

Signed-off-by: Jason Mobarak <x@jason.mobarak.name>
[no_auto_peer test by: Javier Cardona <javier@cozybit.com>
Signed-off-by: Javier Lopez <jlopex@gmail.com>
2014-11-20 16:31:12 +02:00
Jouni Malinen
077f13c3e1 tests: Move parallel-vm.py test control to the host
This allows all VMs to be used at the end of a test sequence by
assigning test cases to VMs based on which VM is available for a new
test case rather than splitting the full task at the beginning and
potentially getting stuck with the last VM running long test cases for
significantly longer than another VM that gets shorter duration tests
assigned to it.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-19 15:20:53 +02:00
Jouni Malinen
44ac019c96 tests: Report kernel panic clearly in parallel-vm.py
Previously, it was possible for a kernel panic to be missed since the
only sign of it in stdout was reduced number of passed test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:34:54 +02:00
Jouni Malinen
7e6942253c tests: Use a single timestamp in all vm-run.sh instances
This avoids possible mismatches in directory and log file timestamps if
the UNIX timestamp (seconds) changes during the startup sequence.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-11-16 22:24:18 +02:00
Ilan Peer
006a1c4d3b tests: Make vm-run.sh arguments non-positional
This was currently breaking parallel-run.*, as it was passing
--split num/num parameters (intended for rnu-tests.py)
to vm-run.sh which broke the --codecov and --timewrap options.

Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2014-11-01 17:55:23 +02:00
Ilan Peer
40c57fa88f tests: Update vm README
Update the code coverage documentation to also specify the
source base directory for the code coverage generation.

Signed-off-by: Ilan Peer <ilan.peer@intel.com>
2014-11-01 17:08:29 +02:00
Johannes Berg
1e3c7e82b5 tests: Disable support_p2p_device by default
When loading the hwsim module, disable support_p2p_device by default.
This will also become the default in the kernel, but until then it
makes sure it's not turned on by default.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2014-10-22 10:53:52 +03:00