Commit graph

1101 commits

Author SHA1 Message Date
Jouni Malinen
dd859e20ac tests: Make p2p_channel test cases more robust
Wait for a CTRL-EVENT-REGDOM-CHANGE even before returning from
set_country() to avoid issues with test operations being executed before
all components have had chance to update their regulatory domain
information. Some of these test cases could fail under heavy load.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
1f4de34ec1 tests: Maintain test result counters through the run
This avoids having to go through the full output buffer from each
VM for each update during a test run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
8041102874 tests: Process VM stdout output in full lines
Merge partial lines together before processing them in parallel-vm.py.
This avoids issues in cases where the stdout read gets split into pieces
that do not include the full READY/PASS/FAIL/SKIP information. In
addition, strip unnecessary whitespace (mainly, '\r') from the log
lines.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
1942b68172 tests: Disable terminal echo for stdin control (-i)
This makes vm-run.sh output cleaner to parse for parallel-vm.py.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
2e0d48e5c8 tests: Do not print to stdout during test execution
This cleans up vm-run.sh output and makes information more easily
available from the *.log files.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:37:12 +02:00
Jouni Malinen
d3fb9c1403 tests: Add debug logging for parallel-vm.py
parallel-vm.log is now written with details of test execution steps and
results. This makes it easier to debug if something goes wrong in VM
monitoring. The --debug option can be used to enable verbose debugging.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-25 16:36:51 +02:00
Jouni Malinen
3eb1db0377 tests: Retry failed cases automatically in parallel VM run
parallel-vm.py is now retrying failed cases once at the end of the run.
If all the failed test cases passed on the second attempt, that is noted
in the summary output. Results are also indicated as the exit value from
the run: 0 = all cases passed on first run, 1 = some cases failed once,
but everything passed after one retry, 2 = some cases failed did not
succeed at all.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 22:25:29 +02:00
Jouni Malinen
a628eeb42b tests: Add more long duration test cases to parallel-vm.py list
This adds the remaining test cases that took more than 15 seconds to run
into the list of test cases to run at the beginning of the execution to
avoid these being left at the end when only some of the VMs may be
running.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 20:34:13 +02:00
Jouni Malinen
6d930a1eef tests: Optimize ap_qosmap test cases
Avoid unnecessary DATA_TEST_CONFIG calls and wlantest_cli invocations to
speed up the test cases. This drops ap_qosmap execution time from about
14 seconds to under 3 seconds.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 17:46:34 +02:00
Jouni Malinen
4905f1ebfe tests: Optimize multi_check() waits in ap_bss_add* and ap_multi_bss*
There is no need to wait for the longer fixed duration for each STA when
an event message or an earlier wait for another STA can be used instead.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 17:20:18 +02:00
Jouni Malinen
610cfff5d7 tests: Make scan_hidden* more robust
Clear cfg80211 BSS table more carefully after the scan_hidden* test
cases. At least scan_hidden_many could have left behind a hidden SSID
entry that could cause problems for following scan_bss_operations test
case.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 16:15:11 +02:00
Jouni Malinen
e0cfd223c0 tests: Fix mesh no_auto_peer=1 test cases to allow enough time for retry
If the initial Authentication frame was too early for the peer (i.e.,
NEW_PEER_CANDIDATE event arrived only after the Authentication frame),
wpas_mesh_open_no_auto and wpas_mesh_secure_no_auto test cases were
failing since they waited only for 10 seconds for the connection to be
completed while the retry timer was set to 10-20 seconds on the
authenticator side.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 13:10:22 +02:00
Jouni Malinen
9828b70ebd tests: Fix wpas_ctrl_data_test with Linux 3.19-rc1
The kernel commit 'packet: make packet_snd fail on len smaller than l2
header' started rejecting <= 14 octet raw packet socket transmission.
This test case was testing with 14 ocets and that is now rejected by the
kernel. While this may be a kernel side issue, use one octet longer test
data for now to avoid undesired FAIL cases in hwsim tests.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-23 12:27:14 +02:00
Jouni Malinen
b2eb3fe21c tests: Fix ap_ht_40mhz_intolerant_ap
The previous design did not actually break from the wait loop when the
AP changed back to 40 MHz channel and as such, ended up waiting the full
30 second time. Furthermore, the five second delay time for returning
back to 40 MHz was not sufficiently long to test behavior correctly
since the STA did not have any chances of returning the next coex report
before the AP had returned to 40 MHz. Increase the AP wait time to 15
seconds so that the once per 10 seconds OBSS scan from the STA gets in
before changing back to 40 MHz channel (after the 40 MHz intolerant AP
gets disabled).

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 22:07:54 +02:00
Jouni Malinen
6ac593ec47 tests: Optimize run-tests.py test case dict iteration
Use a single iteration through the module dictionary rather than
iterating over the keys and separately fetching values to get the
function.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 20:08:11 +02:00
Jouni Malinen
4cfe369b98 tests: Optimize stop.sh wait times
There is no need to wait for fixed 0.5 seconds for the processes to
exit.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 19:37:08 +02:00
Jouni Malinen
b350009f8c tests: Optimize start.sh execution time
There is no need to wait for one extra second to chown log files when
running as root which is always the case in VM execution.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 19:22:42 +02:00
Jouni Malinen
a69d0cdc72 tests: Merge grpform_cred_ready_timeout test cases into a single one
These test cases had a long 120 seconds wait for the GO Negotiation
initiator to time out. This can be done using two devices in parallel to
save two minutes from total test execution time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 15:54:30 +02:00
Jouni Malinen
9be2b811d4 tests: Mesh max peering limit
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 11:53:02 +02:00
Jouni Malinen
4b9481bcd8 tests: Remove unnecessary wrap_wpas_mesh_test() complexity
This function did not seem to do anything else apart from making it less
obvious that hwsim_utils.test_connectivity() is called.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-22 11:52:54 +02:00
Jouni Malinen
8b2600329f tests: Mesh beacon_int parameter
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 23:45:29 +02:00
Jouni Malinen
5554fbef46 tests: Make parallel VM start a bit more resource friendly
Wait one second between each kvm start to avoid hitting large number of
processes trying to start in parallel. This allows the VMs to be started
more efficiently for parallel-vm.py runs with large number of VMs.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 22:04:54 +02:00
Jouni Malinen
b3f4134912 tests: Clear cfg80211 BSS table at the end of scan_hidden
It was possible for the hidden SSID entry to leak into following test
cases and result in failures, e.g., in this sequence:

scan_hidden
scan_reqs_with_non_scan_radio_work
radius_auth_unreachable
wep_open_auth

BSS <BSSID> picked incorrect BSS entry within wpa_supplicant due to the
old zero-length SSID showing up from the earlier hidden SSID case.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 21:58:47 +02:00
Jouni Malinen
2ffbf34db0 tests: Optimize test case execution order for multiple VMs
Move test cases with long duration to the beginning as an optimization
to avoid last part of the test execution running a long duration test
case on a single VM while all other VMs have already completed their
work.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 18:20:15 +02:00
Jouni Malinen
01c87519a3 tests: Do not mark proxyarp_open failed if ebtables is not available
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 17:14:00 +02:00
Jouni Malinen
f24489d94f tests: Use new scripts for vm-run.sh codecov
Now there is only one set of commands to maintain. The separate reports
for individual components have not been of much use in the past, so they
are dropped for now.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 16:41:59 +02:00
Jouni Malinen
cb0555f7be tests: EAP-IKEv2 protocol tests
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-21 16:16:26 +02:00
Jouni Malinen
f41f670ea5 tests: ERP with EAP-IKEv2
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 23:48:53 +02:00
Jouni Malinen
f1381f99e4 tests: Mesh ht_mode options
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 21:27:22 +02:00
Jouni Malinen
f86868c253 tests: wdev_id in STATUS-DRIVER
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 21:18:26 +02:00
Jouni Malinen
542452a93d tests: hostapd behavior with interface in different bridge
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 21:01:31 +02:00
Jouni Malinen
a42f88f8e7 tests: bg_scan_period parameter configuration
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 19:44:40 +02:00
Jouni Malinen
6343f24d35 tests: IBSS cfg80211 workarounds
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 19:36:41 +02:00
Jouni Malinen
cece9b7df7 tests: Dynamic BSS add operation reusing existing interface
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 19:04:18 +02:00
Jouni Malinen
8916857da6 tests: TDLS with VHT
This adds some more coverage, but the TDLS setup is not for VHT due to
missing support in mac80211.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 18:44:48 +02:00
Jouni Malinen
4466fe6541 tests: VHT and 40 MHz channel
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 18:44:48 +02:00
Jouni Malinen
d4d80e9a14 tests: SMPS AP configuration options
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 18:44:48 +02:00
Jouni Malinen
598555c7e8 tests: IEEE 802.1X with static WEP
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 18:44:48 +02:00
Jouni Malinen
1e7528298a tests: cfg80211 scan-for-auth workaround with WEP keys
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 17:28:53 +02:00
Jouni Malinen
a5687274e1 tests: cfg80211 scan-for-auth workaround failing
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 17:22:18 +02:00
Jouni Malinen
b97a7bf672 tests: Large number of BSS add operations with hostapd
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 17:13:00 +02:00
Jouni Malinen
863a54093d tests: rfkill block prior to wpa_supplicant start
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 16:46:39 +02:00
Jouni Malinen
d92da8a23e tests: Radar detection on non-HT and on HT40- channels
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 13:50:19 +02:00
Jouni Malinen
cac5526390 tests: Report ROAM failure in pmksa_cache_opportunistic_multiple_sta
There is no need to wait for the timeout if the ROAM command itself
failed. This could happen if an earlier test case had left hidden SSIDs
in the cfg80211 BSS table.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 13:37:08 +02:00
Jouni Malinen
1264b68355 tests: Clear cfg80211 scan results after wext_hidden
Leaving hidden SSIDs in the cfg80211 BSS table can result in errors in
the following test cases, so use special care to clear all BSS entries
at the end of the wext_hidden test case.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 13:36:01 +02:00
Jouni Malinen
5f35a5e27f tests: Add wait_connected() and wait_disconnected() helpers
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 13:10:09 +02:00
Jouni Malinen
e0cccf26a4 tests: Add --codecov support to parallel-vm.py
This allows code coverage report to be generated must faster with the
help of parallel VMs executing test cases.

Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-20 01:24:28 +02:00
Jouni Malinen
42ce1a27e0 tests: Connection with large number of scan_ssid=1 network blocks
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-19 23:30:58 +02:00
Jouni Malinen
ed930f064c tests: P2P_LISTEN and scan
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-19 23:17:06 +02:00
Jouni Malinen
558ac77c58 tests: setband and scans
Signed-off-by: Jouni Malinen <j@w1.fi>
2014-12-19 23:08:10 +02:00