Commit graph

393 commits

Author SHA1 Message Date
Jouni Malinen
b553eab1a9 tests: Verify FT-over-DS
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-29 10:00:31 +02:00
Jouni Malinen
4a5a5792f1 tests: Verify PeerKey handshake
NOTE: Actual use of the direct link (DLS) is not supported in
mac80211_hwsim, so this operation fails at setting the keys after
successfully completed 4-way handshake. This test case does allow the
key negotiation part to be tested for coverage, though.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-28 16:32:54 +02:00
Jouni Malinen
4472aafbe0 tests: Generate a combined code coverage report
This combines coverage from all three separate reports into a single
report.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-28 16:32:54 +02:00
Jouni Malinen
75b2b9cfb0 tests: Verify EAP fast reauthentication
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-28 16:32:54 +02:00
Jouni Malinen
698f8324c6 tests: Verify PEAPv0 Crypto-Binding
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-28 16:32:54 +02:00
Jouni Malinen
6435799b85 tests: Add WNM test cases
These are not yet complete, but at least some frame exchanges can
be verified for WNM operations.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 20:04:27 +02:00
Jouni Malinen
bcbd74155e tests: Mark kernel issues more clearly in the log file
It was previously not obvious from the <test case>.log file that a test
case was marked failed based on kernel issues. Make this very clear to
avoid wasting time on figuring out what caused the failure.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 18:11:07 +02:00
Jouni Malinen
3f33b3ad8c tests: Collect code coverage separately from each component in vm
Use a more robust design for collecting the gcov logs from the case
where test cases are run within a virtual machine. This generates a
writable-from-vm build tree for each component separately so that the
lcov and gcov can easily find the matching source code and data files.
In addition, prepare the reports automatically at the end of the
vm-run.sh --codecov execution.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 18:11:07 +02:00
Jouni Malinen
629dbdd36a tests: Wait for AP-ENABLED
It takes some time for hostapd to complete AP startup. In some cases,
this could potentially result in station starting a scan before the AP
is beaconing or ready to reply to probes. To avoid such race conditions,
wait for AP-ENABLED before going ahead with the test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-27 18:11:07 +02:00
Jouni Malinen
04e6278848 tests: Verify that WPS re-provisioning replaces old network block
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-12-26 20:50:28 +02:00
Jouni Malinen
9d2cb3ec94 Make CONFIG_TESTING_OPTIONS=y enable all testing options
This makes it easier to enable various testing parameters and
functionality in build configuration.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-12-26 20:50:28 +02:00
Jouni Malinen
cf9189b94c tests: Verify scan behavior
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 20:50:23 +02:00
Jouni Malinen
cef16c47fb tests: Include additional ANQP elements for testing coverage
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 18:27:48 +02:00
Jouni Malinen
dcd6816817 tests: Verify Interworking network selection with other EAP types
This adds number of test cases to go through various possible EAP
configurations for Interworking network selection.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
0aca5d1361 tests: Verify NAI Realm list with multiple realms in one entry
In addition, use TTLS/PAP to increase testing coverage.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
e209eb989c tests: Verify Interworking network selection based on roaming consortium
In addition, use PEAP as the EAP method to increase testing coverage.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
459e96cd3b tests: Interworking network select with simulated SIM/USIM
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
84262feffd tests: Verify behavior on malformed GAS responses
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
fe871e48b8 tests: Verify failure status code processing in GAS/ANQP
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
d9474958d7 tests: Clean up GAS/ANQP message processing
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
18dd2af04b tests: Use helper functions for GAS test cases
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
bfe375ec77 tests: Verify behavior on incorrect GAS response type
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
2cace98ee1 tests: Add a GAS query timeout test case
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
e4a44b3c09 tests: Add a GAS comeback delay test case
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:45 +02:00
Jouni Malinen
8426a1474a tests: Verify Wi-Fi Display operations
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:44 +02:00
Jouni Malinen
3a64d523fa tests: Increase test coverage on BSS command
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:44 +02:00
Jouni Malinen
2cdd91d8b1 tests: Verify sharing and unsharing of ANQP results
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:44 +02:00
Jouni Malinen
8175854e6e tests: Use shorter Interworking element for larger coverage
Since the venue information in the Interworking element is optional,
include both the cases of it included and not included in the test
cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:44 +02:00
Jouni Malinen
0589f40146 tests: Use a single channel scan to avoid unnecessary full scans
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 16:55:44 +02:00
Jouni Malinen
715bf904ff tests: Verify Interworking Probe Request filtering
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 13:37:06 +02:00
Jouni Malinen
f4bfa2d27f tests: Allow gcov to be used when running test cases within vm
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-26 13:37:06 +02:00
Jouni Malinen
0eff1ab3cb tests: Add test cases for various ciphers
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-25 11:17:32 +02:00
Jouni Malinen
65249f6c4a tests: Verify QoS Mapping results in Data frames
This verifies that IP packets with various DSCP values are mapped to the
correct TID both with default mapping and with custom QoS mappings.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-12-24 20:21:58 +02:00
Johannes Berg
625188e5bf hwsim tests: Handle regdomain requests in vm scripts
In order to handle regulatory domain requests, crda needs to be
installed on the host, but we also need to install a uevent helper in
the VM so that it gets executed (since we don't run udev).

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-12-24 06:49:57 +02:00
Jouni Malinen
3208c3e970 tests: Add QoS Map test case
This verifies that QoS Map configuration and update can be completed.
Actual QoS mapping for Data frames within mac80211/cfg80211 is not yet
verified.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-12-23 13:50:54 +02:00
Johannes Berg
667a158d08 tests: VM test script: Copy gcov data if present
If there's code coverage analysis data, copy it out of the VM
to be able to analyse it later. Also add a description to the
README file about how to use it.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-12-07 17:45:12 -08:00
Jouni Malinen
50acc38469 tests: Remove special start.sh option for concurrent P2P tests
These were not really used in practice and better quality test cases for
concurrent P2P operations are now available in tests_p2p_concurrency.py
using the standard test framework design. Remove the special concurrent
option for start.sh and the run-all.sh concurrent and
concurrent-valgrind operations to clean up scripts.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-24 21:20:15 +02:00
Jouni Malinen
41af1305d7 tests: Add test cases for concurrent P2P operations
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-24 21:20:15 +02:00
Jouni Malinen
19ae1d0788 tests: Verify P2P operations with a separate group interface
This extends P2P test coverage to include the case of separate group
interface use with autonomous GO and group formation through GO
negotiation.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-24 15:32:52 +02:00
Jouni Malinen
01370f49a6 tests: Verify that re-invoked P2P group is identified as persistent
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-21 11:39:12 +02:00
Johannes Berg
8c30ab492c hwsim tests: Check kernel messages for lockdep complaints
Lockdep complaints are never good, so check for them in the
kernel messages, not just for warnings and bugs.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-20 16:39:24 +02:00
Jouni Malinen
848bb8dea9 tests: Use per-test case log/capture file from wlantest
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-17 21:47:06 +02:00
Jouni Malinen
393e90191e tests: Verify GO channel switch behavior
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-17 17:46:23 +02:00
Johannes Berg
a572f0b960 hwsim tests: Print failed tests nicer
There's no reason to format the failed tests as a python
list, just print a (space-separated) list of test names.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-17 16:46:41 +02:00
Jouni Malinen
0c9057970b tests: Do not override existing .config from build.sh
This allows the build.sh script to be used to rebuild binaries based on
existing configuration in addition to the initial case of preparing
suitable build configuration.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-09 15:40:06 +02:00
Jouni Malinen
b4b89ae3aa tests: Remove duplicated mkdir for LOGDIR
LOGDIR directory is created automatically if LOGDIR variable was not
set. If the variable is set, that directory is expected to exist.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-09 15:34:29 +02:00
Michal Kazior
d65a593b1e tests: Use proper admin group in start.sh
wpa_supplicant configuration files had a hardcoded GROUP=admin. The
start.sh script failed on a system without admin group (some systems
have it named adm).

Generate configuration files with appropriate GROUP in the log dir
and use them.

Signed-hostap: Michal Kazior <michal.kazior@tieto.com>
2013-11-09 15:31:35 +02:00
Jouni Malinen
483691bd73 tests: Verify GAS query while associated
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-08 00:41:58 +02:00
Jouni Malinen
836a374566 tests: Verify GAS fragmentation
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-08 00:22:33 +02:00
Jouni Malinen
469f5f3c93 tests: Use only new scan results for Hotspot 2.0 test cases
It looks like old scan results (different SSID for the same BSSID) could
cause test failures.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-08 00:22:33 +02:00
Johannes Berg
4ecf11c559 hwsim tests: Make channel configuration for VM tests easier
Add a CHANNELS configuration to the script running the VM
that can be added to the vm-config file to allow running
the tests with hwsim devices supporting more than a single
channel.

Eventually, with the (hopefully) upcoming dynamic work in
mac80211_hwsim, this might go away entirely, but for now
this allows testing more code paths.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-07 14:01:06 +02:00
Jouni Malinen
b4264f8f67 tests: Verify disallow_aps with Interworking
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-07 00:12:30 +02:00
Jouni Malinen
6a0b400273 tests: Verify no-duplicate-networks with Interworking connection
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-07 00:12:30 +02:00
Jouni Malinen
dd6f538ea5 tests: Verify BSS additional/removal during HT co-ex scan
This verifies that hostapd can add and remove a secondary BSS
during an ongoing HT co-ex scan.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-06 15:52:40 +02:00
Jouni Malinen
742117ad88 tests: BSS removal during ongoing HT or ACS scan
These verify that hostapd is able to remove a BSS while the radio is
still in the progress of the initial channel parameter determination.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-06 01:11:25 +02:00
Jouni Malinen
a364852be8 tests: Adjust BSS add/remove test cases for primary BSS constraint
Since removal of the primary BSS is now going to remove all the BSSs
for a radio, these two test cases need changes to not trigger false
failures.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-06 01:08:54 +02:00
Jouni Malinen
66f0bdb465 tests: Handle hostapd dying more gracefully
Catch exceptions from operations that try to remove hostapd interface
and rename the log file. If these operations fail due to socket
connection issues, hostapd has likely died or gotten stuck somewhere.
Report the test case as a failure and stop test run cleanly.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-05 13:21:58 +02:00
Johannes Berg
5061a301e5 hwsim tests: Check kernel messages for warnings/bugs
When a test passes but the kernel printed warnings, consider
the test to have failed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-05 10:17:35 +02:00
Jouni Malinen
d355372ce4 tests: HS 2.0 excluded SSID
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-04 14:48:33 +02:00
Jouni Malinen
d1ba402fe5 tests: HS 2.0 required roaming consortium
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-04 14:48:28 +02:00
Jouni Malinen
2232edf8e2 tests: Use dictionary as add_cred_values() parameter
This makes it more convenient to use this function for cases that modify
credential parameters.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-04 13:12:34 +02:00
Jouni Malinen
5c5656c37a tests: Do not require hlr_auc_gw for HS 2.0 username/password
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-11-04 13:12:28 +02:00
Jouni Malinen
8454e1fd33 tests: Add preliminary version of DFS test cases
These are still disabled by default since they require mac80211_hwsim
changes that are not yet in the upstream kernel and because the test
cases do not fully work yet. Anyway, they are quite convenient for
development use, so good to have them available in the main repository.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 21:57:39 +02:00
Jouni Malinen
1b3926ef73 tests: Add a test case for HT40 co-ex scan
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 21:30:31 +02:00
Jouni Malinen
b47750beba tests: Add test cases for automatic channel selection
This extends the Hostapd class to support monitor interface events and
STATUS command similarly to the WpaSupplicant class so that internal
hostapd state can be verified in more detail.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 21:30:31 +02:00
Jouni Malinen
77990cd7a9 tests: Add test case for multi-BSS configuration file
Verify that a single configuration file can be used to initiate multiple
BSSes in hostapd and that these BSSes can then be dynamically removed.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 20:20:50 +02:00
Jouni Malinen
e114c49cfc tests: Add an EAP-TLS test case
This fixes the user.key file (incorrect key was copied previously) and
adds a test case for EAP-TLS with WPA2-Enterprise.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 19:51:06 +02:00
Jouni Malinen
22b99086ce tests: Add more EAP test cases
This increases EAP method coverage for WPA2-Enterprise to include
EAP-pwd, EAP-GPSK, EAP-SAKE, EAP-EKE, EAP-IKEv2, EAP-PAX, and EAP-PSK.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 19:51:06 +02:00
Jouni Malinen
b5476d940e tests: Add run-tests.py --no-reset
This can be used to request the previously used default behavior where
the devices are not stopped at the end of a test case if a single test
case is run.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 11:31:48 +02:00
Jouni Malinen
5230bf64e8 tests: hostapd behavior on dynamic addition of invalid config
Verify that hostapd handles ENABLE command on invalid configuration
correctly and allows the configuration to be fixed or the interface to
be removed.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 11:31:48 +02:00
Jouni Malinen
1640a2e424 tests: Add test cases for SAE
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-03 11:31:48 +02:00
Jouni Malinen
f5f59e4f57 tests: Clean up output formatting with str.format()
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 17:07:13 +02:00
Jouni Malinen
bafd8bfa57 tests: Remove trailing whitespace
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 15:40:36 +02:00
Jouni Malinen
f01edbcfac tests: Rename stop-wifi.sh to stop.sh
This makes script naming more consistent with start.sh.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 15:39:25 +02:00
Jouni Malinen
78060b25a1 tests: Simplify test_ap_bss_add_remove implementation
Use lists and loops to avoid duplicated operations.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 13:53:23 +02:00
Jouni Malinen
57661377df tests: Use a helper function to rename wpa_supplicant/hostapd logs
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 13:04:06 +02:00
Jouni Malinen
8f47f31dd8 tests: Always write run-tests.py debug logs a file
Remove the -l command like option from run-tests.py and always enable
writing of debug level logs to files. The stdout debug verbosity is
controlled independently of the debug log files.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 12:20:59 +02:00
Jouni Malinen
75428961d9 tests: Split hostapd debug log into per test case files
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 11:53:38 +02:00
Jouni Malinen
d956022c11 tests: Ignore old scan results in test_grpform_pbc
This is needed to avoid invalid PBC session overlap detection when
the previous test case used active PBC mode and the old BSS entry
in cfg80211 may still be valid when starting the next test case.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 11:27:16 +02:00
Jouni Malinen
c65f23ab64 tests: Use single channel scans to speed up test cases
A full scan in these specific test cases does not add anything to the
coverage, so use a single channel scan for the station connection to
remove undesired extra time needed for a full scan.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 11:22:16 +02:00
Jouni Malinen
de684a2b83 tests: Add run-tests.py --shuffle-test
This optional argument can be used to randomize the order in which the
test cases are run. This can provide more coverage on testing
interactions of common use cases in various different sequences. Such
issues have already been found even with the fixed order of test cases,
but being able to reorder the tests makes this more efficient.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-11-02 10:30:37 +02:00
Johannes Berg
fdbc092389 hwsim tests: Add build script
This is easier than to copy/paste from the README. We may
want to extend it later to change the .config for some
common differences between systems (e.g., libnl/libbfd).

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:17:46 +02:00
Johannes Berg
a5d7da3fd3 hwsim tests: Prefill database in VM tests
In some cases, e.g., with the VM tests if the VM crashes, it
can be useful to know which tests should have run but didn't
(or didn't finish). In order to catch these more easily, add
an option to prefill the database with all tests at the very
beginning of the testing (in a new NOTRUN state) and use the
option in the VM tests.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:17:44 +02:00
Johannes Berg
63f83fac0d hwsim tests: Create results database in VM tests
Create a results.db in the output directory when running
the tests in a VM. To make that easier, create the tables
in the python script if they don't exist.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:16:28 +02:00
Johannes Berg
4a1b8901ab hwsim tests: Refactor test results reporting
Refactor the test reporting to treat the different results
(success/skip/failure) identically. This makes the timing
seem a bit longer, but cleans up the code which will allow
for adding more checks (e.g., on the captured data files)
later.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:03:14 +02:00
Johannes Berg
1cd3eae362 hwsim tests: Allow setting KERNEL and KVMARGS
Rather than just having KERNELDIR, allow setting KERNEL directly.
Also remove the -s option that prevents running multiple machines
at the same time, but add a KVMARGS= variable that can be used to
restore that if needed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-11-02 10:01:09 +02:00
Jouni Malinen
a633397756 tests: Test dynamic BSS addition/removal
test_ap_bss_add_remove verifies hostapd behavior when BSSes are
added/removed in multi-BSS configuration.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 23:04:50 +02:00
Jouni Malinen
a1ddc79adf tests: Remove separate error/failed text file
The new sqlite database for results deprecated this.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:39:09 +02:00
Jouni Malinen
904b6e7e75 tests: Remove separate results text file
The new sqlite database for results deprecated this.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:36:26 +02:00
Jouni Malinen
b72c300267 tests: Use a symlink for default LOGDIR between scripts
This allows run-tests.py to use the same logs/<date> default logdir as
start.sh which is quite convenient for manual test runs.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 14:27:57 +02:00
Jouni Malinen
353e5067ff tests: Split wpa_supplicant debug logs into per test case files
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 13:19:38 +02:00
Jouni Malinen
c9aa430896 tests: Split run-tests.py logger info into per test case files
The run-tests.py -l argument does not take an argument value anymore.
Instead, debug output is directed to a separate file <test>.log for each
test case.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 12:51:08 +02:00
Jouni Malinen
e955226208 tests: Remove the 'test_' prefix from test module names
This is unnecessary extra complexity for user, so use the 'test_' prefix
only internally within the python scripts and file names.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:49:58 +02:00
Jouni Malinen
4cd8343f4d tests: Remove the 'test_' prefix from test names
This is unnecessary extra complexity for user and reports, so use the
'test_' prefix only internally within the python scripts.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:43:45 +02:00
Jouni Malinen
d05159ccc5 tests: Replace tcpdump with wlantest file write
There is no need to run a separate tcpdump process to capture frames
from hwsim0 since wlantest is already doing that can write the results
to a file.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:35:02 +02:00
Jouni Malinen
79b16a5c0d tests: Remove last-debug creation
This file was used for adding debug info into the buildbot logs. The new
sqlite database-based design will replace that, so there is no need to
create last-debug and getting rid of it will make it easier to split the
run log into per-test case files.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:26:52 +02:00
Jouni Malinen
6a5b4b1a66 tests: Fix sqlite run column to be an integer
Commit 781b65cfbb ended up accidentally
changing this from an integer to a string. Fix this by not converting
the variable into a string.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
970d3b096f hwsim tests: Add scripts to run in a VM
Instead of running on the host, it can be useful to run in a
VM, particularly to test kernel rather than userspace changes,
so add a few scripts that allow doing so easily.

The basic idea is that the VM kernel is the same architecture
as the host kernel, so the host's root filesystem can be used
(in read-only mode) to run everything. Only a log filesystem
is mounted read-write and will get all the test output.

The kernel console output is collected to a special 'console'
file in the logs directory and kernel crashes are detected.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
e8e238c436 hwsim tests: Silence chown warnings
If running in a VM with the logs going to a host filesystem
chown isn't actually possible, silence any warnings from it.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
044dd1c267 hwsim tests: Silence modprobe warnings
If module loading isn't enabled (e.g., in VM tests) then
don't try to load/unload the modules.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
8f946511c9 hwsim tests: Reset at the end of a test
Resetting at the beginning causes the reset logging/tracing
data to leak from the previous test into the next, and the
last one being missed at all - reset at the end of each run
instead. Also reset before all tests just in case running a
test actually crashed the python script.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
0d5a54048b hwsim tests: Allow collecting dmesg
In addition to tracing, allow collecting dmesg. There's no
provision for actually looking at it and finding problems
in it yet though.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:08:16 +02:00
Johannes Berg
0141fa5244 hwsim tests: Pass --logdir to run-tests.py
Instead of passing the log directory for each option
(-l, -r, -e, and -T) pass it once and make the other
options just take the filename (optionally, even).

This will also make it easier to extend later.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:07:52 +02:00
Johannes Berg
958bf1ba39 hwsim tests: Move logging into timestamped subdirectory
Instead of logging many files called "<timestamp>-*", log into
a new "<timestamp>/*" in the logs/ subdirectory and only put
the last-debug file into logs/. If a LOGDIR is specified in the
environment, instead just put everything into that directory
(so the caller should make sure to give it a timestamp or so)
and skip the creation of last-debug entirely.

Also clean up a bit and pass the LOGDIR from run-all.sh to
start.sh rather than having start.sh create the timestamp and
run-all.sh detect the latest one when having run start.sh.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 11:07:26 +02:00
Johannes Berg
435e37df7c hwsim tests: Allow run-tests.py to start tracing
In order to get tracing per test, allow run-tests.py to start
and stop tracing per test case. This is implemented using a
python 'with' context so it starts/stops automatically at the
right spots.

Instead of starting global tracing, also use it from run-all.sh
and put the trace files into the log dir.

Note that this only works right if you use a separate log dir
for all test runs as the trace files aren't timestamped.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:47:44 +02:00
Johannes Berg
0648c3b8f5 hostapd: Add -T Linux tracing option
Just like wpa_supplicant, give hostapd the -T option to
send all debug messages into the Linux tracing buffer.
Enable this option for hwsim test builds by default.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:47:44 +02:00
Johannes Berg
f62ec3696d hwsim tests: Pass run-all arguments on
Allow run-all to pass on extra arguments from the command
line that aren't built by the script itself.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:29:05 +02:00
Johannes Berg
aee9446d0c hwsim tests: Pass commit as an argument
Rather than saving the current commit to a file, pass it
as an argument to run-tests.py.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-31 10:29:00 +02:00
Jouni Malinen
3b464a3c21 tests: Silence chmod errors
These will fail if used on a readonly file system and are only needed
for valgrind runs.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-30 19:58:23 +02:00
Johannes Berg
f72434b43e tests: Refactor run-all.sh
Reuse the code rather than duplicating the implementation
of starting the tests. To make that easier, allow passing
multiple modules with -f to run-tests.py.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:58:23 +02:00
Johannes Berg
95194219df tests: Reduce duplication in startup code
Instead of hard-coding four different cases, use variables
(and printf) to reduce the duplication.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:58:23 +02:00
Johannes Berg
0b1faa8bf1 tests: Use argparse module with hwsim
Instead of re-implementing a command-line parser, use the
argparse module.

The only real change (I hope) is that the test module must
now be given to the -f option without the .py suffix.

Also, --help now works, and if a test module/test name is
given that doesn't exist, the valid list is printed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:58:23 +02:00
Johannes Berg
2f3a3edd5d tests: Import sqlite3 only if used
Might help some systems that don't have/need it.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:20:07 +02:00
Johannes Berg
781b65cfbb tests: Use proper SQL construction in hwsim test reporting
Don't construct SQL strings on the fly but instead use the
argument placeholders.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 19:19:07 +02:00
Johannes Berg
5fecc0f525 tests: Allow specifying LOGDIR for hwsim scripts
Instead of hardcoding logs/, allow putting LOGDIR into
the environment. This helps run tests in a VM where the
rootfs might be mounted read-only.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-30 15:55:58 +02:00
Jouni Malinen
789b9f1d52 tests: Enable verbose logger_stdout debug in hostapd
This makes hostapd debug logs more helpful for debugging.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-30 00:38:57 +02:00
Jouni Malinen
709f18d501 tests: Add a test for dynamic hostapd config change
test_ap_change_ssid verifies that the SET and RELOAD control interface
commands can be used to change the SSID of a hostapd-controlled WPA2-PSK
AP, i.e., for a case where the PSK needs to be derived again after the
SSID change.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-29 17:17:04 +02:00
Jouni Malinen
8fba2e5d42 tests: Add Hotspot 2.0 test cases for connecting with username
The test_ap_hs20_username* test cases verify that a username/password
credential can be used for Hotspot 2.0 connection and that the network
type is reported correctly.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-29 14:38:31 +02:00
Jouni Malinen
59f8a3c69d tests: Add external_sim test cases with roaming network
test_ap_hs20_ext_sim_roaming is otherwise identical with
test_ap_hs20_ext_sim, but the network is operated by a roaming partner.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-29 14:23:31 +02:00
Jouni Malinen
bbe867672e tests: Add helper functions for Hotspot 2.0 test cases
These make it easier to add new test cases for Interworking network
selection.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-29 14:20:29 +02:00
Jouni Malinen
f4defd91cd tests: Verify that STATUS sp_type is valid for Interworking
This verifies that the network type (home/roaming/unknown) is available
and set properly in the test_ap_hs20_ext_sim test case.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-29 14:12:23 +02:00
Johannes Berg
b3e2ce3bae tests: Do not require wpaspy installation
Since the scripts must be run from the source directory to
find the tests to run, they can use a relative path to the
wpaspy module instead of requiring it to be installed.

Signed-hostap: Johannes Berg <johannes.berg@intel.com>
2013-10-27 18:28:28 +02:00
Jouni Malinen
f3be6eea9b tests: Allow test case descriptions to be written into database
"run-tests.py -S <db file> -L" can now be used to update a database
table with the current set of test cases and their descriptions.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-27 12:41:10 +02:00
Jouni Malinen
b74b7e87bb tests: Add support for sqlite results database
This is more convenient to use directly than going through the
text-based results file.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-27 10:04:47 +02:00
Jouni Malinen
00606de898 tests: Unload cfg80211 on stop-wifi.sh
cfg80211 may keep some state (e.g., regulatory domain), so make sure
this gets cleared between each full testing cycle.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-27 00:08:51 +03:00
Jouni Malinen
e201e5ab61 tests: Add P2P channel selection test cases for group re-invocation
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-26 23:50:11 +03:00
Jouni Malinen
799716a8f1 tests: Add P2P channel selection test cases with world roaming
This verifies that passive scan 5 GHz channels are not selected with and
without p2p_add_cli_chan=1.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-26 23:49:54 +03:00
Jouni Malinen
46f2cfcefc tests: Add P2P channel selection test cases for group formation
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-26 23:49:10 +03:00
Jouni Malinen
e7ecab4a3b Use ARRAY_SIZE() macro
Replace the common sizeof(a)/sizeof(a[0]) constructions with a more
readable version.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-26 17:49:05 +03:00
Jouni Malinen
b9ab2f2e5c tests: Add pre-authorized invitation-to-active-group
test_p2p_go_invite_auth is similar to test_p2p_go_invite with the main
difference being in the peer device pre-authorizing the invitation
instead of processing invitation at upper layers after having received
it.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-21 18:24:20 +03:00
Jouni Malinen
ebc61dc2db tests: Add test cases for GAS operations
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-21 18:24:20 +03:00
Jouni Malinen
72c052d525 tests: Add domain_suffix_match validation
This verifies that domain_suffix_match works correctly both for the
matching and mismatching cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 21:38:02 +03:00
Jouni Malinen
c7afc0789c tests: Add negative TLS test case to verify trust root validation
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 21:38:02 +03:00
Jouni Malinen
d93a240731 tests: Stop test run on NOTE command failure
There is no point trying to go through a test case if the NOTE command
to write TEST-START entry does not succeed. This avoids some excessive
waits on buildbot trying to forcefully kill the programs on its timeout
if wpa_supplicant gets stuck waiting for something (like the current
issue with libnl events and commands having a chance of hitting a
blocking wait on netlink messages).

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 21:38:02 +03:00
Jouni Malinen
033e48ee2d tests: Force GO to have an old entry in test_p2p_go_invite
This adds more coverage to testing by forcing the GO to be found with an
older entry in the BSS table and with that entry having a different
operating channel. Such a case has found issues with PD retries and
scanning if incorrect frequency and SSID is selected for the group.
Instead of relying on the old BSS entry with different operating channel
to happen based on a specific test case sequence, force this to happen
for this test case every time.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 21:38:01 +03:00
Jouni Malinen
efd43d85bd tests: Test case for Hotspot 2.0 with external SIM processing
This verifies network selection using 3GPP Network Information and
connection using EAP-SIM with GSM authentication handled by an external
program.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-10-20 13:41:23 +03:00
Jouni Malinen
0076a18f39 tests: Remove cred before changing domain
This is in preparation for supporting multiple domain values after which
setting the domain value does not replace the old value, but adds a new
one.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-18 14:13:45 +03:00
Jouni Malinen
dfe6ce4eeb tests: Make sure hlr_auc_gw exits and clean up sockets
If a test run is terminated forcefully, hlr_auc_gw could have been left
running and/or some of the control interface files could still remain.
This could result in hlr_auc_gw not starting again for the next test
run.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-17 21:05:06 +03:00
Jouni Malinen
0d4c5494bc tests: Skip EAP-SIM/AKA/AKA' if hlr_auc_gw not available
Do not fail the EAP-SIM/AKA/AKA' test cases if hlr_auc_gw is not
available since hlr_auc_gw is not considered required part for the test
setup.

Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
2013-10-17 21:04:55 +03:00
Jouni Malinen
f9174c3b4e tests: Flush stdout after each status line print
This makes it more convenient to follow live test execution status.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 20:42:37 +03:00
Jouni Malinen
9626962d2f tests: Add test cases for WPA2-Enterprise
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 20:35:26 +03:00
Jouni Malinen
479cbb3892 tests: Start RADIUS authentication server
This can be used to run WPA2-Enterprise test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 19:14:16 +03:00
Jouni Malinen
997ebcfa5c tests: Auto detect group name
Some older systems used admin group, but adm group seems to be more
common nowadays, so detect this automatically rather than assume admin
group is used.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:41:17 +03:00
Jouni Malinen
66767f0f4a tests: Merge start-p2p-concurrent.sh functionality into start.sh
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:37:25 +03:00
Jouni Malinen
0d84c400f9 tests: Add mechanism for test cases to indicate a skipped run
This can be used by test cases that depend on some external component
that may not always be available to indicate clearly that a test case
was skipped rather than passed or failed.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:21:42 +03:00
Jouni Malinen
9b00f24a76 tests: run-tests.py -L to print out list of test cases
This can be used to get a summary of all the available test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:11:48 +03:00
Jouni Malinen
48d8a43a28 tests: Adjust debug verbosity levels to get rid of extra info
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 16:07:01 +03:00
Jouni Malinen
883ab1b9f1 tests: Document automated mac80211_hwsim test setup
This provides guidance on how to build the test setup, how to run the
tests, and how to add new test cases.

Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 15:55:43 +03:00
Jouni Malinen
5f3eddac8e tests: Add test cases for P2P service discovery
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 13:15:26 +03:00
Jouni Malinen
7ebf841fc5 tests: Test GO functionality with legacy STAs
Signed-hostap: Jouni Malinen <j@w1.fi>
2013-09-29 12:42:05 +03:00