hostap/tests/hwsim
Jouni Malinen 5653301409 tests: Prepare ap_blacklist_all for implementation change
The blacklist design will be modified in the following commits and that
would result in this validation step based on the older implementation
starting to fail. Remove this check to avoid such testing failures.

Signed-off-by: Jouni Malinen <j@w1.fi>
2020-10-10 18:34:59 +03:00
..
auth_serv tests: EAP-TEAP with client certificate in Phase 1 2020-06-20 18:08:12 +03:00
tnc tests: Use QUIET=1 option to make build.sh output much shorter 2014-12-29 15:49:05 +02:00
vm tests: Enable MACsec testing by default in the VM kernel 2020-04-04 17:25:30 +03:00
.gitignore tests: Add sigma_dut to .gitignore 2019-02-05 23:51:36 +02:00
build.sh tests: Remove unnecessary "config exists" debug prints from build.sh 2019-08-24 12:18:40 +03:00
check_kernel.py tests: Catch various lockdep warnings 2017-07-08 15:18:40 +03:00
devdetail.xml tests: hs20-osu-client testing for PPS MO parsing 2018-06-21 19:18:54 +03:00
devinfo.xml tests: hs20-osu-client testing for PPS MO parsing 2018-06-21 19:18:54 +03:00
dictionary.radius tests: Hotspot 2.0 Terms and Conditions signaling - CoA 2018-04-25 12:57:46 +03:00
example-hostapd.config tests: SAE-PK 2020-06-02 23:48:37 +03:00
example-setup.txt tests: Document rfkill workaround for systemd for hwsim tests 2017-03-29 15:03:16 +03:00
example-wpa_supplicant.config tests: SAE-PK 2020-06-02 23:48:37 +03:00
fst_module_aux.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
fst_test_common.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
hostapd.py tests: Skip test cases where hostapd does not support TKIP 2020-04-17 23:51:58 +03:00
hostapd.vlan
hostapd.vlan2 tests: hostapd.vlan with bridge ifname 2018-12-21 12:02:57 +02:00
hostapd.wlan3.vlan tests: RSN pre-auth and PMKSA caching with per_sta_vif 2016-02-28 15:43:24 +02:00
hostapd.wlan4.vlan tests: RSN pre-auth and PMKSA caching with per_sta_vif 2016-02-28 15:43:24 +02:00
hostapd.wpa_psk
hwsim.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
hwsim_utils.py tests: Factor out multicast connectivity check 2020-02-15 18:36:29 +02:00
multi-bss-acs.conf tests: Use SAE instead of WPA/TKIP in ap_multi_bss_acs 2020-04-17 23:51:58 +03:00
multi-bss-iface-per_sta_vif.conf tests: Verify ap_vlan_iface_cleanup_multibss with per_sta_vif 2016-02-17 11:46:13 +02:00
multi-bss-iface.conf tests: Verify correct VLAN operation in multi-BSS multi-VLAN case 2015-06-14 13:40:50 +03:00
multi-bss.conf
netlink.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
nl80211.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
owe-bss-1.conf tests: Add the forgotten files for owe_transition_mode_multi_bss 2017-10-11 12:16:19 +03:00
owe-bss-2.conf tests: Add the forgotten files for owe_transition_mode_multi_bss 2017-10-11 12:16:19 +03:00
p2p0.conf
p2p1.conf
p2p2.conf
p2p_utils.py tests: Fix thread handling in P2P GO Negotiation test cases 2019-12-27 18:14:09 +02:00
pps-mo-1.xml tests: hs20-osu-client testing for PPS MO parsing 2018-06-21 19:18:54 +03:00
radius_das.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
README tests: Document building of hostapd_cli for hwsim builds 2018-02-17 18:15:31 +02:00
remotehost.py tests: remote: Allow passing of parameters with monitor interface 2020-10-09 11:21:05 +03:00
rfkill.py tests: rfkill with python3 compatible version 2019-02-04 12:26:33 +02:00
run-all.sh tests: Report authentication server memory leaks more visible 2019-03-08 16:52:52 +02:00
run-tests.py tests: Handle unexpected get_driver_status_field() error 2020-05-17 13:04:26 +03:00
start.sh tests: Check for mac80211_hwsim module before loading it 2020-06-10 21:12:44 +03:00
stop.sh tests: Optimize stop.sh wait times 2014-12-22 19:37:08 +02:00
test_ap_acs.py tests: OWE transition mode with owe_transition_ifname 2020-05-26 16:23:39 +03:00
test_ap_ciphers.py tests: Unify IPv6 disabling 2020-05-17 01:31:19 +03:00
test_ap_config.py tests: Use test-name.* style files more consistently 2020-05-17 16:03:29 +03:00
test_ap_csa.py tests: Move csa_supported() into utils.py 2020-04-18 11:35:32 +03:00
test_ap_dynamic.py tests: Use test-name.* style files more consistently 2020-05-17 16:03:29 +03:00
test_ap_eap.py tests: More explicit TLS version enabling in version tests 2020-09-08 19:32:36 +03:00
test_ap_ft.py tests: Add description for test cases that were missing it 2020-05-17 01:31:19 +03:00
test_ap_hs20.py tests: wpa_supplicant global pmf parameter 2020-06-09 00:59:26 +03:00
test_ap_ht.py tests: Move csa_supported() into utils.py 2020-04-18 11:35:32 +03:00
test_ap_mixed.py tests: Skip more tests based on missing TKIP support 2020-04-17 23:51:58 +03:00
test_ap_open.py tests: Verify Deauthentication frame transmission to STA in PS mode 2020-05-04 00:37:45 +03:00
test_ap_params.py tests: Wait a bit to get Beacon frames into capture 2020-04-25 17:23:24 +03:00
test_ap_pmf.py tests: WPA2-PSK AP with PMF association comeback (WPS) 2020-10-06 22:46:26 +03:00
test_ap_psk.py tests: nl80211 control port in AP mode disabled/enabled 2020-06-21 18:07:54 +03:00
test_ap_qosmap.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_ap_roam.py tests: Prepare ap_blacklist_all for implementation change 2020-10-10 18:34:59 +03:00
test_ap_tdls.py tests: Set VHT160 or VHT160-80PLUS80 capability 2020-06-10 21:12:44 +03:00
test_ap_track.py tests: Fix ap_track_sta_force_2ghz false negative by using common finalizer 2019-06-26 20:42:20 +03:00
test_ap_vht.py tests: Use bytes in a few places 2020-06-10 21:12:44 +03:00
test_ap_vlan.py tests: Use proper temp files for dynamically created config 2020-02-23 18:20:50 +02:00
test_ap_wps.py tests: Move check_sae_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_authsrv.py tests: Update authsrv_oom to match implementation changes 2019-08-20 02:59:06 +03:00
test_autoscan.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_bgscan.py tests: bgscan parameter update 2020-10-09 16:50:36 +03:00
test_cert_check.py tests: Test cases for X.509 certificate checking 2019-04-09 16:24:38 +03:00
test_cfg80211.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_connect_cmd.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_dbus.py tests: Empty network profile parameters will be valid 2020-10-09 16:50:36 +03:00
test_dfs.py tests: Use function decorator to clean up --long processing 2020-04-18 11:35:32 +03:00
test_dpp.py tests: DPP-CHIRP-RX reception 2020-10-07 21:31:46 +03:00
test_eap.py tests: EAP-TEAP with client certificate in Phase 1 2020-06-20 18:08:12 +03:00
test_eap_proto.py tests: Provide digestmod to hmac.new() for Python 3.8 2020-05-03 11:58:54 +03:00
test_erp.py tests: Additional EAP-TEAP coverage 2019-08-07 01:19:00 +03:00
test_ext_password.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_fils.py tests: Extended Key ID 2020-03-23 11:47:31 +02:00
test_fst_config.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_fst_module.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_gas.py tests: GAS/ANQP query without scan 2020-08-13 17:51:12 +03:00
test_hapd_ctrl.py tests: Skip more tests based on missing TKIP support 2020-04-17 23:51:58 +03:00
test_he.py tests: Use bytes in a few places 2020-06-10 21:12:44 +03:00
test_hostapd_oom.py tests: Use function decorator to clean up --long processing 2020-04-18 11:35:32 +03:00
test_hs20_filter.py tests: Add description for test cases that were missing it 2020-05-17 01:31:19 +03:00
test_hs20_pps_mo.py tests: Check for domain_suffix_match in HS 2.0 PPS MO testing 2020-01-26 16:56:06 +02:00
test_ibss.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_ieee8021x.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_kernel.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_macsec.py tests: Replace tcpdump with wlantest 2019-12-27 23:43:09 +02:00
test_mbo.py tests: MBO and WPA2 without PMF on misbehaving AP 2019-09-21 12:56:19 +03:00
test_module_tests.py tests: eloop socket re-open from timeout/socket handler 2015-07-23 18:39:02 +03:00
test_monitor_interface.py tests: Use frame injection in monitor_iface_unknown_sta 2020-03-26 17:51:03 +02:00
test_multi_ap.py tests: Skip more tests based on missing TKIP support 2020-04-17 23:51:58 +03:00
test_nfc_p2p.py tests: Mark 525 tests as remote compatible 2016-06-27 21:47:37 +03:00
test_nfc_wps.py tests: Skip more tests based on missing TKIP support 2020-04-17 23:51:58 +03:00
test_oce.py tests: OCE association in open network 2019-12-12 02:00:26 +02:00
test_ocv.py tests: OCV and missing PMF or OCI 2020-10-08 23:07:43 +03:00
test_offchannel_tx.py tests: Import p2p_utils instead of functions from old locations 2015-11-25 15:47:16 +02:00
test_owe.py tests: Additional coverage for OWE PMKSA caching 2020-08-22 14:03:30 +03:00
test_p2p_autogo.py tests: P2P autonomous GO and interface being removed 2020-10-01 15:48:05 +03:00
test_p2p_channel.py tests: P2P with 6 GHz disabled 2020-10-08 23:29:50 +03:00
test_p2p_concurrency.py tests: Move clear_scan_cache() and set_world_reg() to utils.py 2020-04-18 11:35:32 +03:00
test_p2p_device.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_p2p_discovery.py tests: Test p2p_long_listen longer than remain-on-channel 2020-01-04 23:07:31 +02:00
test_p2p_ext.py tests: Remove group at the end of p2p_ext_vendor_elem_invitation 2019-03-17 17:06:04 +02:00
test_p2p_grpform.py tests: Use function decorator to clean up --long processing 2020-04-18 11:35:32 +03:00
test_p2p_invitation.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_p2p_messages.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_p2p_persistent.py tests: Fix exception generation in persistent_group_per_sta_psk 2020-02-15 18:36:29 +02:00
test_p2p_service.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_p2p_set.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_p2p_wifi_display.py tests: Remove unused import statements 2019-02-02 11:35:13 +02:00
test_p2ps.py tests: Flush correct iface in P2PS stale group removal tests 2019-04-06 17:06:38 +03:00
test_pmksa_cache.py tests: PMKSA cache control interface for external management (FT) 2020-06-06 16:48:57 +03:00
test_radio_work.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_radius.py tests: Provide digestmod to hmac.new() for Python 3.8 2020-05-03 11:58:54 +03:00
test_rfkill.py tests: Wait for AP-STA-CONNECT before running connectivity test 2019-08-05 00:10:32 +03:00
test_rrm.py tests: Set VHT160 or VHT160-80PLUS80 capability 2020-06-10 21:12:44 +03:00
test_sae.py tests: SAE with OCV and fetching PMK 2020-08-03 15:39:45 +03:00
test_sae_pk.py tests: SAE-PK with invalid password on AP 2020-09-11 15:53:56 +03:00
test_scan.py tests: Unify ignore_old_scan_res resetting 2020-05-17 01:31:19 +03:00
test_sigma_dut.py tests: sigma_dut DPP AP as TCP Enrollee/initiator 2020-08-14 17:31:57 +03:00
test_ssid.py tests: Move clear_scan_cache() and set_world_reg() to utils.py 2020-04-18 11:35:32 +03:00
test_sta_dynamic.py tests: External MAC address change for connection 2020-01-02 12:58:58 +02:00
test_suite_b.py tests: Suite B using PMKSA caching and roaming 2020-05-25 16:20:33 +03:00
test_tnc.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
test_wep.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_wext.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_wmediumd.py tests: Remove a multi-line test description 2019-11-23 17:19:49 +02:00
test_wnm.py tests: Move HwsimSkip processing into start_wnm_ap() 2019-12-27 09:38:25 +02:00
test_wpas_ap.py tests: Move check_wep_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_wpas_config.py tests: Move check_sae_capa() into utils.py 2020-04-18 11:35:32 +03:00
test_wpas_ctrl.py tests: iftype parameter with GET_CAPABILITY key_mgmt 2020-05-12 17:33:54 +03:00
test_wpas_mesh.py tests: Move clear_scan_cache() and set_world_reg() to utils.py 2020-04-18 11:35:32 +03:00
test_wpas_wmm_ac.py tests: Python coding style cleanup (pylint3 bad-whitespace) 2019-03-16 18:52:09 +02:00
tshark.py tests: Optimize tshark operations for new versions 2019-12-27 00:02:05 +02:00
utils.py tests: SAE-PK 2020-06-02 23:48:37 +03:00
w1fi_logo.png
wlantest.py tests: sigma_dut controlled AP and beacon protection 2020-03-26 12:50:00 +02:00
wpasupplicant.py tests: DPP enterprise provisioning 2020-06-16 18:26:09 +03:00
wps-ctrl-cred tests: wpa_supplicant config file parsing/writing with WPS 2016-05-02 11:08:25 +03:00
wps-ctrl-cred2 tests: wpa_supplicant config file parsing/writing with WPS 2016-05-02 11:08:25 +03:00
wps-mixed-cred
wps-wep-cred

Automated hostapd/wpa_supplicant testing with mac80211_hwsim
------------------------------------------------------------

This directory contains testing infrastructure and test cases to run
automated tests of full hostapd and wpa_supplicant functionality. This
testing is done with the help of mac80211_hwsim which is Linux kernel
driver that simulates IEEE 802.11 radios without requiring any
additional hardware. This setup most of the hostapd and wpa_supplicant
functionality (and large parts of the Linux cfg80211 and mac80211
functionality for that matter) to be tested.

mac80211_hwsim is loaded with five simulated radios to allow different
device combinations to be tested. wlantest is used analyze raw packets
captured through the hwsim0 monitor interface that capture all frames
sent on all channels. wlantest is used to store the frames for
analysis. Three wpa_supplicant processes are used to control three
virtual radios and one hostapd process is used to dynamically control
the other two virtual radios. wpa_supplicant/hostapd test functionality
is used to verify that data connection (both unicast and broadcast)
works between two netdevs.

The python scripts and tools in this directory control test case
execution. They interact wpa_supplicant and hostapd through control
interfaces to perform the operations. In addition, wlantest_cli is used
to verify that operations have been performed correctly and that the
network connection works in the expected way.

These test cases are run automatically against the hostap.git commits
for regression testing and to help in keeping the hostap.git master
branch in stable state. Results from these tests are available here:
http://buildbot.w1.fi/hwsim/


Building binaries for testing
-----------------------------

You will need to build (or use already built) components to be
tested. These are available in the hostap.git repository and can be
built for example as follows:

cd ../../wpa_supplicant
cp ../tests/hwsim/example-wpa_supplicant.config .config
make clean
make
cd ../hostapd
cp ../tests/hwsim/example-hostapd.config .config
make clean
make hostapd hostapd_cli hlr_auc_gw
cd ../wlantest
make clean
make

Alternatively, the build.sh script here can be used to run these steps
with conditional creation of .config files only if they do not exist.

The test scripts can find the binaries in the locations where they were
built. It is also possible to install wlantest_cli somewhere on the path
to use pre-built tools.

Please note that some of the configuration parameters used to enable
more testing coverage may require development packages that may not be
installed by default in many distributions. For example, following
Debian/Ubuntu packages are likely to be needed:
- binutils-dev
- libsqlite3-dev
- libpcap-dev

example-setup.txt provides more complete step-by-step example on how a
test setup can be built.


wpaspy
------

The python scripts use wpaspy.py to interact with the wpa_supplicant
control interface, but the run-tests.py script adds the (relative)
path into the environment so it doesn't need to be installed.


mac80211_hwsim
--------------

mac80211_hwsim kernel module is available from the upstream Linux
kernel. Some Linux distributions enable it by default. If that's not the
case, you can either enable it in the kernel configuration
(CONFIG_MAC80211_HWSIM=m) and rebuild your kernel or use Backports with
CPTCFG_MAC80211_HWSIM=m to replace the wireless LAN components in the
base kernel.


sudo
----

Some parts of the testing process requires root privileges. The test
scripts are currently using sudo to achieve this. To be able to run the
tests, you'll probably want to enable sudo with a timeout to not expire
password entry very quickly. For example, use this in the sudoers file:

Defaults        env_reset,timestamp_timeout=180

Or on a dedicated test system, you could even disable password prompting
with this in sudoers:

%sudo   ALL=NOPASSWD: ALL


Other network interfaces
------------------------

Some of the test scripts are still using hardcoded interface names, so
the easiest way of making things work is to avoid using other network
devices that may use conflicting interface names. For example, unload
any wireless LAN driver before running the tests and make sure that
wlan0..4 gets assigned as the interface names for the mac80211_hwsim
radios. It may also be possible to rename the interface expectations in
run-tests.py to allow other names to be used.

Please also note that some commonly enabled tools, like NetworkManager,
may end up trying to control new network interfaces automatically. This
can result in conflicts with the test scripts and you may need to
disable such network services or at least mark the mac80211_hwsim wlan#
interfaces as umanaged. As an example, this can be done in
/etc/NetworkManager/NetworkManager.conf with following addition:

[keyfile]
unmanaged-devices=mac:02:00:00:00:00:00;mac:02:00:00:00:01:00;mac:02:00:00:00:02:00;mac:02:00:00:00:03:00;mac:02:00:00:00:04:00


Running tests
-------------

Simplest way to run a full set of the test cases is by running
run-all.sh in tests/hwsim directory. This will use start.sh to load the
mac80211_hwsim module and start wpa_supplicant, hostapd, and various
test tools. run-tests.sh is then used to run through all the defined
test cases and stop.sh to stop the programs and unload the kernel
module.

run-all.sh can be used to run the same test cases under different
conditions:

# run normal test cases
./run-all.sh

# run normal test cases under valgrind
./run-all.sh valgrind

# run normal test cases with Linux tracing
./run-all.sh trace

# run normal test cases with multi channel support (see details below)
./run-all.sh channels=<num of channels>

run-all.sh directs debug logs into the logs subdirectory (or $LOGDIR if
present in the environment). Log file names include the current UNIX
timestamp and a postfix to identify the specific log:
- *.log0 = wpa_supplicant debug log for the first radio
- *.log1 = wpa_supplicant debug log for the second radio
- *.log2 = wpa_supplicant debug log for the third radio
- *.hostapd = hostapd debug log
- hwsim0 = wlantest debug log
- hwsim0.pcapng = capture with all frames exchanged during the tests
- *.log = debug prints from the test scripts
- trace.dat = Linux tracing record (if enabled)
- hlr_auc_gw - hlr_auc_gw (EAP-SIM/AKA/AKA' authentication) log
- auth_serv - hostapd as RADIUS authentication server log


For manual testing, ./start.sh can be used to initialize interfaces and
programs and run-tests.py to execute one or more test
cases. run-tests.py output verbosity can be controlled with -d (more
verbose debug output) and -q (less verbose output) on the command
line. "-f <module name>" (pointing to file test_<module name>.py) can be
used to specify that all test cases from a single file are to be
run. Test name as the last command line argument can be specified that a
single test case is to be run (e.g., "./run-tests.py ap_pmf_required").

Notice that some tests require the driver to support concurrent
operation on multi channels in order to run. These tests will be skipped
in case the driver does not support multi channels. To enable support
for multi channel, the number of supported channel is passed as an
argument to run-all.sh or start.sh


Adding/modifying test cases
---------------------------

All the test cases are defined in the test_*.py files. These are python
scripts that can use the local helper classes to interact with the test
components. While various python constructs can be used in the scripts,
only a minimal level of python knowledge should really be needed to
modify and add new test cases. The easiest starting point for this is
likely to take a look at some of the example scripts. When working on a
new test, run-tests.py with -d and the test case name on the command
line is a convenient way of verifying functionality.

run-tests.py will automatically import all test cases from the test_*.py
files in this directory. All functions starting with the "test_" prefix
in these files are assumed to be test cases. Each test case is named by
the function name following the "test_" prefix.


Results database
----------------

run-tests.py can be requested to write results from the execution of
each test case into an sqlite database. The "-S <path to database>" and
"-b <build id>" command line arguments can be used to do that. The
database must have been prepared before this, e.g., with following:

cat | sqlite3 /tmp/example.db <<EOF
CREATE TABLE results (test,result,run,time,duration,build,commitid);
CREATE INDEX results_idx ON results (test);
CREATE INDEX results_idx2 ON results (run);
CREATE TABLE tests (test,description);
CREATE UNIQUE INDEX tests_idx ON tests (test);
CREATE TABLE logs (test,run,type,contents);
CREATE INDEX logs_idx ON logs (test);
CREATE INDEX logs_idx2 ON logs (run);
EOF