hostap/tests/hwsim
Jouni Malinen bccd22f356 tests: EAP-FAST protocol testing
Signed-off-by: Jouni Malinen <j@w1.fi>
2016-02-20 18:25:13 +02:00
..
auth_serv tests: Renew expired certificates 2016-02-19 18:44:39 +02:00
tnc tests: Use QUIET=1 option to make build.sh output much shorter 2014-12-29 15:49:05 +02:00
vm Revert "tests: vm: Output everything on console" 2015-12-30 21:51:52 +02:00
bss-1.conf tests: Test dynamic BSS addition/removal 2013-10-31 23:04:50 +02:00
bss-2.conf tests: Test dynamic BSS addition/removal 2013-10-31 23:04:50 +02:00
bss-3.conf tests: Test dynamic BSS addition/removal 2013-10-31 23:04:50 +02:00
bss-ht40-1.conf tests: Verify BSS additional/removal during HT co-ex scan 2013-11-06 15:52:40 +02:00
bss-ht40-2.conf tests: Verify BSS additional/removal during HT co-ex scan 2013-11-06 15:52:40 +02:00
build.sh tests: Set FIPSLD_CC=gcc (if not set) to make CONFIG_FIPS=y use easier 2015-08-01 16:57:04 +03:00
check_kernel.py tests: Catch RTNL assertions in the kernel check 2015-08-24 19:28:12 +03:00
dictionary.radius tests: Verify dynamic_vlan=required is honored with macaddr_acl=2 2015-04-25 10:28:19 +03:00
example-hostapd.config tests: Add FST module tests 2015-07-16 18:26:16 +03:00
example-setup.txt tests: Document dependency on python-crypto 2015-03-07 23:47:02 +02:00
example-wpa_supplicant.config tests: wpa_supplicant AP mode with ACS 2015-12-24 22:24:05 +02:00
fst_module_aux.py tests: Dump control interface sockets during FST operations 2015-12-30 13:54:18 +02:00
fst_test_common.py tests: Add FST module tests 2015-07-16 18:26:16 +03:00
hostapd.accept tests: Add test cases for AP VLAN 2014-03-08 11:35:08 +02:00
hostapd.macaddr tests: hostapd MAC ACL file entry removal 2014-03-30 17:06:34 +03:00
hostapd.py tests: Speed up hostapd_oom_loop tests 2015-10-04 18:52:38 +03:00
hostapd.vlan tests: VLAN using vlan_file mapping 2014-03-30 17:06:34 +03:00
hostapd.wlan3.vlan tests: Verify RSN preauth with dynamic VLANs 2015-04-13 17:08:10 +03:00
hostapd.wlan4.vlan tests: Verify RSN preauth with dynamic VLANs 2015-04-13 17:08:10 +03:00
hostapd.wpa_psk tests: Verify different ways of configuring passphrase/PSK 2014-02-04 13:10:57 +02:00
hwsim.py tests: Introduce hwsim radio context manager 2014-10-21 17:35:20 +03:00
hwsim_utils.py tests: Add test to check disconnect in powersave 2015-02-21 16:07:52 +02:00
multi-bss-acs.conf tests: Add test cases for automatic channel selection 2013-11-03 21:30:31 +02:00
multi-bss-iface-per_sta_vif.conf tests: Verify ap_vlan_iface_cleanup_multibss with per_sta_vif 2016-02-17 11:46:13 +02:00
multi-bss-iface.conf tests: Verify correct VLAN operation in multi-BSS multi-VLAN case 2015-06-14 13:40:50 +03:00
multi-bss.conf tests: Add test case for multi-BSS configuration file 2013-11-03 20:20:50 +02:00
netlink.py hwsim tests: Add support for the chanctx flag when creating radios 2014-02-25 16:53:40 +02:00
nl80211.py tests: cfg80211 DEL_STATION issued externally to hostapd 2015-01-29 16:26:31 +02:00
p2p0.conf tests: Add start/stop scripts and configuration files 2013-03-02 11:39:54 +02:00
p2p1.conf tests: Add start/stop scripts and configuration files 2013-03-02 11:39:54 +02:00
p2p2.conf tests: Add start/stop scripts and configuration files 2013-03-02 11:39:54 +02:00
p2p_utils.py tests: Detect invitation GO issues more quickly 2015-12-18 16:14:55 +02:00
radius_das.py tests: Remove unnecessary interpreter line from most python files 2014-02-21 20:21:53 +02:00
README tests: Add step-by-step guide for setting up test framework 2015-01-18 18:11:11 +02:00
rfkill.py tests: Add rfkill module 2015-01-08 20:56:47 +02:00
run-all.sh tests: Properly pass the num of channels to start.sh 2015-08-10 21:10:56 +03:00
run-tests.py tests: run-tests: Print more details about NameError 2015-11-28 00:21:20 +02:00
start.sh tests: Minimal testing of OCSP stapling with ocsp_multi 2015-12-23 00:32:52 +02:00
stop.sh tests: Optimize stop.sh wait times 2014-12-22 19:37:08 +02:00
test_ap_acs.py tests: Skip WPA(V1) test cases in FIPS mode 2015-08-02 16:52:56 +03:00
test_ap_ciphers.py tests: Fix ap_cipher_tkip_countermeasures_sta test 2015-10-16 20:11:05 +03:00
test_ap_config.py tests: hostapd configuration file error cases 2014-03-29 10:36:25 +02:00
test_ap_csa.py tests: AP Channel Switch, one switch with only ECSA IE 2015-10-03 21:37:28 +03:00
test_ap_dynamic.py tests: Make tests using multi_check() more robust 2015-12-04 21:03:43 +02:00
test_ap_eap.py tests: Verify fast_max_pac_list_len=0 special case 2016-02-20 10:06:02 +02:00
test_ap_ft.py tests: fail_test instead of alloc_fail for aes_{encrypt,decrypt}_init 2016-01-14 20:10:16 +02:00
test_ap_hs20.py tests: REQ_HS20_ICON and DEL_HS20_ICON 2015-12-19 18:34:01 +02:00
test_ap_ht.py tests: Allow fallback to 20 MHz in ap_ht40_5ghz_invalid_pair 2016-02-05 17:51:24 +02:00
test_ap_mixed.py tests: Skip WPA(V1) test cases in FIPS mode 2015-08-02 16:52:56 +03:00
test_ap_open.py tests: Prepare ap_open_out_of_memory for modified rfkill init path 2015-12-18 22:26:18 +02:00
test_ap_params.py tests: Make ap_max_num_sta_no_probe_resp more robust 2015-12-06 13:38:18 +02:00
test_ap_pmf.py tests: WPA2-PSK AP with PMF required and PMF disabled on STA 2015-10-25 19:37:17 +02:00
test_ap_psk.py tests: Make key-lifetime-in-memory more robust for GTK check 2015-12-14 17:23:47 +02:00
test_ap_qosmap.py tests: Convert test skipping to use exception 2015-01-08 22:43:47 +02:00
test_ap_roam.py tests: Roam between two WPA2-PSK APs and try to hit a disconnection race 2016-01-15 19:34:58 +02:00
test_ap_tdls.py tests: TDLS with VHT 80, 80+80, and 160 2015-12-17 21:20:02 +02:00
test_ap_track.py tests: AP and STA tracking with passive scan 2015-09-05 20:29:01 +03:00
test_ap_vht.py tests: Verify SIGNAL_POLL values in ap_vht80 2015-12-17 21:20:02 +02:00
test_ap_vlan.py tests: Verify connectivity with untagged/tagged VLAN mixed configuration 2016-02-17 11:46:13 +02:00
test_ap_wps.py tests: Allow PIN generation failure during OOM in ap_wps_random_ap_pin 2016-02-19 18:44:39 +02:00
test_autoscan.py tests: AUTOSCAN reconfiguration while in SCANNING state 2014-12-11 22:57:36 +02:00
test_bgscan.py tests: Additional bgscan coverage 2015-01-06 02:49:13 +02:00
test_cfg80211.py tests: cfg80211 DEL_STATION issued externally to hostapd 2015-01-29 16:26:31 +02:00
test_connect_cmd.py tests: Clean up at the end of connect_cmd_disconnect_event 2015-12-30 13:29:32 +02:00
test_dbus.py tests: Fix wpas_ctrl_country and dbus_country with valid db.txt 2015-12-12 12:45:30 +02:00
test_dbus_old.py tests: Regression test for D-Bus setting scan_freq and freq_list 2015-08-28 00:20:30 +03:00
test_dfs.py tests: Remove unnecessary use of sudo from test cases 2015-02-07 15:37:13 +02:00
test_eap_proto.py tests: EAP-FAST protocol testing 2016-02-20 18:25:13 +02:00
test_erp.py tests: Additional EAP-Finish local error coverage 2016-02-07 21:05:02 +02:00
test_ext_password.py tests: Skip EAP-TTLS/CHAP, MSCHAP, MSCHAPV2 test cases in FIPS mode 2015-08-02 16:52:56 +03:00
test_fst_config.py tests: Use codecov build hostapd/wpa_supplicant in FST tests 2015-07-19 23:10:55 +03:00
test_fst_module.py tests: Use logger.info() instead of print in FST test cases 2015-12-30 13:40:58 +02:00
test_gas.py tests: Add AP Location Public Identifier into gas_anqp_extra_elements 2015-12-21 11:56:56 +02:00
test_hapd_ctrl.py tests: Update hapd_ctrl_set_error_cases RTS/fragmentation threshold 2015-10-28 20:53:15 +02:00
test_hostapd_oom.py tests: Speed up hostapd_oom_loop tests 2015-10-04 18:52:38 +03:00
test_ibss.py tests: IBSS on VHT 80+80 MHz channel 2015-11-26 18:47:40 +02:00
test_ieee8021x.py tests: Skip IEEE 802.1X dynamic WEP tests in FIPS mode 2015-08-02 16:52:56 +03:00
test_module_tests.py tests: eloop socket re-open from timeout/socket handler 2015-07-23 18:39:02 +03:00
test_monitor_interface.py tests: Stop connection at the end of monitor_iface_unknown_sta 2014-11-30 15:53:11 +02:00
test_nfc_p2p.py tests: Speed up discovery_group_client and nfc_p2p_client 2016-01-06 18:56:31 +02:00
test_nfc_wps.py tests: Clear ignore_old_scan_res explicitly in test cases where it is used 2015-12-28 17:48:01 +02:00
test_offchannel_tx.py tests: Import p2p_utils instead of functions from old locations 2015-11-25 15:47:16 +02:00
test_p2p_autogo.py tests: Make autogo_join_auto_go_neg more robust 2015-12-02 12:37:12 +02:00
test_p2p_channel.py tests: Move P2P-REMOVE-AND-REFORM-GROUP cases more robust 2015-12-24 13:40:54 +02:00
test_p2p_concurrency.py tests: Move P2P helper functions to a separate file 2015-10-25 19:37:17 +02:00
test_p2p_device.py tests: P2P group formation timeout with cfg80211 P2P Device 2015-11-30 21:38:23 +02:00
test_p2p_discovery.py tests: Speed up discovery_group_client and nfc_p2p_client 2016-01-06 18:56:31 +02:00
test_p2p_ext.py tests: Move P2P helper functions to a separate file 2015-10-25 19:37:17 +02:00
test_p2p_grpform.py tests: Use full prefix of the P2P-GO-NEG-FAILURE 2015-12-31 22:41:21 +02:00
test_p2p_invitation.py tests: Update group ifname in p2p_go_invite_auth 2015-02-05 13:38:10 +02:00
test_p2p_messages.py tests: Move P2P helper functions to a separate file 2015-10-25 19:37:17 +02:00
test_p2p_persistent.py tests: P2P persistent group re-invocation with peer having dropped info 2016-02-02 13:27:32 +02:00
test_p2p_service.py tests: Use global control interface in test_p2p_service 2015-02-05 13:48:40 +02:00
test_p2p_set.py tests: Fix p2p_set_discoverability waiting for CTRL-EVENT-CONNECTED 2015-06-19 11:27:15 +03:00
test_p2p_wifi_display.py tests: Fix wifi_display_parsing 2015-10-31 16:31:03 +02:00
test_p2ps.py tests: Make P2PS join-a-group cases more robust 2015-12-30 19:14:40 +02:00
test_peerkey.py tests: Skip WPA(V1) test cases in FIPS mode 2015-08-02 16:52:56 +03:00
test_pmksa_cache.py tests: Verify correct VLAN after RSN pre-authentication 2016-02-17 11:46:13 +02:00
test_radio_work.py tests: RADIO_WORK error cases 2014-12-12 00:27:06 +02:00
test_radius.py tests: RADIUS Accounting and non-ASCII SSID 2015-12-24 12:19:41 +02:00
test_rfkill.py tests: Add rfkill tests for P2P Device operations 2015-12-18 22:26:18 +02:00
test_sae.py tests: Make key-lifetime-in-memory more robust for GTK check 2015-12-14 17:23:47 +02:00
test_scan.py tests: Make scan test cases more robust by allowing retries 2015-12-30 19:52:42 +02:00
test_ssid.py tests: Fix scan result clearing in ssid_hidden* 2015-12-20 17:25:41 +02:00
test_sta_dynamic.py tests: Remove unnecessary use of sudo from test cases 2015-02-07 15:37:13 +02:00
test_suite_b.py tests: Run Suite B test cases with OpenSSL 1.1.0 2015-08-02 17:11:47 +03:00
test_tnc.py tests: Speed up TNC test cases with the use of a single channel scan 2015-12-20 17:25:41 +02:00
test_wep.py tests: Add wait_connected() and wait_disconnected() helpers 2014-12-20 13:10:09 +02:00
test_wext.py tests: Skip WPA(V1) test cases in FIPS mode 2015-08-02 16:52:56 +03:00
test_wmediumd.py tests: Add a simple wmediumd test 2015-11-27 21:11:53 +02:00
test_wnm.py tests: WNM BSS Transition Management with invalid operating class 2016-02-05 17:07:48 +02:00
test_wpas_ap.py tests: wpa_supplicant AP mode - unexpected P2P IE in Association Request 2016-01-01 17:22:21 +02:00
test_wpas_config.py tests: Verify that ip_addr_* gets written to config file 2016-01-15 20:25:38 +02:00
test_wpas_ctrl.py tests: Fix wpas_ctrl_oom 2016-01-15 16:27:13 +02:00
test_wpas_mesh.py tests: Secure mesh network plink counting during reconnection 2016-02-06 21:25:52 +02:00
test_wpas_wmm_ac.py tests: WMM-AC reassociation-to-same-BSS test 2015-01-04 18:59:07 +02:00
tshark.py tests: Use wlantest without capture file write buffering 2015-11-27 00:12:38 +02:00
utils.py tests: EAP-TTLS local error cases 2015-12-20 17:25:41 +02:00
w1fi_logo.png tests: HS 2.0 OSU and icon fetch 2014-03-02 10:35:34 +02:00
wlantest.py tests: Re-association to same BSS to toggle PMF status 2015-02-19 16:37:12 +02:00
wpasupplicant.py tests: EAP-TLS error cases 2016-02-02 00:39:39 +02:00
wps-mixed-cred tests: WPS mixed-WPA/WPA2 credential merging 2014-08-29 20:25:18 +03:00
wps-wep-cred tests: WPS 2.0 rejection of WEP credential 2014-03-05 12:40:54 +02:00

Automated hostapd/wpa_supplicant testing with mac80211_hwsim
------------------------------------------------------------

This directory contains testing infrastructure and test cases to run
automated tests of full hostapd and wpa_supplicant functionality. This
testing is done with the help of mac80211_hwsim which is Linux kernel
driver that simulates IEEE 802.11 radios without requiring any
additional hardware. This setup most of the hostapd and wpa_supplicant
functionality (and large parts of the Linux cfg80211 and mac80211
functionality for that matter) to be tested.

mac80211_hwsim is loaded with five simulated radios to allow different
device combinations to be tested. wlantest is used analyze raw packets
captured through the hwsim0 monitor interface that capture all frames
sent on all channels. wlantest is used to store the frames for
analysis. Three wpa_supplicant processes are used to control three
virtual radios and one hostapd process is used to dynamically control
the other two virtual radios. wpa_supplicant/hostapd test functionality
is used to verify that data connection (both unicast and broadcast)
works between two netdevs.

The python scripts and tools in this directory control test case
execution. They interact wpa_supplicant and hostapd through control
interfaces to perform the operations. In addition, wlantest_cli is used
to verify that operations have been performed correctly and that the
network connection works in the expected way.

These test cases are run automatically against the hostap.git commits
for regression testing and to help in keeping the hostap.git master
branch in stable state. Results from these tests are available here:
http://buildbot.w1.fi/hwsim/


Building binaries for testing
-----------------------------

You will need to build (or use already built) components to be
tested. These are available in the hostap.git repository and can be
built for example as follows:

cd ../../wpa_supplicant
cp ../tests/hwsim/example-wpa_supplicant.config .config
make clean
make
cd ../hostapd
cp ../tests/hwsim/example-hostapd.config .config
make clean
make hostapd hlr_auc_gw
cd ../wlantest
make clean
make

Alternatively, the build.sh script here can be used to run these steps
with conditional creation of .config files only if they do not exist.

The test scripts can find the binaries in the locations where they were
built. It is also possible to install wlantest_cli somewhere on the path
to use pre-built tools.

Please note that some of the configuration parameters used to enable
more testing coverage may require development packages that may not be
installed by default in many distributions. For example, following
Debian/Ubuntu packages are likely to be needed:
- binutils-dev
- libsqlite3-dev
- libpcap-dev

example-setup.txt provides more complete step-by-step example on how a
test setup can be built.


wpaspy
------

The python scripts use wpaspy.py to interact with the wpa_supplicant
control interface, but the run-tests.py script adds the (relative)
path into the environment so it doesn't need to be installed.


mac80211_hwsim
--------------

mac80211_hwsim kernel module is available from the upstream Linux
kernel. Some Linux distributions enable it by default. If that's not the
case, you can either enable it in the kernel configuration
(CONFIG_MAC80211_HWSIM=m) and rebuild your kernel or use Backports with
CPTCFG_MAC80211_HWSIM=m to replace the wireless LAN components in the
base kernel.


sudo
----

Some parts of the testing process requires root privileges. The test
scripts are currently using sudo to achieve this. To be able to run the
tests, you'll probably want to enable sudo with a timeout to not expire
password entry very quickly. For example, use this in the sudoers file:

Defaults        env_reset,timestamp_timeout=180

Or on a dedicated test system, you could even disable password prompting
with this in sudoers:

%sudo   ALL=NOPASSWD: ALL


Other network interfaces
------------------------

Some of the test scripts are still using hardcoded interface names, so
the easiest way of making things work is to avoid using other network
devices that may use conflicting interface names. For example, unload
any wireless LAN driver before running the tests and make sure that
wlan0..4 gets assigned as the interface names for the mac80211_hwsim
radios. It may also be possible to rename the interface expectations in
run-tests.py to allow other names to be used.

Please also note that some commonly enabled tools, like NetworkManager,
may end up trying to control new network interfaces automatically. This
can result in conflicts with the test scripts and you may need to
disable such network services or at least mark the mac80211_hwsim wlan#
interfaces as umanaged. As an example, this can be done in
/etc/NetworkManager/NetworkManager.conf with following addition:

[keyfile]
unmanaged-devices=mac:02:00:00:00:00:00;mac:02:00:00:00:01:00;mac:02:00:00:00:02:00;mac:02:00:00:00:03:00;mac:02:00:00:00:04:00


Running tests
-------------

Simplest way to run a full set of the test cases is by running
run-all.sh in tests/hwsim directory. This will use start.sh to load the
mac80211_hwsim module and start wpa_supplicant, hostapd, and various
test tools. run-tests.sh is then used to run through all the defined
test cases and stop.sh to stop the programs and unload the kernel
module.

run-all.sh can be used to run the same test cases under different
conditions:

# run normal test cases
./run-all.sh

# run normal test cases under valgrind
./run-all.sh valgrind

# run normal test cases with Linux tracing
./run-all.sh trace

# run normal test cases with multi channel support (see details below)
./run-all.sh channels=<num of channels>

run-all.sh directs debug logs into the logs subdirectory (or $LOGDIR if
present in the environment). Log file names include the current UNIX
timestamp and a postfix to identify the specific log:
- *.log0 = wpa_supplicant debug log for the first radio
- *.log1 = wpa_supplicant debug log for the second radio
- *.log2 = wpa_supplicant debug log for the third radio
- *.hostapd = hostapd debug log
- hwsim0 = wlantest debug log
- hwsim0.pcapng = capture with all frames exchanged during the tests
- *.log = debug prints from the test scripts
- trace.dat = Linux tracing record (if enabled)
- hlr_auc_gw - hlr_auc_gw (EAP-SIM/AKA/AKA' authentication) log
- auth_serv - hostapd as RADIUS authentication server log


For manual testing, ./start.sh can be used to initialize interfaces and
programs and run-tests.py to execute one or more test
cases. run-tests.py output verbosity can be controlled with -d (more
verbose debug output) and -q (less verbose output) on the command
line. "-f <module name>" (pointing to file test_<module name>.py) can be
used to specify that all test cases from a single file are to be
run. Test name as the last command line argument can be specified that a
single test case is to be run (e.g., "./run-tests.py ap_pmf_required").

Notice that some tests require the driver to support concurrent
operation on multi channels in order to run. These tests will be skipped
in case the driver does not support multi channels. To enable support
for multi channel, the number of supported channel is passed as an
argument to run-all.sh or start.sh


Adding/modifying test cases
---------------------------

All the test cases are defined in the test_*.py files. These are python
scripts that can use the local helper classes to interact with the test
components. While various python constructs can be used in the scripts,
only a minimal level of python knowledge should really be needed to
modify and add new test cases. The easiest starting point for this is
likely to take a look at some of the example scripts. When working on a
new test, run-tests.py with -d and the test case name on the command
line is a convenient way of verifying functionality.

run-tests.py will automatically import all test cases from the test_*.py
files in this directory. All functions starting with the "test_" prefix
in these files are assumed to be test cases. Each test case is named by
the function name following the "test_" prefix.


Results database
----------------

run-tests.py can be requested to write results from the execution of
each test case into an sqlite database. The "-S <path to database>" and
"-b <build id>" command line arguments can be used to do that. The
database must have been prepared before this, e.g., with following:

cat | sqlite3 /tmp/example.db <<EOF
CREATE TABLE results (test,result,run,time,duration,build,commitid);
CREATE INDEX results_idx ON results (test);
CREATE INDEX results_idx2 ON results (run);
CREATE TABLE tests (test,description);
CREATE UNIQUE INDEX tests_idx ON tests (test);
CREATE TABLE logs (test,run,type,contents);
CREATE INDEX logs_idx ON logs (test);
CREATE INDEX logs_idx2 ON logs (run);
EOF