hostap/tests/cipher-and-key-mgmt-testing.txt

Cipher suite (CCMP, TKIP, GCMP, ..) and key management testing
==============================================================

wpa_supplicant and hostapd include number of extensions that allow
special test builds to be used for testing functionality related to
correct implementation of IEEE 802.11. These extensions allow behavior
to be modified and invalid operations to be performed to verify behavior
of other devices in unexpected situations. While most of the testing
extensions are focused on the fully automated testing framework with
mac80211_hwsim (see tests/hwsim subdirectory), many of these can be used
for over-the-air testing of the protocol as well.

Since some of the testing extensions can result in exposing key
information or allowing non-compliant behavior, these changes are
disabled in default wpa_supplicant and hostapd builds for production
purposes. Testing functionality can be enabled by adding
CONFIG_TESTING_OPTIONS=y into build configuration (hostapd/.config and
wpa_supplicant/.config).


Testing setup
-------------

These tests can be run as black-box testing without having to modify the
tested device at all or without knowing details of its
functionality. The test commands in wpa_supplicant/hostapd control
interfaces are used to perform unexpected operations and normal data
traffic is used to verify reaction of the tested device to such
operations.

In theory, the test functionality is available with most drivers
supported by wpa_supplicant/hostapd, but the most reliable results are
likely available through ath9k-based devices. If you are using something
else, it is strongly recommended that you'll run the first tests with
sniffer captures and verify that the test tools are behaving correctly.

wpa_supplicant is used to control a test device in station mode to test
an AP and hostapd is similarly used to control a test device in AP mode
to test a station.

Various data traffic generators could be used to test the behavior, but
this document focuses on using ping to test unicast traffic and arping
to test broadcast traffic. To keep things simple and to reduce
interference from unrelated traffic, the steps here assume static IPv4
addresses are used and IPv6 is disabled.

The tests here use WPA2-Personal for simplicity. WPA2-Enterprise and
other cipher suites can also be tested for more complete coverage.

Example hostapd.conf for the test tool in AP mode:

driver=nl80211
hw_mode=g
channel=1
ieee80211n=1
interface=wlan0
ctrl_interface=/var/run/hostapd
ctrl_interface_group=adm
ssid=test-psk
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
wpa_passphrase=12345678

Example wpa_supplicant.conf for the test tool in station mode:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=adm

network={
    ssid="test-psk"
    key_mgmt=WPA-PSK
    psk="12345678"
}

The examples in this document assume following IPv4 address
configuration:

Test tool (either AP or station mode): 192.168.1.1/24
Device under test: 192.168.1.2/24


Data traffic tests
------------------

ping is used to test whether unicast frames go through on the data
link. It should be noted that ping may need to use broadcast ARP at the
beginning if the other device is not yet in the ARP table, so working
broadcast and unicast connectivity may be needed to get this started.

Example:

$ ping -n -c 5 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=43.7 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=67.9 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=900 ms
64 bytes from 192.168.1.2: icmp_seq=4 ttl=64 time=5.81 ms
64 bytes from 192.168.1.2: icmp_seq=5 ttl=64 time=135 ms

--- 192.168.1.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 5.811/230.605/900.223/337.451 ms

This shows working unicast data connectivity.

$ ping -n -c 5 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.

--- 192.168.1.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4033ms

This shows not working unicast data connectivity.


arping is used to test broadcast connectivity.

Example:

$ arping -b -I wlan0 192.168.1.2 -c 5
ARPING 192.168.1.2 from 192.168.1.1 wlan0
Unicast reply from 192.168.1.2 [<DUT MAC address>]  119.695ms
Unicast reply from 192.168.1.2 [<DUT MAC address>]  144.496ms
Unicast reply from 192.168.1.2 [<DUT MAC address>]  166.788ms
Unicast reply from 192.168.1.2 [<DUT MAC address>]  2.283ms
Unicast reply from 192.168.1.2 [<DUT MAC address>]  2.234ms
Sent 5 probes (5 broadcast(s))
Received 5 response(s)

This shows working broadcast data connectivity.

$ arping -b -I wlan0 192.168.1.2 -c 5
ARPING 192.168.1.2 from 192.168.1.1 wlan0
Sent 5 probes (5 broadcast(s))
Received 0 response(s)

This shows not working broadcast data connectivity.

If testing results do not look consistent, the testing state can be
cleared by disconnection and reconnecting the station (the test tool or
the DUT) to the network.


Sniffer and wlantest
--------------------

It is useful to get a wireless sniffer capture from the operating
channel of the AP to be able to confirm DUT behavior if any of the data
tests indicate reason to believe something is not working as expected.

wlantest (from the wlantest directory of hostap.git) can be used to
decrypt and analyze a sniffer capture. For example:

wlantest -r wlan0.pcap -n decrypted.pcap -p 12345678

The debug prints and comments in the generated file indicate where
unexpected behavior has been detected, e.g., when the test tool ends up
clearing its packet number to test replay protection. That can help in
checking whether the DUT actually replies to a frame that it was
supposed to drop due replay.


Testing replay protection on a station device
---------------------------------------------

Start hostapd and use hostapd_cli on the test device to control testing
operations. Connect the DUT to the network.

<3>AP-STA-CONNECTED <DUT MAC address>

This indicates that the connection was completed successfully.

Verify that broadcast and unicast traffic works correctly (if not,
something is wrong in the test setup and that needs to be resolved
before being able to run any tests).

Verify that unicast traffic works and issue the following command in
hostapd_cli:

> raw RESET_PN <DUT MAC address>
OK

Verify that unicast traffic does not work anymore. If it does, the DUT
does not implement replay protection correctly for unicast frames. Note
that unicast traffic can recover once the packet number from the test
device increases beyond the value used prior to that RESET_PN command.


Verify that broadcast traffic works and issue the following command in
hostapd_cli:

> raw RESET_PN ff:ff:ff:ff:ff:ff
OK

Verify that broadcast traffic does not work anymore. If it does, the DUT
does not implement replay protection correctly for broadcast
frames. Note that broadcast traffic can recover once the packet number
from the test device increases beyond the value used prior to that
RESET_PN command.


Testing replay protection on an AP device
-----------------------------------------

Start the AP (DUT) and start wpa_supplicant on the test device to
connect to the network. Use wpa_cli to control the test device.

<3>SME: Trying to authenticate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
<3>CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
<3>Trying to associate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
<3>Associated with <DUT MAC address>
<3>WPA: Key negotiation completed with <DUT MAC address> [PTK=CCMP GTK=CCMP]
<3>CTRL-EVENT-CONNECTED - Connection to <DUT MAC address> completed [id=0 id_str=]

Verify that unicast traffic works and issue the following command in
wpa_cli:

> raw RESET_PN
OK

Verify that unicast traffic does not work anymore. If it does, the DUT
does not implement replay protection correctly. Note that unicast
traffic can recover once the packet number from the test device
increases beyond the value used prior to that RESET_PN command.

IEEE 802.11 protocol uses unicast frames in station-to-AP direction, so
there is no need to test AP replay protection behavior separately with
the broadcast IPv4 traffic (which would be converted to unicast frames
on the link layer).


Testing GTK reinstallation protection on a station device (group handshake)
---------------------------------------------------------------------------

Use the procedure describe above for testing replay protection, but with
the following hostapd_cli commands:

Test broadcast connectivity; should work

> raw RESEND_GROUP_M1 <DUT MAC address>
OK
> raw RESET_PN ff:ff:ff:ff:ff:ff
OK

Test broadcast connectivity; should not work; if it does, the device
does not implement protection for delayed retransmission of Group Key
Message 1/2.


Testing GTK reinstallation protection on a station device (4-way handshake)
---------------------------------------------------------------------------

Use the procedure described above for testing replay protection for
broadcast traffic, but with the following hostapd_cli commands:

Test broadcast connectivity; should work

> raw RESEND_M3 <DUT MAC address>
OK
> raw RESET_PN ff:ff:ff:ff:ff:ff
OK

Test broadcast connectivity; should not work; if it does, the device
does not implement protection for delayed retransmission of 4-way
handshake EAPOL-Key Message 3/4.

Variant 1: Include extra Message 1/4

Otherwise same as above, but replace RESEND_M3 command with:

> raw RESEND_M1 <DUT MAC address>
OK
> raw RESEND_M3 <DUT MAC address>
OK

Variant 2: Include two extra Message 1/4

Otherwise same as above, but replace RESEND_M3 command with:

> raw RESEND_M1 <DUT MAC address> change-anonce
OK
> raw RESEND_M1 <DUT MAC address>
OK
> raw RESEND_M3 <DUT MAC address>
OK


Testing TK reinstallation protection on a station device (4-way handshake)
--------------------------------------------------------------------------

Use the procedure described above for testing replay protection for
unicast traffic, but with the following hostapd_cli commands:

Test unicast connectivity; should work

> raw RESEND_M3 <DUT MAC address>
OK
> raw RESET_PN <DUT MAC address>
OK

Test unicast connectivity; should not work; if it does, the device
does not implement protection for delayed retransmission of 4-way
handshake EAPOL-Key Message 3/4.

Variant 1: Include extra Message 1/4

Otherwise same as above, but replace RESEND_M3 command with:

> raw RESEND_M1 <DUT MAC address>
OK
> raw RESEND_M3 <DUT MAC address>
OK

Variant 2: Include two extra Message 1/4

Otherwise same as above, but replace RESEND_M3 command with:

> raw RESEND_M1 <DUT MAC address> change-anonce
OK
> raw RESEND_M1 <DUT MAC address>
OK
> raw RESEND_M3 <DUT MAC address>
OK


Testing ANonce generation on an AP device
-----------------------------------------

Start the AP (DUT) and start wpa_supplicant on the test device to
connect to the network. Use wpa_cli to control the test device.

<3>SME: Trying to authenticate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
<3>CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
<3>Trying to associate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
<3>Associated with <DUT MAC address>
<3>WPA: Key negotiation completed with <DUT MAC address> [PTK=CCMP GTK=CCMP]
<3>CTRL-EVENT-CONNECTED - Connection to <DUT MAC address> completed [id=0 id_str=]

Show the ANonce from the first 4-way handshake, request PTK rekeying,
and show the ANonce from the second 4-way handshake:

> GET anonce
df8c61d1f1f7aca9f1739dd888199547f4af2b8b07f8bf15b45ea271da0072b2
> raw KEY_REQUEST 0 1
OK
> GET anonce
d8ddcb716f28abfdf1352a05d51e7a70f58802122e99d13c730c3c0f09594aac

If the ANonce values are same, the AP did not update the ANonce for
rekeying (it should have as shown in the example above).


Testing FT Reassociation Request frame retransmission on an AP device
---------------------------------------------------------------------

This test case requires a sniffer to be used and manually analyzed.

Enable FT on the DUT AP (likely two AP devices needed), connect test
tool to the AP using FT protocol (e.g., connect to another AP first and
then use the "ROAM <BSSID>" command), and do the following steps:

- verify unicast traffic from the AP to test station (either ping from
  the AP or from a device behind the AP); this needs to work
- wpa_cli "raw RESEND_ASSOC"
- verify unicast traffic from the AP to test station (either ping from
  the AP or from a device behind the AP); this is likely to fail, but
  the real analysis is done based on the sniffer capture

In the sniffer capture, find the last Reassociation Request frame from
the test station (this is more or less identical to the previous one and
the only one that should not have Authentication frame exchange before
it). Look at the last used PN in a unicast Data frame from the AP to the
test station before the last Reassociation Request frame and the PN in
the following unicast Data frame after the last Reassociation Request
frame. If the PN goes down (e.g., is reset to 1), this would be a sign
of a likely security vulnerability. The AP's TK configuration should be
verified (i.e., whether it is configuring the same TK again and then
allowing it to be used with reused PN values).