Commit graph

111 commits

Author SHA1 Message Date
Johannes Berg
f124367afb tests: parallel-vm: allow running without curses
Allow running without curses, in which case the log is simply written to
stdout instead of a file. This is useful for automated (but parallel)
testing. Note that in most cases, you'd want to specify --debug, and so
I added a .rstrip() there on the lines to clean that up a bit.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2020-01-20 21:17:24 +02:00
Jouni Malinen
a6ba5cca45 tests: Allow more VMs to be started in parallel
Check the number of CPUs to determine how many VMs can be started in
parallel.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 20:31:33 +02:00
Jouni Malinen
c64b6f62cd tests: Use python selector in the parallel-vm.py main loop
This gets rid of the loop that was polling for things to do every 0.25
seconds and instead, reacts to any data from VMs as soon as it becomes
available. This avoids unnecessary operations when no new data is
available and avoids unnecessary waits when new data becomes available
more quickly.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 17:12:34 +02:00
Jouni Malinen
fd0465b85e tests: Print a summary of SKIP reasons
This makes it easier to determine reasons for test cases being skipped.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 16:03:50 +02:00
Jouni Malinen
179279eb34 tests: Move VM starting delay control into parallel-vm.py
This is more efficient since we can now start only the necessary number
of VMs instead of always forcing all VMs to start with one second delay.
This can also control the starting delay by keeping at most two VMs
starting at a time instead of using the hardcoded one second wait for
each consecutive VM.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-12-27 16:03:50 +02:00
Jouni Malinen
9dc07a1d55 tests: Fix 'unexpected exit' detection in parallel-vm.py
Now that parallel-vm.py is actually stopping VMs as soon as they are not
needed for retries, it is not really an unexpected exit to see a VM exit
while test cases remain in the queue as long as at least that many VMs
remain running. Get rid of confusing 'unexpected exit' status from the
UI in such cases.

Fixes: 4aaddecdd8 ("tests: Handle test retries through the same queue")
Signed-off-by: Jouni Malinen <j@w1.fi>
2019-11-23 17:09:56 +02:00
Pradeep Kumar Chitrapu
0a809529d0 tests: HE AP Functionality
Signed-off-by: Pradeep Kumar Chitrapu <pradeepc@codeaurora.org>
2019-10-25 19:29:53 +03:00
Jouni Malinen
d137275131 tests: Remove openssl header files from code coverage report
Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-08-06 13:21:56 +03:00
Jouni Malinen
4aaddecdd8 tests: Handle test retries through the same queue
This removes the separate rerun step from the parallel-vm.py processing
and instead, simply requeues the failed test cases into the same queue
that is used for the initial run. This is simpler and more efficient
since reruns start as soon as any VM is ready for processing them
instead of having to wait for all VMs to complete the first round.
Furthermore, this allows VMs to be stopped sooner when no more test
cases remain and that is helpful especially with the time travel patches
that make the wait-for-next-test step in the VM use all available CPU.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-07-27 20:19:28 +03:00
Johannes Berg
2080f4c779 tests: uml: Request non-raw serial ports
When the serial ports are set into raw mode on stdio (fd:0,fd:1)
then Ctrl-C is sort of passed through, but not effective. Request
non-raw mode to avoid that and let us cancel test execution with
Ctrl-C properly (both in parallel-vm.py and vm-run.sh cases).

Note that this requires a currently out-of-tree patch, but so
does the virtual time. If the patch is not applied, the command
line argument is ignored.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-05-28 13:34:44 +03:00
Jouni Malinen
d55035c935 tests: Use time-travel=inf-cpu with UML
This is the current proposed way of configuring virtual time.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-05-27 22:43:04 +03:00
Jouni Malinen
8528ad0c60 tests: Enable virtual time in UML
This speeds up test execution significantly by removing unnecessary
waiting for things to happen since the kernel log is allowed to jump
forward whenever there is nothing to do.

Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-05-06 20:29:16 +03:00
Jouni Malinen
1cefd89b37 tests: Support UML kernel as a replacement for KVM
This allows UML builds to be used in running user mode without having to
run the full x86 kernel in virtual machine.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-05-05 00:44:04 +03:00
Jouni Malinen
fab49f6145 tests: Python coding style cleanup (pylint3 bad-whitespace)
Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-03-16 18:52:09 +02:00
Jouni Malinen
a2d300765b tests: Make parallel-vm.py log easier to use for debugging failures
Print a list of full paths to log files from failed test cases both the
parallel.log and stdout so that they can be easily opened for analysis.
In addition, renumber the VM lines in the <timestamp>-parallel.log to
match the i+1 numbering used in the log directories and UI that is
tracking test execution.

Signed-off-by: Jouni Malinen <jouni@codeaurora.org>
2019-03-15 21:10:00 +02:00
Lubomir Rintel
192d061add tests: Drop testing of the old D-Bus interface
This old interface has been obsoleted and should not have been used
since 2010, so remove testing for it in preparation to dropping the
interface completely from wpa_supplicant.

Signed-off-by: Lubomir Rintel <lkundrak@v3.sk>
2019-03-09 18:23:09 +02:00
Masashi Honma
432496cdd8 tests: Switch default python version to 3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-17 17:26:01 +02:00
Masashi Honma
51c83edfca tests: Catch only specific exceptions
This allows unexpected cases to terminate parallel-vm.py without being
hidden by the exception handler.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-11 12:42:52 +02:00
Masashi Honma
45d3e2edbd tests: Explicitly flush stdin for python3
Without this flush(), test does not run.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
3069be8fe3 tests: Encode VM input for python3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
689a956049 tests: Decode VM output for python3
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Masashi Honma
0da6d93dbf tests: Change handling of reading non blocked empty stream for python3
The result of reading non blocked empty stream is different between
python2 and 3. The python2 sends "[Errno 11] Resource temporarily
unavailable" exception. The python3 could read "None" without
exception, so handle this "None" case as well.

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-02-10 01:43:50 +02:00
Johannes Berg
e5a410f472 tests: Make /etc/alternatives work in VM
In recent Debian versions, ebtables is an alias managed by
the alternatives(8) mechanism. This means /usr/sbin/ebtables
is a symlink to /etc/alternatives/ebtables, which in turn
links to the real binary.

As we mount a tmpfs over /etc, we cannot access this.

Fix this by bind-mounting the real /etc to /tmp/etc and
adding a symlink from /etc/alternatives to this.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-02-05 23:51:37 +02:00
Johannes Berg
e339511007 tests: Optionally start telnet server inside VMs
If telnetd is installed and --telnet <port> is passed on the
vm-run.sh command line, start a telnet server (directly connected
to bash, no login) inside the VM(s) to be able to look into them
when something is wrong. Use a user network in qemu with a single
host forward from the specified port for this, listening only on
'localhost'.

Please note that this provides unauthenticated access to the guest
system from anything that can open a TCP connection on the host system.
The guess system does have access to reading all files on the host that
the user account running kvm has access to (and even write access if the
default ROTAG ,readonly parameter is cleared). In other words, this
option should not be used on any multiuser systems where kvm is run
under user accounts that are not dedicated for testing purposes (i.e.,
do not have access to any files that should not be readable to
everyone).

This needs CONFIG_VIRTIO_NET=y in the guest kernel.

For parallel-vm.py, the --telnet argument specifies the base port
and each VM index (0, 1, ...) is added to it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-02-05 14:26:36 +02:00
Johannes Berg
b0797ec955 tests: Suppress annoying console reset from VMs
Recently, qemu/seabios grew an annoying console/terminal reset,
which also causes my terminal to be left in a state where long
lines don't work well and less gets confused because of this.

Suppress this by suppressing all output from qemu before a new
magic string printed from inside.sh.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-02-05 13:35:15 +02:00
Johannes Berg
8bdefe511c tests: vm-run: Create a symlink to the latest logs
This is useful when running a test multiple times, looking at
log output etc. to not have to pick out the right directory
each and every time.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2019-02-01 23:00:42 +02:00
Masashi Honma
255f3846a9 tests: ap_eap test requires more memory in VM
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-01-26 13:05:44 +02:00
Masashi Honma
89896c0008 tests: Use python3 compatible print statement
This patch is made by using 2to3 command.

$ find . -name *.py | xargs 2to3 -f print -w -n

Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
2019-01-26 12:37:25 +02:00
Jouni Malinen
ad171ccd82 tests: Update README to refer to parallel-vm.py
parallel-vm.sh was removed earlier.

Signed-off-by: Jouni Malinen <j@w1.fi>
2019-01-22 13:27:31 +02:00
Jouni Malinen
43d174b8e2 tests: Automatic channel selection, HT scan, and DFS
Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-23 17:25:11 +02:00
Jouni Malinen
2cef6f6e89 tests: Enable dynamic debug from cfg80211/mac80211
These debug logs were lost due to CONFIG_DYNAMIC_DEBUG=y.

Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-22 00:18:33 +02:00
Jouni Malinen
3a80672e22 tests: Remove parallel-vm.sh
parallel-vm.py has obsoleted this a long time ago and there is no need
to maintain two scripts for doing more or less the same thing.

Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-08 15:35:47 +02:00
Jouni Malinen
e4ce86f98b tests: Add dfs_etsi to the long_tests list
Signed-off-by: Jouni Malinen <j@w1.fi>
2018-12-08 15:33:31 +02:00
Andrei Otcheretianski
79bbafd78c tests: Store the correct PID in hostapd-test.pid file
The hwsim's start.sh script spawns hostapd process using "sudo".
Since sudo forks a child process, $! holds the pid of sudo itself.
Fix that by storing the PID of the child process instead.
Since in VM "sudo" is replaced with a dummy script, pass an additional
argument to run-all.sh and start.sh scripts to indicate that they are
running inside a VM.

This is needed to fix ap_config_reload and ap_config_reload_file test
cases on some platforms where sudo is apparently not relaying the
signals properly.

Signed-off-by: Andrei Otcheretianski <andrei.otcheretianski@intel.com>
2018-11-22 15:53:30 +02:00
Lubomir Rintel
c6ba02154e tests: Enable dynamic debugging for mac80211_hwsim
mac80211_hwsim module typically dumps a lot of details into the kernel
message buffer. While it's probably okay in a dedicated VM, it's way too
chatty in other setups.

The kernel allows fine-tuning logging via the dynamic debugging
facility. Let's enable all logging locations in the mac80211_hwsim
module so that we don't loose debugging output when the kernel adopts
the dynamic debug mechanism for the driver.

Signed-off-by: Lubomir Rintel <lkundrak@v3.sk>
2017-11-26 11:47:46 +02:00
Johannes Berg
18cdbb3c80 tests: Add a script to aid bisecting Linux kernel with hwsim VM
I find myself writing a version of this script every now and
then, but there's little point in that - just add one to the
tree so we can use it again.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2017-10-21 12:04:53 +03:00
Johannes Berg
100d334f43 tests: Print higher debug level on console
We capture the dmesg that contains everything, but if a test
causes a kernel crash we will miss all logging at higher levels
like debug. Change the printk level to catch all of that too.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2017-07-08 15:19:24 +03:00
Jouni Malinen
8b19401679 tests: Comment out CONFIG_DEBUG_KOBJECT_RELEASE from default config
This kernel debugging option adds multiple seconds of extra latency to
interface removal operations. While this can be worked around by
increasing timeouts in number of test cases, there does not seem to be
any clean way of working around this for PMKSA cacheching test with
per-STA VIFs (e.g., pmksa_cache_preauth_vlan_used_per_sta_vif).

To avoid unnecessary test failures, remove CONFIG_DEBUG_KOBJECT_RELEASE
from the default config. If someone wants to test with this kernel debug
option, it can be enabled for custom kernel builds while understanding
that it can result in false failure reports and significantly extended
time needed to complete full testing run.

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-01-29 16:06:44 +02:00
Jouni Malinen
326720c1db tests: Split proxyarp test cases into IPv4 and IPv6 parts
This is useful for now since the IPv6 support for proxyarp is not yet
included in the upstream kernel. This allows the IPv4 test cases to pass
with the current upstream kernel while allowing the IPv6 test cases to
report SKIP instead of FAIL.

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-01-29 14:32:17 +02:00
Jouni Malinen
371920b941 tests: Step-by-step guide for testing in VM
This describes example steps on how to get the VM testing setup with
parallel VMs configured with Ubuntu Server 16.04.1.

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-01-29 14:32:17 +02:00
Jouni Malinen
b85eaca0ab tests: Update example VM kernel config
This is now based on Linux 4.10.0-rc5.

Signed-off-by: Jouni Malinen <j@w1.fi>
2017-01-29 01:32:25 +02:00
Jouni Malinen
e4b5f889b4 tests: Remove src/common/cli.c from code coverage report
This file is used only by hostapd_cli and wpa_cli and neither of those
are currently included in code coverage reporting. Avoid dropping the
coverage numbers by code that cannot be reached due to not being
included in the programs that are covered.

Signed-off-by: Jouni Malinen <j@w1.fi>
2016-12-26 14:30:50 +02:00
Jouni Malinen
803d01901b tests: DFS CAC functionality on channel 104 HT40-
Signed-off-by: Jouni Malinen <j@w1.fi>
2016-03-27 20:30:40 +03:00
Jouni Malinen
56dfb604a4 Revert "tests: vm: Output everything on console"
This reverts commit be9fe3d8af. While I
did manage to complete multiple test runs without failures, it looks
like this change increases full test run duration by about 30 seconds
when using seven VMs. The most visible reason for that seems to be in
"breaking" active scanning quite frequently with the Probe Response
frame coming out about 40 ms (or more) after the Probe Request frame
which is long enough for the station to already have left the channel.

Since this logging change is not critical, it is simplest to revert it
for now rather than make changes to huge number of test cases to allow
more scan attempts to be performed before timing out.

Signed-off-by: Jouni Malinen <j@w1.fi>
2015-12-30 21:51:52 +02:00
Johannes Berg
be9fe3d8af tests: vm: Output everything on console
When running tests, make printk put all messages, including debug
messages, onto the serial console to go into the console file.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-12-24 21:27:24 +02:00
Johannes Berg
05736da8c6 tests: vm: Honor EPATH in uevent.sh
This is useful for testing CRDA since it means you can use EPATH to
redirect the test scripts to a different crda binary.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-12-18 00:24:51 +02:00
Jouni Malinen
45e8a45bb5 tests: Show vm-run.sh command line for re-running failure sequences
Now that vm-run.sh supports a long list of test cases without crashing
the VM kernel, there is no need to use the "parallel-vm.py -1 1 <tests>"
workaround. Print the re-run example commands with vm-run.sh instead. In
addition, add the --long argument if it was specified for the test run
to avoid skipping test cases in the re-run case.

Signed-off-by: Jouni Malinen <jouni@qca.qualcomm.com>
2015-11-30 19:42:56 +02:00
Johannes Berg
23fcfd6004 tests: Allow passing more arguments to vm-run.sh
The script is currently limited by the maximum kernel command line
length and if that's exceeded the kernel panics at boot. Fix this by
writing the arguments to a file and reading it in the VM.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-30 14:03:28 +02:00
Johannes Berg
680ce356c0 tests: Honor HWSIM_TEST_LOG_DIR variable in VM runs
If /tmp has a relatively small size limit, or multiple people run the
tests on the same machine, using the same output directory can easily
cause problems.

Make the test framework honor the new HWSIM_TEST_LOG_DIR environment
variable to make it easier to avoid those problems.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-27 21:11:53 +02:00
Johannes Berg
f2a438f51d tests: Print test name to kernel log buffer
Print the test name to the kernel log buffer so debugging
cross-test issues becomes easier.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
2015-11-24 16:28:24 +02:00