| 24h-stress-test |
Run stress_ng, stress_oom and stress_network tests |
functional, performance |
| Firmware-Test-Suite |
FirmWare Test Suite (fwts) |
functional |
| S-suite |
The S-suite is a suite of I/O-performance benchmarks.This test uses the S-suite to run, for every I/O scheduleravailable for the target device, two main benchmarks:1. A throughput benchmark, which measures throughput with random and sequential sync I/O. In this respect, sync I/O is the kind of I/O for which it is most difficult to reach a higher throughput.2. A responsiveness benchmark, which measures the start-up time of various popular application, in the presence of sequential I/O in the background (sequential is the kind of background I/O that makes it more difficult to guarantee a high responsiveness |
I/O performance |
| aklite-apps |
Run apps without registration The test checks if docker-compose apps can be started with 'aklite-apps'. When using APPS parameter the script will runaklite-apps run --apps ${APPS}When using DEFAULT_APPS the script will runaklite-apps run |
functional |
| aklite-download-interrupt |
This test confirms that aktualizr-lite download was successfully resumed after timeout caused by missing networking on DUT. |
functional |
| aklite-download-interrupt |
This test stops at the beginning of download update by aktualizr-lite. It can be used in a test job with user commands to disable DUT networking and force timeout during OTA download. |
functional |
| aklite-download-resume |
This test checks for ostree download timeout in the aktualizr-lite when there is no network connectivity at DUT. |
functional |
| aklite |
Check basic information provided by aktualizr-lite. This includes systemd service status, device registration status and actual target. If device is registered, device name is retrieved. |
functional |
| apache-apache-bench |
Apachebench is a benchmark created to measure the webserver performance on HTTP protocol. This test runs apachebench against apache webserver |
functional |
| apps-early-start |
Run apps without registration using systemd Setting DEFAULT_APPS tells the script which docker-compose apps should be running. |
functional |
| armnn-mlperf-benchmark |
Carry out armnn-mlperf benchmark test. |
functional |
| armnn-unit-tests |
Carry out armnn unit tests. |
functional |
| badblocks |
Run a read-only test for bad blocks |
functional |
| block-device |
Test a partition on a block device.Check that files can be written and read reliably, measure the timings for read and write, and run the bonnie++ block device test on the device. |
blkdev, bonnie++, performance, functional |
| blogbench |
Blogbench is a portable filesystem benchmark that tries to reproduce the load of a real-world busy file server. |
performance |
| bootrr |
Run bootrr https://github.com/andersson/bootrr |
functional |
| busybox |
Busybox smoke tests. |
functional |
| crypto |
Verify that every crypto algorithms pass their selftests |
crypto |
| cryptsetup |
Cryptsetup benchmark tests |
crypto |
| cyclicdeadline |
cyclicdeadline is a test that is similar to cyclictest but instead of using SCHED_FIFO and nanosleep() to measure jitter, it uses SCHED_DEADLINE and has the deadline be the wakeup interval. |
performance, preempt-rt |
| cyclictest |
cyclictest measures event latency in Linux kernel by measuring the amount of time that passes between when a timer expires and when the thread which set the timer actually runs. |
performance, preempt-rt |
| dd-speed-test |
dd write/read speed test. |
performance |
| deqp-runner |
The drawElements Quality Program (deqp) Testing allows to test OpenGL API's, uses OpenGL ES CTS from https://github.com/KhronosGroup/VK-GL-CTS. |
functional |
| device-read-perf |
Use 'hdparm -t' to perform timings of block device reads for benchmark and comparison purposes. |
performance |
| device-tree |
Device tree test to check the folder structure. The test verifies that device-tree is available and contains model name of the board. |
functional |
| disable-aklite-reboot |
Disable automatic reboot after update. This is required when running long test. It prevents the board from rebooting when the tests didn't yet complete. |
functional |
| disk-partitioning-test |
Disk partitioning test. |
functional |
| docker-networking |
Docker networking test. It will try to find the 'bridge' network, start a container and ping it. Container by default should use 'bridge' network. |
functional |
| docker-prune |
Prune app image after disabling app |
functional |
| docker-prune |
Download OTA update and prune old container images |
functional |
| docker |
Docker smoke test. When docker not pre-installed, docker-ce will be installed from upstream on supported distributions. |
functional |
| dsdbench |
dsdbench runs benchmarks and tests for storage driver configurations to help figure out how the configuration will perform and which known issues the daemon may be affected by in this configuration. |
performance, functional |
| el2go-registration |
Basic test suite EdgeLock2Go integration. The test enables EL2GO registration to the FoundriesFactory. If the device is successfully registered, the test passes. Running the test requires FoundriesFactory with EL2GO integration and a device with SE050 secure element.Optionally AWS IoT integration can be tested by setting AWS_ENDPOINT and AWS_CONTAINER variables. It is recommended to use Foundries.io awsiot-optee container with corresponding LmP release. The container sources can be found on github: https://github.com/foundriesio/containers/tree/master/awsiot-opteeAWS_ENDPOINT is the URL returned by: aws iot describe-endpoint --endpoint-type iot:Data-ATS --query 'endpointAddress' --output textIt is recommented to reset SE050 secure element after the test. This is done by setting RESET_SE05X to True. |
functional |
| ethernet |
This test checks if Ethernet is up and prints IP address. |
functional |
| fdisk |
Create partitions with fdisk command and format with file system type |
functional |
| fio-test |
FIO or Flexible IO is a versatile IO workload generator test on Linux. |
performance |
| fs-resize |
Test if rootfs was properly resizedThis test checks whether rootfs occupies the whole block device. It tests the scenario when rootfs should be resized after first boot. |
functional |
| fuego-dut |
Run fuego tests with LAVA. Refer to http://fuegotest.org/ |
functional |
| fuego-host |
Run fuego tests with LAVA. Refer to http://fuegotest.org/ |
functional |
| glmark2 |
Glmark2 test for Linux is an OpenGL 2.0 and ES 2.0 benchmark. The Glmark2 test suite is used to measure different aspects of OpenGL (ES) 2.0 performance like video, graphics and display. |
performance |
| gpiod |
Run a gpiod suite |
functional |
| gst-validate |
A suite of tools to run integration tests for Gstreamer, For more information: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/ |
functional |
| guest-complete |
Signal tests inside guest complete and power off |
functional, virtualization |
| hackbench |
Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. It's main job is to create a specified number of pairs of schedulable entities (either threads or traditional processes) which communicate via either sockets or pipes and time how long it takes for each pair to send data back and forth. |
performance, virtualization, preempt-rt |
| hci-smoke |
Basic HCI smoke tests The test will run the following commands: - hciconfig up/down - hcitool scan - hcitool lescan (when le mode is enabled) - hcitool inq When testing LE mode some helper BT device must be in DUT's proximity. |
functional |
| httperf-nginx-linux-ip |
Server running plain NGiNX, no acceleration |
|
| httperf-nginx-ofp-odp-dpdk-git |
Server running OFP+ODP+DPDK NGiNX from git |
|
| httperf-nginx-ofp-odp-dpdk |
Server running OFP+ODP+DPDK NGiNX |
|
| httperf-runner |
Run httperf against a web webserver to measure performance |
|
| igt-test |
Run igt-gpu-tools with a given test list |
functional |
| ima-smoke |
Check if IMA is enabled in the kernel The test only checks kernel configs presence, IMA initialization and presence of measurements file. There is no comparison between measurements taken in different boot stages. There is also no requirement to use TPM. |
functional |
| imagenet-resnet50 |
imagenet-resnet50 job, running resnet50 benchmark through mlperf. |
functional |
| imagenet-ssd-resnet32 |
imagenet-ssd-resnet32 job, running resnet32 benchmark through mlperf. |
functional |
| install-android-platform-tools |
Install android platform tools provided by Google from https://developer.android.com/studio/releases/platform-tools |
functional |
| iozone |
IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. |
performance |
| iperf |
iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks. |
performance |
| iperf |
iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks. |
performance |
| kernel-compilation |
Compile kernel with defconfig on ARM64/ARM platform and measure how long it takes. |
functional, performance |
| kernel-config-checker |
Check the kernel config file for the presence of required CONFIG_* values. |
functional, performance |
| kselftest-chroot |
Run a chroot |
functional |
| kselftest |
Linux kernel self test |
functional |
| kunit-tests |
KUnit is a lightweight unit testing and mocking framework forthe Linux kernel. KUnit consists of a kernel component, whichprovides a set of macros for easily writing unit tests. Testswritten against KUnit will run on kernel boot when built-in |
functional |
| kvm-unit-tests |
Build and run kvm-unit-tests natively |
functional |
| kvm |
Start KVM for secondary testing |
functional, virtualization |
| lamp |
A LAMP stack is a group of open source software that is typically installed together to enable a server to host dynamic websites and web applications. |
functional |
| lapack |
Build and run lapack (Linear Algebra PACKage) tests natively |
functional |
| lemp |
LEMP stack is a group of open source software to get web server up and running. The acronym stands for Linux, nginx, MySQL and PHP. |
functional |
| libcurl |
Test exercising libcurl support for certificates without SAN |
functional |
| libhugetlbfs |
Build and run libhugetlbfs tests. Runs through $WORD_SIZE bit libhugetlbfs on target system. The default WORD_SIZE is 64 bit to accomodate 64 bit systems but for 32 bit systems the WORD_SIZE value should be set to 32 from the json file passed to the scheduler. One needs to run with a kernel that supports huge pages |
functional |
| linpack |
Linpack benchmark calculates fLoating point operations per second(FLOPS). |
performance |
| lisa-postprocessing |
Post-processing of results generated by Workload Automation v3. |
|
| lkp |
lkp-tests (Linux Kernel Performance) is an open source tool that allowsdevelopers to evaluate their patches in a thorough way. It integrates 80+popular industry open source test suites and provides a standard interfaceto do installation, execution and result analysis. Besides the outputs frombenchmark itself, it is specialized at collecting data from every aspect ofsystems such as vmstat, which is very handy to assist performance analysisand root causing issues. Lkp-tests is integrated in 0-Day for Linux kernelregression test. |
functional |
| lmbench-memory |
Run lmbench memory bandwidth and latency tests. |
performance |
| lmp-device-register |
Run lmp-device-register and check if the output is corret. It assumes the device is flashed with FoundriesFactory image.Providing token is essential for proper test execution. Part of the test script is attempting to use a static, invalid token.HSM module can be set to either softhsm or hardware backed libckteec: /usr/lib/softhsm/libsofthsm2.so or /usr/lib/libckteec.so.0.1.0UUID is usally generated randomly during test execution. It can be set to a constant value to exercise the backend response. Attempt to re-register the same UUID should fail with proper HTTP code. |
functional |
| lmp-factory-reset-verify |
Verify device factory reset. The end result should be a device filesystem restored to the condition after initial manufacturing. Only a limited subset of files is checked. |
functional |
| lmp-factory-reset |
Perform device factory reset. The end result should be a device filesystem restored to the condition after initial manufacturing. Only a limited subset of files is checked. Reset type is evaluated in the following priority:1. factory_reset 2. factory_reset_keep_sota 3. factory_reset_keep_sota_dockerIf higher priority type is present, lower priority type is ignored.Device can be registered with specifig tag using LABEL variable. Device can be registered with HSM using HSM_MODULE variable. |
functional |
| local-daemon |
docker-ce engine integration test for local daemon. Docker's tests are designed to run inside docker container. This test case clones docker-ce repo and checks out the released tag to test and then lunchs a container for testing. Inside the container, it creats a new binary and runs a local docker daemon, and then runs all tests against the daemon. |
functional |
| local-run |
Run tests from https://git.linaro.org/qa/test-definitions.git with docker. Test runs on test target. |
functional, performance |
| lshw |
lshw is a small tool to extract detailed information on the hardware configuration of the machine. |
functional |
| ltp-chroot |
Run a chroot |
functional |
| ltp-open-posix |
Run LTP open posix test suite |
functional |
| ltp-realtime |
Run LTP realtime test suite on Ubuntu/CentOS/Openembedded. This testsuite contains some functional tests and a few performance and latency measurement tests |
preempt-rt |
| ltp |
Run LTP test suite on Ubuntu |
functional |
| meminfo |
Collect the information in /proc/meminfo after boot. |
functional |
| memtester |
Stress test to find memory subsystem faults |
functional, stress |
| mmc-test |
The Multi Media Card (MMC) is a memory card standard used for solid-state storage This will perform Basic MMC tests |
functional |
| mmtests |
MMTests is a configurable test suite that runs performance tests against arbitrary workloads. This is not the only test framework but care is taken to make sure the test configurations are accurate, representative and reproducible. Reporting and analysis is common across all benchmarks. Support exists for gathering additional telemetry while tests are running and hooks exist for more detailed tracing using ftrace or perf. |
|
| network-manager-smoke |
The test runs basic nmcli subcommands |
functional |
| network-tests-basic |
Basic network test commands |
functional |
| ngingx-apache-bench |
Apachebench is a benchmark created to measure the webserver performance on HTTP protocol. This test runs apachebench against apache nginx |
performance |
| offline-run |
Perform 2nd part of offline-update with aklite-offline |
functional |
| offline-update |
Perform 1st part of offline-update with aklite-offline |
functional |
| openjdk-smoke-test |
Install openjdk, compile and run HelloWorld. |
functional |
| openssh-debian |
Normal OpenSSH development produces a very small, secure, and easy to maintain version for the OpenBSD project. The OpenSSH Portability Team takes that pure version and adds portability code so that OpenSSH can run on many other operating systems More details in wiki: http://www.openssh.com/portable.html |
functional |
| openssl-speed |
Use openssl speed command to test the performance of cryptographic algorithms |
performance |
| optee-xtest-qemu |
OP-TEE sanity test suite using QEMU |
functional |
| optee-xtest |
OP-TEE sanity test suite |
functional, performance |
| ostree |
Test if OSTree hash on the device matches the expected value |
functional |
| ota-apps |
Download OTA apps update. Use eiter proper or corrupt app |
functional |
| ota-reboot-verify |
Verify system booted without upgrade after reboot |
functional |
| ota-rollback-download |
Download OTA update and corrupt kernel or uboot to force rollback. |
functional |
| ota-rollback-verify |
Verify system booted without upgrade |
functional |
| ota-static-delta-upgrade-download |
Download OTA assuming static delta is available |
functional |
| ota-update-reboot |
Verify system booted with upgrade. In case of u-boot upgrade it should boot to primary partition with proper firmware version. |
functional |
| ota-update-verify |
Verify system booted with upgrade |
functional |
| ota-upgrade-download |
Download OTA update and (optionally) force u-boot upgrade. |
functional |
| over-ssh |
Run tests from https://git.linaro.org/qa/test-definitions.git with docker. Test run will be triggered over ssh |
functional, performance |
| overlayfs |
Run overlay filesystem test from unionmount testsuite. Kernel config CONFIG_OVERLAY_FS=y is required for this test. |
functional |
| packetdrill |
The packetdrill scripting tool enables quick, precise tests for entireTCP/UDP/IPv4/IPv6 network stacks, from the system call layer down to the NIChardware. packetdrill currently works on Linux, FreeBSD, OpenBSD, and NetBSD.It can test network stack behavior over physical NICs on a LAN, or on a singlemachine using a tun virtual network device.The code is GPLv2. Currently the source for the testing tool and a number oftest scripts is in the git repository. We will continue to post more tests fromour team's Linux TCP test suite (described in our USENIX paper), as timepermits. |
functional |
| perf |
Perf is a profiler tool for Linux and is used to collect and analyze performance and trace data. This test runs following perf commands: record(record events for later reporting), report(break down events by process, function, etc), stat(obtain event counts) and test(Run sanity tests). |
performance |
| pi-stress |
pi_stress checks Priority Inheritence Mutexes and their ability to avoid Priority Inversion from occuring by running groups of threads that cause Priority Inversions. |
functional, preempt-rt |
| piglit |
Piglit is an open-source test suite for OpenGL implementations. For more information: https://piglit.freedesktop.org/ |
functional |
| pkcs11-tests |
Basic test suite for pkcs11 operations.The main motivation behind the tests is checking the interactionwith SE05x TPM devices. Tests can also be used with softhsm ifTPM is not availableExample use with TPM: ./pkcs11.sh -p 'pkcs11-tool --module /usr/lib/libckteec.so.0.1'Example use with softhsm: ./pkcs11.sh -p 'pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so' -t falsessscli tool comes from NXP and is not packaged for any distributionat the moment. It needs to be preinstalled in the OS in order to useSE05x TPM for this test. |
functional |
| pm-qa |
Test power management (PM-QA). Currently, the test runs cpufreq, cpuidle, cpuhotplug, thermal and cputopology by default and individual test can be run by setting TESTS parameter in test job definition too. Test case documentation available in wiki: https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/TestSuite/PmQaSpecification |
functional, performance |
| pmqtest |
Pmqtest measures the latency of interprocess communication with POSIX messages queues. The test starts pairs of threads that are synchronized via mq_send/mw_receive() and measures the latency between sending and receiving the message. |
performance, preempt-rt |
| pmsematest |
ptsematest starts threads that are synchronized via pthread_mutex_unlock()/pthread_mutex_lock() and measures the latency between releasing and getting the lock. |
performance, preempt-rt |
| pointer-tagging-tests |
This test suite is designed to verify that the AArch64 tagged-addressing feature is properly supported. Tagged addressing is guaranteed to be available in the ARMv8 architecture |
functional |
| pritee_test_utility |
PlayReady Interface for Trusted Execution Environment Test Utility |
functional |
| ptest |
OpenEmbedded Package tests support (ptest) The pacakge tests from OpenEmbedded enables the support to run package unittests into the target device. For more information: https://wiki.yoctoproject.org/wiki/Ptest |
functional |
| pyarmnn-benchmark |
Part of ml-bench, pyarmnn-bench testing validity of pyarmnn pip wheels. |
functional |
| rcutorture |
RCU or Read-Copy Update Torture test for Linux Kernel. Kernel config required for this test CONFIG_RCU_TORTURE_TEST=m |
functional |
| rt-migrate-test |
rt-migrate-test verifies the RT threads scheduler balancing. |
performance, preempt-rt |
| rteval |
Rteval is a python program written to evaluate the performance of arealtime Linux kernel on a particular hardware platform. The programunpacks source code for two loads; hackbench and a Linux kernelcompile, then loops running hackbench and a parallel build of theLinux kernel. While the loads are running, the cyclictest program isrun to measure realtime performance under load. When the specified runduration is met, the loads are stopped and cyclictest outputs measuredtimer latency values in histogram format, which is analyzed byrteval. Rteval then writes an XML file to disk with information aboutthe system (gotten through DMI tables), the raw histogram datacollected during the run and the statistical analysis of the run. |
performance |
| rtla-osnoise |
The rtla osnoise tool is an interface for the osnoise tracer. The osnoise tracer dispatches a kernel thread per-cpu. These threads read the time in a loop while with preemption, softirq and IRQs enabled, thus allowing all the sources of operating system noise during its execution. The osnoise’s tracer threads take note of the delta between each time read, along with an interference counter of all sources of interference. At the end of each period, the osnoise tracer displays a summary of the results. |
performance, preempt-rt |
| rtla-timerlat |
The rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency. |
performance, preempt-rt |
| secure-boot-enabled |
Validate UEFI SecureBoot is enabled by checking the value of variable8be4df61-93ca-11d2-aa0d-00e098032b8c-SecureBoot. This long variable ispredefined in the UEFI standard and it will never change. Refer tohttps://uefi.org/specs/UEFI/2.10/03_Boot_Manager.html#globally-defined-variables |
functional |
| signaltest |
signaltest is a RT signal roundtrip test software. |
performance, preempt-rt |
| sigwaittest |
sigwaittest starts two threads or, optionally, forks two processes that are synchronized via signals and measures the latency between sending a signal and returning from sigwait(). |
performance, preempt-rt |
| smoke-tests-basic |
Run basic smoke tests. |
functional |
| spectre-meltdown-checker |
Run spectre meltdown checker |
functional |
| ssd-mobilenet |
ssd-mobilenet job, running ssd-mobilenet benchmark through mlperf. |
functional |
| stream-uniprocessor |
STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s). This test case run STREAM on a single processor. |
performance |
| svsematest |
svsematest starts two threads or, optionally, forks two processes that are synchronized via SYSV semaphores and measures the latency between releasing a semaphore on one side and getting it on the other side. |
performance, preempt-rt |
| switch-apps |
Switch list of the active apps. Use fioconfig callback handler in the process. |
functional |
| sysbench |
SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. Current features allow to test fileio, cpu, memory, threads, mutex and oltp. |
performance |
| sysfs-bus-iio-smoke |
Range checks for particular properties in the sysfs bus iio subsystem. |
functional, performance |
| timesyncd-test |
The test checks whether the systemd-timesyncd is running and uses NTP. It should only be used when it's expected to use NTP. As an additional check the test will contact default NTP server to check whether the time is synced properly. |
functional |
| toolchain-smoke |
Simple test that checks whether gcc compilation works |
functional |
| tvm-benchmark |
Part of ml-bench, tvm-bench tests various ml benchmarks using tvm. |
functional |
| tvm-unit-tests |
Carry out tvm unit tests from the TVM/tests/scripts dir from https://github.com/apache/tvm. The purpose is to check the validity of PR's and ensure that there are no regressions. Apache TVM is a compiler stack for deep learning systems. Current UnitTests being tested include cpp and crt unit tests. |
functional |
| uart-loopback |
UART functional test. This runs a functional test across two connected UARTs. It tests baud rates from 9600 to 230400, a selection of parity, CTS/RTS, and stop bit settings. |
functional, uart, loopback |
| ui-browser-test |
ui browser tests by using robot frame work Test runs chromium, chrome and firefox browser test cases. |
functional |
| unixbench |
The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system. |
performance |
| usb-test-basic |
Basic USB test for Linaro Ubuntu images. The test examines all available USB devices and prints supported USB protocols and speed. |
functional |
| v4l2-compliance |
v4l2 Compliance test suite |
functional |
| vdsotest |
vdsotest is a utility for testing and benchmarking a Linux VDSO.The "vDSO" (virtual dynamic shared object) is a small sharedlibrary that the kernel automatically maps into the address spaceof all user-space applications. Applications usually do not needto concern themselves with these details as the vDSO is mostcommonly called by the C library. This way you can code in thenormal way using standard functions and the C library will takecare of using any functionality that is available via the vDSO. |
performance, functional |
| verify_isolation |
Verify kernel configuration. Isolate application and verify that kernel tick/resch interrups do not occur more than 1/sec. |
functional |
| widevine_unittest |
Widevine Content Decryption Modules (CDMs) are how streaming services protect content using HTML5 video to web browsers without relying on an NPAPI plugin like Flash or Silverlight. This test runs widevine_ce_cdm_unittest |
functional |
| wireguard-fioconfig |
Generate wireguard public key before server starts and check that key is updated correctly via fioconfig. |
functional |
| wlan-download |
Basic WLAN download test, requires ip, iw, wpa_supplicant and dhclient |
functional |
| wlan-smoke |
Basic WLAN smoke tests |
functional |
| workload-automation-linux |
Workload Automation on Linux |
performance |
| workload-automation-linux |
Workload Automation on Linux |
performance |
| xenomai4 |
Run 'evl test' from Xenomai4 |
functional, performance |
| xfstests |
xfstests is a test suite for validating the reliability and stability of file systems in the Linux kernel. |
functional |