Test Definitions¶

A test suite works with and without LAVA. The following two sets of automated tests are supported.

automated/linux/
automated/android/

For each test case, both the test script and the corresponding test definition files are provided in the same folder and are named after the test case name. Test scripts are self-contained and work independently. Test definition files in YAML format are provided for test runs with local test-runner and within LAVA.

Installation¶

Installing the latest development version:

git clone https://github.com/Linaro/test-definitions
cd ./test-definitions
. ./automated/bin/setenv.sh
pip install -r ${REPO_PATH}/automated/utils/requirements.txt

If the above succeeds, try:

test-runner -h

Running test¶

Running test script¶

linux¶

cd ./automated/linux/smoke/
./smoke.sh

Skip package installation:

./smoke.sh -s true

android¶

cd ./automated/android/dd-wr-speed/
./dd-wr-speed.sh

Specify SN when more than one device connected:

./dd-wr-speed.sh -s "serial_no"

Specify other params:

./dd-wr-speed.sh -i "10" -p "/dev/block/mmcblk1p1"

Using test-runner¶

single test run¶

test-runner -d ./automated/linux/smoke/smoke.yaml

skip package install:

test-runner -d ./automated/linux/smoke/smoke.yaml -s

running test plan¶

Run a set of tests defined in agenda file:

test-runner -p ./plans/linux-example.yaml

Apply test plan overlay to skip, amend or add tests:

test-runner -p ./plans/linux-example.yaml -O test-plan-overlay-example.yaml

Collecting result¶

Using test script¶

Test script normally puts test log and parsed results to its own output directory. e.g.

automated/linux/smoke/output

Using test-runner¶

test-runner needs a separate directory outside the repo to store test and result files. The directory defaults to $HOME/output and can be changed with -o <dir>. test-runner converts test definition file to run.sh and then parses its stdout. Results will be saved to results.{json,csv} by test. e.g.

/root/output/smoke_9879e7fd-a8b6-472d-b266-a20b05d52ed1/result.csv

When using the same output directory for multiple tests, test-runner combines results from all tests and save them to ${OUTPUT}/results.{json,csv}. e.g.

/root/output/result.json

More details on test-runner usage in test-runner docs

Generating documentation¶

Full docs are generated from existing YAML files. Resulting markdown files are not stored in the repository.

Using uv¶

uv run --with pyyaml --with zensical -- python generate_test_docs.py
uv run --with zensical -- zensical serve

Using pip¶

create and activate virtualenv

python3 -m venv venv
source venv/bin/activate

install requirements
```
pip install -r requirements-docs.txt
```
generate test docs from YAML definitions
```
python generate_test_docs.py
```
run zensical
```
zensical serve
```

This will start a local http server on http://127.0.0.1:8000.

To build static HTML instead:

zensical build

Files are stored in the 'site' directory.

Contributing¶

Please use Github for pull requests: https://github.com/Linaro/test-definitions/pulls

https://git.linaro.org/qa/test-definitions.git is a read-only mirror. New changes in the github repo will be pushed to the mirror every 10 minutes.

Refer to test writing guidelines to modify or add test.

Changes need to be able to pass sanity check, which by default checks files in the most recent commit:

./sanity-check.sh

To develop locally, there are Dockerfiles in test/ that can be used to simulate target environments. The easiest way to use is to run test.sh [debian|centos]. test.sh will run validate.py, and then build the Docker environment specified, run plans/linux-example.yaml, and then drop into a bash shell inside the container so that things like /root/output can be inspected. It is not (yet) a pass/fail test; merely a development helper and validation environment.

For full documentation visit test-definitions.readthedocs.io.