picofuse

src/test/README.md
1 # Test Runner
2 
3 <!-- @brief: Runs picofuse tests on real Pico hardware through a debug probe. -->
4 
5 ## Introduction
6 
7 `testrunner` runs a picofuse system test compiled for a Pico board through
8 CTest, on real hardware, via a debug probe. It's a small host-only tool
9 (never cross-compiled for `PICO_BOARD`) that:
10 
11 1. Flashes a given `.elf` onto the target via `openocd`.
12 2. Resets the target and lets it run.
13 3. Watches the target's own output for one of two markers -
14  `"[TEST] [EXIT] "` (printed by every test on a clean pass) or `"[PANIC] "`
15  (printed by `sys_panicf()` - a failed `test_assert()`, or a genuine crash) -
16  and reports pass/fail back to CTest accordingly.
17 4. Fails the test if neither marker appears before a timeout.
18 
19 Output is read one of two ways:
20 
21 - **[RTT](https://www.segger.com/products/debug-probes/j-link/technology/about-real-time-transfer/) (default)** - `openocd` opens an RTT server against the target's
22  `SEGGER RTT` control block and `testrunner` connects to it over TCP. No
23  extra wiring required beyond the debug probe's SWD connection.
24 - **Serial** (`--serial <device>`) - `openocd` programs and resets the
25  target, then exits immediately (no SWD session stays open), and
26  `testrunner` reads the target's UART output directly over a serial device
27  instead - e.g. a debug probe's separate UART-bridge interface. Useful when
28  you want the target running with no debugger attached at all (RTT
29  requires an active SWD session to poll memory; serial doesn't).
30 
31 On a host (non-`PICO_BOARD`) build, `picofuse_test()` skips `testrunner`
32 entirely and runs the compiled test executable directly.
33 
34 ## Dependencies
35 
36 `testrunner` shells out to `openocd` - it isn't a library dependency, but it
37 must be on `$PATH` (or passed via `--openocd`) for any `PICO_BOARD` test to
38 run. Use the Raspberry Pi fork's build, not a distro/mainline OpenOCD
39 package: download it from
40 [github.com/raspberrypi/pico-sdk-tools/releases](https://github.com/raspberrypi/pico-sdk-tools/releases).
41 It carries the RP2040/RP2350 target support and RTT server this tool
42 depends on, which mainline OpenOCD may lack or only partially support.
43 
44 ## Usage
45 
46 ### Flags
47 
48 ```
49 Usage: testrunner [options] <elf-file>
50 
51  --help, -h <bool> (default: true, negate: --no-help)
52  --openocd <string> (default: openocd)
53  --interface <string> (default: interface/cmsis-dap.cfg)
54  --target <string> (required)
55  --serial <string> (default: none -- read via OpenOCD's RTT server)
56  --baud <uint> (default: 115200)
57  --timeout <uint> (default: 10)
58  --verbose, -v <bool> (default: false)
59 ```
60 
61 - `--openocd` - path or bare name of the `openocd` binary (resolved against
62  `$PATH` the same way `execvp()` would if it doesn't contain a `/`).
63 - `--interface` - OpenOCD interface config, e.g. `interface/cmsis-dap.cfg`
64  for a CMSIS-DAP debug probe.
65 - `--target` - OpenOCD target config, e.g. `target/rp2040.cfg` or
66  `target/rp2350.cfg`. Required.
67 - `--serial` - when set, switches from RTT to reading a UART device
68  directly (see above). Omit to use RTT.
69 - `--baud` - baud rate for `--serial` mode.
70 - `--timeout` - seconds allowed for flashing plus waiting for a pass/fail
71  marker, combined.
72 - `--verbose` - passes `openocd`'s own stdout/stderr through instead of
73  discarding it; useful when a test is misbehaving and you need to see what
74  OpenOCD itself is doing (flash errors, RTT control block not found, etc).
75 
76 Example, run directly against real hardware:
77 
78 ```sh
79 build/src/test/testrunner --target target/rp2350.cfg \
80  build-pico/test/sys_019.elf
81 ```
82 
83 ### How to write a `picofuse_test` in CMakeLists.txt
84 
85 Tests live under `test/<name>/`, one `main.c` per test, and are registered
86 in `test/CMakeLists.txt` with the `picofuse_test()` macro:
87 
88 ```cmake
89 # One-line comment describing what this test actually exercises.
90 picofuse_test(sys_019
91  sys_019/main.c
92 )
93 ```
94 
95 `picofuse_test(<target> <source>...)`:
96 
97 - `<target>` becomes both the CMake target name and the CTest test name
98  (the convention is `sys_NNN` for `picofuse/sys` coverage, `hw_NNN` for
99  `picofuse/hw`), and must be unique across the whole suite.
100 - `<source>...` are paths relative to `test/` (the calling
101  `CMakeLists.txt`'s directory), same as any other CMake `SOURCES` list -
102  normally just the one `main.c`.
103 - It links `picofuse-hw` (which itself pulls in `picofuse-sys`), so both
104  APIs are always available regardless of what the test actually uses.
105 - On a `PICO_BOARD` build it produces a `.elf` and registers a CTest test
106  that runs it through `testrunner` (see above) against
107  `target/<chip>.cfg` (`rp2040`/`rp2350` chosen automatically from
108  `PICO_RP2040`/`PICO_RP2350`). On a host build it registers a CTest test
109  that just runs the compiled executable directly.
110 
111 Guard a test with `if(NOT DEFINED PICO_BOARD)` / `if(NOT APPLE)` /
112 etc. when it only makes sense on some platforms - see the existing
113 `test/CMakeLists.txt` for examples (host-only argument-parsing tests,
114 Darwin having no GPIO backend, and so on). Prefer leaving a test registered
115 and failing over silently gating it out, unless there's a specific reason
116 documented in a comment above the guard.
117 
118 ### How to structure a unit test in `.c`
119 
120 Each test's `main.c` uses one of two macros from `include/test/test.h` in
121 place of a raw `main()`:
122 
123 - `test_main_sys(arena_size) { ... }` - wraps `sys_init()`/`sys_exit()`
124  around the body, and prints the `"[TEST] [INIT] <env>"` /
125  `"[TEST] [EXIT] <env>"` markers `testrunner` looks for.
126  `arena_size` is forwarded to `sys_init()`: `0` leaves `sys_malloc()` and
127  friends routed to the system allocator, a nonzero value configures a
128  fixed-size default arena instead.
129 - `test_main_hw(arena_size) { ... }` - the same, but also wraps
130  `hw_init()`/`hw_exit()` (inside the `sys_init()`/`sys_exit()` pair, since
131  `hw` depends on `sys`) - use this instead when the test needs
132  `picofuse/hw`.
133 
134 Inside the body, use:
135 
136 - `test_assert(condition)` - panics via `sys_panicf()` (printing
137  `"[PANIC] "` plus the failed condition, file, and line) if `condition` is
138  false. Always checked, regardless of `NDEBUG`.
139 - `test_assert_strequal(actual, expected)` - like `test_assert`, but for
140  comparing two null-terminated strings, with both values included in the
141  panic message on mismatch.
142 
143 A minimal test looks like:
144 
145 ```c
146 #include <picofuse/sys.h>
147 #include <test/test.h>
148 
149 test_main_sys(0) {
150  sys_mutex_t *mutex = sys_mutex_init();
151  test_assert(mutex != NULL);
152  test_assert(sys_mutex_lock(mutex));
153  test_assert(sys_mutex_unlock(mutex));
154  sys_mutex_deinit(mutex);
155 }
156 ```