Skip to main content

Test Infrastructure

This guide explains how each file in a test directory works and how they interact during make TEST=<id>.

Files in a Test Directory​

Most tests have this core set of files. Some lessons check with a script of their own instead of comparing files with cmp: Lesson 6's Makefile runs check.py, for example. Others add scripts, such as cycles.py, which prints a table of measured cycles.

my-test/
+-- kernel.w          # Tungsten source(s) - your PE program
+-- bench.paint       # Paint layout - PE placement and routing
+-- Makefile          # Build rules - how to compile, simulate, and verify
+-- test.rc           # Test instances - parameter sets to test
+-- ref.py            # Reference - Python golden model

1. Makefile — The Build and Test Engine​

The Makefile ties everything together. Here's an annotated template:

# The flute build rules: paint (which compiles each .w with Carbide), then
# the simulator
include $(FLUTEMK)

# Pull in the test harness (provides check, cleanup, TEST= selection;
# test/$(MODE) is defined below)
include $(TESTMK)

# ----- Simulation flags -----
# cycles=N sets max simulation cycles before timeout; a run that needs more
# fails like a wrong answer, with its output only partly written. CYCLES ?=
# lets a test.rc row or the command line set the cap.
CYCLES ?= 1000
SIMFLAGS=cycles=$(CYCLES) $(SIMOPTS)

# ----- Paint flags -----
# PAINTVARS: which variables from test.rc to pass to Paint
PAINTVARS=N W
# PAINTFLAGS: flags for the paint tool
#   lib=..         — find .w files in parent dir (needed for VPATH build)
#   verbose        — print compilation details
#   arch=${ARCH}   — architecture from test.rc
PAINTFLAGS=lib=.. $(PAINTOPTS) verbose arch=${ARCH}
PIPELINEFLAGS=arch=${ARCH}

# ----- Mode: sim (simulation) is default -----
MODE ?= sim

# ----- Standard targets -----
all: test
clean: cleanup
test: test/$(MODE)

# ----- Test comparison -----
# Pass: out-ref.bin == out-sim.bin (cmp exits 0)
# Fail: cmp prints the byte offset of first difference
test/$(MODE): out-ref.bin out-$(MODE).bin
	cmp $^

# ----- Data extraction -----
# Extract initial data from compiled image
in.bin: bench.img bench.map
	miget image=$< map=bench.map symbol=send.a data=$@

# Extract simulation results from core dump
out-%.bin: bench-%.core bench.map
	miget image=$< map=bench.map symbol=recv.a data=$@

# Generate reference output
out-ref.bin: in.bin
	python3 ../ref.py $< $@

Self-Contained Makefile Pattern​

For examples where inputs are hardcoded in kernels (common with MP):

include $(FLUTEMK)
include $(TESTMK)

CYCLES ?= 2000
SIMFLAGS=cycles=$(CYCLES) $(SIMOPTS)
PAINTFLAGS=lib=.. $(PAINTOPTS) verbose arch=${ARCH}
PIPELINEFLAGS=arch=${ARCH}

MODE ?= sim

all: test
clean: cleanup
test: test/$(MODE)

test/$(MODE): out-ref.bin out-$(MODE).bin
	cmp out-ref.bin out-$(MODE).bin

# Self-contained: ref.py generates output without input files
out-ref.bin:
	python3 ../ref.py

out-%.bin: bench-%.core bench.map
	miget image=$< map=bench.map symbol=recv.result data=$@

Multi-File Comparison Pattern​

When verifying several PEs separately (e.g., each PE's block of a sort):

REFS=l0-ref.bin l1-ref.bin

test/$(MODE): $(REFS) l0-$(MODE).bin l1-$(MODE).bin
	cmp l0-ref.bin l0-$(MODE).bin
	cmp l1-ref.bin l1-$(MODE).bin

$(REFS):
	python3 ../ref.py

l0-%.bin: bench-%.core bench.map
	miget image=$< map=bench.map symbol=l0.result data=$@

l1-%.bin: bench-%.core bench.map
	miget image=$< map=bench.map symbol=l1.result data=$@

See the topic Sorting networks along a row for this pattern in action.

Build Artifacts​

FileDescription
bench.imgCompiled machine image (CS-3 image with all PE code and data)
bench.mapSymbol table — maps names to ( W X Y ): word address, then fabric X and Y
bench-sim.coreCore dump after simulation completes. make deletes it when the row finishes, unless a rule of the Makefile names it; Lesson 4 shows how to make it again
bench-sim.logSimulation log (wavelet traffic, cycle counts)
in.binInput data extracted from the image
out-sim.binOutput data extracted from the core dump
out-ref.binExpected output from ref.py

Key Make Variables​

VariableSet ByDescription
$(FLUTEMK)Environment, from the experimental SDK's setupPath to flute.mk
$(TESTMK)Environment, from the experimental SDK's setupPath to test.mk
$(CYCLES)Makefile, test.rc row or command lineThe cycle cap, in templates that write CYCLES ?=
$(SIMOPTS)User/CIExtra simulation options
$(PAINTOPTS)User/CIExtra paint options
$(ARCH)test.rcArchitecture (sdr)
$(PIPELINEFLAGS)MakefilePipeline flags (typically arch=${ARCH})
$(TEST)UserWhich test instances to run: a test.rc line number, a row name, or *
$(MODE)UserExecution mode (default: sim)

2. test.rc — Test Instances​

Each line defines a test configuration:

tutorial N=4 W=4 ARCH=sdr
smoke    N=8 W=4 ARCH=sdr
full     N=16 W=8 ARCH=sdr

The Makefile above lists N W in PAINTVARS, so every row sets both. Each row runs in its own directory, test-<label>. Give every row its own label: a label used on two rows names neither of them, their directories become test-1/ and test-2/, after their line numbers, and make TEST=<label> runs both.

With CYCLES ?= in the Makefile, a row can set its own cap: large N=4096 CYCLES=40000 ARCH=sdr.

Format​

LABEL PARAM=VALUE ... ARCH=arch

Labels​

LabelDescription
tutorialThe first row of every lesson; the run instructions use make TEST=tutorial
smokeQuick sanity test
fullThorough test — may take longer
# at line startRow disabled (reported SKIP under TEST='*')

Running Specific Tests​

make TEST=1          # runs line 1: tutorial N=4 W=4 ARCH=sdr
make TEST=smoke      # runs the row named smoke: smoke N=8 W=4 ARCH=sdr
make TEST='*'        # runs every row that is not disabled

Under TEST='*', the first failing row stops the run, and later rows are not reported; make -k TEST='*' runs them all.

A row's test-<label>/ directory is built once. After you edit a source, bench.paint, ref.py, the Makefile or test.rc, run make clean first, or make reports the old result.

How Parameters Flow​

test.rc row     ->  make variables  ->  PAINTVARS    ->  bench.paint
N=4 ARCH=sdr        N=4, ARCH=sdr       PAINTVARS=N      N is 4, so
                                                         code('k [N:N])
                                                         sets param N = 4

3. bench.paint — Layout and Routing​

The paint file describes:

  1. PE placement — where each PE lives on the fabric
  2. Code attachment — which .w file runs on each PE
  3. Parameter binding — connecting paint variables to tungsten params/sockets
  4. Routing — which colors flow in which directions

See the Paint Syntax Quick Reference for full details.

Compilation Chain​

bench.paint
    v (paint tool)
    +-- bench.obj    (machine image - code + data for all PEs; milink and
    |                 weight update it, and the build copies it to bench.img)
    +-- bench.map    (symbol table)
    +-- <name>_artefacts/*.yeti
                     (generated Yeti assembly, one per code() variant)

4. ref.py — Reference Implementation​

A Python script that computes the expected output.

Pattern A: CLI-arg based​

Receives command-line arguments from the Makefile:

import sys
import numpy as np

# Arguments from Makefile: python3 ref.py in.bin out.bin [extra_args...]
in_name = sys.argv[1]
outfile = sys.argv[2]

# Read input data (match the dtype to your Tungsten type)
a = np.fromfile(in_name, dtype=np.int16)

# Apply the same computation as your kernel
b = a + 1

# Write expected output
b.astype('int16').tofile(outfile)

Pattern B: Self-contained​

For MP and other examples with hardcoded inputs:

import numpy as np

# Mirror the hardcoded data from the Tungsten kernel
values = [10, 20, 30, 40]
result = sum(values)

ref = np.array([result], dtype=np.uint16)
ref.tofile("out-ref.bin")
print(f"Expected: {result}")

This pattern is simpler and avoids the need for in.bin extraction. The topic Collectives over messages uses it, with the values that its test.rc row gives the layout.

Tips​

  • Use np.float16 for hp, np.float32 for sp, np.int16 for xp.
  • For multi-input tests, pass multiple input files: 
    out-ref.bin: in_data.bin in_cmd.bin
    	python3 ../ref.py $^ $@
    data = np.fromfile(sys.argv[1], dtype=np.float16)
    cmd  = np.fromfile(sys.argv[2], dtype=np.float16)
    out  = sys.argv[3]

5. miget and miset — Symbol-Based Data Access​

miget: Read Data from Image or Core Dump​

# Read initial data from compiled image
miget image=bench.img map=bench.map symbol=a data=out.bin

# Read results from simulation core dump
miget image=bench-sim.core map=bench.map symbol=recv.a data=out.bin

# Print data as text instead of binary
miget image=bench.img map=bench.map symbol=a text

miset: Write Data into Image​

# Write data into the image before simulation
miset image=bench.img map=bench.map symbol=a data=input.bin order=n \
      out=bench.img

Symbol Path Syntax​

For layouts with several PEs, symbols are prefixed with the region name:

PaintSymbol
name('recv) with xp a[n=N]recv.a
name('compute) with hp A[n=N][m=W]compute.A

Execution Flow Summary​

+-------------------------------------------------------------+
| make TEST=1 MODE=sim                                        |
+-------------------------------------------------------------+
|                                                             |
| 1. Parse test.rc line 1 -> set make variables               |
|                                                             |
| 2. Paint: bench.paint -> bench.obj + bench.map              |
| (Carbide compiles the .w files that code() names)           |
|                                                             |
| 3. milink: link annotations -> bench.obj; bench.img copies  |
| it, and the Makefile may miset inputs into it               |
|                                                             |
| 4. miget: bench.img -> in.bin  (extract input data)         |
|                                                             |
| 5. ref.py: in.bin -> out-ref.bin  (reference output)        |
|                                                             |
| 6. mielf, simulator, elfmi: bench.img -> bench-sim.core     |
|                                                             |
| 7. miget: bench-sim.core -> out-sim.bin  (extract output)   |
|                                                             |
| 8. cmp: out-sim.bin vs out-ref.bin  (pass/fail)             |
|                                                             |
+-------------------------------------------------------------+

References​