Skip to main content

Debugging Kernels

When a test fails (cmp reports a difference), you need to figure out why your kernel's output doesn't match the reference. This guide covers the tools available for debugging.

Quick Debugging Checklist​

  1. Rebuild from clean: after any edit, make clean before rerunning; an existing test-<label>/ is not rebuilt, so its result may be stale
  2. Check the reference: Is ref.py correct?
  3. Inspect data: Use miget with text mode to print values
  4. Run debug build: make clean, then make TEST=<id> GOAL=debugx
  5. Launch Portrait: Visualize execution in the browser
  6. Check pipeline traces: See instruction-by-instruction execution
  7. Check wavelet traffic: See what data flowed between PEs

1. Inspecting Data with miget​

Before launching heavy debug tools, check what data looks like:

View initial data (from compiled image)​

# Print array 'a' as human-readable text
miget image=bench.img map=bench.map symbol=a text

# For multi-tile: include the region name
miget image=bench.img map=bench.map symbol=send.a text

# Dump as binary file
miget image=bench.img map=bench.map symbol=a data=initial.bin

View simulation results (from core dump)​

Make deletes the core, and in some lessons the map, as intermediate files once the test has run. Rebuild and keep both, then work in the test directory (test-1/ instead of test-tutorial/ in lessons with two tutorial rows):

make test-tutorial/bench-sim.core test-tutorial/bench.map
cd test-tutorial

# Print the result array after simulation
miget image=bench-sim.core map=bench.map symbol=recv.a text

# Compare with initial data
miget image=bench.img map=bench.map symbol=send.a text
miget image=bench-sim.core map=bench.map symbol=recv.a text

Interpreting miget text output​

0000 0001 0002 0003

text prints only the values, as 16-bit hexadecimal words (width= sets how many per line). Use decimal or hp instead of text for decimal or half-precision output.

Ordering control​

# Specify dimension ordering
miget image=bench.img map=bench.map symbol=A order="n m" text

2. Debug Build with GOAL=debugx​

The debug build produces extra artifacts needed for Portrait and pipeline visualization:

make clean
make TEST=1 GOAL=debugx

Clean first: debugx reuses an existing simulation log, which then has no pipeline or landing records.

This generates (inside the row's test directory, test-tutorial/ in most tutorials):

FileDescription
pipeline.binBinary pipeline trace (instruction execution per cycle)
region.jsonTile geometry and region tree
listing.jsonSource code ↔ assembly mapping
landing.jsonWavelet traffic traces
debuginfo.jsonSource-level debug information
bench-sim.logFull simulation log

3. Portrait — Web-Based Visualization​

Portrait is an interactive browser tool for inspecting execution.

Launching​

# After running GOAL=debugx, navigate to the test directory
cd test-tutorial/
portrait

Portrait prints the address to open in a web browser:

http://hostname:8080/paint.html

Portrait Panels​

PanelDescription
Floor PlanSVG view of the tile layout. Click a tile to inspect it.
Region TreeHierarchical view of paint regions.
Source CodeShows the Tungsten source or the Yeti assembly, chosen from a drop-down.
Pipeline ViewCycle-by-cycle instruction execution for the selected tile.
Tile ReportSummary of a selected tile's configuration.
Route PlanVisual routing for the selected tile's colors.
Wavelet LogTimeline of wavelet traffic on the selected tile.

Common Portrait Workflow​

  1. Open Floor Plan — verify tiles are where you expect them.
  2. Click a tile — inspect its source, pipeline, and wavelet traffic.
  3. Check Pipeline View — look for stalls, unexpected instruction sequences.
  4. Check Wavelet Log — verify data arrived at the right time, right value.

Portrait Command-Line Options​

portrait port=9090          # use a different port
portrait quiet=1            # do not print the address

4. Pipeline Traces​

The pipeline trace shows cycle-by-cycle execution for a specific tile. Portrait fetches this data on-demand when you click a tile.

What to look for​

IssueSymptom in Pipeline View
DeadlockTile stalls at a receive instruction, never proceeds
Wrong orderData processed in unexpected sequence
Missing dataFewer wavelets received than expected
Infinite loopExecution never terminates (hits cycle limit)

Cycle Budget​

Set cycles=N in the Makefile's SIMFLAGS, keeping its $(SIMOPTS). If the simulation hits the cycle limit without completing, the core dump will show the state at timeout. Increase cycles if your kernel legitimately needs more time.

5. Wavelet Traffic Analysis​

In Portrait​

The Wavelet Log panel shows all wavelets that arrived at or departed from a tile, with timestamps and values.

From the Simulation Log​

With debug=landing, the simulation log records each wavelet landing:

make clean
make TEST=1 SIMOPTS=debug=landing
grep ' landing ' test-tutorial/bench-sim.log | head -20

Inspect memory at a cycle with meminsp​

Use the image, map, and log from the debug run to inspect a tile's memory:

meminsp img_path=bench.img map_path=bench.map log_path=bench-sim.log \
    tile_str=T1.0

At the prompt, list the available symbols, then inspect one at a cycle:

symbols
seek 100
print recv.result
diff recv.result @0 @200

Replace recv.result with your symbol name. See the meminsp reference for watchpoints and other commands.

6. Common Debugging Scenarios​

Scenario: Test fails with cmp error​

# 1. Check what output you got (rebuild the core first, as in section 1)
miget image=bench-sim.core map=bench.map symbol=recv.a text

# 2. Check what you expected (out-ref.bin is written by ../ref.py)
xxd out-ref.bin | head

# 3. Compare byte-by-byte
cmp -l out-sim.bin out-ref.bin | head

Scenario: Simulation times out (hits cycle limit)​

# Increase cycles in the Makefile, keeping $(SIMOPTS)
SIMFLAGS=cycles=10000 $(SIMOPTS)

Or check for deadlocks — is a tile waiting on data that never arrives? Run debugx and check Portrait's pipeline view:

make clean
make TEST=1 GOAL=debugx

Scenario: Array has wrong values​

# Check initial values — is the initializer correct?
miget image=bench.img map=bench.map symbol=a text

# Check values after simulation (rebuild the core first, as in section 1)
miget image=bench-sim.core map=bench.map symbol=a text

# Compare initial vs final
diff <(miget image=bench.img map=bench.map symbol=a text) \
     <(miget image=bench-sim.core map=bench.map symbol=a text)

Scenario: Routing problem (data not arriving)​

# 1. Check the colors along the route in bench.paint
# Without a colorpair swap, the sender's tx color must equal the
# receiver's rx color; with one, a sender on a can reach a receiver on b

# 2. Verify direction codes
# Sender: [C>R] means data goes right
# Receiver: [L>C] means data comes from left
# These must be consistent with tile positions (hstack = left-to-right)

Scenario: Switch positions not advancing (flip / serial drain)​

If you use multi-position routing ([C>R ; L>R]) with flip, data from interior tiles may not arrive at the receiver.

# 1. Check that flip is LOCAL:
#    - flip advances the SENDING tile's own input switch
#    - It does NOT advance downstream tiles
#    - With [C>R ; L>R]: tile starts injecting (C>R),
#      after flip switches to passthrough (L>R)
#    - Rightmost tile drains first, then tiles further left

# 2. Common mistakes:
#    - [L>R ; C>R] with flip: tile starts in passthrough, but
#      flip advances the LOCAL switch, not the downstream one.
#      Use this pattern with control(ADV) wavelets instead, as in
#      the Switch protocols topic.
#    - [C>R ; L>R] with flip: correct for serial drain.
#      Tiles closest to receiver drain first.

See lesson 10, Switches for a working example of flip-based switch ordering.

Scenario: Floating-point mismatch​

Half-precision (hp) has limited precision. Small rounding differences between hardware (16-bit FP) and Python (np.float16) can cause cmp to fail even when the logic is correct.

# In ref.py, use float16 arithmetic throughout:
a = np.fromfile(in_name, dtype=np.float16)
result = (a * np.float16(2.0))  # keep in float16
result.astype('float16').tofile(outfile)

Scenario: MP message not arriving​

Message passing debugging:

# 1. Verify all tiles in the MP path have mp(bank) in paint
#    Missing mp() on relay tiles is a common mistake

# 2. Check header construction:
#    - dest_col must be << 4 in w[1] to place it in header bits 31:20
#    - north (0x80) only if the destination has a smaller row ID than the
#      sender, south (0x40) only if it has a larger one; both clear for a
#      same-row destination
#    - length field must match the number of data wavelets

# 3. Verify wavelets are 32-bit (full wavelets)
#    Sending raw xp on an MP color sends a half-wavelet, which is undefined
#    behavior on a message-passing color
#    Fix: send a u32, built with a union, u32 shifts or a (u32) cast

# 4. Check message ordering when receiving from multiple senders:
#    MP messages from different tiles may arrive in arbitrary order.
#    Use payload tagging (embed an ID of the sender) to identify the source.
#    The Collectives over messages topic tags its messages this way.

7. Useful Debug Flags​

Makefile level​

# More simulation cycles
SIMFLAGS=cycles=10000 $(SIMOPTS)

Command line​

# Record wavelet landings (debug=landing) or the pipeline (debug=hwtile)
# in the simulation log; clean first so the simulation reruns
make clean
make TEST=1 MODE=sim SIMOPTS=debug=landing

# Run debug build, after cleaning for the same reason
make clean
make TEST=1 GOAL=debugx

Also See​

References​