Lesson 1 — Hello, wafer!

This lesson builds, runs and checks the smallest complete program: one PE adds 1 to each element of an array. Before the code, here is how a program for the Cerebras machine is put together.

What you will be able to do

• Build and run a one-PE program with make, and tell a passing test from a failing one.
• Say what each line of a Tungsten kernel and of its Paint layout does.
• Find an array in bench.map and read its values with miget.
• Recognize the two silent traps in this program: a let without its ;, and an initializer that is longer than its array.

The machine, the languages and the compiler

The tutorial programs the Cerebras Wafer-Scale Engine (WSE), here the WSE-3, the processor of a CS-3 system. The WSE is a grid of processing elements (PEs). Each PE has three parts:

• a compute engine (CE), a processor that runs the PE's program;
• 48 KB of memory, which holds the program's instructions and its data together;
• a router, which connects the PE to its four neighbours.

A PE shares no memory with any other PE. PEs cooperate only by sending data to each other through their routers, from lesson 4 on.

A program for the WSE is written in two languages:

• Tungsten describes what one PE does: its data, its loops, and what it sends and receives. A Tungsten source file (.w) is a program for one PE, called a kernel.
• Paint places kernels on a rectangular region of the wafer. It says which PEs run each kernel, gives each kernel its parameters, and connects the kernels to each other. One kernel can run on many PEs at once.

The Tungsten compiler is called Carbide (carbide). Paint runs Carbide on each kernel of a layout and combines the results into one machine image for the whole region. Each lesson's Makefile runs Paint and the simulator for you, so the lessons never call carbide or paint by hand. To run them yourself, see Compiler usage and Paint's command-line options.

The program

One PE holds an array a of N 16-bit integers and adds 1 to each element:

+-------------------------------+
| PE at column 0, row 0         |
|   CE:     runs add.w          |
|   memory: a = {0, 1, 2, 3}    |
|   router: not used            |
+-------------------------------+

The test row tutorial sets N=4, and the result is a = {1, 2, 3, 4}. The row n8 sets N=8, and the result is a = {1, 2, 3, 4, 1, 1, 1, 1}.

The kernel: add.w

xp param N;

xp a[n=N] = {0, 1, 2, 3};

let i in [0, N);
∀i a[i] <- a[i] + 1;

Line	Meaning
xp param N;	A compile-time parameter N. Paint gives it a value; the program cannot change it. xp is the 16-bit signed integer type. Lesson 2 lists the other types.
xp a[n=N] = {0, 1, 2, 3};	An array a of N elements. n names its dimension and N is its size. The initializer gives the first four elements; any elements after them start at 0.
let i in [0, N);	An iterator i over the half-open range 0 to N−1.
∀i a[i] <- a[i] + 1;	For every i, a[i] gets a[i] + 1.

The statements at the top of a kernel run once, from top to bottom. The kernel ends when its last statement ends.

<- and = are the same assignment: a[i] = a[i] + 1; compiles to the same code. Both spellings appear in the lessons. An array's initializer list needs =: xp a[n=4] <- {0, 1, 2, 3}; is a syntax error. A scalar declaration takes either sign: xp s <- 5; and xp s = 5; are the same.

∀ ("for all") runs the statement after it once for each value of the iterator. A loop can also declare its range inline, with no let and no ∀, in plain ASCII:

i in [0, N) a[i] <- a[i] + 1;

The Unicode support table gives the ASCII spelling of each symbol.

The ; after let matters

The ; ends the let declaration. Without it, let i in [0, N) becomes a loop over the statement that follows, so the next line runs once for each i:

let i in [0, N)
∀i a[i] <- a[i] + 1;

This still compiles, with no error or warning. With N=4 the ∀i statement now runs four times, and a ends as {4, 5, 6, 7}.

The layout: bench.paint

let xadd = tile()
xadd : code('add [N:N])

Line	Meaning
let xadd = tile()	tile() makes a region of one PE, at column 0, row 0. let xadd = gives the region a name in Paint.
xadd : code('add [N:N])	The PE runs the kernel add, from add.w. [N:N] sets the kernel's parameter N to Paint's variable N, which the test row supplies.

The layout has one PE, so it has no colors or routes. Lesson 4 adds them.

The test

test.rc

tutorial N=4 ARCH=sdr
n8       N=8 ARCH=sdr

Each line is one test. Its first word, such as tutorial, labels it. N=4 sets the variable N, and ARCH=sdr builds for the WSE-3. Every row must set each variable the Makefile names in PAINTVARS. A row without N= stops Paint with Command-line error: 'source' not specified.

The Makefile

include $(FLUTEMK)
include $(TESTMK)

BENCH_NAME=add

SIMFLAGS=cycles=1000 $(SIMOPTS)
PAINTVARS=N
PAINTFLAGS=lib=.. $(PAINTOPTS) verbose arch=${ARCH}
PIPELINEFLAGS=arch=${ARCH}

MODE ?= sim

all: test
clean: cleanup
test: test/$(MODE)

test/$(MODE): out-ref.bin out-$(MODE).bin
	cmp $^

in.bin: bench.img bench.map
	miget image=$< map=bench.map symbol=a data=$@

out-%.bin: bench-%.core bench.map
	miget image=$< map=bench.map symbol=a data=$@

out-ref.bin: in.bin
	python3 ../ref.py $< $@

• include $(FLUTEMK) brings in the rules that build and simulate a layout, and include $(TESTMK) the rules that run the rows of test.rc.
• Each row builds in its own directory, such as test-tutorial/, so lib=.. and ../ref.py point back at the lesson's files.
• PAINTVARS=N passes the row's N to Paint.
• SIMFLAGS=cycles=1000 stops the simulator after 1,000 cycles. This kernel needs far fewer: its last write to memory happened at cycle 14. A cap that is too small cuts off a program that needs more cycles. The test checks memory when the simulator stops, so a pass does not prove that the kernel finished.
• PIPELINEFLAGS is for the debug views of Debugging kernels.

What make runs, in order

For the row tutorial, in test-tutorial/:

1. Paint reads bench.paint and calls Carbide to compile add.w with N=4. It writes the machine image bench.obj and the symbol map bench.map.
2. milink finishes the image, and the build copies it to bench.img.
3. miget reads a from bench.img, the memory as it is before the run, into in.bin.
4. ref.py reads in.bin, adds 1, and writes the expected result to out-ref.bin.
5. The simulator runs bench.img and writes the final memory of every PE to bench-sim.core.
6. miget reads a from bench-sim.core into out-sim.bin.
7. cmp compares out-ref.bin with out-sim.bin. The test passes when they are equal, byte for byte.
8. make deletes bench-sim.core and bench.obj, which it treats as intermediate files. bench.img, bench.map, in.bin, out-sim.bin and make-log stay.

One line of bench.map

bench.map lists every symbol of the image. The line for a in test-tutorial/bench.map is:

a         [ n<4              ] ( 0x0+n        0 0 )

It says that a has one dimension, n, of size 4, and that element n is the 16-bit word at address 0x0+n of the PE at column 0, row 0. miget uses this line to find a. The other lines name the PE's registers and the compiler's own symbols.

ref.py

import sys
import numpy as np

in_name = sys.argv[1]
outfile = sys.argv[2]

a = np.fromfile(in_name, dtype=np.int16)
b = np.array([x + 1 for x in a])

b.astype('int16').tofile(outfile)

The reference reads the initial array from the image, not from add.w, adds 1 in Python, and writes the expected output.

What the hardware does

The CE runs the kernel from its own memory. Memory is addressed in 16-bit words, not bytes: the 48 KB are 24,576 words, at addresses 0x0000 to 0x5FFF. bench.map shows that a sits at word 0, and the kernel's instructions sit in the same memory. The memory map shows the rest of the address space.

Carbide compiles the whole loop ∀i a[i] <- a[i] + 1; into a single instruction, which adds 1 to all N elements. Lesson 3 shows how to read that instruction in the compiler's listing, and what it costs.

When the last statement has run, the PE has nothing left to do. When the simulation ends, the simulator writes each PE's memory to the core file, which miget reads.

Run it

Copy the SDK's tutorial, <sdk>/share/tutorial in the SDK's directory <sdk>, to a directory of your own, as the run guide shows. Every later lesson works in this copy. Then run the row labelled tutorial:

cp -r <sdk>/share/tutorial tutorial
cd tutorial/main-1-one-pe/01-hello-wafer
make TEST=tutorial

TEST= picks rows of test.rc: TEST=tutorial the row labelled tutorial, TEST=2 the second row, and TEST='*' every row (see Test infrastructure).

A passing test prints one line, such as 01-hello-wafer : 0:03.73 PASS tutorial [ N=4 ARCH=sdr ].

To see the values themselves, make the core file again in the row's directory and read it:

cd test-tutorial
make -f ../Makefile VPATH=.. TEST=tutorial bench.map bench-sim.core
miget image=bench-sim.core map=bench.map symbol=a decimal
cd ..

The second line makes bench-sim.core and bench.map again and keeps them. miget prints 1 2 3 4. Its decimal output is unsigned: a negative xp such as -2 prints as 65534.

A failing test prints FAIL instead. test-tutorial/make-log holds every command make ran and its output, including cmp's report of the first byte that differs, for example out-ref.bin out-sim.bin differ: byte 1, line 1. To see a failure, change + 1 to + 2 in add.w, run make clean, and run the test again: ref.py still adds 1. Then restore + 1 in add.w and run make clean before you go on.

Pitfalls

• Run make clean after every edit. make reuses the build in test-tutorial/ when it exists, and reports the old result.
• The ; after let, shown above.
• An initializer is not checked against the array's size. A list longer than the array is cut off without a diagnostic (see the exercise).
• A list of one value fills every element, unlike C: xp a[n=4] = {5}; holds 5, 5, 5, 5. The Tungsten overview's Rules That Differ from C lists this rule with the others.

Exercises

1. Run the n8 row with make TEST=n8, and explain the last four values of a.
2. Add a row n3 N=3 ARCH=sdr to test.rc. Predict a, then run the row. Carbide drops the fourth initializer value without a diagnostic, and the test still passes. Why does it pass, and why is that dangerous? (Look at where ref.py gets its input.)
3. Delete the ; after let i in [0, N), predict the result, run make clean, and run the tutorial row.

Next

Lesson 2 — Values, control flow and functions writes ordinary code on one PE: types, if, while and functions.

References

• Parameters
• Arrays and initializers
• Iterators and loops
• Named iterators (let)
• Assignment
• Unicode support
• Compiler usage (carbide)
• Regions: tile()
• The : operator
• Attaching code: code()
• Makefile integration (PAINTVARS)
• The bench.map file format
• Test infrastructure