Paint

This document is intended to provide an introduction to using the paint engine from a user perspective. It does not fully document how paint works or the internals of the engine, although will make mentions where relevant to help understanding.

Introduction
Reading Paint
Basic syntax
Primitives
- Arithmetic
- Strings
- Conditionals
- Debug
- Layouts
- Code
- Colors
- Routing
- Auto-routing
- Link
  - Filter example
Running paint
Examples

Introduction

At a high-level, paint is a description language for the instantiation, layout, and interconnection of Cerebras kernels. For example, we can use it to attach a Tungsten kernel to a rectangular array of PEs, configure routes on specified tiles, or specify locations and orientations of regions.

Paint builds a tree-like data structure from the bottom up. The leaves of the trees represent pes on the fabric, with interior nodes allowing us to build more complex representations through replication, rotation, joins, etc. When applying a new primitive, the property will apply to the subtree whose root is the node corresponding to the primitive.

Paint has no loops or branches, by design. Repetition comes from replication primitives such as hstackrep, and setif chooses between two values (see Conditionals).

Reading Paint

Paint reads like a small Lisp with the parentheses moved after the name. A file is a sequence of expressions. Most of them build a region, a tree of tiles, or annotate one with code, colors and routes, and the value of the last expression is the layout paint builds. A handful of rules cover almost every line:

You see	It means
`f(a b c)`	a call with three arguments: no commas, and no space before `(`
`f(a b(c))`	two arguments, the second the call `b(c)`; `f(a b (c))` passes three
`'name`	the string `name`: letters, digits and `_ - . : @ * '`, up to any other character
`let x = e`	declares `x`; a later `x = e` updates it
`x : f(a b)`	`x = f(x a b)`: applies `f` to `x` and keeps the result
`[1 2 3]`, `[k:v]`	an array, and a dictionary
`[L>CR]`	a route rule: in from the left, out to the core and the right
`a + b`, `n//2`	arithmetic, the same as `add(a b)` and `idiv(n 2)`
`define f(a b) { … }`	a function, whose value is its last expression
`(* … *)`	a comment

Here is one tile of the nine-point heat equation's layout, heateq.paint, read line by line:

let c0 = color()          (* a color; the solver picks its number *)
let x00 = rect(0 0 1 1)   (* a 1 × 1 region at (0, 0) *)

(* Run kernel heateq9pt on x00, its sockets bound to colors and its
   parameters set: x00 = code(x00 'heateq9pt [...]) *)
x00 : code('heateq9pt [left:c0 right:c1 up:c2 down:c3 n:n p:p pad:pad])

x00 : paint(c0 [L>C])     (* on c0, wavelets from the left go to the core *)
x00 : paint(c1 [C>R])     (* on c1, the core's wavelets go right *)

let corerect = union(x00 x01 x10 x11)  (* the 2 × 2 block as one region *)
corerect : hstackrep(nx/2)             (* repeated nx/2 times side by side *)

/ divides exactly, so nx/2 is a whole number only because the heat equation's nx is even; nx//2 truncates toward zero. Rules that differ from C and Python lists the other surprises.

Basic syntax

The rest of this section covers each part of the syntax in turn.

Comments in paint use (* to open and *) to close, e.g.:

(* this is a 
   multi-line comment *)

Tokens

Numbers are decimal (42, 1.5) or hexadecimal integers (0x2A). Integers are 32-bit signed and floats 32-bit. There are no negative literals: write 0-1 for −1.

Variables start with a letter or _ and can contain letters, digits and _, with apostrophes only at the end (x').

Strings use double quotes, e.g. "example". A single opening quote also starts a string: 'example. It takes letters, digits and _ - . : @ * ', and stops at any other character, so 'a-1 is the string a-1. Parentheses after the quote hold a string with spaces, e.g. '(hello world). Names, tags and kernel names are usually written 'name.

Routes are specified by listing the router’s source and target directions: [D>U] is a route from down to up. [L>CR] is a route from left to center (offramp) and right. The center contains the tile’s compute element. Paint numbers rows as the fabric does, from the north edge: a PE's row ID is its paint y, and row 0 is the northernmost row. Paint calls the +y direction up, so U is the router's south port and D its north port. R is east and L is west.

Expressions

Infix math expressions are possible using +, -, *, /, %, //, and ==.

Parentheses can be used for precedence and for function calls. However, because there are no commas to separate function parameters, it is required that the open parenthesis to start a function’s argument list does not have a preceding whitespace. f(a b c) passes three arguments to f; f(a b (c)) is the same. f(a b(c)) passes two arguments to f, and one argument to b.

Function definitions

Functions can be defined with the define keyword:

define name(arg1 arg2 ...) {
  (* body *)
}

Assignments

Use let to declare local variables and regions, including inside functions. For example, let a = 4 + 1 declares a; a later a = a + 1 updates it. Bare assignment cannot declare a binding (x = tile() fails with can not assign undeclared variable x); declare it first:

let x = tile()

The : operator provides a short-hand for curried assignment: a : b(c) has the same meaning as a = b(a c).

Includes

Other paint files can be included by using the include keyword. Paint will search the library paths included by the lib argument for the specified file and throw an error if not found.

include "other.paint" (* will search libs for this file *)

Arrays

Arrays are defined using square-bracket syntax:

let array = [1 1 2 3 5]

There are helper functions to create common arrays (cliff, range). The arithmetic operators do not apply to arrays. There are several primitives that help manipulate the array:

let len = arraylen(array): Returns the size of array.
let val = arrayidx(array i): Returns the value at index i.
arraydelete(array i): Deletes the ith entry of array.
arrayupdate(array i 10): Updates the value of the ith entry of array to 10.

Dictionaries

Dictionaries are defined using square-bracket key:value syntax:

let dict = [key1:value1 key2:value2 ...]

It is then possible to use the query() command to populate values from this dictionary, with default-values provided for those keys not present. This can be useful for compacting many function arguments into a single one, similar to a kwarg. For example:

define module(options) {
    let [a b c] = query(options [a:1 b:2 c:3])
}

Rules that differ from C and Python

You write	You get
`let a = -1`	A parse error: there is no unary minus. Write `0-1`.
`[0 -2 4]`, `f(x -1)`	`a -1` inside a list is `a - 1`: the array `[-2 4]`, and a call with one argument. Write `[0 (0-2) 4]`.
`max (3)`	Two expressions, `max` and `(3)`: `let n = max (3)` makes `n` the function `max`. With more values inside, such as `rect (0 0 2 2)`, paint stops with `list is not a member of ex` and no line number.
`(0-7)//2`, `(0-7)%2`	−3 and −1: `//` truncates toward zero and `%` takes the sign of the left operand, as in C.
`2147483647 + 1`	−2147483648: integers wrap at 32 bits without a message.
`hstackrep(r 1.5*2)`	An error: a count argument takes an integer, and `int(x)` converts a float.
`define f(n) { setif(lt(n 2) 1 n*f(n-1)) }`	A crash: both `setif` branches are evaluated, so a function cannot end its own recursion. Build repetition with `hstackrep`, `vstackrep`, `rep`, `arrayplace` or `range`.
`let b = a` then `arrayupdate(b 0 9)`	`a` changes too: arrays are shared, not copied. `arrayupdate` and `arraydelete` change the array in place and return nothing, so do not assign their result. An all-constant literal such as `[1 2 3]` is shared as well, so updating it inside a function changes it for later calls. Copy with `arrayslice(a 0 arraylen(a))` first.
`module([b:20 cc:30])`	`a=1 b=20 c=3` for the `module` above: `query` ignores keys missing from its defaults without a message, so a misspelt key gets the default.
`range(10 0 0-1)`	`[ ]`: `range` counts up only.

Primitives

Paint provides a library of primitives which are intended to allow the user to build complex tree representations. They fall into different families of primitives which are separated and explained below.

Arithmetic

add(x y): Is an alternative to + for addition.
sub(x y): Is an alternative to - for subtraction.
mul(x y): Is an alternative to * for multiplication.
div(x y): Is an alternative to / for division.
assert(x m): Asserts that x is non-zero, and stops with error message m if it is zero.
cmpeq(x y): Is an alternative to == for equality.
idiv(x y): Is an alternative to // for integer division, which truncates toward zero.
mod(x y): Is an alternative to %; the remainder takes the sign of x.
max(x y ...): Returns the max element from the arguments.
min(x y ...): Returns the min element from the arguments.
lt(x y): Returns 1 if x is less than y, and 0 otherwise.
pi(x y ...): Returns the product of arguments.
split(x y): Returns normal value when splitting x elements into y buckets via a cliff.
cliff(x y): Helper to get a cliff distribution with x elements into y buckets. Returns an array of length 2 where the first entry is the number of elements in the normal buckets and the second entry is the number of elements in the last bucket. Throws if the cliff is invalid (i.e. last bucket has 0 or less elements).
int(x): Casts x to a signed int, where x can be either floating point or integer type.
range(<start> stop <step>): Creates the array start, start+step, … below stop. If start is not passed in it is assumed to be 0. If step is not passed in it is assumed to be 1. Unlike Python's range, it counts up only: a negative step gives an empty array.

Strings

join(delim a b ...): Creates a new string joining a, b, … using delim as the delimiter.
symjoin(a b): Creates the kernel symbol $a.b for link. If either a or b is empty it will omit joining them. The result is a symbol, not a string, so name() rejects it; join("." a b) gives the string a.b.
strcmp(a b): return 0 iff a==b

Conditionals

The conditional support is limited. Primitive setif can be used to use expressions conditionally. Syntax is: setif(<int condition 0 or 1> <if expression> <else expression>). There is no type checking. So if and else parts may or may not be of different types. Note, although only one of the value is assigned, both the expressions, if and else, are evaluated irrespective of the condition. So both need to be valid expressions. setif can be used to conditionally call functions, setup regions, assign values, or like a standard if else. However, it the programmer's responsibility to ensure correct types.

Debug

show(): Prints the arguments to stdout.

Layouts

These primitives allow the user to specify placement and orientation of regions.

To create a region:

tile() or tile(x y) : Specifies a pe with co-ordinates x, y (or (0, 0) if empty).
rect(x y wd ht <xdim> <ydim>): Creates a rectangular region of pes. This is implemented as a tile with replicates in the x and y direction. The two dimensions can be optionally named. These names can then be used in the order argument for specifying which dimensions you would like to read or set in miget or link, respectively.

We can name a given region by applying the name annotation:

name(r example): Annotates the region r with name example.

We can translate, rotate, or replicate these regions:

place(r dx dy): Places the region r with origin dx and dy. This is relative to any previous layout call, rather than absolute. For example, if we called place(r 4 0) and place(r 2 2) successively on the region r then that region would now be placed with an offset of (6 2).
hflip(r): Mirrors the region r left to right, inside its bounding box.
vflip(r): Mirrors the region r along y, inside its bounding box.
rotate(r D): Rotates the region r by D degrees counter-clockwise (R90 turns +X into +Y). We expect D to be one of the variables R0,R90, R180, R270

Updated `place` syntax (with rotation)

place now supports an explicit rotation argument. You can use it either as an attribute or as a function:

Attribute form:
odd_tile : place(0 1 R0)
Functional form:
let odd_tile = place(odd_tile 0 1 R0)

Where R0 means no rotation, R90 means rotate 90°, etc.

We can join regions together:

union(r1 r2 ...): Returns the region formed by taking a union of the region arguments. The order of arguments specifies the order of children. This is important when scanning child regions for certain properties (transports, paint, etc.). If searching for a property, then the children searched first will be found. If applying a property that could overwrite, then the children searched last will have the final word. Having multiple children with the same name is not recommended.
hstack(r1 r2 ...): Keeps r1 where it is and puts each next region against the right (east) edge of the one before. Only x changes: each region keeps its own y.
vstack(r1 r2 ...): Keeps r1 where it is and puts each next region above the one before (+Y). Only y changes: each region keeps its own x.
hstackrep(r nrep <space> <name>): Repeatedly stacks the region r nrep times in the horizontal direction (i.e. EAST). Optionally we can space the repetitions and attach a name to the replicated dimension. It is supported to pass in 0 for nrep which allows conditionally including a region.
vstackrep(r rep <space> <name>): Repeatedly stacks the region r nrep times in the vertical direction (+Y). Optionally we can space the repetitions and attach a name to the replicated dimension. Similarly to hstackrep, you can pass in 0 for nrep which allows conditionally including a region.
arrayplace(r [i j ...] <[geovarname:geovar_array_name]>): Repeatedly places the region r with an array of offsets in the y-axis. For placing along the x-axis the approach is to offset in y-axis and then rotate; this requires setting up the pes 90º offset to ensure that when rotated they are correct. Optional argument geovar_array values are assigned to the geovar geovarname in the repeated regions sequentially. The copies are named x0, x1, ..., so their symbols read as x0.a. Note the sequence of statements is important: create tile, add code, replicate with arrayplace, and all 3 need to operate on the same region. e.g. the following works:

    let code_tile = tile()
    code_tile : code("grad_filter" [IDX: @idx])
    let ids = range(0 N)
    let kernel = arrayplace(code_tile ids [idx:ids])

However, the following code or its combinations do not:

    let code_tile = tile()
    let ids = range(0 N)
    let kernel = arrayplace(code_tile ids [idx:ids])
    kernel : code("grad_filter" [IDX: @idx])    

Paint builds this one without a message, but every copy compiles with IDX=0: a geovar set on part of a region before code() is applied to the whole is ignored (see Geovariables).

diagplace(r nrep dx dy): Repeatedly places the region r nrep times along a diagonal where each successive placement is incremented by dx and dy in the x and y-axis, respectively. The repetitions are named x0, x1, .... This allows terminals to be replicated inside a given region.
diagarrayplace(r dx [i j ...]): Repeatedly places the region r with an array of offsets in the y-axis and where each successive placement is also increment by dx in the x-axis.
rep(r name nrep dx dy): Replicate the region r nrep times with a step of dx and dy in the x and y-axis, respectively. This differs from diagplace since each replicate is not a separately named region. The dimension will be called name and can be similarly then queried via any command that supports the order argument (link, miget, see rect() above).

Code

We can apply microcode to a region via the code() primitive:

a : code('test [param1:value1 param2:value2 ...])

This attaches code to the subtree below the attribute. Paint searches the library paths for test.w (Tungsten) or test.yeti (Yeti, the assembly language the Tungsten compiler emits), trying the suffixes named by compiler= first. It does not compile .casm, .csl, or .mk sources. Parameters and sockets are passed by name inside [...].

We do not support overlapping code regions on a PE. Paint rejects a union of overlapping regions that both have code attached with code(). It does not reject a second code() on the same region: both kernels are placed at the same addresses and one overwrites the other, so call code() once per region.

Geovariables

A given parameter value for a code region may vary over the region. Rather than requiring specifying the code region on each set of pes with distinct values, paint supports passing in a variably-valued argument called a geovariable (or geovar). A geovar consists of separate geospaces with a given geovalue.

Consider this simple example:

let a = tile(0 0)
a : geovar('example 1)
let b = tile(1 0)
b : geovar('example 2)
let all = rect(0 0 2 1)
all : code('kernel [id:@example])
all : union(a b)

The rect all consists of tile a and b on (0, 0) and (1, 0), respectively. We apply the same microcode to both tiles but with a different value for the geovar example. We pass in the argument to the parameter list using an @ symbol to indicate that it is a geovar.

Here code() sits on all, a region beside a and b. The other working form applies the geovar after code(), as arrayplace does with copies that already carry code. A geovar set on part of a region before code() is applied to the whole is ignored without a message, and every PE compiles with one value: with the geovars above, let all = union(a b) followed by all : code('kernel [id:@example]) compiles both tiles with id=1.

A geovar must cover the complete area on which the corresponding code attribute is being applied. Paint checks this, and fails with Geovar example does not cover region and the uncovered range when the geovar misses part of that area. The check counts each region the geovar is set on as covering every pairing of the columns and rows its tiles occupy, so it misses a gap inside a region replicated along a diagonal: rep(tile() 'i 2 1 1) holds (0, 0) and (1, 1), and the check also counts (1, 0) and (0, 1) as covered. Those two tiles then receive the geovar's value.

Arcvar

An arc variable (arcvar) is sent over a routing path. The variable is defined at the origin of a routing path and available for use at the destination.

The Language Reference gives the syntax: an arcvar is defined on a terminal and read, with the ^ prefix, by the region at the other end of its transport.

Symbol variable

A symbol is a reference to a variable in a kernel’s symbol table. For example, a variable defined in microcode. It can be used to provide initial values for a memory region in a kernel.

Colors

In paint the user will declare symbolic colors and use them to annotate routes and in terminals. The paint engine has a built-in solver that will then assign physical color IDs to these symbolic colors, ensuring to avoid conflicts and taking any constrains into account.

We can declare symbolic colors to be used via:

color(<d>): Specifies a symbol color variable with an optionally hard-coded physical value. Paint warns about a hard-coded value; let the solver choose. A PE has 24 colors, and some have fixed jobs; Colors in the Language Reference lists them.

For example, we declare a color:

let tx = color()

To indicate that two colors should form a colorpair:

colorpair(r col1 col2): Annotates the region r to indicate that col1 and col2 are the swap of one another, i.e. the following relationship holds col1 == col2 ^ 1.

This is then fed as part of the input to the coloring algorithm.

Routing

Paint allows users to manually specify the route config on regions. The syntax is:

paint(r color routerule): Annotates the region r with the given routerule on color c.

For example:

let tx = color()
let a = tile(0 0)
a : paint(tx [D>C])

This sets the routing on tile (0, 0) to accept from DOWN and send to CENTER. The available directions are down (D), up (U), right (R), left (L), and center (C).

Boundary conditions

We can set the routing to have special treatment on the boundary pes using the | axe-operator. This allows us to avoid accepting or sending outside of a rectangle. For example:

let tx = color()
let rx = color()
let r = rect(0 0 5 5)
r : paint(rx [L|L>R])
r : paint(tx [L>R|R])

In this example r will not have rx set to accept from the LEFT for the leftmost column and will not have tx set to send to the RIGHT for the rightmost column. We can pass in multiple directions to this boundary “noexit” / “noentry” treatment.

The edges are those of each rectangle the rule is painted on, such as one rect, one tile or one hstackrep block, not the outline of a region joined with union, hstack or vstack. On such a joined region paint also gets every rectangle after the first wrong, and gives no message: on hstack(tile() tile() tile()), [L|L>R] turns off the input of all three tiles. Apply | rules to one rect() that covers the grid.

Color swaps

A color swap makes a color take its input from its partner color, the one numbered c ^ 1, so wavelets change color on the router without a core relaying them. Use colorpair to name the partner. The swap replaces > in the route rule:

H swaps the left and right (east/west) inputs.
V swaps the up and down (north/south) inputs.
X swaps all four and the input from the core (onramp).

An input that is not swapped still reads the color's own wavelets, so a swap usually needs its partner to swap as well. A swap belongs to the color on that router, so every switch position of a rule must use the same modifier. Paint does not check this: it takes the modifier from the first position and ignores the others without a message. For example, one socket r can send right and receive from the left:

row : paint(rp [CXR|R])   (* rp carries what the core sends on r *)
row : paint(r  [L|LXC])   (* r takes rp from the left, for the core *)
row : colorpair(r rp)

Color Swapping in the architecture notes draws this and a second use, a moat that turns a route around.

Switching

We can annotate up to four switch positions inside the routerule array. For example:

r : paint(tx [L>R; L>U; D>U; ring pop=all])

Sets the initial switch position to accept from LEFT and send to RIGHT, then to accept from LEFT and send UP, and finally to accept from DOWN and send to UP. ring and pop=all are modifiers, not a fourth position.

The router moves to the next position when a control wavelet with an advance command passes through it, or when its own core sends the last word of a flip send; a reset command returns it to the first position. The control wavelet itself leaves in the old direction. Only 16 of the 24 colors can switch (0 to 9, 12, 13, 16, 17, 20 and 21), and paint gives each rule with positions one of them. Switch Positions in the Language Reference gives the details.

Paint does not reject a fifth position: it writes past the four slots and changes the outputs of the earlier positions.

The switch is configured to operate in ring mode, if the ring is omitted it will not operate in ring mode.

We can also control the pop mode of the switch: whether a control wavelet leaving the router has its first command removed, so the next router acts on the next command. The possible settings are:

pop=all: unconditional pop.
pop=adv: pop on advance.
pop=advnop: pop on advance or nop.
if none of the above are passed in then there is no pop configured.

Teardown

We can annotate for the color to begin in teardown by adding tip after the final ;. In teardown, the color accepts no wavelets on that router until a program on the tile clears the color's teardown-in-progress bit. Paint leaves every color that has no rule on a tile in this state, with its input disabled:

a : paint(tx [D>C; tip])

Primitives

There are primitives available via the library to paint more complicated routing patterns:

paintuppertriangle(r color routerule xoffsets yoffsets xend): Adds new tiles painted with routerule to r, at absolute coordinates; the tiles r already has are not painted. For the ith entry the new tiles lie on row yoffsets[i], from xoffsets[i] to xend, both exclusive. For example, with yoffsets = [0 2 3], xoffsets = [0 3 4], xend = 5, setting a L>R route would give the following routing pattern:

y  x  0   1   2   3   4 
  --- L>R L>R L>R L>R
  --- --- --- --- ---
  --- --- --- --- L>R
  --- --- --- --- ---     <-- note, the xoffset is the final col, so no routing

The yoffsets indicate that rows 0, 2, & 3 will have routing. The xoffsets indicate that we will route from columns 0, 3, & 4. Therefore the “diagonal” points are (0,0), (3, 2), & (4, 3). Since xend = 5 we will route the upper-triangle to the right of those diagonal points, i.e. (1,0) - (4,0), (4,2) - (4,2), . The last row produces no routing because neither the diagonal point nor the xend are included.

Auto-routing

As well as the ability to manually draw routes via the paint() primitive, paint also supports letting the internal router connect up pes.

Regions contain terminals (ports). These terminals will then be connected by applying a transport region on top of them.

Terminals

We define a terminal using the following syntax:

terminal(region name tag side color shape split chunk [revpin])
- The tag is a way of annotating the “type” of data flowing through a terminal (e.g. act, wgt, cmd). Terminals that are connected via transports will only route if they have matching tags.
- The shape array relates to the shape of the tensor flowing through the terminal.
- The split array relates to how we split the tensor across pes. The length of the split must match the length of the shape. The product of the split is the number of pins that should be used.
- It is not enforced, but connected terminals should advertise the same shape and split.
- chunk is recorded with the terminal (in regiondump) and not used by the router; the examples pass 1.
- revpin indicates that the pins of the terminal should be reversed.
- Paint gives the two terminals of a transport the same physical color, as txact and rxact below get.

For example,

let txact = color()
let a = tile(0 0)
a : paint(txact [C>R])
a : terminal('tx 'act RIGHT txact [1 1] [1 1] 1)
a : name('a)

This creates a terminal called tx on the region named a which has a shape of [1 1] and a split of [1 1] (i.e. a single pe is used). The terminal uses the txact color and leaves from the right side of the region.

The region name must be applied after the terminal, otherwise the terminal will be not attached to the region named a but the one higher in the tree. A good way to think about this is as if you are building the tree from the ground up and properties apply successively on top of each other.

A terminal consists of a set of “pins” which are the set of pes that will be routed to/from. These are sometimes also referred to as “hot pes”. Pins do not need to be advertised. Paint will try to detect the pins based on the side and color used. It will search for pes on the side advertised sending on the specified color in that direction. A region with no such paint() rule makes paint stop with has no pins defined.

Replication of a region with a terminal via rep() will not cause the terminal to be replicated. Instead the region will still only contain that same terminal. However, if a region is replicated using arrayplace() or a similar method will repeatedly places the region with a name, then each of those replicas will contain a terminal. If you want a terminal to take up the whole of the region after replication, the terminal should be applied after replication.

Terminals can be advertised with one of the five directions as an edge. However, the behavior of CENTER differs from the others:

RIGHT/LEFT/UP/DOWN: Searches for pes using the specified color in the specified direction. The router will not modify the routing settings for that pe, instead beginning/ending on the pe adjacent in the given direction. In the RIGHT example above, this means the routing would begin on (1, 0).
CENTER: Searches for pes using the specified color either onramp [C>] or offramp [>C]. It will then begin/end the routing on that pe. This allows the router to choose to leave the pe in any direction, rather than the one specified in a cardinal direction. However, this means the router will overwrite any other routing set on the active pin. Continuing the example above, we can add an input terminal using CENTER:

let rxact = color()
let b = tile(5 0)
b : paint(rxact [>C])
b : terminal('rx 'act CENTER rxact [1 1] [1 1] 1)
b : name('b)

Since the terminal is advertised as CENTER the router would end on (5, 0).

Transports

We can connect two terminals via a transport region. The syntax is:

transport(src dst name)

The transports create a new region which should then be unioned with the regions containing the terminals. For example, continuing the example above :

let arc = transport('a.tx 'b.rx 'arc)
let all = union(a b arc)

Running paint and using the portmap option gives the following output:

$ paint source=test.paint portmap
(TRANSPORT
  (TERMINAL 'a.tx.act [ 1/1 1/1 ] RIGHT @(1 0) 0 [ 0])
  (TERMINAL 'b.rx.act [ 1/1 1/1 ] CENTER @(5 0) 1 [ (0,0)]))

It is important to note a few things to ensure success:

Terminals must use the same tag to route.
The region that contains the terminal must be named, otherwise there is no way to reference it. Paint will search for a child region with the specified name and expect that a terminal of the specified name exists. If not, it will give a message saying that the port(s) were not found and ignore that transport.
A transport names each terminal by its path of region names, such as 'a.tx, or 'c.b.rx for a terminal of b inside c.
Paint still lets the build succeed after it drops a transport. Besides a wrong name, name() applied before terminal() and two different tags also give the port(s) of arc a.tx -> b.rx not found message; a transport left out of the final union gives none. The image then has no route for the transport, and the program stalls when it runs, so check the build log for that line.
paint has no notion of fabric size or bounds it should be constrained to. Therefore it is possible that the terminals are placed in a manner such that the router will go outside off-fabric. Be wary of such cases.

Aliases

At times, you may want a parent region to offer a terminal of one of its children under its own name, so that a transport need not know what is inside the parent. To achieve this we make use of aliases (port mapping) which allow us to annotate a “dummy” terminal onto a parent region that simply maps to the child terminal. The syntax is:

alias(r name child_terminal): Creates a terminal called name on region r which maps directly to child_terminal.

As an example, we can introduce another region c into the example above to disrupt the hierarchy:

let txact = color()
let a = tile(0 0)
a : paint(txact [C>R])
a : terminal('tx 'act RIGHT txact [1 1] [1 1] 1)
a : name('a)

let rxact = color()
let b = tile(5 0)
b : paint(rxact [>C])
b : terminal('rx 'act CENTER rxact [1 1] [1 1] 1)
b : name('b)

let c = rect(5 0 2 1)
c : union(b)
c : alias('rx 'b.rx)
c : name('c)

let arc = transport('a.tx 'c.rx 'arc)
let all = union(a c arc)

Running this through paint with the portmap option we can see the alias taking effect:

$ paint source=test.paint portmap
(TRANSPORT
  (TERMINAL 'a.tx.act [ 1/1 1/1 ] RIGHT @(1 0) 0 [ 0])
  (TERMINAL 'c.b.rx.act [ 1/1 1/1 ] CENTER @(5 0) 1 [ (0,0)]))

This output is telling us that there are two terminals a.tx.act and c.b.rx.act connected via a transport. The input terminal was advertised in the code as c.rx but in the printout here we can see that paint has expanded the alias to give the full path c.b.rx.act. Inside the square-brackets is the shape / split for each dimension of the tensor. The number of pins detected will be the product of the divisors.

Following this is the side of the terminal.

Finally, we have the description of where the terminal is located. The co-ordinate prefixed by symbol @ is the origin of the terminal (in the global co-ordinate space) and the entries in the final square brackets are the offsets from that origin. For a cardinal side terminal this will be a single integer offset in the direction of the side. For a CENTER terminal this will be an (x,y) offset for each pin.

The number between the origin and the offsets identifies the terminal's symbolic color, numbered in declaration order. It is not the physical color: here both terminals are routed on one physical color.

Routing algorithm

Currently the built-in router to the paint engine is a fairly simplistic heuristic-based bus router. It cannot support all cases and should not be expected to. It does support straight line and orthogonal routes, knowing when it will need to use the color swap to overlap wires. You do not need to advertise a colorpair() attribute for paint to know when the input/output terminal colors must be the swap of one another.

It does not support multicast routes and so any attempts to do this manually by annotating multiple transports from the same source terminal should be attempted with caution. Successive calls to the router with the same color will “paint over” the previous routing. For example, if you have a set of transports added via a union that use the same color, the right-most transport in the union will have the final say on any pes which overlap.

Link

The code() region allows the user to pass in different parameter values to the underlying source code via the use of geovar() as detailed above. However, each unique combination of parameters results in a separate compiler invocation, which can lead to longer compile times.

The link() primitive is an alternative post-processing fix-up command that is provided to get around this issue. Instead of having to pass parameter values via the code() primitive, it allows the user to specify a symbol, a range of memory and pes to apply this primitive to, and an executable to generate values to set those memory values for.

The basic syntax for the link command is:

link(region symbol order program program_params [disable])

more commonly used as:

region : link(symbol order program program_params [disable])

where:

region is the paint region to apply the link command on,
order specifies the order to access the different memory addresses,
program specifies the executable to call to generate the words that will be set to the address values; any executable on PATH works,
program_params specifies the parameters to pass to the program, and
disable is an optional argument that turns the command off without removing it.

The symbol is looked up from the region link is applied to: on region grid, $aa.cell names grid.aa.cell.

milink runs the program as program key=value ... out=/dev/fd/N, one argument per entry of program_params; numbers and colors arrive in hex, such as color=0x5. The program writes raw little-endian 16-bit words to the out= file, one word per element of the dimensions in the order string. milink warns supplied data is too short or too long when the count differs, and fails when the program exits with a nonzero status.

The link annotations are collected by paint and annotated into the .note files produced. After paint writes the image (bench.obj in the build), the build runs milink on it to apply them, for example:

milink arch=sdr image=bench.img out=bench.img map=bench.map note=bench.note [nofixup]

Notes:

The nofixup argument allows an optimization when no link() command reads and fixes up an existing value, rather than just setting a value.
The program_params supports passing in color variables which will be evaluated to their assigned physical color.

Filter example

As an example of the link() command we can consider one common use-case which is for configuring the filter settings across a range of pes. The use of the link() primitive is convenient in this scenario since:

Each pe will have a separate maxpass value and passing this value in via a param in the code() region would be cumbersome.
The maxpass values will follow some arithmetic pattern that we can allow a simple program to generate for us.

For a region r:

     r : link($filter0_cfg "y n" "counter_filter" [
         color: rx
         ...
     ])

The $ character specifies that $filter0_cfg refers to a symbol.
"y n" specifies the order in which we should access the address.
counter_filter is the executable that will be called. It stands for a program you provide; the toolset does not ship one.
We will pass in whatever physical color rx is assigned to as the value for the argument color to counter_filter. The other arguments are omitted here.

To understand better what the “order” refers to, consider the map file entry for this symbol:

kernel.core.filter0_cfg [ y<9 x<6 n<4 ] ( 0x7f70+n 36+x 1+y )

This symbol is therefore spread across x and y in physical space, and n in memory. The rightmost entry is the fastest-moving and if counter_filter returns words: x1, x2, x3, x4, etc. then we will first set those four values for x=36, y=1. The next four values get set for x=36, y=2, then x=36, y=3, and so on. Since x is not provided in the order but is present in the symbol, the same values are written to every x column.

A good strategy to debug is to call the executable directly to see what values get returned and compare this with the address values returned from using a miget query with the same order symbol and order specified.

Running paint

paint can be run from the command line using the paint command.

Paint has several command line options which you can use and we won’t go through them all here. The main options to be familiar with:

source: the input Paint file
lib: library path to look at for include files and code() kernels. This argument can be used multiple times for different libraries.
image: the output machine image file (a Cerebras machine image, not an ELF file)
map: the output symbol table
portmap: dumps to ostream the connectivity. This can be useful for debugging routing failures.
compiler: source suffix priority for code() attachments (w, yeti).
toolname w carbide: the compiler paint runs for sources with that suffix. Give it for both w and yeti.
toolflag w "flags": flags to pass to the compiler for .w sources, the one toolname w names.
regiondump=file.json: writes a JSON representation of the region tree.
noroute: disables automatic transport routing (enabled by default).

You can see a summary of most options by running the command below. It omits some, such as listing, debuginfo= and threads=, which the Language Reference describes.

paint help

Therefore, to build a simple paint test-case with Carbide, then apply its link() notes:

paint lib=. arch=sdr toolname w carbide toolname yeti carbide \
  source=bench.paint image=bench.obj map=bench.map note=bench.note
milink image=bench.obj out=bench.obj map=bench.map note=bench.note

A tutorial Makefile runs both steps for you through the build system's flute.mk.

Examples

Most examples are inline in their sections. The tutorial has complete, runnable examples.

Interleaving two kernels in a row of size H

(* interleave.paint — interleave two kernels along a row of size H *)

(* ---- user knobs ---- *)
let H         = 16          (* row length *)
let KERNEL_A  = "kernel_a"  (* compiles kernel_a.w *)
let KERNEL_B  = "kernel_b"  (* compiles kernel_b.w *)

(* convenience counts *)
let N_EVEN = idiv(H + 1 2)  (* ceil(H/2) *)
let N_ODD  = idiv(H     2)  (* floor(H/2) *)

(* ----- build even sites (x = 0,2,4,...) ----- *)
let even_tile = tile(0 0)
even_tile : code(KERNEL_A [])                 (* attach microcode before replication *)
let evens = rep(even_tile 'i N_EVEN 2 0)      (* step by +2 in x *)

(* ----- build odd sites (x = 1,3,5,...) ----- *)
let odd_tile = tile(1 0)
odd_tile : code(KERNEL_B [])
let odds = rep(odd_tile 'j N_ODD 2 0)         (* step by +2 in x *)

(* ----- final region ----- *)
let interleaved = union(evens odds)
interleaved : name('row)

How it works

rep(region 'idx nrep dx dy) names the replicated dimension (here 'i/'j), which miget, miset and link orders can use. It is not a variable: code() cannot read it, so a per-copy value needs a geovar.
We seed the even series at (0,0) and step by dx=2.
We seed the odd series at (1,0) (already offset by 1) and also step by dx=2.
union(evens odds) merges both into a single row named 'row.

Introduction​

Reading Paint​

Basic syntax​

Tokens​

Expressions​

Function definitions​

Assignments​

Includes​

Arrays​

Dictionaries​

Rules that differ from C and Python​

Primitives​

Arithmetic​

Strings​

Conditionals​

Debug​

Layouts​

Updated place syntax (with rotation)​

Code​

Geovariables​

Arcvar​

Symbol variable​

Colors​

Routing​

Boundary conditions​

Color swaps​

Switching​

Teardown​

Primitives​

Auto-routing​

Terminals​

Transports​

Aliases​

Routing algorithm​

Link​

Filter example​

Running paint​

Examples​

Interleaving two kernels in a row of size H​

Introduction

Reading Paint

Basic syntax

Tokens

Expressions

Function definitions

Assignments

Includes

Arrays

Dictionaries

Rules that differ from C and Python

Primitives

Arithmetic

Strings

Conditionals

Debug

Layouts

Updated `place` syntax (with rotation)

Code

Geovariables

Arcvar

Symbol variable

Colors

Routing

Boundary conditions

Color swaps

Switching

Teardown

Primitives

Auto-routing

Terminals

Transports

Aliases

Routing algorithm

Link

Filter example

Running paint

Examples

Interleaving two kernels in a row of size H