Fabric adder and message passing

29. Fabric Adder

The fabric adder is a hardware unit in the router that performs in-fabric floating-point reduction. When enabled on a color, each tile takes one fp32 operand from the fabric (arriving via an NSEW port) and one fp16 operand from the CE (the local compute element), adds them, and sends the fp32 sum out one NSEW exit port. This allows a chain of tiles to perform a running sum reduction without software involvement beyond supplying the local fp16 contribution.

Operation

On a tile with the fabric adder enabled:

1. An fp32 wavelet arrives from a fabric direction (e.g., from the left via L).
2. The CE sends one fp16 wavelet on the same color. The rule does not name C: the adder takes the CE operand by itself, and [CL>R; fabadder] is rejected (Multiple router inputs set).
3. The hardware computes fp32 + fp16 → fp32 and sends the result out the configured fabric exit port (e.g., rightward via R).

There is no ordering requirement — the sum is computed when both operands are available. Operands are buffered in separate queues (one for fabric, one for CE). So each fabric wavelet waits for one CE operand: an adder tile whose CE sends nothing stalls the chain.

After >, name exactly one of R, L, U or D: the CE cannot receive a sum. Paint accepts [L>C; fabadder] and [L>RU; fabadder] without an error, and the hardware result is undefined. The tile that keeps the sum uses a plain rule such as [L>C].

The CE must send each contribution as one hp value. The adder reads a 32-bit wavelet from the CE as two packed fp16 operands, so an sp contribution compiles and runs but gives a wrong sum: with sp 2.0 and 3.0 the example below gives 100.0.

Hardware rules on an adder tile:

• Control wavelets arriving on NSEW fabric ports pass through without modification, out the same port as the sums.
• A control wavelet from the CE gives undefined results, so the CE sends only fp16 data on the adder color.
• The adder color carries no other traffic.
• Denormal fp32 inputs from the fabric and denormal sums are flushed to zero; fp16 denormals from the CE are kept.
• Rounding is always to nearest even.

Paint syntax

The fabric adder is enabled by appending a flag keyword inside the routing brackets:

(* fp16 + fp32 → fp32 addition *)
t : paint(c [L>R; fabadder])

Flag	Mode	Description
fabadder	F16	fp32(fabric) + fp16(CE) → fp32

Color constraint

The fabric adder is only available on specific physical colors:

• SDR: color 10 only.

When a fabric adder flag is specified, paint automatically pins the virtual color to the required physical color. You do not need to manually specify color(10). Pinning a fabric adder color to another number makes paint abort with double free or corruption. Color 10 cannot switch, so a fabadder rule has one position.

FP16 data type

The float16_dtype paint option sets the format of the CE operand. It is required whenever a rule uses fabadder: without it, paint stops with Invalid float16 data type 0 used with SDR fabric adder. Tungsten hp is IEEE FP16, so use float16_dtype=10 unless the kernel builds the bits of another format itself.

Value	Type	Exponent	Mantissa	Bias
10	F16 (IEEE 754)	5 bits	10 bits	15
16	CBfloat16	6 bits	9 bits	31
15	BFloat16	8 bits	7 bits	127 (fixed)

PAINTFLAGS=... float16_dtype=10    # use IEEE FP16 for CE operand

Typical chain topology

A fabric adder reduction chain consists of:

1. Head tile — sends the initial fp32 value into the fabric with a normal routing rule (no fabadder flag):

   head : paint(c [C>R])

2. Interior tiles — each adds its local fp16 contribution to the running fp32 sum:

   mid : paint(c [L>R; fabadder])

3. Tail tile — receives the final fp32 result:

   tail : paint(c [L>C])

  head          mid1          mid2          tail
┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
│ CE──►R ──┼──┤►L+CE──►R ─┼──┤►L+CE──►R ─┼──┤►L──►CE │
│ (fp32)   │  │ (fp16)   │  │ (fp16)   │  │ (fp32)  │
│ [C>R]    │  │ [L>R;fab]│  │ [L>R;fab]│  │ [L>C]   │
└──────────┘  └──────────┘  └──────────┘  └──────────┘

Example

A 4-tile chain where the head sends 100.0 as fp32, and two interior tiles add 2.0 and 3.0 as fp16. The same parameterized kernel is used with a different contribution on each tile. The receiver gets 105.0. Run this example with arch=sdr float16_dtype=10.

let c = color()

let head = tile()
head : code('head [tx:c])
head : paint(c [C>R])
head : name('head)

let add0 = tile()
add0 : code('add [tx:c VALUE:2])
add0 : paint(c [L>R; fabadder])
add0 : name('add0)

let add1 = tile()
add1 : code('add [tx:c VALUE:3])
add1 : paint(c [L>R; fabadder])
add1 : name('add1)

let recv = tile()
recv : code('recv [rx:c])
recv : paint(c [L>C])
recv : name('recv)

hstack(head add0 add1 recv)

Tungsten kernels, one file per code() name.

head.w:

// Sends the initial fp32 value into the chain.
socket tx;

tx[] <- 100.0:sp;

add.w:

// Adds this tile's fp16 VALUE to the running fp32 sum.
hp param VALUE;
socket tx;

hp a = VALUE;
tx[] <- a;

recv.w:

// Receives the final fp32 sum.
socket rx;
sp result[1];

main {
    result[0] <- rx[];
}

The tutorial topic Reductions in the router: the fabric adder is a complete working example.

---

30. Message Passing

Message passing is a hardware routing mode that allows any tile to send a multi-wavelet message to any other tile using coordinate-based routing. Unlike normal color-based routing (where paths are statically configured per color), message-passing wavelets carry a destination header and the fabric routes them automatically.

Overview

Normal color routing:

• Path configured at build time via paint() rules.
• Each tile's rules for a color fix where it goes; together they can branch ([L>CR]), and switch positions can change them at run time.
• All tiles on the path must be configured.

Message passing:

• Destination encoded in a header wavelet.
• Hardware routes X-first, then Y to reach the target.
• No per-destination paint rules needed — only MP bank enable.
• Any-to-any communication within the enabled region.

Banks

Message passing uses two independent banks of 5 reserved colors:

Bank	Physical Colors	CE Color
Bank 0	14, 15, 16, 17, 18	16
Bank 1	19, 20, 21, 22, 23	21

Each bank dedicates one color per input direction (E, N, CE, S, W). The CE color (16 or 21) is the offramp — regardless of which direction a message arrived from, it is delivered to the PE on the CE color. The socket in the kernel binds to this CE color.

Allocating Banks

let b0 = bank(0)       (* allocate MP bank 0: colors 14-18 *)
let b1 = bank(1)       (* allocate MP bank 1: colors 19-23 *)

bank(n) allocates all 5 colors for bank n at once and returns a bank handle. Only values 0 and 1 are valid.

Getting the CE Color

let b0color = color(b0)   (* CE onramp color for bank 0 = physical 16 *)
let b1color = color(b1)   (* CE onramp color for bank 1 = physical 21 *)

When color() receives a bank handle, it returns the CE color for that bank. This color is used in code() parameter binding to connect a kernel socket to the message-passing channel.

Enabling Message Passing on Tiles

let t = tile()
t : mp(b0)              (* enable MP bank 0 on this tile *)
t : mp(b1)              (* enable MP bank 1 on this tile *)

The mp() directive:

1. Programs the router on every PE in the region for all 5 bank colors with the MP routing mode.
2. Sets the bank-enable bit in the Fabric Misc Config Register (bit 2 for bank 0, bit 3 for bank 1).

Message passing also uses the Row ID and Column ID registers. Paint writes them on every tile of a layout, with or without mp(), so each PE knows its own fabric coordinates.

For a replicated region with mp(), each dimension must step along at most one axis (dx == 0 or dy == 0). Paint rejects a dimension that steps in both axes because message-passing configuration cannot represent it.

Important: Every tile on the routing path between sender and receiver must have mp() enabled for the same bank, even tiles that only forward messages (pass-through tiles).

Binding Sockets

t : code('kernel [sock:b0color])

The kernel's socket is bound to the bank's CE color, just like binding to any other color. Inside the kernel, the socket sends and receives raw wavelets (header + payload).

Complete Example

(* Two tiles: sender sends a message to receiver via MP bank 0 *)
let b0 = bank(0)
let b0color = color(b0)

let sender = tile()
sender : code('sender [sock:b0color])
sender : mp(b0)
sender : name('sender)

let receiver = tile()
receiver : code('receiver [sock:b0color])
receiver : mp(b0)
receiver : name('receiver)

hstack(receiver sender)

If there are intermediate tiles between sender and receiver, they must also have mp(b0) applied. The intermediate tiles do not need a socket bound to the bank color — the hardware forwards messages through the router automatically.

Hardware Routing

The fabric routes messages using the following algorithm:

1. X-first: Route east or west until the column matches the destination col_id.
2. Y-next: Route north or south until the row matches the destination row_id.
3. Deliver: Offramp to the destination PE's compute element via the CE color (16 or 21).

The sender must set header bit 7 when the destination row is north of its own row (a smaller Paint y) and bit 6 when it is south (a larger y); on the same row it sets neither. The fabric takes the Y direction from these bits, so with a missing or wrong bit the message does not reach its destination.

Constraints

• Atomic messages: Once a tile begins sending a message, it must finish. Partial messages will lock the routing path.
• Control wavelet functions are off: teardown, the no-CE bit and control-wavelet switching do not work on MP colors. The control bit still marks the end of a message.
• No half-wavelets: Sending half-wavelets on MP colors is undefined behavior.
• Two banks maximum: Only bank 0 and bank 1 are available.
• Reserved colors: each bank takes 5 colors, 14–18 for bank 0 and 19–23 for bank 1, reducing the colors available for normal routing. They include 4 of the 16 switching colors: 16, 17, 20 and 21.

Message Header Format

See the Tungsten Language Reference — Message Passing for the 32-bit header word format and Tungsten send/receive patterns.

The tutorial topic Message passing: routing by address is a complete working example.