Sockets, FIFOs and rings

9. Sockets

Sockets are the primary mechanism for inter-PE (processing element) communication. They represent endpoints for sending and receiving data wavelets across the fabric network.

A wavelet carries 32 bits. An sp or u32 send fills one wavelet. An xp or hp send is 16 bits, and two consecutive 16-bit sends can share one wavelet: four xp sends of 1, 2, 3 and 4 read back as the two u32 values 0x20001 and 0x40003. Nothing checks that the receiver reads the type the sender sent, so read each value with the type and width it was sent with, in the same order.

A socket's fabric color comes from the Paint layout: code('mycode [tx:txcolor]) binds socket tx to the color txcolor, which Paint passes as set=tx={.color=N,.bus=B}, with N the color's number (see Code Attachment and Colors in the Paint Language Reference). A socket left unbound fails the compile with socket [tx] is not initialized.

Declaration

socket rx;              // type inferred from usage
socket tx;
socket rx, tx;          // multiple sockets

Note: Declare sockets without a type, not as xp socket or hp socket. The compiler infers socket types from context, or types can be explicitly cast at the usage site.

Design Principle: Sockets should always be declared without a type prefix. Use explicit casts at the point of use when the type is ambiguous.

Socket Dimensions

socket data[n=10];      // dimensioned socket

Socket Attributes

Attributes are specified in double brackets:

socket rx [[iq=0]];
socket buffered_rx [[doubled=1]];
socket stored_rx [[dir=in]];

Common attributes:

• dir=in — Lets the socket be stored in a descriptor (see Streams); storing a socket without it is rejected.
• doubled=1 — Reserve a paired input queue for additional receive buffering. It does not widen the fabric data path. Without it, the allocator doubles the receive sockets that have no iq only when all receive sockets fit in pairs; otherwise it assigns one queue each.
• iq — Input queue assignment, 0 to 7

Each receive socket takes one input queue, or two when doubled; a socket that only sends takes none. A PE has 8 input queues, so a kernel can have at most 8 receive sockets; a ninth fails with Allocator needs [9] input queues (available: 8).

Sending Data

Write to a socket using the stream access syntax (empty brackets):

socket tx;
xp a[n=4] = {1, 2, 3, 4};

// Send array elements one at a time
i in [0, 4) tx[] <- a[i];

Receiving Data

Read from a socket:

socket rx;
xp a[n=4];

// Receive into array
i in [0, 4) a[i] <- rx[];

Sending Structured Wavelets

You can pack multiple values into a single wavelet for dispatch:

// Send a wavelet with named fields
tx[] <- wavelet(index, value);

// The receiver dispatches on the name 'wavelet'
forever dispatch(rx) {
    wavelet(xp i, hp x) {
        data[i] <- x;
    }
}

wavelet(...) is an RPC, so it compiles only when the receiving kernel has a dispatch with a wavelet handler; otherwise the sender fails with RPC function declaration for [wavelet] not found in scope. To send an index and a value to a receiver that reads a[].index and a[].data, build the 32-bit word with a union instead, the index in the upper half (see Raw Wavelet Send/Receive via Union).

---

10. FIFOs

FIFOs (First-In, First-Out queues) provide on-PE buffering backed by local memory.

Declaration

xp mem[n=N];
xp fifo ff = {.mem = mem};   // FIFO backed by array 'mem'

Typed FIFOs:

hp fifo_mem[n=100];
hp fifo hpff = {.mem = fifo_mem};

Usage

Write to a FIFO:

i in [0, N) ff[] <- data[i];

A local function call can supply the value directly. The function is evaluated locally, then its result is written to the FIFO:

function produce(xp value) {
    return value + 1;
}

i in [0, N) ff[] <- produce(data[i]);

Read from a FIFO:

i in [0, N) result[i] <- ff[];

A FIFO holds as many elements as its backing array. A write to a full FIFO waits until a read frees a slot, and a read from an empty FIFO waits for a write. Serial code has nothing else to run while it waits, so writing more than the capacity before reading, or reading more than was written, stops the PE at that statement. Keep the counts within the capacity, or put the writer and the reader in different arms of a parallel block.

FIFOs are commonly used with parallel blocks for producer-consumer patterns:

xp fifo afifo = {.mem = a};

parallel {
    i in [0, N) afifo[] <- rx[];   // producer: receive from socket into FIFO
    i in [0, N) tx[] <- afifo[];   // consumer: send from FIFO to socket
}

FIFO Count

You can query the number of elements in a FIFO:

xp count;
count = ff.count;

---

11. Rings

Rings are circular buffers backed by local memory. A FIFO is one too, but a ring has no full or empty state: a write never waits and overwrites the oldest data, and a read never waits and returns whatever its slot holds.

Declaration

xp ring_mem[n=N];
xp ring r = {.mem = ring_mem};

Usage

A loop that writes a ring and a later loop that reads it each use their own ring over the same memory, one per head:

xp ring wr = {.mem = ring_mem};   // write head
xp ring rd = {.mem = ring_mem};   // read head, over the same memory

// Write to the ring: 2*N values wrap around, so the ring keeps the last N
i in [0, 2*N) wr[] <- rx[];

// Read from the ring
j in [0, N) result[j] <- rd[];

Each head starts at slot 0 of the backing array and wraps at its end, independently of the other head. After 6 writes into a 4-word ring, 8 reads give 5, 6, 3, 4, 5, 6, 3, 4. The example above reads the last N values in order only because it writes exactly 2*N. Using one ring for both the writes and the reads is rejected (each head needs its own).

Multiple rings can share different backing memory:

xp a[n=N], b[n=N];
xp ring ring1 = {.mem = a};
xp ring ring2 = {.mem = b};