---
name: kcoral-client
description: >-
  Write client code for KCoral's POST /execute instruction
  protocol. Use when writing, reviewing, or debugging programs that upload
  kernels (TIRx, CUDA C, CuTeDSL, Triton), create tensors, compile, check
  correctness, or benchmark on a remote GPU server, or when using the
  kcoral Client and Program API.
---

# KCoral client

Facts needed to write a protocol-conformant client. `docs/client-guide/protocol.md` is the
authoritative field-level specification; where this page and that file
disagree, that file wins.

## Model

- The client endpoints are `POST /execute` and `GET /health`.
- A request body is a **program**: an ordered list of instructions executed
  top to bottom on a GPU or CPU worker.
- There is no session state. Handles (`id`s) live for one request; a second
  `execute` shares nothing with the first, so every program uploads everything
  it needs.
- `run` computes a value but does not send it back. The response carries only
  what `return` selects.
- A reference has the exact form `{"$ref": "<id>"}` and must point to an
  earlier instruction.
- The `GET /health` response includes the GPU `target` (e.g. `{"arch": "sm_100a"}`),
  installed `versions`, and `load`: `request_capacity` (occupied + free capacity),
  `requests_in_progress` (assigned requests), and `requests_waiting`
  (requests awaiting assignment).

## Python client

For simple tasks, use `@client.function(timeout=30)` and call `.remote()` for the
decoded value, or `.execute()` for the full `ProgramResult`. Define the function
in a file, import dependencies inside it, and pass inputs explicitly. Configure
the server on `Client` and keep it open for remote calls. See
[Remote functions](../../../docs/client-guide/remote-functions.md)
for supported inputs and execution boundaries.

Upload a harness, select its entry point, run it and return what you want to inspect:

```python
from kcoral import Client, Program

program = Program()
module = program.upload(kind="module", source="""
import torch

def main(n):
    x = torch.arange(n, dtype=torch.float32, device="cuda")
    y = x + 1
    torch.testing.assert_close(y, x + 1)
    return {"ok": True, "output": y}
""")
main = program.get_function(module=module, name="main")
report = program.run(fn=main, args=[256])
program.return_(key="report", value=report)
with Client("http://127.0.0.1:8000") as client:
    result = client.execute(program, timeout_seconds=120)
print(result.results)
```

Uploaded Python can perform tasks such as compilation, tensor allocation,
validation, measurement, or invoking scripts and CLI tools. `run.fn` references
an earlier callable; a callable returned by a `run` can be used by a later call.

API surface:

```python
Program.upload(kind="module", source=...) -> Register
Program.upload(kind="tensor", value=..., dtype=None, shape=None) -> Register
Program.upload(kind="bytes", value=...) -> Register
Program.upload(kind="library", value=...) -> Register
Program.upload_file(blob=..., path=...) -> Register
Program.upload_folder(folder, *, path=...) -> None
Program.get_function(module=..., name=..., cpu_only=False) -> Register
Program.run(fn=..., args=[]) -> Register
Program.return_(key=..., value=...) -> None
Program.return_file(key=..., path=...) -> None    # str or Register resolving to str
Program.return_folder(key=..., path=...) -> None  # str or Register resolving to str

Client(base_url, *, headers=None, connect_timeout_seconds=10)
Client.execute(program, *, timeout_seconds=None, output_limit_bytes=None, gpu_count=None) -> ProgramResult
Client.health() -> dict
Client.target() -> dict   # e.g. {"arch": "sm_100a"}
Client.close() -> None
```

`Program` automatically generates an ID for each value-producing instruction.
Use the optional `id` field to customize it.

The client derives `blob`, `dtype`, and `shape` from a tensor `value`. It sends
no blob parts at first, retries a `CACHE_MISS` with the missing parts, and falls
back to resending every local blob if the cache changes between the two
requests. Returned tensors decode to CPU `numpy.ndarray` (`bfloat16` and
`float8_*` via `ml_dtypes`).

`upload_folder` snapshots a local directory into ordinary file upload
instructions; retries send the same instructions with different blob parts.
It includes hidden files, rejects links, special files and repeated directories,
and omits empty directories and original permissions/timestamps. A failed call
leaves the program unchanged. Caching is automatic and best-effort.

`return_file` and `return_folder` capture workspace-relative paths at that
instruction; `path` may be an earlier register containing a path string.
Use `result[key].save(destination)` to save locally. File results also provide
`read_bytes()`; folder results provide `files` and `directories`. Folders include
hidden files and empty directories. Missing paths, symlinks, and special files
fail collection. Earlier returns survive later ordinary instruction failures.
Saves require an existing parent; file replacement needs `overwrite=True`, and
folder destinations must be new. Transfers are buffered and subject to server
limits. See [client usage](../../../docs/client-guide/writing-a-program.md#return-files-and-folders).

## Instructions

### `upload`

Each field is required for the listed kinds:

| Field | Kinds | Notes |
|---|---|---|
| `id` | all | Unique handle name |
| `kind` | all | `"module"`, `"tensor"`, `"bytes"`, `"file"`, or `"library"` |
| `source` | module | Python source executed to define a module |
| `blob` | tensor, bytes, file, library | SHA-256 of the raw bytes |
| `path` | file | Relative path in the request working directory |
| `dtype` | tensor | Tensor data type |
| `shape` | tensor | Tensor shape |

A module upload binds its full source namespace. Use `get_function` to select a
named Python object. Uploaded Python is ordinary code
executed on the worker (torch included), so a plain function works as a
reference baseline.

A `bytes` upload binds the blob's bytes unchanged. They stay in CPU memory and
can be passed to uploaded Python code, which suits files and other binary
formats the server should parse.

A `library` upload is a prebuilt TVM FFI shared library loaded with
`tvm_ffi.load_module`; use `get_function` to bind one of its exported functions.

A `file` upload binds its normalized relative path as a string. Use
`upload_file(blob=..., path=...)` in the Python client, where `blob` is bytes-like;
the wire field holds its SHA-256. The server copies it into the request's private
working directory with mode `0600` and removes that directory after execution.
Its content can remain in the disk cache across requests and server restarts.

### `get_function`

Select an object from an uploaded module with `get_function(module=..., name=...)`.
The returned `Register` stores its instruction ID and can be passed as `fn` to `run`.

`get_function(..., cpu_only=True)` declares that later calls to this function
use no GPU. Before its `run`, a GPU worker synchronizes and releases the lease.
A detected CUDA call fails with `gpu_access`. This is a best-effort check.
The flag applies to the selected function's calls; the Python module upload
executes top-level code under the GPU lease.

### `run`

`{"op": "run", "id": ..., "fn": {"$ref": "function_id"}, "args": [...]}`
calls an earlier callable. In Python, pass the `Register` returned by
`get_function` (or a `run` that returned a callable); the client encodes its ID
as `{"$ref": "function_id"}`. Top-level argument references resolve to values;
other JSON values pass as literals.

### `return`

`{"op": "return", "key": ..., "value": {"$ref": id}}` — selects a handle for
the response `results` object. A `return` that already ran keeps its entry
even if a later instruction fails, so returning early checkpoints partial
work.

## Multi-GPU programs

Use `Client.execute(program, gpu_count=N)` for 1–8 GPUs on one direct server.
Upload definitions and files, select an entry point with `get_function`, then
execute with `run`. Single- and multi-GPU workers share instruction-level leasing:
`cpu_only` synchronizes and releases the entire set, and later GPU instructions
reacquire the same devices. The script owns communication and must join GPU
subprocesses before returning. Spawn workers need an importable `.py` file.
Do not mark a GPU launcher CPU-only. See the [two single-file examples](../../../docs/client-guide/writing-a-program.md#run-a-multi-gpu-program)
and [execution protocol](../../../docs/client-guide/protocol.md#multi-gpu-execution).

## Compilation and measurement helpers

For convenience, uploaded Python can import `compile_tirx` and `benchmark` from
`kcoral.builtins`. You can also use your own compilation and measurement harness.
See the [API reference](../../../docs/python-api/index.rst#gpu-utilities)
for signatures and options, and the
[compilation tutorial](../../../docs/tutorials/benchmark-kernel.md#choose-where-to-compile)
for GPU-server, CPU-server and local compilation examples.

## Tensors

Two ways to provide tensors are to upload Python that initializes them remotely,
or to upload local tensors with `kind="tensor"`.

- `kind="tensor"` accepts a NumPy array, a torch tensor, any DLPack object,
  or raw bytes with `dtype` and `shape`.
- Accepted dtypes: `bool`, `uint8`, `int8`, `int16`, `int32`, `int64`,
  `float16`, `float32`, `float64`, `bfloat16`, `float8_e4m3fn`,
  `float8_e5m2`.
- To hand a returned `bfloat16` array to torch:
  `torch.from_numpy(value.view(np.uint8)).view(torch.bfloat16)`.

## Correctness and timing

Your uploaded code can configure, for example, tolerances, assertions, profiler
invocation, warmup, and measurement methodology. Return reports and relevant
artifacts explicitly. Synchronize and wait for GPU subprocesses before
returning or releasing a lease.

## Request timing

The execution response contains top-level `queue_ms`, `elapsed_ms`,
`lease_wait_ms`, and `lease_held_ms` fields. The Python client exposes them as
attributes of the `ProgramResult` returned by `Client.execute`, for example
`result.lease_held_ms`.

`queue_ms` measures the wait for a worker. `elapsed_ms` starts once a worker is
assigned and includes waiting for the GPU lease (`lease_wait_ms`), holding it
(`lease_held_ms`), and other worker work. Holding the lease reserves the GPU;
it does not imply continuous GPU activity. Kernel latency is measured separately
by `benchmark` or your own measurement code.

## Outcomes

- `options`: `timeout_seconds` (server defaults: 300, maximum 900),
  `output_limit_bytes` (default 1 MiB per stream). Values above a maximum are
  clamped. `stdout`/`stderr` come back with the response.
- `COMPLETED` — every instruction ran; `results` holds the returned values.
- `FAILED` — one instruction failed and the rest were skipped; returns that
  already ran stay in `results`. `error` carries `kind` (`parse`, `compile`,
  `runtime`, `gpu_access`, `correctness`, `serialization`, `unavailable`,
  `engine`), `message`, `instruction_index`, `instruction_id`, `traceback`.
- `CACHE_MISS` — blobs missing; the Python client retries with the missing
  blobs, then with all local blobs if that retry misses again.
- Exceptions: `KCoralError` (carries `status_code` and `kind`; 503
  means no worker free, 504 means `timeout_seconds` hit), `TransportError`
  (no HTTP response could be obtained; execution may have occurred),
  `ProtocolError` (malformed response or failed cache recovery).

## Non-Python clients

The wire format is `multipart/form-data` with a `program` part
(`application/json`) plus one `blob:<sha256>` part
(`application/octet-stream`) per tensor, bytes, file, or library blob, where
`<sha256>` is the lowercase hex SHA-256 of the part bytes. Blobs are cached by
hash: on `status: CACHE_MISS`, resend the program with the parts listed in
`missing_blobs`, and resend every blob if that retry misses again. Responses
containing tensors or bytes are multipart with a `result` JSON part and
`return:<index>` binary parts. The typed value encoding, blob-cache rules, and
full error table are in `docs/client-guide/protocol.md`.

## References

- `docs/client-guide/protocol.md` — field-level wire specification: request envelope,
  value encoding, library upload build routes (TVM FFI, TIRx
  `export_library`, CuTeDSL `--enable-tvm-ffi`) and their link flags, full
  HTTP error table.
- `docs/client-guide/writing-a-program.md` — narrative guide: server-side compile vs prebuilt
  library trade-offs, measurement guidance.
- `examples/basics/first_program.py` — runnable: upload a function and tensor, then
  execute and return the result.
- `examples/benchmark_kernel/benchmark_kernel.py` — CPU-only TIRx compilation and GPU measurement on one server.
- `examples/cpu_compile/library_upload.py` — client build and library upload.
- `examples/cpu_compile/cpu_compile_gpu_execute.py` — CPU build and GPU execution.