flide serve¶
flide serve loads a checkpoint and serves it on the /v1/systemone API.
flide serve --model Qwen/Qwen3.5-4B --host 127.0.0.1 --port 8000
--model takes a Hugging Face repository id, downloaded on first use, or a local directory.
Model¶
| Option | Default | What it does |
|---|---|---|
--model M |
required | a Hub repository id or a local directory |
--format F |
from the checkpoint | the model family; flide formats lists them |
--format-arg KEY=VALUE |
an option of the family, listed on its model page; repeat for more | |
--name N |
from --model |
the model name the API reports |
--calibration FILE |
next to the weights | OneJev's calibration file |
--device D |
cuda:0 |
the GPU to run on |
--dtype T |
bfloat16 |
the dtype of the weights and activations |
--quantize fp8 |
off | Qwen3.5 decoder weights in FP8, on GPUs of compute capability 8.9 or newer; the numbers move a little |
--media-root DIR |
none | the directory local media paths resolve against; without it, only URLs and base64 |
Limits¶
| Option | Default | What it does |
|---|---|---|
--max-branch-tokens N |
32768 | longest state plus question, in tokens |
--max-request-tokens N |
65536 | longest request, the state once plus every question, in tokens |
A request over a limit gets a 422 that says how long it was.
Speed and memory¶
| Option | Default | What it does |
|---|---|---|
--session-cache-gb G |
0 | GPU memory for the states of earlier requests; a request that extends an earlier one resumes from it |
--session-min-tokens N |
2048 | the shortest state the session cache keeps |
--batch-invariant |
off | the same bits for a request whatever it is batched with, at some cost in speed |
--no-cuda-graphs |
off | runs every request eagerly |
--planners N |
4 | CPU threads that render and tokenize requests |
--frontends N |
0 | HTTP front-end processes, each with its own planners; 0 serves from the GPU process |
--batch-window-ms MS |
0 | how long a GPU round waits for more requests after the first one arrives |
--gather-ms MS |
2 | how long a GPU round waits for requests that are still being planned |
Network¶
| Option | Default | What it does |
|---|---|---|
--host H |
0.0.0.0 |
the interface to listen on; 127.0.0.1 keeps the server local |
--port P |
8000 | the port to listen on |
--access-log |
off | logs every request |
Agents and the session cache¶
An agent's requests grow. Each step sends the trajectory so far, which repeats the previous step's state and adds a
little. With --session-cache-gb, the server keeps the states of earlier requests in that much GPU memory, and a
request whose state starts with a saved one reads only the new part.
flide serve --model OmniJev/OneJev1.6-0.8B --session-cache-gb 40
POST /flush_cache empties the cache, for example between benchmark runs.