Engine

Engine loads a checkpoint in the current process and answers requests without a server.

from flide import Engine
from flide.api.schema import SystemOneRequest

engine = Engine("Qwen/Qwen3.5-0.8B", backbone_options={"cuda_graphs": True})
request = SystemOneRequest.model_validate({
    "state": "Order #1182 arrived with a cracked screen. The customer asks for a refund.",
    "questions": {
        "action": {"type": "choice", "instructions": "What should support do next?",
                   "criteria": {"refund": "refund the order", "replace": "send a replacement",
                                "escalate": "ask a human"}},
        "urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"},
    },
})
response, meta = engine.decide(request)
print(response.model_dump()["answers"])
print(meta["path"])

decide returns the response the server would send and the server's own record of the request. meta["path"] is graph when the request ran in a CUDA graph, as it does under flide serve. To answer a file of requests, use flide run, which keeps the GPU busy with packed micro-batches.

Arguments

Argument	Default	What it does
model	required	a Hub repository id or a local directory
format	from the checkpoint	the model family
device	"cuda:0"	the GPU to run on
dtype	"bfloat16"	the dtype of the weights and activations
backbone_options	{}	{"cuda_graphs": True} captures CUDA graphs as flide serve does; {"quantize": "fp8"} loads Qwen3.5 in FP8
session_cache_gb	0	GPU memory for the states of earlier requests
batch_invariant	False	the same bits for a request whatever it is batched with

Any other keyword goes to the family, such as style="sglang" for a chat model or calibration= for OneJev.

CUDA graphs are off unless backbone_options turns them on. Without them every request runs eagerly and more slowly.