A local inference server that keeps several models resident side by side under a fixed memory quota. It is a library, not a daemon — nothing starts a server, opens a port, or writes a config file.
links:
You import it, declare a memory budget, and name the models you want to call. Vramen holds each one in its own process for as long as the budget allows, hands it to callers a lease at a time, and evicts the least recently used idle model when something else needs the room.
from vramen import CausalModel, EncoderModel, InferenceModelResourceManager
from vramen import qwen_chat_prompt
manager = InferenceModelResourceManager(quota_gb=24.0)
writer = CausalModel(
"Qwen/Qwen3-8B", qwen_chat_prompt, manager, mem_required_gb=17.0
)
embedder = EncoderModel(
"Qwen/Qwen3-Embedding-0.6B", manager, mem_required_gb=2.0
)
answer = writer.complete("You are terse.", "Name three seabirds.", max_new_tokens=64)
vectors = embedder.encode(["a passage of a story", "another passage"])Both models load on first use and stay loaded.
uv add vramen # or: pip install vramenRequires Python 3.12 or newer, and pulls in torch, transformers, accelerate and tqdm. Image generation is an extra — uv add 'vramen[image]' — which adds diffusers and Pillow.
A model is loaded inside a child process and answers requests over a pair of queues. When it is evicted the process exits, which is the only way to be certain the weights and the allocator’s arenas are actually gone. A model that crashes takes its own process down and raises ModelNotAvailable to the caller, not the host.
You tell each model how much room it needs, and the manager admits models while the declared sizes fit inside the quota. It does not measure a model and then decide. That is what makes admission predictable: the same set of models always fits, or always does not, regardless of what the allocator happened to do last time.
A call holds a lease on its model for its duration. A model under lease cannot be evicted, so a swap waits for the generation in flight rather than killing it. Idle models are the only eviction candidates, and they go oldest first.
Causal, encoder, seq2seq and text-to-image models sit under the same quota — any transformers checkpoint and any diffusers pipeline. Runs on Apple Silicon via MPS, and on Linux with CUDA.