Open models¶
Any server that speaks the OpenAI chat API works: vLLM, SGLang, Ollama, LM Studio, llama.cpp's server. vLLM is the usual choice on a GPU cluster.
Serving with vLLM¶
vllm serve Qwen/Qwen3.5-9B --port 8000 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser qwen3 --max-model-len 32768
llm = make_llm("vllm:Qwen/Qwen3.5-9B@http://gpu012:8000/v1",
extra_body={"chat_template_kwargs": {"enable_thinking": False}})
Then check everything works with simsim smoke vllm:...@http://gpu012:8000/v1 --no-thinking.
Tool-call parsers¶
Native tool calling needs the parser that matches the model family's chat template:
| Model family | --tool-call-parser |
|---|---|
Qwen3.5 / Qwen3-Coder (XML <function=...> calls) |
qwen3_coder |
Qwen2.5, Qwen3 (<tool_call> JSON) |
hermes |
| Llama 3.1–3.3 | llama3_json |
| Mistral / Ministral | mistral |
If a model or server has no tool calling, use Agent(..., action_format="react").
Thinking models¶
--reasoning-parser qwen3 (or the parser for your model) makes the reasoning
arrive separately (turn.reasoning). Either give thinking enough max_tokens, or
turn it off per request with
extra_body={"chat_template_kwargs": {"enable_thinking": False}}. Surveys need it
off.
Embedding models¶
vllm serve BAAI/bge-small-en-v1.5 --port 8002 --runner pooling
from simsim import OpenAIEmbedder
embedder = OpenAIEmbedder("BAAI/bge-small-en-v1.5", base_url="http://gpu012:8002/v1")
On a Slurm cluster¶
scripts/serve_vllm.slurm in the repository serves a model as a batch job:
mkdir -p logs
VLLM=/path/to/venv/bin/vllm CACHE=/scratch/$USER/cache TOOL_PARSER=qwen3_coder \
EXTRA_ARGS="--reasoning-parser qwen3" sbatch scripts/serve_vllm.slurm Qwen/Qwen3.5-9B
grep Serving logs/vllm-*.out # -> http://gpuXXX:8000/v1
Adjust the #SBATCH lines to your cluster. Things that commonly go wrong:
- MIG GPU slices: Slurm sets
CUDA_VISIBLE_DEVICESto a MIG UUID, which vLLM cannot parse. The script remaps it to0(the job only sees its slice). - Caches in your home directory fill small quotas; the script points Hugging
Face, vLLM, Triton and Torch caches at
CACHE. - Offline compute nodes: download weights on a login node first, then set
HF_HUB_OFFLINE=1. Vision-language checkpoints (such as Qwen3.5) also need theirpreprocessor_config.jsonfiles even for text-only use. - Rate-limited downloads from shared cluster IPs: set
HF_TOKEN. - Newer architectures may need a newer
transformersthan your vLLM environment has (the error names an unknownmodel_type).
Concurrency¶
vLLM batches concurrent requests, so throughput grows with the number of agents
acting at once. Use Simulation(simultaneous=True, max_concurrency=32) for
populations and Experiment.run(workers=8..32) across runs. On one 40 GB A100
slice, a 4B model ran 50 simultaneous agents for 150 turns in about 40 seconds.