Experiments and logs¶
An Experiment runs a fresh simulation for every condition × seed:
from simsim import Experiment, grid, NoMemory, FullHistory
def build(cond, seed): # returns a new Simulation
llm = make_llm(cond["model"], seed=seed)
agent = Agent("assistant", llm, memory=cond["memory"]())
return Simulation(TaskSequence(tasks, tools=make_tools), [agent], seed=seed)
exp = Experiment(build,
grid(model=["openai:gpt-4o-mini", "vllm:Qwen/Qwen3.5-9B@http://node:8000/v1"],
memory={"none": NoMemory, "full": FullHistory}),
seeds=range(5), out="runs/memory_v1")
rows = exp.run(workers=8)
grid crosses factors into named conditions. A list's values name themselves
(the part after the last /); a dict maps short names to values. The condition
names (model=gpt-4o-mini,memory=none) become one column per factor in the
analysis tables.
Create LLM objects inside build so token and cost counts are per run.
Output files¶
runs/memory_v1/
conditions.json the conditions and their parameters
results.jsonl one row per run: condition, seed, steps, termination,
metrics, probes, usage (tokens, cost, cache hits)
model=gpt-4o-mini,memory=none/
seed0.jsonl the full log of one run
Each line of a run log is one event: start (agents, personas, models, memory and
component types, seed), llm_call (the exact messages sent and the response,
reasoning, tool calls, usage), tool_call, turn (observation, public text,
private fields, components, errors), repair, probe, survey,
intervention, evaluation, and end (metrics, usage, memory summary).
Everything a reviewer could ask about a run is in its log.
Resuming and failures¶
Running the same experiment again skips runs already in results.jsonl, so an
interrupted job continues where it stopped. A run that raises is recorded with an
error field and its traceback in the log, and is retried next time; the other
runs are unaffected.
Parallelism¶
workers runs that many simulations at once in threads (model calls are
network-bound). For a local vLLM server, 8–32 workers keep the GPU busy; for APIs,
set rpm/max_concurrency on the model as well.
Costs¶
Pass price=(in, out) (dollars per million tokens) to make_llm and runs report
cost_usd; or compute it later with simsim.analysis.cost_report. Use
CachedLLM to rerun analyses without paying again.