Reporting in papers¶
LLM simulations are sensitive to small choices (recent work finds the base model often matters more than any other design decision), and reviewers increasingly ask for the details. SimSim records all of them; this page lists what to report and where to find it.
Checklist¶
| Report | Where SimSim keeps it |
|---|---|
| Models, exact versions or checkpoints, and how they were served | start event (llm per agent), conditions.json |
| Sampling: temperature, max tokens, seeds | start event; seeds in results.jsonl |
| Number of runs per condition | results.jsonl (one row per run) |
| Agent prompts: persona, instructions, the template | llm_call events contain the exact messages |
| Memory and components used | start event (memory, components) |
| Environment rules, turn order, stopping rule | your code/config; termination per run |
| Measures: probe and survey wording, judge model and rubric | probe/survey events, the config |
| Failure rates: schema repairs, unparsed replies, crashed runs | repair events, turn.error, error rows |
| Token use and cost | usage in results.jsonl; cost_report |
| Validation of judges or simulated behavior against humans | your own study |
Reproducibility¶
- Pin SimSim:
pip install "simsim @ git+https://github.com/chahatraj/SimSim@v0.7.1". - Pin models: dated API snapshots (
gpt-4o-mini-2024-07-18) or a specific open checkpoint revision. - Seeds: with
seedset, each request gets a deterministic seed derived from the run seed and the prompt. API providers do not guarantee determinism even then, and vLLM batching can change results slightly. - Cache: wrap models in
CachedLLMand share the cache file; anyone rerunning your code with the same cache replays your exact runs. - Release logs:
results.jsonlplus the run logs are the full record; the viewer (simsim view) is an easy supplementary artifact.
Validity¶
Simulation results describe the models, not people, unless validated. Common practice:
- vary the base model (more than one family) and report all of them;
- vary prompts and persona wording, and check that effects survive;
- compare against human data where a ground truth exists (survey marginals, behavioral experiments);
- check that measures are not artifacts: e.g. surveys with probabilities rather than one sampled answer, judges validated on a human-rated sample.
Citing SimSim¶
The repository has a CITATION.cff; GitHub's "Cite this repository" button
gives BibTeX.