Skip to content

LLM Clients & Dry Runs

In SocioVerse the LLM sits in exactly one place: the decision model. decide_batch(observations, memories) is where B = f(P, E) gets computed, and whether that computation is a rule, a statistical model, or LLM calls is the study's choice. Everything else — environment, population, store, reports — runs identically either way.

Configuration

LLM-driven decision models read their settings from the environment or from the gitignored .env (see Installation for the lookup order). Real-LLM runs need the llm extra (pip install -e ".[llm]"), which brings the OpenAI-compatible client.

variable meaning
SV_LLM_API_KEY the key for the simulated agents' calls (OPENAI_API_KEY honored as fallback)
SV_LLM_BASE_URL optional endpoint override — any OpenAI-compatible gateway (unset: the official OpenAI API)
SV_LLM_MODEL optional; read by scripts/demo_offline.py, the Chicago adapter and studies whose simulation.json names no model. The shipped studies pin decision_args.model 
cp .env.example .env   # fill in your key; .env is gitignored

These variables configure the simulation's LLM only. The coding agent that runs the /sv-* skills (Claude Code, or the experimental Codex CLI path) uses its own login — see what powers the workflow skills.

The call pattern

The engine hands decide_batch the whole step — every agent's observation — and expects one Action per agent back. "Batch" describes that interface, not how the LLM is called. A well-behaved LLM decision model:

  • renders each observation's 4-quadrant view into the prompt (Observation.rendered is the conventional slot for the prepared text);
  • conditions on the agent's AgentMemory (a rolling window of its own past actions — "I already moved twice, I'll stay");
  • in a from-scratch study, calls the LLM once per agent, sending the step's calls concurrently on a bounded thread pool (max_workers in decision_args), and parses — or falls back to the rule — per agent (see studies/opinion_diffusion/model.py). It never asks for one JSON reply covering the whole population: long structured replies get truncated, and one bad reply fails the step;
  • in a legacy adapter, keeps the wrapped engine's own call pattern (for example the Chicago model's archetype-grouped calls), so its validated dynamics are unchanged;
  • returns typed Actions that apply() can fold back into E.

Deterministic dry runs

You should be able to prove the plumbing — loop, artifacts, store, metrics — without spending a token.

The general path: llm_kind: "scripted". Every from-scratch template registers one decision model with two modes, switched in simulation/simulation.json without a code change:

"decision_args": { "llm_kind": "scripted" }

"scripted" runs the study's deterministic rule (free, exactly reproducible from the seed); "openai" role-plays each agent through the SV_LLM_* channel and falls back, for that agent only, when a reply does not parse (to the rule in opinion_diffusion; campus_dining_choice keeps the agent's previous choice). scripts/demo_offline.py runs opinion_diffusion this way, and /sv-run offers the scripted mode as the free dry run. A new study built by /sv-build-model follows the same pattern, so the rule always stays available as the fallback path.

Rule-based decision models need nothing at all: the decision is a pure function of the observation.

Wrapped legacy engines: a stand-in client. A study that wraps a legacy simulator keeps that engine's LLM client, so it dry-runs by injecting a deterministic stand-in instead. The shipped instance is chicago_schelling's DeterministicLLMClient (studies/chicago_schelling/adapter/fake_llm.py), passed as llm_client=DeterministicLLMClient() to build_chicago_simulator. It exists only inside that adapter; it is not a general client for other studies. Parity tests use it: the wrapped study asserts it reproduces the original trajectory on the same seed, with the stand-in removing the only source of nondeterminism.

Reproducibility, honestly stated

Live LLM runs are not reproducible

Real LLM calls sample at nonzero temperature: two live runs of the same study will differ. The reproducible objects are (a) dry runs — seeded and deterministic end to end, and (b) any recorded run — its panel store is permanent, and branch replay replays stored actions exactly rather than re-asking the LLM.

Cost control

  • /sv-run shows the run configuration and asks for confirmation before spending — the spend gate.
  • Prefer dry runs while you iterate on artifacts; go live once per meaningful configuration.
  • Branch replay inherits the parent version's steps before the fork step (no LLM), so a counterfactual pays only for the steps after it.