Data services: population and event

A study is B = f(P, E). Two of those four symbols have to come from the real world: P, the fixed population, and the exogenous part of E, the signals the world sends into the simulation while it runs. Neither is produced by the framework. Both are resolved, at build time, by two typed MCP services:

service	resolves	formal name	capability / MCP server id
the population service	P — personas with persistent ids	sv-user-pool	user_pool_survey_mcp
the event service	the exogenous channel of E	sv-event-tool	event_tool

This page is about what the services do and what they leave on disk. Registration, availability checks, confirm/cost metering and the fallback chain live in External Capabilities; the ledger that records what a study actually used lives in Grounding & Provenance.

Why the services sit outside the framework

Three properties follow from keeping them out of the runtime package:

• P and E are independently replaceable. A version can swap the population source without touching the environment, or re-ground the environment without rebuilding the population, because neither is compiled into the engine — each is a bundle on disk produced by a service call.
• One typed interface, two kinds of caller. The same tool surface serves the in-session sv-* skills and any external agent that speaks MCP. There is no private back door the skills use and outsiders do not get.
• A third party can implement a compatible service. The framework depends on the shape of what comes back (records with an agent_id; per-month SourceResult envelopes), not on these particular servers. Point SV_USER_POOL_MCP_URL at your own pool service and nothing in socioverse/ changes.

The services themselves are closed-source and deployed server-side. A local clone reaches them over authenticated HTTP, or degrades — see Using the services from a local clone.

The population service

What is registered

Seven platform ids are registered in the service's config/platforms.yaml, covering five sub-pools (the two internal pools each register a raw and a preprocessed variant):

sub-pool	platform ids	records in the source pool	profile fields	role
internal X/Twitter	x, x_mode_1	973,928	10	anchor; real behavioural text, English prompts
internal RedNote	rednote, rednote_v2	9,158,404	8 (v2)	anchor; Chinese-language context
Nemotron-Personas USA	nemotron_usa	59,994	7	anchor / complement; census-aligned, fully open
MatrAIx Persona 1M	matraix_1m	56,583	20	complement; high-dimensional, restricted terms
PersonaHub	personahub	199,466	0	retrieval and persona-text material only

Each entry declares profile_fields (what you can filter on), numeric_fields, simulation.supported_fields (the subset usable as distributions keys), simulation.prompt_language, and an attribution block. x_mode_1 and rednote_v2 are the preprocessed variants the distribution simulation expects.

The counts above are the size of the registered source pool, not of the SQLite index. How much of each pool a given deployment has indexed — and which platforms are indexed at all — is whatever get_user_pool_index_status reports; the open pools are typically indexed whole, the two large internal pools in slices through their preprocessed variants.

PersonaHub carries no demographic fields

Its profile_fields is empty by design: it is a text-only corpus. It can back search_user_pool and supply persona text, and it cannot back distribution sampling or cross-pool statistical matching. matraix_1m calibrates one-dimensional margins only and declares no licence — both are research/demo tier, and the registry entry says so in usage_restriction.

Ten typed tools, four groups

group	tool	what it returns	key parameters
discovery	list_user_pool_platforms	registered platforms, source file count and size, field config, index state	—
discovery	describe_user_pool_platform	one platform; reads only the first record of the first source file as a schema check	platform_id, include_sample_content
discovery	get_user_pool_field_values	top values of one field, for vocabulary alignment	platform_id, field, use_index, limit ≤ 200
routing	match_user_pool_fields	per-platform coverage for a requested field list, plus a suggestion naming the anchor pool, complement pools per uncovered field, and unresolved_fields	fields, include_values, values_limit, value_probes
retrieval	search_user_pool	personas matching text, exact profile filters and numeric ranges	platform_id, query, profile_filters, numeric_filters, limit ≤ 100, offset, scan bounds
retrieval	get_user_profile	one persona by user_id	platform_id, user_id, include_sample_content
simulation	get_user_pool_simulation_capabilities	which platforms are sampling-ready, their supported fields, prompt language, server-side model, auth mode	—
simulation	run_user_pool_distribution_simulation	sampled respondents + one LLM questionnaire answer each, as raw JSONL	platform_id, distributions, user_count (1–100,000), questionnaire, seed, max_workers ≤ 128, allow_virtual_users, sampling_mode
index	get_user_pool_index_status	index presence, SQLite size, FTS availability, per-platform indexed counts	—
index	build_user_pool_index	build or refresh the SQLite index	platform_id, rebuild, max_records (default 50,000; null = uncapped), batch_size, include_full_content

Eight of the ten are annotated read-only. run_user_pool_distribution_simulation is the metered one — it spends LLM budget in proportion to user_count — and build_user_pool_index writes the index. match_user_pool_fields is read-only and unmetered, which is why routing comes before anything else.

Everything real runs off the SQLite index

Distribution sampling requires the index and never pre-reads the source JSON: on an unindexed platform it does not degrade, it returns status: "not_indexed" and asks you to build the index first. The lookup tools do degrade — search_user_pool, get_user_profile and get_user_pool_field_values fall back to a bounded scan capped by max_scan_records / max_scan_files / max_scan_bytes, and the response is flagged partial: true. A partial answer is a statement about the scan, not about the pool.

The standard flow, in six steps

1. Declare the target marginals — from census tables, survey microdata, or domain knowledge.
2. Route to the best-matching pool or pools with match_user_pool_fields. Probe the exact category labels with value_probes: label vocabularies differ across pools and sampling matches strings exactly, so a misspelled label silently degrades a whole cell to VIRTUAL_*.
3. Sample from the anchor pool and complement the dimensions it lacks from secondary pools by statistical matching on the keys they share (e.g. query the complement pool filtered on the anchor persona's AGE × GENDER).
4. Align by IPF — sampling_mode="preserve_joint" reweights the pool's observed joint distribution onto the declared marginals, and reports achieved_marginals, unsupported_targets, joint_support and ipf_iterations. The default, sampling_mode="independent", draws each field independently and does not preserve cross-field correlation.
5. Synthesize what no pool covers, conditioned on each persona's existing demographics, and record where every attribute came from.
6. Assemble the persona: demographic tags, a natural-language profile, and the study's domain-specific initial state.

allow_virtual_users defaults to true

When a requested distribution cell has no indexed match, the service fills it with clearly labelled VIRTUAL_* respondents so the delivered margins still match what you asked for. Pass allow_virtual_users=false to get the gap reported instead of filled, and read the returned sample either way. A coverage hole is a finding about your population.

Determinism, and why the panel key survives a version

P is materialized once, at initialization, not lazily during the run. /sv-build-population calls the same population.build(seed) the engine would call at t = 0, under the study's declared seed, and writes the result out. Repeating the study reproduces the same set of agents, so agent_id stays a stable primary key for the panel across versions and branches — which is what makes two versions comparable agent by agent rather than only in aggregate.

Researcher-supplied populations are first-class on the same terms: a study can draw P from an uploaded file or from a study-specific builder, and the materialize-once-under-a-seed rule is unchanged.

provider_ref	class	provider_args
file.personas	FilePopulationProvider	{"path": …, "format"?: "parquet"} — CSV/Parquet, must have an agent_id column; group_key / weight optional, the rest become attributes
mcp.socioverse_pool	McpPopulationProvider	{"server": <name from resources.json>, "query": …} — needs a fetch_fn injected by the build skill that performs the tool call

Both live in socioverse/providers.py and yield Persona objects (socioverse/schemas/population.py).

What lands on disk

path	contents
studies/<id>/population/population.json	PopulationBundle — study_id, provider_ref, provider_args, interaction, propagation, materialized_count, and personas inline when the pool is modest (≲1000)
studies/<id>/population/roster.jsonl	the materialized t = 0 roster, one agent per line; the authoritative list for large pools and what the dashboard reads
studies/<id>/resources.json	ResourceManifest.mcp_servers pins the service by name (McpServerDecl). The declaration also has a url field, but the build skills fill it only for a genuinely public endpoint and otherwise leave the address in .env / .mcp.json; the key is never written to a study artifact
studies/<id>/grounding/grounding.json	one fact per load-bearing distribution, source.via = "provider", plus the pool's attribution block

The event service

The catalog

21 collectors, each an independent class behind one interface (collectors/base.py), registered in COLLECTOR_REGISTRY. Ten are default-enabled; sources="default" selects exactly those ten, sources="all" selects all 21.

category (code)	sources	default
macro_market	fred, bls, treasury, nyfed_sce, yahoo_market, alt_fng, credit_spreads, housing_finance, ustr, nyt_archive	all ten enabled
population	census_acs, cps_basic, cps_asec, sipp	off
optional	world_bank, google_trends, gdelt, coingecko, stooq_market, news_alt, household_pulse	off

Sub-series filtering (series="UNRATE,CPIAUCSL") is implemented by fred, bls, credit_spreads, housing_finance and world_bank; other collectors ignore the argument and return their full payload. world_bank is the non-U.S. hook: annual GDP growth, CPI inflation, unemployment, population and GDP per capita for six economies (USA, CHN, JPN, IND, DEU, BRA) plus the world aggregate, CC-BY-4.0, attribution carried in the result's metadata.

Adding a source is one collector class plus one registry line. Nothing else in the platform changes.

Three retrieval tools, plus a search route

tool	signature	notes
list_sources()	—	name, category, description, requires_api_key, api_key_env_var, default_enabled for all 21
get_data(sources, year, month, use_cache=True)	sources = comma-separated names, "default", or "all"	a failing source becomes an entry in errors, never an exception
get_source_detail(source, year, month, series="", use_cache=True)	one source, optional sub-series filter	the call the SocioVerse2 client (socioverse.external_events) uses when sources= is given
search_web(query, max_results=8, days=7, lang="")	provider chain, first available wins: Tavily → Google News RSS → GDELT DOC	live search, not a vintage query

search_web exists because the model-side web search is not always reachable; it routes through keys the deployment controls, the response names the provider actually used, and results are cited with via="event_service" in the grounding ledger. It is not part of the point-in-time path: it answers "what is being said now", not "what was published by month m". ExternalEventsClient never calls it.

The standard flow, in six steps

1. Declare the horizon as a list of "YYYY-MM" and pick the sources.
2. List the catalog to confirm each selected source's coverage and fields.
3. Fetch month by month, so a month's payload holds only what had been published by then.
4. Stamp provenance onto every payload — source, series, vintage.
5. Materialize the stamped payloads into the study workspace as an external-events bundle.
6. Translate the bundle into the two channels of E: broadcasts on the information layer and scheduled events on the physical layer, each bound to the step of its month.

Steps 2–4 happen inside the service; steps 5–6 inside /sv-build-environment.

Resolution happens at build time. The runtime fetches nothing.

The loop in socioverse/engine/loop.py reads the workspace and never reaches a service. That is why a run is self-contained and reproducible, and why a branch — which replays its parent's steps up to its fork_step — is comparable with its parent step for step instead of against a differently-resolved snapshot of the world.

The client: socioverse/external_events.py

from socioverse import materialize_external_events, month_range

path, ev = materialize_external_events(
    study_dir,
    query="US inflation and consumer sentiment, 2024",   # provenance only
    months=month_range("2024-01", "2024-12"),            # REQUIRED
    sources=["fred", "nyt_archive"],
)

month_range(start, end) builds an inclusive "YYYY-MM" list and stops once the list passes 48 entries (49 at most), so a horizon longer than four years has to be fetched in pieces. materialize_external_events(...) writes environment/external_events.json and returns (path, bundle).

ExternalEventsClient walks a three-tier chain and never raises: remote (the deployed service, SV_EVENT_API_URL + SV_EVENT_API_KEY, health-probed at <scheme>://<host>/healthz before any fetching) → local_cache (SV_EVENT_LOCAL_CACHE, laid out <source>/<YYYY-MM>.json) → unavailable. The timeout is per source-month: with sources= given, the remote tier calls get_source_detail once per source per month, so one slow upstream loses only its own key instead of stalling the batch. Without sources= it falls back to one get_data(sources="default") call per month.

months= is mandatory on the remote tier

The deployed service has no query parser — the agent layer was deliberately dropped so that no uncontrolled LLM sits in the data path to E. Without an explicit months=, _fetch_remote raises right after the health probe and before any tool call; fetch() records that as a remote-tier error and drops to the cache tier, so you see it as a missing bundle, not an exception. query is carried for provenance only.

The bundle

ExternalEventsBundle fields, as they appear in environment/external_events.json:

field	contents
query, requested_months, requested_sources	what was asked for
provider	"remote" | "local_cache" | "unavailable"; .available is the property provider != "unavailable"
endpoint, fetched_at	which service answered, and when
results	the payloads, keyed "<source>_<YYYY-MM>"
availability	sources_used, sources_unavailable, and source_status per key
errors	per-tier and per-source-month failure records
summary, web_evidence	legacy fields — never author from these

One real entry, abridged, from a study that pulled fred across 2024–2026 (the stored entry also carries description, fetched_at, error_message, and all thirteen FRED series rather than the four shown):

"fred_2024-01": {
  "source_name": "fred", "category": "macro_market",
  "target_date": "2024-01", "status": "ok", "from_cache": true,
  "data": {
    "snapshot": {"UNRATE": 3.7, "CPIAUCSL": 309.698, "UMCSENT": 79.0, "SP500": 4845.65},
    "yoy":      {"UNRATE": 5.71, "CPIAUCSL": 3.09,   "UMCSENT": 21.73, "SP500": 18.86}
  },
  "metadata": {}
}

Author broadcasts from ev.results, never from ev.summary

summary and web_evidence carry another model's prose. Injecting that text into E means the behaviour you observe is partly a response to a third-party LLM's framing of the world rather than to the world, which contaminates the attribution in B = f(P, E). Read the typed numbers out of results and write the broadcast sentence yourself.

Translating a bundle into E

The bundle is evidence, not environment. Step 6 turns it into two typed object kinds from socioverse/schemas/environment.py, both bound to a step:

channel	class	fields	fires
information	Broadcast	message_id, content, channel, at_step, ttl, audience ("all" → macro, selector dict → local), note	visible for ttl steps from at_step
physical	ScheduledEvent	at_step, target_layer, op (set/add/multiply/add_pct/replace_source), selector, property_name, value, note	at at_step, in the exogenous-update stage, before anyone observes

Both are interventions in the vocabulary of this documentation, and both are validated when the bundle is written: a ScheduledEvent whose target_layer is not a declared layer is rejected, and broadcasts without any modality="information" layer are rejected.

path	contents
studies/<id>/environment/external_events.json	the bundle above — the evidence
studies/<id>/environment/environment.json	EnvironmentBundle — study_id, layers, scheduled_events, information_program.broadcasts, provider_ref, provider_args
studies/<id>/trajectory/study.duckdb	table events(step INTEGER, note VARCHAR) — the interventions that actually fired during the run

Point-in-time retrieval

Every retrieval tool is parameterized by (year, month), and each collector truncates its observation window at the last day of that month — fred, for instance, requests observation_end = date(year, month, last_day) and computes its year-over-year figures against the same cut. The payload for 2024-01 therefore holds no observation dated after January 2024.

Payloads are archived by vintage: FileCache writes <cache_dir>/<source>/<YYYY-MM>.json, the same layout the SV_EVENT_LOCAL_CACHE tier of socioverse.external_events reads, so an offline rebuild of a study reproduces the month-by-month sequence rather than today's numbers.

Any backtest or nowcast depends on this. If the simulation at step t can see a figure published after the real-world date t stands for, the result is a measure of the leak, not of the model. Two consequences worth stating:

• A version that extends the horizon fetches the new months under the same as-of rule; it does not re-resolve the old ones against today.
• An intervention added later — a broadcast declared for month m in a branch — is grounded by the same as-of query as its parent, so the branch is held to the same information discipline as the run it is compared against.

As-of by observation date

The guarantee the collectors implement is on the observation window: nothing dated after the requested month enters the payload. Whether a figure was later revised depends on the upstream API's own revision policy. Record the vintage you used (target_date, fetched_at, from_cache) and the result stays auditable either way.

Governance

Privacy. The population service exposes demographic tags and statistical profiles. Raw platform content and personally identifying information are not part of the tool surface — the index stores truncated sample_content previews rather than full text, and even include_sample_content=true is bounded by max_scan_bytes. All access passes through the typed interface.

Attribution and usage tier. Every registered pool and every signal source carries an attribution block. Open pools declare project, author, license, url and citation; restricted ones additionally declare usage_restriction and caveats — personahub is CC-BY-NC-SA-4.0 and share-alike, so it is research and demonstration only; matraix_1m declares no licence on its dataset card and is treated the same way, with an explicit per-study confirmation before any paid or published run. The attribution block rides back with every simulation response, and the build skill copies it into grounding/grounding.json.

Reports carry a data basis. /sv-report renders the grounding ledger as the report's data-basis and references section, so the pools and signal vintages a result rests on are printed next to the result.

Auditability. Each stage writes one validated artifact; resources.json declares the services, datasets and tools a study depends on; versions.json records every edit; the events table records every intervention that fired. A conclusion can be traced back through the trajectory store to the environment bundle, the population bundle and the intervention schedule that produced it, and through the resource manifest to the pools and vintages behind those.

End to end: US consumer confidence, 2024

1 — Declare the horizon and the sources. Twelve monthly steps, one step per month, fred for the indicator series and nyt_archive for the news framing.

2 — Check the catalog. list_sources() confirms both are registered and that fred needs FRED_API_KEY on the server side; fred also supports sub-series filtering, so UMCSENT and CPIAUCSL can be pulled alone.

3–5 — Fetch, stamp, materialize. One call does all three:

path, ev = materialize_external_events(
    study_dir,
    query="US consumer confidence 2024",
    months=month_range("2024-01", "2024-12"),
    sources=["fred", "nyt_archive"],
)
assert ev.available                      # else: local cache, or the manual path
ev.availability["sources_used"]          # ['fred', 'nyt_archive']

environment/external_events.json now holds 24 keys, fred_2024-01 through nyt_archive_2024-12.

6 — Translate into E. Read the typed numbers and write one broadcast per month, bound to that month's step:

Broadcast(
    message_id="macro_2024_01",
    content="Consumer sentiment reads 79.0 this month, up 21.7% from a year ago. "
            "Headline CPI is 3.1% year over year and mortgage rates sit at 6.69%.",
    channel="news", at_step=0, ttl=1, audience="all",
    note="fred_2024-01: UMCSENT/CPIAUCSL yoy/MORTGAGE30US",
)

Twelve of those go into information_program.broadcasts, the environment bundle declares one modality="information" layer to carry them, and each load-bearing number gets a grounding fact with via="event_service" and its as_of month. Nothing else is needed at run time: at step 0 the exogenous-update stage makes macro_2024_01 visible, every agent observes the same state, and the panel row for that step records what each persona did about it.

→ The shipped study that follows this shape is Consumer Confidence.

Using the services from a local clone

Both services are deployed server-side and are not part of the open-source package. A clone reaches them by declaration, not by vendoring:

# .env  (gitignored — the single source)
SV_EVENT_API_URL=http://your-event-host:9997/event_mcp     # FULL MCP endpoint, path included
SV_EVENT_API_KEY=…
SV_USER_POOL_MCP_URL=http://your-pool-host:9998/user_pool_mcp
SV_USER_POOL_MCP_KEY=…                                      # sent as Authorization: Bearer

python scripts/gen_mcp_json.py      # renders .mcp.json from .env — never hand-edit it

Without credentials, nothing breaks and nothing is silently faked:

• the event client falls through to SV_EVENT_LOCAL_CACHE, and then to provider="unavailable", at which point the build stage searches the web by hand and records each number with via="web_search" plus url and accessed date;
• the population stage falls back to an uploaded file (provider_ref="file.personas") or to grounded synthesis, and notes in the provenance that the pool was unavailable.

On the hosted service both are preconfigured and metered against your quota.

See also

• External Capabilities — the registry entry format, confirm vs cost, and the full fallback chain
• Grounding & Provenance — how the facts these services produce are recorded
• socioverse.external_events reference — the generated API docs
• Quotas & billing — what a metered call costs on the hosted service