Telemetry catalog

Every event type published by quorum-rs::telemetry.

Subject layout

telemetry.agent.{agent_id}.{event_type}

The agent_id is bound by the agent's telemetry JWT, which grants publish permission only for telemetry.agent.{agent_id}.>.

Multi-endpoint deployments. Agents publish to one or more destinations in parallel via [TelemetryEmitterMux]. Each endpoint owns its own NATS connection, credentials, and subject prefix. mux.emit(&event) fans out to every endpoint; mux.emit_for("name", &event) targets one. Endpoint names must be unique within the list — TelemetryEmitterMux::new returns TelemetryMuxError::DuplicateNames on collision.

Per-endpoint subject_prefix defaults to telemetry.agent when omitted; setting it explicitly takes precedence (e.g. tenant.x.telemetry for tenanted deployments). The telemetry JWT is bound to the prefix the agent uses, so any custom prefix must be reflected in the JWT permissions in lockstep.

Config schema

telemetry:
  enabled: true
  endpoints:
    - name: service        # service operator's NATS
      nats_url: nats://orchestrator.example.com:4222
      creds: /etc/nsed/agent-service.creds
      # subject_prefix omitted → defaults to telemetry.agent
    - name: own            # agent operator's own dashboard
      nats_url: nats://my-grafana-stack.local:4222
      creds: /etc/nsed/agent-own.creds
      subject_prefix: tenant.x.telemetry  # explicit override

An empty endpoints list (or block omitted) with enabled: true is allowed and results in no telemetry emitted — the mux falls through silently. Operators relying on telemetry should declare at least one endpoint.

Overlay merges replace the list wholesale — env-specific overlays must restate every endpoint, not just the additions.

Common fields

Every agent-side event carries these fields:

Every event also carries a serde "type" discriminant (e.g. "type": "task_completed").

Events

llm_request_start

Fired when the agent dispatches a call to its LLM provider.

Not sent: message text, system prompt, tool schemas.

llm_request_complete

Fired when the LLM response finishes.

Not sent: response content, tool call arguments, reasoning text.

llm_request_failed

Fired on any call that did not return a well-formed response.

Not sent: error body text, stack trace, raw provider response.

llm_request_stalled

Fired every 30s while an in-flight llm_request_start has no terminal event. Implemented by LlmRequestSpan: start() spawns a heartbeat task; complete() / fail() notify cancellation before emitting the terminal event so no llm_request_stalled is published after the matching terminal. First emission is at +30s (not +0s) — a request that completes inside the first interval emits zero stalled events.

api_error

Fired by the agent's status-server middleware on every HTTP response with status >= 400. Mirrors the orchestrator's api_error event but omits operator_principal — the agent dashboard is loopback-bound and not authenticated as an operator.

tool_call_executed

Fired when the agent runs an internal tool.

Not sent: tool arguments, tool output, search queries.

deliberation_context_assembled

Fired once per propose / evaluate call, recording what prior-round context the agent assembled into its prompt and whether it wrote its scratchpad. Lets a dashboard confirm a serving agent actually inspects its own past proposals/evals across rounds — the same signals quorum smoke-test prints in-process.

retry_loop_attempt

Fired each time the structured-output retry loop iterates.

Not sent: the offending text, the parse error detail.

task_accepted / task_completed / task_failed

Task-level bookends.

Not sent: what the agent proposed.

Population status (current implementation)

The four fields below are typed Option<u64>/Option<u32> and the worker emits null (omitted from the JSON via #[serde(skip_serializing_if = "Option::is_none")]) until the wiring to populate them is in place. Operators should treat absent fields as "not yet measured", not as "measured zero" — those are different signals.

The schema slot exists today so consumers can lock onto the contract; once population lands the values become Some(n) without a wire-shape change.

nats_connection_state

Fired on every state transition of the agent's NATS client.

These events are process-level (no task scope), so the common envelope's job_id, round, and phase are omitted. trace_id keeps the catalog's 32-char lowercase-hex shape but hashes a session-less (agent_id, uuid_v4) seed instead of the task tuple — distinct events get distinct traces (so one agent's state transitions don't alias under a single id), but consumers parsing trace_id see a uniform shape across the whole catalog.

Population status (current implementation)

pending_publish_depth and buffer_bytes are Option<u32> / Option<u64> and the worker emits null (omitted from the JSON) until async_nats::Client::statistics() (or equivalent introspection) is wired. Treat absent fields as "not yet measured", not "buffer is empty".

context_emergency_shrink

Fires once per LLM call where the SDK clamped max_tokens UP to its floor (typically 200). Pairs with llm_request_complete.max_tokens_shrunk_to_floor: both surface the same shrink_info.floor_used condition. Healthy available > floor adaptive shrinks leave both unset.

Not sent: tool result content, message text, prompt fragments. Only tool (a public name) and bytes (a count) are in recent_tool_outputs — content is never disclosed.

The recent_tool_outputs array is currently emitted empty. The per-task running ledger is owned by generate_structured_output and threaded into react_loop as a &mut u64, so structured-output schema retries inherit the prior attempt's bloat. The top-N attribution is added in a follow-up that reuses that same per-task ledger.

claude_subprocess_spawn / _exit / _session_lock_collision

Lifecycle events for provider_type: claude_cli. Emitted around each run_claude_attempt. The session jsonl at ~/.claude/projects/<munged>/<session_id>.jsonl is treated as the implicit lock — non-zero size before spawn means a prior run left state that the next spawn will collide with.

claude_subprocess_spawn:

claude_subprocess_exit (correlates by session_id):

claude_session_lock_collision (fired by spawn on prior-lock discovery):

Not sent: lock file contents, command line, env vars, stdout / stderr text. Only the path-shaped session_id is emitted.

prompt_exposure_detected

Paired with the PromptExposureMiddleware guardrail. Fires when the guardrail sees dictionary hits on terminal-tool content.

Note. The event variant exists in the SDK schema for completeness, but quorum-rs does not ship a default detector that emits it. Agents can attach any OutputLeakDetector implementation via ProposerEvaluatorAgent::with_output_guard(...); whether this event fires depends on the chosen detector.

Not sent: scanned response content, proposal text, evaluation justifications.

Invariant: hit_count == xml_tag_hits + tool_name_hits + instruction_hits + wrong_acronym_hits.

Events NOT emitted

The following are explicitly never sent. Redaction is enforced at the type layer — the struct fields do not exist on any TelemetryEvent variant:

• Prompts, system instructions, personas, thinking rules
• thought_process
• Proposal content, evaluation justifications, claim assessments
• User queries or follow-up nudges
• Raw LLM response bodies or tool outputs
• Token strings, API keys, or secret material
• NATS credentials or session cookies