lib.agent-evals

@lssm-tech/lib.agent-evals

Deterministic agent evaluation runners, model adapters, graders, and recommendation evidence for ContractSpec.

What it provides

• AgentEvaluationRunner for candidate-vs-baseline trial aggregation.
• AgentEvalsFeature package metadata for docs and discovery.
• Scripted and live model runner adapters.
• Exact-match, schema, policy, artifact, and environment-outcome graders.
• An LLM-judge adapter that turns an injected judge client into a grader.
• Replay bundle refs, redaction leak detection, and cost/latency summaries.
• Fail-closed output, blocking-grader, deterministic-safety, and trial-integrity gates for production-shaped candidate comparisons.
• Route- and release-bound simulated qualification decisions with mandatory baseline evidence for model upgrades.
• A checkpointed rollout state machine covering offline, shadow, internal, 1%, 10%, 50%, production, and evidence-backed rollback stages.
• V2 immutable dataset and exact evaluation-subject bindings covering route, model profile, invocation settings, application purpose, evaluator set, environment, and release versions.
• Reproducible verified-success, cost-per-verified-success, p50/p95 latency, structured-output, tool, fallback, provider, and human-review metrics by dataset segment.
• Dataset digests include an application-owned confidence-interval or power analysis method, minimum sample count, and rationale reference. Missing samples, invalid segment weights, uncovered segments, or rebound cases fail before recommendation.
• V2 comparison artifacts project trials to sanitized case/segment refs, numeric metrics, grader verdicts, replay digests, and boolean failure flags; prompts, responses, artifact bodies, metadata payloads, and replay URIs are never included.
• CAS-ready qualification promotion/revocation and verified canary histories; expired, revoked, rebound, discontinuous, or re-digested rollout evidence fails closed.

Model evaluation never grants provider access or effect authority. A candidate becomes live_verified only after every rollout checkpoint has complete health evidence; a newer model is not promoted merely because it exists or scores well on average.

Local commands

• bun run build
• bun run typecheck
• bun run test
• bun run lint:check