app.contractspec-ci-controller

ContractSpec CI controller

This private LSSM application coalesces GitHub delivery events, reserves the LSSM CI budget, and issues one repository-scoped JIT configuration per Railway runner. It is an execution adapter for the provider-neutral delivery protocol; it is not policy or promotion authority.

The default is CI_CONTROLLER_ENABLED=false. In that state the service exposes GET /healthz and an unavailable GET /readyz; every mutation endpoint returns 503 controller_disabled, and GitHub App, webhook, economic-evidence, and runner credentials are not required. This mode exists for infrastructure and health qualification before authority is installed.

Active mode requires CI_CONTROLLER_ENABLED=true plus the complete repository-only GitHub App and controller credentials. Startup fails closed if any active credential is absent. Never substitute a personal or production-wide token.

Budget defaults are $30 target and $45 circuit breaker. Override them with CI_MONTHLY_TARGET_USD and CI_CIRCUIT_BREAKER_USD; the target must remain strictly below the breaker.

The controller listens on IPv6 and IPv4 using the assigned PORT. Its runner label is a single validated CI_RUNNER_LABEL, defaulting to railway-linux. The active readiness response means configuration is loaded, not that a runner pool or end-to-end delivery is qualified.

Dispatch intent is persisted before contacting GitHub. An unconfirmed dispatch retains its budget reservation and blocks that branch across restarts. New attempts receive a durable UUID before dispatch, included with the source SHA in the workflow run name. On restart the controller scans complete bounded GitHub run pages and accepts only a unique matching workflow from its own App in the configured repository. A newer queued head survives acknowledgement of the older attempt. No match, incomplete pagination, identity conflict, or duplicate matches retain the reservation and block retries. Legacy attempts without a UUID still require operator reconciliation. Never erase the queue to retry.

Queue and budget transitions sync the file and directory before acknowledgement. They support one controller writer on durable storage, not multiple active writers. /healthz reports blocked dispatch count. GitHub calls have bounded deadlines. CI_DELIVERY_WORKFLOW_REF selects the trusted workflow branch (default release); deploy the matching workflow correlation changes before enabling the controller. This recovery covers GitHub dispatch acknowledgement, not the runner-service pool.

Only matching-repository branch pushes and open, non-draft, same-repository PR updates enter the queue. Labels and fork metadata do not grant trusted-runner access. Workflow dispatch uses Actions permission with source read-only access.

Budget inputs reject negative/non-finite amounts, historical billing periods, and unexplained decreases. Ordinary positive-cost admission closes at the configured circuit breaker. Provider reservations, actual usage reconciliation, independent watchdog operation and progressive production cutover remain separately qualified platform work.

Run-scoped disposable pool (operator mode)

The src/pool engine persists intent before create/start/delete, reserves cost by stable job identity and verifies resource absence before closing a lease. Unknown creates are recovered by exact generated identity; no observed resource means no retry. Unknown starts are observed rather than reissuing JIT credentials.

Use a dedicated local state directory and an operator-reviewed configuration:

{
  "repository": "owner/repo",
  "runId": 123,
  "runAttempt": 1,
  "workflowSha": "<40-character approved workflow commit>",
  "sourceSha": "<40-character approved source commit>",
  "actorLogin": "ci-app[bot]",
  "workspaceId": "<Railway workspace UUID>",
  "projectId": "<Railway CI project UUID>",
  "environmentId": "<Railway CI environment UUID>",
  "repositoryRoot": "/absolute/repository",
  "stateDirectory": "/absolute/private/run-state",
  "githubAppId": "<App ID>",
  "githubInstallationId": "<installation ID>",
  "githubPrivateKeyPath": "/secure/github-app.pem",
  "maximumConcurrent": 2,
  "maximumRunUsd": 2,
  "estimatedJobUsd": 0.25,
  "maximumLifetimeMs": 1200000
}

Manual Dev CI defaults its ephemeral_pool input to enabled. Other workflows must explicitly enable that input when supported. This selects run/job-bound Railway workers; it does not start the operator automatically. Then run bun run pool:run /absolute/config.json and, as a separate process, bun run pool:watchdog /absolute/config.json. The watchdog can also run once with --once. The operator host needs authenticated gh and railway CLIs and Bun. The main writer holds an OS-backed SQLite lock; restart cannot change the pinned operator scope. Runner build files come from the approved Git object, not a mutable checkout. A separate expiry watchdog records its own cleanup receipts without writing the main ledger. It remains independent of the Railway workers.

Every job uses a run/attempt/job-specific label, with no shared Railway label on its worker. Labels are only routing: an administrator-installed pre-job guard compares GitHub's assigned identity with the launch manifest. Mismatch terminates the runner process group and blocks, preventing even subsequent always() steps. GitHub may download action packages during setup before this hook; this profile is for explicitly reviewed source, not qualification of hostile-code isolation.

The initial profile uses 2 vCPU and 8 GB, one JIT job, no restart, no data volume, no public domain and no controller bootstrap credential in the worker. The operator caps reservations per run and checks observed project usage against the LSSM CI circuit breaker. Holds remain until provider cost reconciliation; deleting a worker does not prove its cost was zero.

This is a local operator capability, not active-active managed scheduling. The shared pool authority port remains mandatory; managed installation must wire current Connect/Runtime Node authority and shared atomic budget/state adapters. The HTTP controller's automatic admission stays separately qualified and disabled until the complete pool cutover requirements pass.

General reviewed dispatches can supply workflowPath and jobBindings in the operator config. Exact source/workflow identity, App actor, repository, run, attempt and job labels remain mandatory. The pinned scope includes the job map; a restart cannot change it. General PR/matrix admission remains a separate qualification.

The single-host scheduler defaults to a 60-second observed quiet window before admission. quietWindowMs accepts 0–600000 milliseconds. Candidate changes reset persisted quiet time; no worker or cost reservation is created while coalescing. A disappearing candidate or a backward clock resets the window. Previously admitted work resumes immediately with its original reservation and authority. This local coalescing behavior still requires live push qualification before activating automatic routing.

Build workload measurements

Dev CI requests a Turbo run summary and projects only source identity, run/attempt, workload profile, elapsed time, task counts, cache-hit counts and the ten slowest tasks. Raw command lines, filesystem locations, environment values and global cache inputs are never published by this projection. Malformed or mismatched summaries fail explicitly. Measurements appear in job output and the GitHub step summary; they are diagnostic evidence, not release or execution authority.

Operator shutdown first requests normal cancellation, then waits up to 90 seconds for terminal status and log availability. Force cancellation requires a fresh verified run/attempt snapshot after that grace period. Final receipts distinguish worker cleanup from log availability. Transient CLI observation failures retry within the original run lifetime; they do not imply the run is terminal.

Manual full-repository Dev CI qualification has a 120-minute build ceiling. Its operator must be configured with an appropriate bounded lifetime (at most two hours); ordinary CI retains its existing timeout. This cold-baseline allowance does not satisfy the affected-change latency target or alter configured budgets.

The first operator admission persists an immutable run deadline before any runner context or allocation. Restart reuses that deadline; missing/corrupt lifetime state after work began fails closed. Legacy interrupted runs must be drained with their original operator before adopting the new lifetime record. Late workers cannot receive a fresh full lifetime: their retirement is capped by the run deadline plus a two-minute cleanup reserve. The watchdog remains active through that retirement bound and its cleanup grace, including after an operator crash.

Operator configuration parsing rejects unsupported event/actor types, absent or invalid workflow paths, mismatched source/workflow commits, and empty, duplicate or malformed job maps before returning an executable configuration. This closes a comma-expression regression that discarded the initial policy predicates. Independent GitHub run/job verification remains required before allocation.

inspectReviewedPullRequestRun is a read-only PR identity boundary. It binds an explicit reviewed repository, actor, PR number, workflow, head, base and synthetic merge commit against two independent run/PR observations. Forks, closed PRs, pull_request_target, substituted associations and changed attempts fail closed. The result explicitly has executionAuthorized: false; allocation requires an explicit operator configuration with matching pullRequest identity and a reviewed runnerContextSha. The operator revalidates current PR metadata on each snapshot. The runner guard checks the exact PR merge ref, GITHUB_SHA and workflow SHA.

The local scheduler accepts discovery.kind: pull_request with a reviewed PR number, head branch, base branch, actor and job map. Each candidate is pinned to current GitHub head/base/merge identities and rechecked before return. It uses the existing quiet window, persistent reservation and exact-run resume. Push discovery remains the default when kind is omitted. A PR lane requires a reviewed operator.runnerContextSha before scheduler startup.

When the PR changes, admission fails closed and the operator retires the old run. The retirement reader verifies immutable run/attempt/repository/actor/PR association without demanding that the PR remain open at its old head. Only shutdown and terminal receipt readback use that reader; allocation continues to require fresh PR state. Live scheduler/guard qualification remains necessary before cutover.

Operator resourceProfile selects a versioned host-owned capacity contract: standard-v1 (default, 2 vCPU / 8 GB), light-v1 (1 vCPU / 2 GB), or build-v1 (4 vCPU / 16 GB). The selected ID and capacity are recorded in profile evidence and bound into restart scope. Workflow labels cannot select capacity. Set a reviewed run reservation and lifetime appropriate to the chosen profile; resource support alone is not cost or workload qualification. Existing operators without the field preserve their original standard capacity.

The reviewed PR canary also runs durable worker concurrency/restart tests, Builder readiness, coding receipt parsing and sandbox CLI recovery checks. It builds crypto-utils and lssm-os-spec prerequisites after the pinned install. Sandbox checks in this lane are subprocess/CLI fixtures, not live Docker containment evidence. The exact self-hosted job label and PR selector remain.

Targeted Next build resource qualification

build-v2 retains the build-v1 allocation of 4 vCPU / 16 GB. It additionally binds CONTRACTSPEC_BUILD_CPU_BUDGET=4 to the operator resource profile. The runner validates this exact version/value pair and forwards only the two non-secret profile variables through its empty job environment. Existing v1 profiles and their persisted scope material remain unchanged; labels or workflow inputs cannot select a larger allocation.

Use .github/workflows/web-landing-build-qualification.yml after integrating the reviewed commit into release. It is manual-only, has no GitHub-hosted fallback, and binds job name Web landing build qualification to key next-build. Start with the existing reviewed CI project/environment, resourceProfile: build-v2, maximumConcurrent: 1, and an explicitly reserved run budget plus independent watchdog. Bind sourceSha and workflowSha to the exact same release commit, workflowPath to this file, the actual run/attempt, and a fresh state directory. Keep the existing maximum lifetime at or below two hours; do not reset a failed run's deadline or reuse its scope to try a different profile.

The workflow first measures the dependency-only Turbo build closure, then runs the complete web-landing package build (including its type gate) in a separate measurement phase. The dependency selector was verified to exclude web-landing itself. The app build is not skipped through a Turbo cache hit. Route counts belong to the exact source; do not freeze the old failed run's 1678 routes when the current release has changed.

Reports bind actual Git HEAD, source SHA, run and attempt. They retain configured capacity, host-visible CPU count, actual cgroup quota/memory limit, CPU use and throttling, OOM-kill deltas, sampled phase memory and kernel lifetime peak. The lifetime peak includes earlier phases; sampled current memory can miss short spikes. Qualification requires exact CPU quota and an exact 16 GB or 16 GiB byte interpretation, which is reported explicitly. Larger, unbounded, missing or changed allocations cannot qualify the 4/16 profile. Failed commands, startup failures and OOMs retain failing measurement records without logging arguments or environment values.

Download web-landing-resources-<run>-<attempt> or read the corresponding bounded JSON log records. Compare the Next phase with the prerequisite phase and Railway usage before/after the scoped run. Reconcile actual costs and verify both runner inventories are clean through the operator/watchdog receipt. This is one workload sample, not full Dev CI, progressive cutover or operational acceptance. Do not increase memory until measured peak/limit evidence justifies a different reviewed profile.

For the bounded correction to repository-wide cache invalidation, see root workspace cache scope^†.

Next workload qualification requires an initially absent .next output directory and newly produced BUILD_ID, prerender and route manifests. Their hashes are recorded alongside source/run-bound resource observations. A zero exit code from Bun help or another no-op cannot qualify the Next phase. Existing outputs fail preflight rather than being silently deleted or reused.

recoverBuildMeasurements recovers the exact measurement JSON records from operator-fetched job logs when artifact transport fails. Callers must verify the repository/run/job provenance separately. Recovery checks source/run/attempt, requires every requested workload, rejects conflicting duplicate records and hashes the retained bytes. It always reports qualificationAccepted: false; recovery cannot turn historical allocation-only or no-op evidence into success.

Run the recovery command with an operator-fetched log from the verified job:

bun src/recover-build-measurements-cli.ts /absolute/job.log /absolute/new-receipt.json SOURCE_SHA RUN_ID ATTEMPT web-landing-dependencies-v1 web-landing-next-v1

The output is exclusive-create with private file permissions; an existing receipt is never overwritten. Only record count and log hash are printed. Review recovered records against workload artifacts and independent provider observations before accepting qualification.

For a fresh GitHub read, bun src/pool/recover-measurements.ts OPERATOR_CONFIG JOB_KEY /absolute/new-receipt.json WORKLOAD... first revalidates the configured repository/run/attempt/workflow/source and selects one matching terminal job. It then downloads that job's logs through the bounded CLI transport and writes private, exclusive-create recovery evidence. This runs after shutdown and never holds cleanup open. A failed provider read leaves recovery unverified; use an already-retained, independently identified log only through the explicit file recovery command.

Trusted push to Delivery bridge

The existing single-host scheduler now accepts an opt-in delivery configuration that coalesces a trusted branch push, durably correlates an App-owned Delivery dispatch, and starts the existing job-bound operator/watchdog pair. The HTTP controller remains disabled; no provider deployment is implied. Run schedule.ts CONFIG.json --check before explicit --watch qualification with a one-run budget. See the exact qualification procedure^† for the config, runtime test coverage, receipts and ambiguity/recovery rules.