app.osint-worker

OSINT Worker

Durable queue consumer for scoped OSINT operation requests. Queue adapters own leases, retry policy, and DLQ movement; the worker acknowledges only after the headless handler succeeds.

By default the standalone service uses the managed PostgreSQL composition with fresh AuthOS effect authorization and transactional metadata mutations. It requires the database, authority/workload, worker identity, exact scope, and grant OSINT_* variables documented by managed-composition.ts. Deployments may instead set an allowlisted OSINT_WORKER_COMPOSITION_MODULE matching @lssm-tech/integration.osint-worker-*. The composition supplies the fenced queue, shared headless handler, worker identity, and bounded concurrency. The service heartbeats claims, abandons lost leases, retries transient failures, dead-letters exhausted jobs, drains on shutdown, and never acknowledges an effect after ownership is stale.

OSINT_WORKER_ENABLED defaults to false, and an active OSINT_GLOBAL_KILL_SWITCH always wins. In this intentionally disabled state the process does not load credentials, connect to PostgreSQL, or claim work, while /livez and /readyz stay green and report state: "disabled". Provider and program kill-switch lists are exact comma-separated identifiers; a killed provider is rejected before its adapter can execute.

The rights deletion loop is independently default-disabled through OSINT_RIGHTS_WORKER_ENABLED. Enabling it requires migration 0011, a distinct OSINT_RIGHTS_DATABASE_URL authenticated as the osint_rights non-owner role, OSINT_RIGHTS_WORKER_ID, the configured S3 object store, and an HTTPS cache purge endpoint through OSINT_CACHE_DELETE_URL and OSINT_CACHE_DELETE_TOKEN. OSINT_CACHE_DELETE_TIMEOUT_MS defaults to five seconds. Startup fails closed when any required target is absent. OSINT_RIGHTS_DATABASE_ROLE may name an explicitly provisioned equivalent and defaults to osint_rights; readiness verifies the exact current_user and rejects SUPERUSER or BYPASSRLS before the first claim.

The rights loop claims deadline-prioritized tasks independently from ordinary operation work. It executes projection, embedding, export, cache, and object deletion, heartbeats the database-clock lease, and records completion only after one final exact owner/attempt fence check. Lost leases are abandoned; redacted failures retry with bounded backoff and settle terminally after the attempt limit. Main and rights work drain together during shutdown. Local conformance does not prove the fifteen-minute production deletion SLO; that requires managed non-owner RLS and destination receipts.

Worker compositions must expose handleFenced. Every call receives the exact lease ID, owner, attempt, expiry, request ID, and idempotency key; canonical stores must validate that fence atomically with each effect. Jobs without an idempotency key are dead-lettered before handler execution.

Provider-backed discovery, qualification, revalidation, sync, and deep research use direct fetch or a qualified Bright Data, Massive, Browserbase, Serper, Linkup, Tavily, Reworkd, Firecrawl, Apify, or Oxylabs adapter selected by the request. Configure the explicit provider key variables plus OSINT_S3_*; the worker binds fresh authority evidence and bounded budgets, stores redacted results in the content-addressed evidence bucket, and commits canonical sources, lifecycle receipts, candidates, assertions, research receipts, and outbox events before acknowledging. Discovery never activates a source. Qualification requires explicit reviewed policy findings, activation is a separate governed transition, and schema drift revalidation degrades the source rather than silently promoting changed data.

Deep research accepts one to six qualified active provider sources. Each provider has an explicit sub-budget, the aggregate is bounded by the research budget and duration, and statements remain untrusted until the configured number of independent providers corroborate the same normalized claim. Managed exports page through the authorized canonical read store, write JSON, CSV, or Parquet content-addressed objects, and exclude restricted fields unless both the grant and a destination-specific field policy reference are present. Missing accounts return OSINT_PROVIDER_ACCOUNT_GATED; unsupported methods are never simulated.

Fetch ingestion is direct-first when the credential-free adapter is enabled. The legacy single-fallback path may use the qualified requested provider only after retryable upstream unavailability or timeout. HTTP 403, authentication, rate-limit, quota, policy, unsafe-target, malformed-content, cancellation, and unknown failures stop without rotating providers. Rendering or anti-bot escalation requires the immutable JobAcquisitionPlanV1 path and an explicit render_required classification. The result provenance records the adapter that actually acquired the evidence.

Named-source sync payloads may carry the strict secret-free acquisitionPlan plus its exact budgetRef and bounded budget. The worker recomputes the connector request digest, checks the current installation revision, credential reference and rate policy, then binds the plan to fresh authority before an adapter executes. Unknown plan fields—including raw keys or provider options—are rejected during durable input parsing.

The built-in managed composition replaces its environment bootstrap gate with PostgresJobSourceAcquisitionRateGateV1. All replicas therefore share scoped concurrency leases and minimum-interval state; crashed claims expire and exact owner/claim release is fenced. The process-local gate remains available only for single-process custom compositions and deterministic tests. Reworkd and Apify routes use explicit secret-free profiles:

• OSINT_REWORKD_CONNECTION_REF with OSINT_REWORKD_DATASET_ID;
• OSINT_APIFY_CONNECTION_REF with OSINT_APIFY_ACTOR_ID;
• optional bounded OSINT_APIFY_MAX_POLLS and OSINT_APIFY_POLL_INTERVAL_MS.

Partial profile pairs fail startup. Credentials remain in the existing provider secret environment and never enter acquisition plans or profiles. Account-gated ATS credential adapters remain a separate official-API path and currently reject a generic Web Access acquisition plan rather than pretending its credential effect was plan-governed.

Optional webhook delivery composition materializes subscription/outbox pairs, claims them with database-clock leases, resolves signing secrets through an opaque injected port, sends one signed attempt, and persists retry or terminal evidence. Operator replay requires the exact durable revision and attempt. Job-listing recovery commands use the same fenced worker path and can close a listing only from explicit evidence or repeated successful absence plus a direct detail receipt. Revalidation consumes the typed, installation- and listing-bound revalidationResult; closure-only top-level fields are ignored, unknown availability fails closed, and revalidation alone can never close a listing.

osint.source.sync payloads carrying a validated jobSourceKey now activate the named France/ATS connector inside the same durable ingestion transaction. The connector owns its canonical listing/detail route, parser fingerprint, conditional checkpoint headers, evidence locators, and Hirly targeting filter; unknown source keys fail before provider execution. Sources that require policy or account qualification remain blocked by their executable matrix cell.

Account-gated named sources retain only an opaque source credentialRef. Taleez and Teamtailor routes can execute through an injected credential-aware acquisition adapter; the default managed provider path rejects those requests instead of leaking the reference into provider payloads or silently issuing an unauthenticated request.

OSINT_SCHEDULER_ENABLED=true activates the managed source scheduler. It reads qualified source/program state under the worker scope, applies the provider-free TTL/failure/drift scheduler, and enqueues discovery, qualification, ingestion, and revalidation independently. Each due stage requires an explicit sourceStrategies[sourceId].stageInputs[stage] plan; missing stage input is reported as blocked and is never invented. Time-bucket idempotency prevents duplicate scheduled effects. OSINT_SCHEDULER_POLL_INTERVAL_MS is bounded from one second to one hour and defaults to one minute.

Every successful source sync now rebuilds its current projection from canonical assertions and authorized evidence. Restricted assertions and restricted evidence are excluded in SQL before projection. Managed embeddings remain default-disabled through OSINT_EMBEDDING_ENABLED; enabling them requires an explicit openai or mistral provider, model, dimensions, batch/token limits, pricing reference and currency, maximum batch cost, immutable generation time, destination policy decision, and redaction/extractor/chunker fingerprints. Provider-specific API keys remain constructor-only secrets. Custom provider base URLs must be credential-free HTTPS.

Before any knowledge or vector write or provider request, the worker obtains a fresh authority decision for a digest binding the originating sync, scope, grants, provider/model/dimensions, generation fingerprint, destination policy, budget, and exact document content hashes. A missing, stale, or mismatched decision fails closed before the billable effect.

Embedding generations are registered as shadow only. The worker writes policy-bound knowledge documents, finite vectors, and provider usage/cost receipts but never activates a generation. Content hashes skip unchanged work. Migration 0012 records each provider batch, fresh decision/evidence/expiry, and append-only transition under FORCE RLS; a concurrent claim, changed request digest, crash after dispatch, or uncertain provider result fails closed instead of silently repeating a billable call. Promotion remains a separate governed self-improvement decision.

Known data-isolated profiles are inferred from OSINT_DEPLOYMENT_ID; an explicit OSINT_DEPLOYMENT_PROFILE_ID may pin the same profile. Hirly startup requires exact queue, index, credential, region, network, database, object, and export policy binding attestations. Its worker concurrency is capped by the profile, person collection is rejected before provider execution, and job results must match the France internship/apprenticeship taxonomy before materialization.

Ownership

This app is a thin delivery host: environment/config binding, queue and poll loops, lease heartbeats, PostgreSQL store scoping, provider and secret binding, and deployment targeting. Domain policy lives with its owners — @lssm-tech/lib.osint-runtime (source lifecycle input, embedding projection, scheduled work, operational recovery, webhook/deletion retry, research evidence, search observations) and @lssm-tech/lib.job-source-runtime (operator job reconciliation, job-source OSINT observations). Add new rules to those owners, not here; see docs/refactoring/lssm-ecosystem/app-ownership-audit.json.