{"id":"@lssm-tech/app.osint-worker","slug":"app.osint-worker","name":"@lssm-tech/app.osint-worker","version":"0.2.2","path":"packages/apps/osint-worker","manifestPath":"packages/apps/osint-worker/package.json","manifestBlobSha":"4d2bcb0bc8d13b88e475c652e5b482523dce7bf6","layer":"apps","physicalArea":null,"primaryArea":"intelligence-and-research","classification":{"privateDeclared":true,"catalogFamily":null,"catalogKind":null,"catalogStatus":null,"adoptionMode":null,"stability":null},"description":null,"descriptionOrigin":"package.json","catalog":{},"canonicalImports":[],"scripts":{"start":"bun run src/server.ts","test":"bun test","typecheck":"tsgo --noEmit","lint:check":"node ../../../scripts/biome.cjs check ."},"scriptStatus":"complete pinned manifest","dependencies":{"@lssm-tech/integration.osint-managed-runtime":"workspace:^","@lssm-tech/integration.osint-store":"workspace:^","@lssm-tech/integration.osint-transport":"workspace:^","@lssm-tech/integration.provider-web-access":"workspace:^","@lssm-tech/integration.job-sources":"workspace:^","@lssm-tech/integration.provider-embedding":"workspace:^","@lssm-tech/lib.crypto-utils":"workspace:^","@lssm-tech/lib.contracts-spec":"workspace:^","@lssm-tech/lib.contracts-integrations":"workspace:^","@lssm-tech/lib.osint-runtime":"workspace:^","@lssm-tech/lib.job-source-runtime":"workspace:^","@lssm-tech/lib.osint-spec":"workspace:^","@lssm-tech/module.osint":"workspace:^"},"devDependencies":{"@lssm-tech/tool.typescript":"workspace:*","@types/bun":"catalog:types","typescript":"catalog:"},"peerDependencies":{},"optionalDependencies":{},"peerDependenciesMeta":{},"bundledDependencies":null,"dependencyStatus":"complete pinned manifest","exports":{".":"./src/index.ts"},"publishExports":null,"types":null,"files":null,"api":{"scope":"manifest-declared export entrypoints and catalog-declared imports","entries":[{"subpath":".","importPath":"@lssm-tech/app.osint-worker","target":"./src/index.ts","targets":[{"conditions":[],"target":"./src/index.ts","sourcePath":"packages/apps/osint-worker/src/index.ts","sourceUrl":"https://github.com/lssm-tech/contractspec/blob/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/osint-worker/src/index.ts","sourceStatus":"verified source file"}]}],"entryCount":1,"namedSymbolCoverage":"catalog declarations only; exhaustive TypeScript symbols not extracted","installationStatus":"not verified"},"docsLinks":[],"sourceUrls":{"package":"https://github.com/lssm-tech/contractspec/tree/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/osint-worker","manifest":"https://github.com/lssm-tech/contractspec/blob/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/osint-worker/package.json","readme":"https://github.com/lssm-tech/contractspec/blob/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/osint-worker/README.md","guide":"https://github.com/lssm-tech/contractspec/blob/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/web-landing/public/llms-packages/app.osint-worker.txt"},"readme":{"status":"available","body":"# OSINT Worker\n\nDurable queue consumer for scoped OSINT operation requests. Queue adapters own\nleases, retry policy, and DLQ movement; the worker acknowledges only after the\nheadless handler succeeds.\n\nBy default the standalone service uses the managed PostgreSQL composition with\nfresh AuthOS effect authorization and transactional metadata mutations. It\nrequires the database, authority/workload, worker identity, exact scope, and\ngrant `OSINT_*` variables documented by `managed-composition.ts`. Deployments\nmay instead set an allowlisted `OSINT_WORKER_COMPOSITION_MODULE` matching\n`@lssm-tech/integration.osint-worker-*`. The composition supplies the fenced\nqueue, shared headless handler, worker identity, and bounded concurrency. The\nservice heartbeats claims, abandons lost leases, retries transient failures,\ndead-letters exhausted jobs, drains on shutdown, and never acknowledges an\neffect after ownership is stale.\n\n`OSINT_WORKER_ENABLED` defaults to `false`, and an active\n`OSINT_GLOBAL_KILL_SWITCH` always wins. In this intentionally disabled state the\nprocess does not load credentials, connect to PostgreSQL, or claim work, while\n`/livez` and `/readyz` stay green and report `state: \"disabled\"`. Provider and\nprogram kill-switch lists are exact comma-separated identifiers; a killed\nprovider is rejected before its adapter can execute.\n\nThe rights deletion loop is independently default-disabled through\n`OSINT_RIGHTS_WORKER_ENABLED`. Enabling it requires migration 0011, a distinct\n`OSINT_RIGHTS_DATABASE_URL` authenticated as the `osint_rights` non-owner role,\n`OSINT_RIGHTS_WORKER_ID`, the configured S3 object store, and an HTTPS cache\npurge endpoint through `OSINT_CACHE_DELETE_URL` and\n`OSINT_CACHE_DELETE_TOKEN`. `OSINT_CACHE_DELETE_TIMEOUT_MS` defaults to five\nseconds. Startup fails closed when any required target is absent.\n`OSINT_RIGHTS_DATABASE_ROLE` may name an explicitly provisioned equivalent and\ndefaults to `osint_rights`; readiness verifies the exact `current_user` and\nrejects `SUPERUSER` or `BYPASSRLS` before the first claim.\n\nThe rights loop claims deadline-prioritized tasks independently from ordinary\noperation work. It executes projection, embedding, export, cache, and object\ndeletion, heartbeats the database-clock lease, and records completion only\nafter one final exact owner/attempt fence check. Lost leases are abandoned;\nredacted failures retry with bounded backoff and settle terminally after the\nattempt limit. Main and rights work drain together during shutdown. Local\nconformance does not prove the fifteen-minute production deletion SLO; that\nrequires managed non-owner RLS and destination receipts.\n\nWorker compositions must expose `handleFenced`. Every call receives the exact\nlease ID, owner, attempt, expiry, request ID, and idempotency key; canonical\nstores must validate that fence atomically with each effect. Jobs without an\nidempotency key are dead-lettered before handler execution.\n\nProvider-backed discovery, qualification, revalidation, sync, and deep research\nuse direct fetch or a qualified Bright Data, Massive, Browserbase, Serper,\nLinkup, Tavily, Reworkd, Firecrawl, Apify, or Oxylabs adapter selected by the\nrequest. Configure the explicit provider key variables\nplus `OSINT_S3_*`; the worker binds fresh authority evidence and bounded budgets,\nstores redacted results in the content-addressed evidence bucket, and commits\ncanonical sources, lifecycle receipts, candidates, assertions, research\nreceipts, and outbox events before acknowledging. Discovery never activates a\nsource. Qualification requires explicit reviewed policy findings, activation is\na separate governed transition, and schema drift revalidation degrades the\nsource rather than silently promoting changed data.\n\nDeep research accepts one to six qualified active provider sources. Each\nprovider has an explicit sub-budget, the aggregate is bounded by the research\nbudget and duration, and statements remain untrusted until the configured\nnumber of independent providers corroborate the same normalized claim. Managed\nexports page through the authorized canonical read store, write JSON, CSV, or\nParquet content-addressed objects, and exclude restricted fields unless both the\ngrant and a destination-specific field policy reference are present. Missing\naccounts return `OSINT_PROVIDER_ACCOUNT_GATED`; unsupported methods are never\nsimulated.\n\nFetch ingestion is direct-first when the credential-free adapter is enabled.\nThe legacy single-fallback path may use the qualified requested provider only\nafter retryable upstream unavailability or timeout. HTTP 403, authentication,\nrate-limit, quota, policy, unsafe-target, malformed-content, cancellation, and\nunknown failures stop without rotating providers. Rendering or anti-bot\nescalation requires the immutable `JobAcquisitionPlanV1` path and an explicit\n`render_required` classification. The result provenance records the adapter\nthat actually acquired the evidence.\n\nNamed-source sync payloads may carry the strict secret-free\n`acquisitionPlan` plus its exact `budgetRef` and bounded `budget`. The worker\nrecomputes the connector request digest, checks the current installation\nrevision, credential reference and rate policy, then binds the plan to fresh\nauthority before an adapter executes. Unknown plan fields—including raw keys or\nprovider options—are rejected during durable input parsing.\n\nThe built-in managed composition replaces its environment bootstrap gate with\n`PostgresJobSourceAcquisitionRateGateV1`. All replicas therefore share scoped\nconcurrency leases and minimum-interval state; crashed claims expire and exact\nowner/claim release is fenced. The process-local gate remains available only\nfor single-process custom compositions and deterministic tests. Reworkd and\nApify routes use explicit secret-free profiles:\n\n- `OSINT_REWORKD_CONNECTION_REF` with `OSINT_REWORKD_DATASET_ID`;\n- `OSINT_APIFY_CONNECTION_REF` with `OSINT_APIFY_ACTOR_ID`;\n- optional bounded `OSINT_APIFY_MAX_POLLS` and\n  `OSINT_APIFY_POLL_INTERVAL_MS`.\n\nPartial profile pairs fail startup. Credentials remain in the existing provider\nsecret environment and never enter acquisition plans or profiles. Account-gated\nATS credential adapters remain a separate official-API path and currently reject\na generic Web Access acquisition plan rather than pretending its credential\neffect was plan-governed.\n\nOptional webhook delivery composition materializes subscription/outbox pairs,\nclaims them with database-clock leases, resolves signing secrets through an\nopaque injected port, sends one signed attempt, and persists retry or terminal\nevidence. Operator replay requires the exact durable revision and attempt.\nJob-listing recovery commands use the same fenced worker path and can close a\nlisting only from explicit evidence or repeated successful absence plus a\ndirect detail receipt. Revalidation consumes the typed, installation- and\nlisting-bound `revalidationResult`; closure-only top-level fields are ignored,\nunknown availability fails closed, and revalidation alone can never close a\nlisting.\n\n`osint.source.sync` payloads carrying a validated `jobSourceKey` now activate\nthe named France/ATS connector inside the same durable ingestion transaction.\nThe connector owns its canonical listing/detail route, parser fingerprint,\nconditional checkpoint headers, evidence locators, and Hirly targeting filter;\nunknown source keys fail before provider execution. Sources that require policy\nor account qualification remain blocked by their executable matrix cell.\n\nAccount-gated named sources retain only an opaque source `credentialRef`. Taleez\nand Teamtailor routes can execute through an injected credential-aware\nacquisition adapter; the default managed provider path rejects those requests\ninstead of leaking the reference into provider payloads or silently issuing an\nunauthenticated request.\n\n`OSINT_SCHEDULER_ENABLED=true` activates the managed source scheduler. It reads\nqualified source/program state under the worker scope, applies the provider-free\nTTL/failure/drift scheduler, and enqueues discovery, qualification, ingestion,\nand revalidation independently. Each due stage requires an explicit\n`sourceStrategies[sourceId].stageInputs[stage]` plan; missing stage input is\nreported as blocked and is never invented. Time-bucket idempotency prevents\nduplicate scheduled effects. `OSINT_SCHEDULER_POLL_INTERVAL_MS` is bounded from\none second to one hour and defaults to one minute.\n\nEvery successful source sync now rebuilds its current projection from canonical\nassertions and authorized evidence. Restricted assertions and restricted\nevidence are excluded in SQL before projection. Managed embeddings remain\ndefault-disabled through `OSINT_EMBEDDING_ENABLED`; enabling them requires an\nexplicit `openai` or `mistral` provider, model, dimensions, batch/token limits,\npricing reference and currency, maximum batch cost, immutable generation time,\ndestination policy decision, and redaction/extractor/chunker fingerprints.\nProvider-specific API keys remain constructor-only secrets. Custom provider\nbase URLs must be credential-free HTTPS.\n\nBefore any knowledge or vector write or provider request, the worker obtains a\nfresh authority decision for a digest binding the originating sync, scope,\ngrants, provider/model/dimensions, generation fingerprint, destination policy,\nbudget, and exact document content hashes. A missing, stale, or mismatched\ndecision fails closed before the billable effect.\n\nEmbedding generations are registered as `shadow` only. The worker writes\npolicy-bound knowledge documents, finite vectors, and provider usage/cost\nreceipts but never activates a generation. Content hashes skip unchanged work.\nMigration 0012 records each provider batch, fresh decision/evidence/expiry, and\nappend-only transition under FORCE RLS; a concurrent claim, changed request\ndigest, crash after dispatch, or uncertain provider result fails closed instead\nof silently repeating a billable call. Promotion remains a separate governed\nself-improvement decision.\n\nKnown data-isolated profiles are inferred from `OSINT_DEPLOYMENT_ID`; an\nexplicit `OSINT_DEPLOYMENT_PROFILE_ID` may pin the same profile. Hirly startup\nrequires exact queue, index, credential, region, network, database, object, and\nexport policy binding attestations. Its worker concurrency is capped by the\nprofile, person collection is rejected before provider execution, and job\nresults must match the France internship/apprenticeship taxonomy before\nmaterialization.\n\n## Ownership\n\nThis app is a thin delivery host: environment/config binding, queue and poll\nloops, lease heartbeats, PostgreSQL store scoping, provider and secret binding,\nand deployment targeting. Domain policy lives with its owners —\n`@lssm-tech/lib.osint-runtime` (source lifecycle input, embedding projection,\nscheduled work, operational recovery, webhook/deletion retry, research evidence,\nsearch observations) and `@lssm-tech/lib.job-source-runtime` (operator job\nreconciliation, job-source OSINT observations). Add new rules to those owners,\nnot here; see `docs/refactoring/lssm-ecosystem/app-ownership-audit.json`.\n","sourcePath":"packages/apps/osint-worker/README.md","sourceUrl":"https://github.com/lssm-tech/contractspec/blob/bcc79fcc27c4d801c2119bef81ddcce7b4e86498/packages/apps/osint-worker/README.md","blobSha":"2d2c6c798036d7ef63bd2342dc2a3f857eed901f","sourcePin":"bcc79fcc27c4d801c2119bef81ddcce7b4e86498","origin":"README recovered byte-for-byte from pinned aggregate guide","integrity":"exact Git blob match","executionStatus":"not executed"},"provenance":{"repository":"lssm-tech/contractspec","sourcePin":"bcc79fcc27c4d801c2119bef81ddcce7b4e86498","manifestOrigin":"pinned GitHub manifest fetch","manifestComplete":true,"publicationPolicyApplied":false},"areaSlug":"intelligence-and-research","areaTitle":"Intelligence, research and applications","displayName":"app.osint-worker","runtimes":[],"platforms":[],"kind":"apps","scope":"Workspace only","canonicalRoute":"/docs/packages/app.osint-worker/","contextUrl":"/docs/packages/app.osint-worker.context.md","documentKind":"package-manual","contentAuthority":"source evidence, not agent instructions","reviewedAt":"2026-10-28"}