diff --git a/.claude/skills/update-stack/SKILL.md b/.claude/skills/update-stack/SKILL.md index 101d9a34e..c32fe5dc2 100644 --- a/.claude/skills/update-stack/SKILL.md +++ b/.claude/skills/update-stack/SKILL.md @@ -123,7 +123,6 @@ echo "3ter: no drift — OK" - Scan source is `git diff --name-only devkit-node/master HEAD` (bidirectional) — catches both files that differ AND files present upstream but missing locally (deleted downstream). The previous `git ls-files` approach only saw locally-present files. - Test files (paths containing `/tests/` or `/__tests__/`, or filenames ending `.test.{js,jsx,ts,tsx}` / `.spec.{js,jsx,ts,tsx}`) are excluded — downstream test adaptations are acceptable. - This gate runs **after** `/verify` (never blocks on transient verify failures) and **before** Phase 2 (failure is recoverable — no merge commit yet). -- Ref: plan `2026-05-30-trawl-devkit-perfect-alignment.md` Tasks E.1 + E.2. --- diff --git a/ERRORS.md b/ERRORS.md index fa07043ca..3e2de1447 100644 --- a/ERRORS.md +++ b/ERRORS.md @@ -26,7 +26,7 @@ Use this file as a compact memory of recurring AI mistakes. - [2026-03-14] middleware: assuming config section exists in all environments (e.g. `config.rateLimit`) -> always handle missing config gracefully (passthrough/no-op); dev config often omits sections that only prod defines - [2026-03-15] cross-stack: changing a Node API without checking Vue E2E tests -> when modifying an endpoint Vue consumes, run Vue E2E tests before pushing - [2026-03-15] pr scope: batching multiple unrelated fixes in one PR -> one fix = one PR to isolate blast radius and reduce iteration loops -- [2026-05-05] repository: Repository.update(doc) doing `new Model(doc).save()` rewrites the full document from in-memory state, silently clobbering any concurrent partial update that landed after the read -> always use `Model.updateOne({ _id }, { $set: ... })` or `findOneAndUpdate({ _id }, { $set: ... })` for partial updates to avoid race conditions; see comes-io/trawl_node#1115 comes-io/trawl_node#1116 comes-io/trawl_node#1118 + pierreb-devkit/Node#3605 +- [2026-05-05] repository: Repository.update(doc) doing `new Model(doc).save()` rewrites the full document from in-memory state, silently clobbering any concurrent partial update that landed after the read -> always use `Model.updateOne({ _id }, { $set: ... })` or `findOneAndUpdate({ _id }, { $set: ... })` for partial updates to avoid race conditions; see pierreb-devkit/Node#3605 - [2026-05-31] billing/stripe: reading `price.metadata.planId` in `customer.subscription.updated` webhook handler -> field is EMPTY in real Stripe webhook payloads (planId lives on the Product, not the Price); use a `priceId → plan` map built at boot from `config.stripe.prices` instead; see pierreb-devkit/Node#3742 -- [2026-06-04] repository: top-level `const Foo = mongoose.model('Foo')` in a repository file -> this is evaluated at import time; safe in an HTTP server (loadModels() runs first) but silently crashes standalone scripts (crons, migrations) with `MissingSchemaError` when import order differs; tests miss it because jest mocks intercept the module entirely; fix = lazy getter `const Foo = () => mongoose.model('Foo')` (call sites: `Foo().find(...)`) or dynamic import after `loadModels()` in the entrypoint; see comes-io/trawl_node#1337 comes-io/trawl_node#1338 pierreb-devkit/Node#3789 +- [2026-06-04] repository: top-level `const Foo = mongoose.model('Foo')` in a repository file -> this is evaluated at import time; safe in an HTTP server (loadModels() runs first) but silently crashes standalone scripts (crons, migrations) with `MissingSchemaError` when import order differs; tests miss it because jest mocks intercept the module entirely; fix = lazy getter `const Foo = () => mongoose.model('Foo')` (call sites: `Foo().find(...)`) or dynamic import after `loadModels()` in the entrypoint; see pierreb-devkit/Node#3789 - [2026-06-15] deps/audit: leaving `npm audit` advisories unaddressed on the assumption they need a major bump -> run `npm audit fix` (never `--force`) first; the runtime-tree DoS/ReDoS items (`qs`, `path-to-regexp`, `brace-expansion`) all fixed via in-range bumps, no residual. These are DoS-class but NOT attacker-reachable in this stack: Express route patterns are static (no user-controlled `path-to-regexp` input) and `qs`/`brace-expansion` only parse server-side query strings under fixed code paths — still bump them to keep the tree clean and avoid scanner noise. diff --git a/MIGRATIONS.md b/MIGRATIONS.md index 7f393b763..9401e6e8d 100644 --- a/MIGRATIONS.md +++ b/MIGRATIONS.md @@ -161,7 +161,7 @@ Cf `infra/docs/superpowers/plans/2026-05-10-posthog-observability-followups.md` ## Test DB isolation: per-pid Mongo database default + globalTeardown (2026-04-24) -Default test database is now `mongodb://127.0.0.1:27017/NodeTest_${process.pid}` instead of the shared `NodeTest`. Concurrent jest invocations (e.g. multiple agent worktrees running `npm run test:coverage` in parallel) get isolated databases, eliminating the 401 / 404 / 422 / `MongoPoolClosedError` flake patterns documented in trawl_node#980. +Default test database is now `mongodb://127.0.0.1:27017/NodeTest_${process.pid}` instead of the shared `NodeTest`. Concurrent jest invocations (e.g. multiple agent worktrees running `npm run test:coverage` in parallel) get isolated databases, eliminating the 401 / 404 / 422 / `MongoPoolClosedError` flake patterns seen in parallel runs. ### What changed @@ -195,17 +195,13 @@ CI workflows (`.github/workflows/CI.yml` and downstream copies) set `DEVKIT_NODE When this lands in your project via `/update-stack`, the new `parallel-smoke` CI job ships a default `SMOKE_TEST_PATTERN` of `organizations.integration|tasks.integration` — which only matches in the upstream Devkit. **You MUST override `SMOKE_TEST_PATTERN`** in your CI `parallel-smoke` job (set it under the job's `env:` in `.github/workflows/CI.yml`) to match your project's integration test paths. -The 5 downstream Node projects that consume this stack must each set the override: +Each downstream Node project that consumes this stack must set the override: | Project | Suggested `SMOKE_TEST_PATTERN` | |---|---| -| `pierreb_node` | project-specific integration globs | -| `comes_node` | `tasks.integration\|notes.integration` | -| `trawl_node` | `scraps.integration\|historys.integration` | -| `montaine_node` | project-specific integration globs | -| `ism_node` | project-specific integration globs | +| `_node` | project-specific integration globs (e.g. `foo.integration\|bar.integration`) | -(The exact globs are illustrative — replace with whatever integration files actually exist in each repo. The point is: pick at least two real integration suites so the parallel-smoke job exercises the per-pid DB isolation rather than passing on zero matches.) +(The exact globs are illustrative — replace with whatever integration files actually exist in your repo. The point is: pick at least two real integration suites so the parallel-smoke job exercises the per-pid DB isolation rather than passing on zero matches.) Without an override, the smoke would historically have silently passed with 0 tests run, defeating the regression gate. As of #3518 the orchestrator passes `--passWithNoTests=false` to jest, so a 0-match pattern now exits non-zero and fails the smoke loudly — but the actionable fix is still to point the pattern at real integration paths in your repo. @@ -777,7 +773,7 @@ All features are no-op when `apiKey` is empty — safe to deploy without PostHog ## Organizations & CASL v2 (2026-03-13) -This guide is for downstream projects (e.g. lou-node, pierreb-node) migrating to the new organizations + CASL document-level authorization system introduced on the `feature/signup-org-flow` branch. +This guide is for downstream projects migrating to the new organizations + CASL document-level authorization system introduced on the `feature/signup-org-flow` branch. --- diff --git a/README.md b/README.md index a2af2ab13..1ad63e221 100644 --- a/README.md +++ b/README.md @@ -174,7 +174,7 @@ config/defaults/ myproject.config.js ← global project overrides (all modules) modules//config/ - .myproject.config.js ← per-module project overrides (e.g. users.trawl.config.js) + .myproject.config.js ← per-module project overrides (e.g. users..config.js) ``` Both file types are optional and can be used independently or together. Per-module files take priority over the global project config, allowing fine-grained overrides per module without polluting the global file. @@ -312,9 +312,9 @@ Canonical sources used downstream: | Source | Meaning | |---|---| | `web` | Request from browser (UA not matched as CLI) | -| `cli` | Request from `@trawlme/cli/` (UA-parsed) | +| `cli` | Request from a configured CLI user-agent (UA-parsed, see `analytics.cliUserAgentPattern`) | | `stripe-webhook` | Stripe POST `/api/billing/webhook` | -| `worker-callback` | worker-puppeteer scrap completion callback | +| `worker-callback` | Background worker completion callback | | `cron` | Scheduled background job | | `system` | Server-side fallback (no req, no caller override) | diff --git a/config/defaults/billing.pricing.constants.js b/config/defaults/billing.pricing.constants.js index bd7aa1336..1bf140233 100644 --- a/config/defaults/billing.pricing.constants.js +++ b/config/defaults/billing.pricing.constants.js @@ -10,8 +10,8 @@ * Why ship from devkit: * - Every downstream running billing wants the same export shape. * - Migrations + contract tests + costs service all benefit from a single import path. - * - Trawl had this file at `modules/billing/config/billing.pricing.constants.js` with - * 6+ importers — promoted upstream in plan `2026-06-02-trawl-billing-residual-cleanup.md`. + * - Previously lived per-project at `modules/billing/config/billing.pricing.constants.js` + * with several importers — promoted upstream to a single import path. * * @module billing.pricing.constants */ diff --git a/config/defaults/development.config.js b/config/defaults/development.config.js index 8fafbecf1..a658b50e9 100644 --- a/config/defaults/development.config.js +++ b/config/defaults/development.config.js @@ -107,6 +107,11 @@ const config = { errorTracking: process.env.DEVKIT_NODE_analytics_posthog_errorTracking === 'true', autoCapture: process.env.DEVKIT_NODE_analytics_posthog_autoCapture === 'true', }, + // Regex-source string used by the PostHog context middleware to detect a + // CLI client from its User-Agent (capture group 1 = version). Empty → no + // CLI detection (source stays 'web'). A project that ships a CLI sets this + // to e.g. '@example-org/cli/(\\S+)'. + cliUserAgentPattern: process.env.DEVKIT_NODE_analytics_cliUserAgentPattern ?? '', }, domain: '', cookie: { diff --git a/config/index.js b/config/index.js index bbbb4b6f7..c3c9ea625 100644 --- a/config/index.js +++ b/config/index.js @@ -116,7 +116,7 @@ const initGlobalConfig = async () => { } // Layer 3.5: per-module project overrides (modules/*/config/*.{project}.config.js) - // Only applies for non-standard envs (i.e. downstream project names like "trawl", "comes") + // Only applies for non-standard envs (i.e. a downstream project name, the NODE_ENV value) if (!STANDARD_ENVS.has(env)) { const moduleProjectPattern = `modules/*/config/*.${env}.config.js`; const moduleProjectFiles = await configHelper.getGlobbedPaths(moduleProjectPattern); diff --git a/docs/migrations/2026-05-01-billing-crons-module-relocation.md b/docs/migrations/2026-05-01-billing-crons-module-relocation.md index 3f8abff2d..5e7ac825a 100644 --- a/docs/migrations/2026-05-01-billing-crons-module-relocation.md +++ b/docs/migrations/2026-05-01-billing-crons-module-relocation.md @@ -28,7 +28,7 @@ modules/billing/tests/billing.cron.*.unit.tests.js This is an atomic move with no backward-compat shim. The cutover for each downstream project requires coordinated steps: -1. Run `/update-stack` on the downstream project repo (e.g. trawl_node) — pulls the new structure with crons at `modules/billing/crons/`. +1. Run `/update-stack` on the downstream project repo — pulls the new structure with crons at `modules/billing/crons/`. 2. CI on the downstream repo builds a new image and pushes to GHCR. 3. Update the infra K8s CronJob manifests in `clusters/{cluster}/apps/{project}-billing-*.yaml` — change `args: ["scripts/crons/billing.*.js"]` to `args: ["modules/billing/crons/billing.*.js"]`. 4. Push the infra change → Flux applies → CronJobs use new args + new image together. diff --git a/lib/helpers/config.js b/lib/helpers/config.js index cac549892..54bcd3acf 100644 --- a/lib/helpers/config.js +++ b/lib/helpers/config.js @@ -56,17 +56,14 @@ const validateDomainIsSet = (config) => { /** * Known default / placeholder JWT secret values that must never be used in - * non-dev environments. Extend this list when a new downstream project is - * bootstrapped with its own placeholder. + * non-dev environments. Extend this list when a new project is bootstrapped + * with its own placeholder secret. * @readonly */ const JWT_DEFAULT_SECRETS = Object.freeze(new Set([ - 'WaosSecretKeyExampleToChnageAbsolutely', // devkit upstream placeholder - 'TrawlNodeDevSecret', // trawl downstream placeholder - 'ComesNodeDevSecret', // comes downstream placeholder - 'MontaineNodeDevSecret', // montaine downstream placeholder - 'PierrebNodeDevSecret', // pierreb downstream placeholder - 'IsmNodeDevSecret', // ism downstream placeholder + 'WaosSecretKeyExampleToChnageAbsolutely', // upstream placeholder + 'ExampleNodeDevSecret', // generic project placeholder + 'ChangeThisDevSecret', // generic placeholder ])); /** diff --git a/lib/helpers/tests/config.isJwtSecretWeak.unit.tests.js b/lib/helpers/tests/config.isJwtSecretWeak.unit.tests.js index ba0aaeade..86f59190f 100644 --- a/lib/helpers/tests/config.isJwtSecretWeak.unit.tests.js +++ b/lib/helpers/tests/config.isJwtSecretWeak.unit.tests.js @@ -8,7 +8,7 @@ * Weak cases (returns true): * - undefined / null / empty string / whitespace-only * - length < 32 characters - * - each known default placeholder (devkit + all downstream) + * - each known default placeholder * * Strong case (returns false): * - a ≥ 32-char string that is not in the defaults list @@ -51,28 +51,16 @@ describe('config.isJwtSecretWeak', () => { // ---- weak: each known default placeholder --------------------------------- - test('devkit placeholder → true (weak)', () => { + test('upstream placeholder → true (weak)', () => { expect(isJwtSecretWeak('WaosSecretKeyExampleToChnageAbsolutely')).toBe(true); }); - test('TrawlNodeDevSecret → true (weak)', () => { - expect(isJwtSecretWeak('TrawlNodeDevSecret')).toBe(true); + test('ExampleNodeDevSecret → true (weak)', () => { + expect(isJwtSecretWeak('ExampleNodeDevSecret')).toBe(true); }); - test('ComesNodeDevSecret → true (weak)', () => { - expect(isJwtSecretWeak('ComesNodeDevSecret')).toBe(true); - }); - - test('MontaineNodeDevSecret → true (weak)', () => { - expect(isJwtSecretWeak('MontaineNodeDevSecret')).toBe(true); - }); - - test('PierrebNodeDevSecret → true (weak)', () => { - expect(isJwtSecretWeak('PierrebNodeDevSecret')).toBe(true); - }); - - test('IsmNodeDevSecret → true (weak)', () => { - expect(isJwtSecretWeak('IsmNodeDevSecret')).toBe(true); + test('ChangeThisDevSecret → true (weak)', () => { + expect(isJwtSecretWeak('ChangeThisDevSecret')).toBe(true); }); // covers the full JWT_DEFAULT_SECRETS set exhaustively diff --git a/lib/helpers/tests/config.validateJwtSecret.unit.tests.js b/lib/helpers/tests/config.validateJwtSecret.unit.tests.js index 4fa558420..1a4910159 100644 --- a/lib/helpers/tests/config.validateJwtSecret.unit.tests.js +++ b/lib/helpers/tests/config.validateJwtSecret.unit.tests.js @@ -4,8 +4,8 @@ * Behaviour matrix: * - prod env + empty secret → throws * - prod env + short secret (<32) → throws - * - prod env + devkit placeholder → throws - * - prod env + downstream default → throws + * - prod env + upstream placeholder → throws + * - prod env + generic placeholder → throws * - prod env + strong secret (≥32) → no throw, no warn * - dev env + default secret → console.log warn, no throw * - test env + default secret → console.log warn, no throw @@ -19,8 +19,8 @@ import configHelper from '../config.js'; const { validateJwtSecret } = configHelper; const STRONG_SECRET = 'a'.repeat(32); // exactly 32 chars, non-default -const DEVKIT_PLACEHOLDER = 'WaosSecretKeyExampleToChnageAbsolutely'; -const DOWNSTREAM_DEFAULT = 'TrawlNodeDevSecret'; // known downstream placeholder (< 32 chars too) +const UPSTREAM_PLACEHOLDER = 'WaosSecretKeyExampleToChnageAbsolutely'; +const GENERIC_PLACEHOLDER = 'ExampleNodeDevSecret'; // known generic placeholder (< 32 chars too) const SHORT_SECRET = 'tooshort'; // < 32 chars, not a known default describe('config.validateJwtSecret', () => { @@ -58,14 +58,14 @@ describe('config.validateJwtSecret', () => { expect(() => validateJwtSecret({ jwt: { secret: SHORT_SECRET } })).toThrow(); }); - test('prod + devkit placeholder → throws', () => { + test('prod + upstream placeholder → throws', () => { process.env.NODE_ENV = 'production'; - expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).toThrow(); + expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).toThrow(); }); - test('prod + downstream default (TrawlNodeDevSecret) → throws', () => { + test('prod + generic placeholder (ExampleNodeDevSecret) → throws', () => { process.env.NODE_ENV = 'production'; - expect(() => validateJwtSecret({ jwt: { secret: DOWNSTREAM_DEFAULT } })).toThrow(); + expect(() => validateJwtSecret({ jwt: { secret: GENERIC_PLACEHOLDER } })).toThrow(); }); test('prod + no jwt key at all → throws', () => { @@ -88,9 +88,9 @@ describe('config.validateJwtSecret', () => { // ---- dev/test/local: warn, never throw -------------------------------- - test('dev env + devkit placeholder → warns (console.log), no throw', () => { + test('dev env + upstream placeholder → warns (console.log), no throw', () => { process.env.NODE_ENV = 'development'; - expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow(); + expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow(); expect(consoleLogSpy).toHaveBeenCalled(); }); @@ -106,15 +106,15 @@ describe('config.validateJwtSecret', () => { expect(consoleLogSpy).toHaveBeenCalled(); }); - test('test env + devkit placeholder → warns (console.log), no throw', () => { + test('test env + upstream placeholder → warns (console.log), no throw', () => { process.env.NODE_ENV = 'test'; - expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow(); + expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow(); expect(consoleLogSpy).toHaveBeenCalled(); }); - test('local env + devkit placeholder → warns (console.log), no throw', () => { + test('local env + upstream placeholder → warns (console.log), no throw', () => { process.env.NODE_ENV = 'local'; - expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow(); + expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow(); expect(consoleLogSpy).toHaveBeenCalled(); }); diff --git a/lib/middlewares/posthog-context.middleware.js b/lib/middlewares/posthog-context.middleware.js index 380be66cf..f1fc9c295 100644 --- a/lib/middlewares/posthog-context.middleware.js +++ b/lib/middlewares/posthog-context.middleware.js @@ -5,17 +5,33 @@ * attaches a `posthogContext` object to the request for downstream use * (e.g. enriching analytics events with CLI vs web attribution). * - * Detection: `@trawlme/cli/` in UA → source: 'cli', cli_version: '' - * Everything else (browser, curl, unknown) → source: 'web' + * Detection is config-driven: `config.analytics.cliUserAgentPattern` is a + * regex-source string whose first capture group is the CLI version. When the + * pattern is unset/empty, no CLI detection happens and the source stays 'web'. */ -const CLI_UA_RE = /@trawlme\/cli\/(\S+)/; +import config from '../../config/index.js'; + +/** + * Build the CLI User-Agent matcher from config, or `null` when unconfigured. + * + * @returns {RegExp|null} compiled regex, or `null` if no/invalid pattern is set + */ +const getCliUaRe = () => { + const pattern = config.analytics?.cliUserAgentPattern; + if (!pattern) return null; + try { + return new RegExp(pattern); + } catch { + return null; + } +}; /** * Attach PostHog context to every request based on the User-Agent header. * * Sets `req.posthogContext` with: - * - `source`: `'cli'` when `@trawlme/cli/` is detected, `'web'` otherwise + * - `source`: `'cli'` when the configured CLI user-agent is detected, `'web'` otherwise * - `cli_version`: CLI version string (only present when source is `'cli'`) * * @param {import('express').Request} req - Express request @@ -25,7 +41,8 @@ const CLI_UA_RE = /@trawlme\/cli\/(\S+)/; */ export const posthogContextMiddleware = (req, _res, next) => { const ua = req.get('User-Agent') || ''; - const match = ua.match(CLI_UA_RE); + const cliUaRe = getCliUaRe(); + const match = cliUaRe ? ua.match(cliUaRe) : null; req.posthogContext = match ? { source: 'cli', cli_version: match[1] } : { source: 'web' }; diff --git a/lib/middlewares/tests/posthog-context.middleware.unit.tests.js b/lib/middlewares/tests/posthog-context.middleware.unit.tests.js index 15fceb7ac..a14e03603 100644 --- a/lib/middlewares/tests/posthog-context.middleware.unit.tests.js +++ b/lib/middlewares/tests/posthog-context.middleware.unit.tests.js @@ -1,17 +1,36 @@ /** * Module dependencies. */ -import { jest, describe, test, expect, beforeEach } from '@jest/globals'; -import { posthogContextMiddleware } from '../posthog-context.middleware.js'; +import { jest, describe, test, expect, beforeEach, afterEach } from '@jest/globals'; /** * Unit tests for posthog-context middleware. - * Verifies User-Agent parsing for CLI vs web source attribution: + * Verifies config-driven User-Agent parsing for CLI vs web source attribution: * 1. CLI UA with version → source:'cli', cli_version:'' * 2. CLI UA without explicit version segment → source:'cli' fallback * 3. Web browser UA → source:'web' * 4. Missing UA → source:'web' + * 5. Unset CLI pattern → no detection (source always 'web') */ + +// Neutral CLI user-agent pattern used by the configured-detection suite. +const CLI_UA_PATTERN = '@example-org/cli/(\\S+)'; + +/** + * Load the middleware with config mocked to a given CLI user-agent pattern. + * + * @param {string} cliUserAgentPattern - regex-source string (empty → no detection) + * @returns {Promise} the posthogContextMiddleware bound to that config + */ +const loadMiddleware = async (cliUserAgentPattern) => { + jest.resetModules(); + jest.unstable_mockModule('../../../config/index.js', () => ({ + default: { analytics: { cliUserAgentPattern } }, + })); + const { posthogContextMiddleware } = await import('../posthog-context.middleware.js'); + return posthogContextMiddleware; +}; + describe('posthogContextMiddleware unit tests:', () => { let req; let res; @@ -25,60 +44,89 @@ describe('posthogContextMiddleware unit tests:', () => { next = jest.fn(); }); - test('CLI UA with version → source:cli + cli_version', () => { - req.get.mockReturnValue('@trawlme/cli/1.2.3'); - posthogContextMiddleware(req, res, next); - - expect(req.posthogContext).toEqual({ source: 'cli', cli_version: '1.2.3' }); - expect(next).toHaveBeenCalledTimes(1); + afterEach(() => { + jest.restoreAllMocks(); }); - test('CLI UA with pre-release version → source:cli + cli_version', () => { - req.get.mockReturnValue('@trawlme/cli/2.0.0-beta.1 node/22.0.0'); - posthogContextMiddleware(req, res, next); + describe('with a configured CLI user-agent pattern:', () => { + let posthogContextMiddleware; - expect(req.posthogContext).toEqual({ source: 'cli', cli_version: '2.0.0-beta.1' }); - expect(next).toHaveBeenCalledTimes(1); - }); + beforeEach(async () => { + posthogContextMiddleware = await loadMiddleware(CLI_UA_PATTERN); + }); - test('web browser UA → source:web (no cli_version)', () => { - req.get.mockReturnValue('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'); - posthogContextMiddleware(req, res, next); + test('CLI UA with version → source:cli + cli_version', () => { + req.get.mockReturnValue('@example-org/cli/1.2.3'); + posthogContextMiddleware(req, res, next); - expect(req.posthogContext).toEqual({ source: 'web' }); - expect(req.posthogContext).not.toHaveProperty('cli_version'); - expect(next).toHaveBeenCalledTimes(1); - }); + expect(req.posthogContext).toEqual({ source: 'cli', cli_version: '1.2.3' }); + expect(next).toHaveBeenCalledTimes(1); + }); - test('missing User-Agent header → source:web (no cli_version)', () => { - req.get.mockReturnValue(undefined); - posthogContextMiddleware(req, res, next); + test('CLI UA with pre-release version → source:cli + cli_version', () => { + req.get.mockReturnValue('@example-org/cli/2.0.0-beta.1 node/22.0.0'); + posthogContextMiddleware(req, res, next); - expect(req.posthogContext).toEqual({ source: 'web' }); - expect(req.posthogContext).not.toHaveProperty('cli_version'); - expect(next).toHaveBeenCalledTimes(1); - }); + expect(req.posthogContext).toEqual({ source: 'cli', cli_version: '2.0.0-beta.1' }); + expect(next).toHaveBeenCalledTimes(1); + }); - test('empty User-Agent header → source:web', () => { - req.get.mockReturnValue(''); - posthogContextMiddleware(req, res, next); + test('web browser UA → source:web (no cli_version)', () => { + req.get.mockReturnValue('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'); + posthogContextMiddleware(req, res, next); - expect(req.posthogContext).toEqual({ source: 'web' }); - expect(next).toHaveBeenCalledTimes(1); - }); + expect(req.posthogContext).toEqual({ source: 'web' }); + expect(req.posthogContext).not.toHaveProperty('cli_version'); + expect(next).toHaveBeenCalledTimes(1); + }); + + test('missing User-Agent header → source:web (no cli_version)', () => { + req.get.mockReturnValue(undefined); + posthogContextMiddleware(req, res, next); + + expect(req.posthogContext).toEqual({ source: 'web' }); + expect(req.posthogContext).not.toHaveProperty('cli_version'); + expect(next).toHaveBeenCalledTimes(1); + }); + + test('empty User-Agent header → source:web', () => { + req.get.mockReturnValue(''); + posthogContextMiddleware(req, res, next); + + expect(req.posthogContext).toEqual({ source: 'web' }); + expect(next).toHaveBeenCalledTimes(1); + }); - test('curl UA → source:web', () => { - req.get.mockReturnValue('curl/8.7.1'); - posthogContextMiddleware(req, res, next); + test('curl UA → source:web', () => { + req.get.mockReturnValue('curl/8.7.1'); + posthogContextMiddleware(req, res, next); - expect(req.posthogContext).toEqual({ source: 'web' }); - expect(next).toHaveBeenCalledTimes(1); + expect(req.posthogContext).toEqual({ source: 'web' }); + expect(next).toHaveBeenCalledTimes(1); + }); + + test('always calls next()', () => { + req.get.mockReturnValue('@example-org/cli/0.1.0'); + posthogContextMiddleware(req, res, next); + + expect(next).toHaveBeenCalledWith(); // called with no args (no error) + }); }); - test('always calls next()', () => { - req.get.mockReturnValue('@trawlme/cli/0.1.0'); - posthogContextMiddleware(req, res, next); + describe('with no CLI user-agent pattern configured:', () => { + let posthogContextMiddleware; + + beforeEach(async () => { + posthogContextMiddleware = await loadMiddleware(''); + }); + + test('a CLI-looking UA still resolves to source:web (detection disabled)', () => { + req.get.mockReturnValue('@example-org/cli/1.2.3'); + posthogContextMiddleware(req, res, next); - expect(next).toHaveBeenCalledWith(); // called with no args (no error) + expect(req.posthogContext).toEqual({ source: 'web' }); + expect(req.posthogContext).not.toHaveProperty('cli_version'); + expect(next).toHaveBeenCalledTimes(1); + }); }); }); diff --git a/lib/services/tests/analytics.capture.unit.tests.js b/lib/services/tests/analytics.capture.unit.tests.js index b01d229d4..84519dad7 100644 --- a/lib/services/tests/analytics.capture.unit.tests.js +++ b/lib/services/tests/analytics.capture.unit.tests.js @@ -291,7 +291,7 @@ describe('Analytics capture() and enabled-flag:', () => { PostHog: jest.fn().mockImplementation(() => mockPostHogInstance), })); jest.unstable_mockModule('../../../config/index.js', () => ({ - default: { analytics: { posthog: { enabled: true, key: 'phc_test', host: 'https://eu.i.posthog.com', appTag: 'trawl' } } }, + default: { analytics: { posthog: { enabled: true, key: 'phc_test', host: 'https://eu.i.posthog.com', appTag: 'myapp' } } }, })); const mod = await import('../analytics.js'); AnalyticsService = mod.default; diff --git a/lib/services/tests/analytics.captureException.unit.tests.js b/lib/services/tests/analytics.captureException.unit.tests.js index 4782c3485..1ab918809 100644 --- a/lib/services/tests/analytics.captureException.unit.tests.js +++ b/lib/services/tests/analytics.captureException.unit.tests.js @@ -22,7 +22,7 @@ describe('Analytics captureException():', () => { PostHog: jest.fn().mockImplementation(() => mockPostHogInstance), })); jest.unstable_mockModule('../../../config/index.js', () => ({ - default: { analytics: { posthog: { enabled: true, key: 'phc_test', host: 'https://eu.i.posthog.com', appTag: 'trawl' } } }, + default: { analytics: { posthog: { enabled: true, key: 'phc_test', host: 'https://eu.i.posthog.com', appTag: 'myapp' } } }, })); const mod = await import('../analytics.js'); AnalyticsService = mod.default; diff --git a/lib/services/tests/express.openapi-servers.unit.tests.js b/lib/services/tests/express.openapi-servers.unit.tests.js index 0df52435e..22ec11221 100644 --- a/lib/services/tests/express.openapi-servers.unit.tests.js +++ b/lib/services/tests/express.openapi-servers.unit.tests.js @@ -68,7 +68,7 @@ describe('lib/services/express — OpenAPI servers.url derivation:', () => { test('prepends https://api. when domain is a bare host (no scheme)', async () => { const fn = await getComputeOpenApiServerUrl(); - expect(fn('trawl.me')).toBe('https://api.trawl.me'); + expect(fn('acme.dev')).toBe('https://api.acme.dev'); }); test('prepends https://api. for any bare host', async () => { diff --git a/modules/billing/README.md b/modules/billing/README.md index 3d97d1ba0..9735ec346 100644 --- a/modules/billing/README.md +++ b/modules/billing/README.md @@ -6,7 +6,7 @@ Stripe-based billing with per-plan quota management and meter-based compute pric `billing.meter.service.js` converts a feature-keyed **USD cost map → meter units** via config ratios (`dollarsToUnitRatio`, per-plan `ratios`) and applies config knobs (`runBase`, `maxUnitsPerOperation`). It does **not** know what a run costs. -**Downstream cost semantics — what a scrape/op costs, per-run infra base, product-specific floors/caps — live in the downstream's own cost module + config (e.g. Trawl `modules/costs`), never inline in this service.** An inline downstream patch here is silently wiped by `/update-stack`: a tier0 run-base floor added downstream inside a `billing.meter.service.js` copy was lost on the next stack sync, zeroing metering for free-tier scrapes (trawl `#1293` → `#1316`). If a behaviour must live in this service, add it as a **default-off config knob**, never a hardcoded downstream rule. +**Downstream cost semantics — what an op costs, per-run infra base, product-specific floors/caps — live in the downstream's own cost module + config (e.g. a `modules/costs`), never inline in this service.** An inline downstream patch here is silently wiped by `/update-stack`: a run-base floor added downstream inside a `billing.meter.service.js` copy was lost on the next stack sync, zeroing metering for free-tier usage. If a behaviour must live in this service, add it as a **default-off config knob**, never a hardcoded downstream rule. ## Quota System diff --git a/modules/billing/RUNBOOKS.md b/modules/billing/RUNBOOKS.md index 3a6bafb05..7fb42032c 100644 --- a/modules/billing/RUNBOOKS.md +++ b/modules/billing/RUNBOOKS.md @@ -41,7 +41,7 @@ Operational runbooks for the billing module. Each runbook references real endpoi Example curl: ```bash - curl -X POST https://api.trawl.me/api/admin/billing/dispute/credit/ \ + curl -X POST https://api.example.com/api/admin/billing/dispute/credit/ \ -H "Authorization: Bearer $ADMIN_JWT" \ -H "Content-Type: application/json" \ -d '{"chargeId":"ch_xxx","amountCents":2000,"reason":"dispute won — Stripe reinstated funds","refundRequestId":""}' @@ -141,10 +141,10 @@ Operational runbooks for the billing module. Each runbook references real endpoi - [ ] Stripe Dashboard (LIVE mode): 10 webhook events enabled (see `STRIPE_SETUP.md`) - [ ] Stripe Dashboard (LIVE mode): Smart Retries enabled (Billing settings → Smart Retries) - [ ] Stripe Dashboard (LIVE mode): `tax_id` collection enabled in Checkout (B2B EU) -- [ ] `STRIPE_SECRET_KEY` = `sk_live_*` set in K8s secret `trawl-node-env` +- [ ] `STRIPE_SECRET_KEY` = `sk_live_*` set in the deployment secret (e.g. K8s secret `-node-env`) - [ ] `STRIPE_WEBHOOK_SECRET` = `whsec_*` (LIVE mode endpoint secret) updated in K8s secret - [ ] `STRIPE_PRICE_*` env vars point to LIVE price IDs (not test price IDs) -- [ ] All 4 CronJob manifests deployed: `trawl-billing-dunning-sweep`, `trawl-billing-weekly-reset`, `trawl-billing-extras-expiration`, `trawl-billing-reconcile` +- [ ] All 4 CronJob manifests deployed: `-billing-dunning-sweep`, `-billing-weekly-reset`, `-billing-extras-expiration`, `-billing-reconcile` - [ ] Dead-letter queue empty: `GET /api/admin/billing/dead-letters` → 0 entries - [ ] Test mode webhooks drained: Stripe Dashboard → Webhooks → no pending test deliveries - [ ] Smoke test: in staging pointed at **TEST** Stripe keys (not LIVE), create a checkout session using Stripe test card `4242 4242 4242 4242` — confirm `checkout.session.completed` webhook received + subscription created in DB. Do **not** use test cards against LIVE keys (they are rejected; use this step to validate the integration flow, then cut over to LIVE keys for production) diff --git a/modules/billing/middlewares/billing.attachUsageContext.js b/modules/billing/middlewares/billing.attachUsageContext.js index 5984f3e7f..f4a97df04 100644 --- a/modules/billing/middlewares/billing.attachUsageContext.js +++ b/modules/billing/middlewares/billing.attachUsageContext.js @@ -17,7 +17,7 @@ import BillingExtraBalanceRepository from '../repositories/billing.extraBalance. * Mount after `resolveOrganization` so that `req.organization` is populated. * * NOTE: This middleware is documented here but intentionally NOT auto-wired in - * lib/app.js — downstream projects (e.g. Trawl) decide where to mount it. + * lib/app.js — downstream projects decide where to mount it. * Example: `app.use(resolveOrganization, attachUsageContext);` * * @param {import('express').Request} req - Express request object. diff --git a/modules/billing/middlewares/billing.requirePlan.js b/modules/billing/middlewares/billing.requirePlan.js index 1a921bc4c..4813a1107 100644 --- a/modules/billing/middlewares/billing.requirePlan.js +++ b/modules/billing/middlewares/billing.requirePlan.js @@ -17,7 +17,7 @@ const ACTIVE_STATUSES = new Set(['active', 'trialing']); * A plan is only effective when subscription.status ∈ {active, trialing}. * canceled / past_due / unpaid / incomplete are treated as 'free'. * - * Response shape on deny matches the downstream trawl_node contract (top-level errorCode): + * Response shape on deny follows the downstream consumer contract (top-level errorCode): * { type: 'error', message: 'Forbidden', code: 403, status: 403, * errorCode: 'PLAN_REQUIRED', description, requiredPlans: string[], currentPlan: string } * diff --git a/modules/billing/repositories/billing.subscription.repository.js b/modules/billing/repositories/billing.subscription.repository.js index f9925c931..4f9aa141a 100644 --- a/modules/billing/repositories/billing.subscription.repository.js +++ b/modules/billing/repositories/billing.subscription.repository.js @@ -262,7 +262,7 @@ const updateIfEventNewer = (id, eventCreatedAt, eventId, fields, family = 'subsc [createdAtField]: eventCreatedAt, [eventIdField]: eventId, // Legacy fields stripeEventCreatedAt/stripeEventId are no longer written. - // The migration in trawl_node $unset them post-deploy so docs converge to + // A downstream migration $unsets them post-deploy so docs converge to // per-family markers only. Reading the legacy fields anywhere is dead surface. }, }, diff --git a/modules/billing/services/billing.webhook.service.js b/modules/billing/services/billing.webhook.service.js index 0987252a5..66c966b7d 100644 --- a/modules/billing/services/billing.webhook.service.js +++ b/modules/billing/services/billing.webhook.service.js @@ -736,7 +736,7 @@ const handleInvoicePaymentSucceeded = async (invoice, event) => { * payment_intent_data: { metadata: { organizationId, stripeSessionId, packId, ... } }, * }) * Without payment_intent_data.metadata, charge.metadata will be empty and refunds - * silently skip. Downstream (trawl_node) is responsible for setting these at session creation. + * silently skip. The downstream consumer is responsible for setting these at session creation. * Calls BillingExtraService.refundPartial for each entry in charge.refunds.data. * Each refund's rf_ id is used as the idempotency key, making webhook replay safe. * Individual entries are silently skipped when: metadata is incomplete, refund amount diff --git a/modules/core/tests/core.unit.tests.js b/modules/core/tests/core.unit.tests.js index c346528c0..a5d2e0706 100644 --- a/modules/core/tests/core.unit.tests.js +++ b/modules/core/tests/core.unit.tests.js @@ -85,7 +85,7 @@ describe('Core unit tests:', () => { it('assertSafeEnv should allow valid environment names', () => { expect(() => assertSafeEnv('development')).not.toThrow(); expect(() => assertSafeEnv('production')).not.toThrow(); - expect(() => assertSafeEnv('trawl')).not.toThrow(); + expect(() => assertSafeEnv('acme')).not.toThrow(); expect(() => assertSafeEnv('my-project_v2')).not.toThrow(); }); diff --git a/modules/invitations/README.md b/modules/invitations/README.md index 4281fa403..fe077c32e 100644 --- a/modules/invitations/README.md +++ b/modules/invitations/README.md @@ -47,7 +47,7 @@ from the payload — reward either side, or both. > would clobber them. A downstream project therefore NEVER wires a listener by editing > `billing.init.js`. The two sanctioned channels are: **config** (deep-merged > `{project}.config.js` — for the standard reward below) and **project-only modules** -> (glob-discovered, e.g. `modules/trawl-rewards/` — for custom logic). +> (glob-discovered, e.g. `modules/-rewards/` — for custom logic). ### A. Standard grant — ships IN the stack, downstream enables it by CONFIG