Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .claude/skills/update-stack/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,6 @@ echo "3ter: no drift — OK"
- Scan source is `git diff --name-only devkit-node/master HEAD` (bidirectional) — catches both files that differ AND files present upstream but missing locally (deleted downstream). The previous `git ls-files` approach only saw locally-present files.
- Test files (paths containing `/tests/` or `/__tests__/`, or filenames ending `.test.{js,jsx,ts,tsx}` / `.spec.{js,jsx,ts,tsx}`) are excluded — downstream test adaptations are acceptable.
- This gate runs **after** `/verify` (never blocks on transient verify failures) and **before** Phase 2 (failure is recoverable — no merge commit yet).
- Ref: plan `2026-05-30-trawl-devkit-perfect-alignment.md` Tasks E.1 + E.2.

---

Expand Down
4 changes: 2 additions & 2 deletions ERRORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Use this file as a compact memory of recurring AI mistakes.
- [2026-03-14] middleware: assuming config section exists in all environments (e.g. `config.rateLimit`) -> always handle missing config gracefully (passthrough/no-op); dev config often omits sections that only prod defines
- [2026-03-15] cross-stack: changing a Node API without checking Vue E2E tests -> when modifying an endpoint Vue consumes, run Vue E2E tests before pushing
- [2026-03-15] pr scope: batching multiple unrelated fixes in one PR -> one fix = one PR to isolate blast radius and reduce iteration loops
- [2026-05-05] repository: Repository.update(doc) doing `new Model(doc).save()` rewrites the full document from in-memory state, silently clobbering any concurrent partial update that landed after the read -> always use `Model.updateOne({ _id }, { $set: ... })` or `findOneAndUpdate({ _id }, { $set: ... })` for partial updates to avoid race conditions; see comes-io/trawl_node#1115 comes-io/trawl_node#1116 comes-io/trawl_node#1118 + pierreb-devkit/Node#3605
- [2026-05-05] repository: Repository.update(doc) doing `new Model(doc).save()` rewrites the full document from in-memory state, silently clobbering any concurrent partial update that landed after the read -> always use `Model.updateOne({ _id }, { $set: ... })` or `findOneAndUpdate({ _id }, { $set: ... })` for partial updates to avoid race conditions; see pierreb-devkit/Node#3605
- [2026-05-31] billing/stripe: reading `price.metadata.planId` in `customer.subscription.updated` webhook handler -> field is EMPTY in real Stripe webhook payloads (planId lives on the Product, not the Price); use a `priceId → plan` map built at boot from `config.stripe.prices` instead; see pierreb-devkit/Node#3742
- [2026-06-04] repository: top-level `const Foo = mongoose.model('Foo')` in a repository file -> this is evaluated at import time; safe in an HTTP server (loadModels() runs first) but silently crashes standalone scripts (crons, migrations) with `MissingSchemaError` when import order differs; tests miss it because jest mocks intercept the module entirely; fix = lazy getter `const Foo = () => mongoose.model('Foo')` (call sites: `Foo().find(...)`) or dynamic import after `loadModels()` in the entrypoint; see comes-io/trawl_node#1337 comes-io/trawl_node#1338 pierreb-devkit/Node#3789
- [2026-06-04] repository: top-level `const Foo = mongoose.model('Foo')` in a repository file -> this is evaluated at import time; safe in an HTTP server (loadModels() runs first) but silently crashes standalone scripts (crons, migrations) with `MissingSchemaError` when import order differs; tests miss it because jest mocks intercept the module entirely; fix = lazy getter `const Foo = () => mongoose.model('Foo')` (call sites: `Foo().find(...)`) or dynamic import after `loadModels()` in the entrypoint; see pierreb-devkit/Node#3789
- [2026-06-15] deps/audit: leaving `npm audit` advisories unaddressed on the assumption they need a major bump -> run `npm audit fix` (never `--force`) first; the runtime-tree DoS/ReDoS items (`qs`, `path-to-regexp`, `brace-expansion`) all fixed via in-range bumps, no residual. These are DoS-class but NOT attacker-reachable in this stack: Express route patterns are static (no user-controlled `path-to-regexp` input) and `qs`/`brace-expansion` only parse server-side query strings under fixed code paths — still bump them to keep the tree clean and avoid scanner noise.
14 changes: 5 additions & 9 deletions MIGRATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ Cf `infra/docs/superpowers/plans/2026-05-10-posthog-observability-followups.md`

## Test DB isolation: per-pid Mongo database default + globalTeardown (2026-04-24)

Default test database is now `mongodb://127.0.0.1:27017/NodeTest_${process.pid}` instead of the shared `NodeTest`. Concurrent jest invocations (e.g. multiple agent worktrees running `npm run test:coverage` in parallel) get isolated databases, eliminating the 401 / 404 / 422 / `MongoPoolClosedError` flake patterns documented in trawl_node#980.
Default test database is now `mongodb://127.0.0.1:27017/NodeTest_${process.pid}` instead of the shared `NodeTest`. Concurrent jest invocations (e.g. multiple agent worktrees running `npm run test:coverage` in parallel) get isolated databases, eliminating the 401 / 404 / 422 / `MongoPoolClosedError` flake patterns seen in parallel runs.

### What changed
Comment on lines 162 to 166

Expand Down Expand Up @@ -195,17 +195,13 @@ CI workflows (`.github/workflows/CI.yml` and downstream copies) set `DEVKIT_NODE

When this lands in your project via `/update-stack`, the new `parallel-smoke` CI job ships a default `SMOKE_TEST_PATTERN` of `organizations.integration|tasks.integration` — which only matches in the upstream Devkit. **You MUST override `SMOKE_TEST_PATTERN`** in your CI `parallel-smoke` job (set it under the job's `env:` in `.github/workflows/CI.yml`) to match your project's integration test paths.

The 5 downstream Node projects that consume this stack must each set the override:
Each downstream Node project that consumes this stack must set the override:

| Project | Suggested `SMOKE_TEST_PATTERN` |
|---|---|
| `pierreb_node` | project-specific integration globs |
| `comes_node` | `tasks.integration\|notes.integration` |
| `trawl_node` | `scraps.integration\|historys.integration` |
| `montaine_node` | project-specific integration globs |
| `ism_node` | project-specific integration globs |
| `<project>_node` | project-specific integration globs (e.g. `foo.integration\|bar.integration`) |

(The exact globs are illustrative — replace with whatever integration files actually exist in each repo. The point is: pick at least two real integration suites so the parallel-smoke job exercises the per-pid DB isolation rather than passing on zero matches.)
(The exact globs are illustrative — replace with whatever integration files actually exist in your repo. The point is: pick at least two real integration suites so the parallel-smoke job exercises the per-pid DB isolation rather than passing on zero matches.)

Without an override, the smoke would historically have silently passed with 0 tests run, defeating the regression gate. As of #3518 the orchestrator passes `--passWithNoTests=false` to jest, so a 0-match pattern now exits non-zero and fails the smoke loudly — but the actionable fix is still to point the pattern at real integration paths in your repo.

Expand Down Expand Up @@ -777,7 +773,7 @@ All features are no-op when `apiKey` is empty — safe to deploy without PostHog

## Organizations & CASL v2 (2026-03-13)

This guide is for downstream projects (e.g. lou-node, pierreb-node) migrating to the new organizations + CASL document-level authorization system introduced on the `feature/signup-org-flow` branch.
This guide is for downstream projects migrating to the new organizations + CASL document-level authorization system introduced on the `feature/signup-org-flow` branch.

---

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ config/defaults/
myproject.config.js ← global project overrides (all modules)

modules/<name>/config/
<name>.myproject.config.js ← per-module project overrides (e.g. users.trawl.config.js)
<name>.myproject.config.js ← per-module project overrides (e.g. users.<project>.config.js)
```

Both file types are optional and can be used independently or together. Per-module files take priority over the global project config, allowing fine-grained overrides per module without polluting the global file.
Expand Down Expand Up @@ -312,9 +312,9 @@ Canonical sources used downstream:
| Source | Meaning |
|---|---|
| `web` | Request from browser (UA not matched as CLI) |
| `cli` | Request from `@trawlme/cli/<version>` (UA-parsed) |
| `cli` | Request from a configured CLI user-agent (UA-parsed, see `analytics.cliUserAgentPattern`) |
| `stripe-webhook` | Stripe POST `/api/billing/webhook` |
| `worker-callback` | worker-puppeteer scrap completion callback |
| `worker-callback` | Background worker completion callback |
| `cron` | Scheduled background job |
| `system` | Server-side fallback (no req, no caller override) |

Expand Down
4 changes: 2 additions & 2 deletions config/defaults/billing.pricing.constants.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
* Why ship from devkit:
* - Every downstream running billing wants the same export shape.
* - Migrations + contract tests + costs service all benefit from a single import path.
* - Trawl had this file at `modules/billing/config/billing.pricing.constants.js` with
* 6+ importers — promoted upstream in plan `2026-06-02-trawl-billing-residual-cleanup.md`.
* - Previously lived per-project at `modules/billing/config/billing.pricing.constants.js`
* with several importers — promoted upstream to a single import path.
*
* @module billing.pricing.constants
*/
Expand Down
5 changes: 5 additions & 0 deletions config/defaults/development.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,11 @@ const config = {
errorTracking: process.env.DEVKIT_NODE_analytics_posthog_errorTracking === 'true',
autoCapture: process.env.DEVKIT_NODE_analytics_posthog_autoCapture === 'true',
},
// Regex-source string used by the PostHog context middleware to detect a
// CLI client from its User-Agent (capture group 1 = version). Empty → no
// CLI detection (source stays 'web'). A project that ships a CLI sets this
// to e.g. '@example-org/cli/(\\S+)'.
cliUserAgentPattern: process.env.DEVKIT_NODE_analytics_cliUserAgentPattern ?? '',
},
domain: '',
cookie: {
Expand Down
2 changes: 1 addition & 1 deletion config/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ const initGlobalConfig = async () => {
}

// Layer 3.5: per-module project overrides (modules/*/config/*.{project}.config.js)
// Only applies for non-standard envs (i.e. downstream project names like "trawl", "comes")
// Only applies for non-standard envs (i.e. a downstream project name, the NODE_ENV value)
if (!STANDARD_ENVS.has(env)) {
const moduleProjectPattern = `modules/*/config/*.${env}.config.js`;
const moduleProjectFiles = await configHelper.getGlobbedPaths(moduleProjectPattern);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ modules/billing/tests/billing.cron.*.unit.tests.js

This is an atomic move with no backward-compat shim. The cutover for each downstream project requires coordinated steps:

1. Run `/update-stack` on the downstream project repo (e.g. trawl_node) — pulls the new structure with crons at `modules/billing/crons/`.
1. Run `/update-stack` on the downstream project repo — pulls the new structure with crons at `modules/billing/crons/`.
2. CI on the downstream repo builds a new image and pushes to GHCR.
3. Update the infra K8s CronJob manifests in `clusters/{cluster}/apps/{project}-billing-*.yaml` — change `args: ["scripts/crons/billing.*.js"]` to `args: ["modules/billing/crons/billing.*.js"]`.
4. Push the infra change → Flux applies → CronJobs use new args + new image together.
Expand Down
13 changes: 5 additions & 8 deletions lib/helpers/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,14 @@ const validateDomainIsSet = (config) => {

/**
* Known default / placeholder JWT secret values that must never be used in
* non-dev environments. Extend this list when a new downstream project is
* bootstrapped with its own placeholder.
* non-dev environments. Extend this list when a new project is bootstrapped
* with its own placeholder secret.
* @readonly
*/
const JWT_DEFAULT_SECRETS = Object.freeze(new Set([
'WaosSecretKeyExampleToChnageAbsolutely', // devkit upstream placeholder
'TrawlNodeDevSecret', // trawl downstream placeholder
'ComesNodeDevSecret', // comes downstream placeholder
'MontaineNodeDevSecret', // montaine downstream placeholder
'PierrebNodeDevSecret', // pierreb downstream placeholder
'IsmNodeDevSecret', // ism downstream placeholder
'WaosSecretKeyExampleToChnageAbsolutely', // upstream placeholder
'ExampleNodeDevSecret', // generic project placeholder
'ChangeThisDevSecret', // generic placeholder
]));

/**
Expand Down
24 changes: 6 additions & 18 deletions lib/helpers/tests/config.isJwtSecretWeak.unit.tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
* Weak cases (returns true):
* - undefined / null / empty string / whitespace-only
* - length < 32 characters
* - each known default placeholder (devkit + all downstream)
* - each known default placeholder
*
* Strong case (returns false):
* - a ≥ 32-char string that is not in the defaults list
Expand Down Expand Up @@ -51,28 +51,16 @@ describe('config.isJwtSecretWeak', () => {

// ---- weak: each known default placeholder ---------------------------------

test('devkit placeholder → true (weak)', () => {
test('upstream placeholder → true (weak)', () => {
expect(isJwtSecretWeak('WaosSecretKeyExampleToChnageAbsolutely')).toBe(true);
});

test('TrawlNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('TrawlNodeDevSecret')).toBe(true);
test('ExampleNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('ExampleNodeDevSecret')).toBe(true);
});

test('ComesNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('ComesNodeDevSecret')).toBe(true);
});

test('MontaineNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('MontaineNodeDevSecret')).toBe(true);
});

test('PierrebNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('PierrebNodeDevSecret')).toBe(true);
});

test('IsmNodeDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('IsmNodeDevSecret')).toBe(true);
test('ChangeThisDevSecret → true (weak)', () => {
expect(isJwtSecretWeak('ChangeThisDevSecret')).toBe(true);
});

// covers the full JWT_DEFAULT_SECRETS set exhaustively
Expand Down
28 changes: 14 additions & 14 deletions lib/helpers/tests/config.validateJwtSecret.unit.tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
* Behaviour matrix:
* - prod env + empty secret → throws
* - prod env + short secret (<32) → throws
* - prod env + devkit placeholder → throws
* - prod env + downstream default → throws
* - prod env + upstream placeholder → throws
* - prod env + generic placeholder → throws
* - prod env + strong secret (≥32) → no throw, no warn
* - dev env + default secret → console.log warn, no throw
* - test env + default secret → console.log warn, no throw
Expand All @@ -19,8 +19,8 @@ import configHelper from '../config.js';
const { validateJwtSecret } = configHelper;

const STRONG_SECRET = 'a'.repeat(32); // exactly 32 chars, non-default
const DEVKIT_PLACEHOLDER = 'WaosSecretKeyExampleToChnageAbsolutely';
const DOWNSTREAM_DEFAULT = 'TrawlNodeDevSecret'; // known downstream placeholder (< 32 chars too)
const UPSTREAM_PLACEHOLDER = 'WaosSecretKeyExampleToChnageAbsolutely';
const GENERIC_PLACEHOLDER = 'ExampleNodeDevSecret'; // known generic placeholder (< 32 chars too)
const SHORT_SECRET = 'tooshort'; // < 32 chars, not a known default

describe('config.validateJwtSecret', () => {
Expand Down Expand Up @@ -58,14 +58,14 @@ describe('config.validateJwtSecret', () => {
expect(() => validateJwtSecret({ jwt: { secret: SHORT_SECRET } })).toThrow();
});

test('prod + devkit placeholder → throws', () => {
test('prod + upstream placeholder → throws', () => {
process.env.NODE_ENV = 'production';
expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).toThrow();
expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).toThrow();
});

test('prod + downstream default (TrawlNodeDevSecret) → throws', () => {
test('prod + generic placeholder (ExampleNodeDevSecret) → throws', () => {
process.env.NODE_ENV = 'production';
expect(() => validateJwtSecret({ jwt: { secret: DOWNSTREAM_DEFAULT } })).toThrow();
expect(() => validateJwtSecret({ jwt: { secret: GENERIC_PLACEHOLDER } })).toThrow();
});

test('prod + no jwt key at all → throws', () => {
Expand All @@ -88,9 +88,9 @@ describe('config.validateJwtSecret', () => {

// ---- dev/test/local: warn, never throw --------------------------------

test('dev env + devkit placeholder → warns (console.log), no throw', () => {
test('dev env + upstream placeholder → warns (console.log), no throw', () => {
process.env.NODE_ENV = 'development';
expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow();
expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow();
expect(consoleLogSpy).toHaveBeenCalled();
});

Expand All @@ -106,15 +106,15 @@ describe('config.validateJwtSecret', () => {
expect(consoleLogSpy).toHaveBeenCalled();
});

test('test env + devkit placeholder → warns (console.log), no throw', () => {
test('test env + upstream placeholder → warns (console.log), no throw', () => {
process.env.NODE_ENV = 'test';
expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow();
expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow();
expect(consoleLogSpy).toHaveBeenCalled();
});

test('local env + devkit placeholder → warns (console.log), no throw', () => {
test('local env + upstream placeholder → warns (console.log), no throw', () => {
process.env.NODE_ENV = 'local';
expect(() => validateJwtSecret({ jwt: { secret: DEVKIT_PLACEHOLDER } })).not.toThrow();
expect(() => validateJwtSecret({ jwt: { secret: UPSTREAM_PLACEHOLDER } })).not.toThrow();
expect(consoleLogSpy).toHaveBeenCalled();
});

Expand Down
27 changes: 22 additions & 5 deletions lib/middlewares/posthog-context.middleware.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,33 @@
* attaches a `posthogContext` object to the request for downstream use
* (e.g. enriching analytics events with CLI vs web attribution).
*
* Detection: `@trawlme/cli/<version>` in UA → source: 'cli', cli_version: '<version>'
* Everything else (browser, curl, unknown) → source: 'web'
* Detection is config-driven: `config.analytics.cliUserAgentPattern` is a
* regex-source string whose first capture group is the CLI version. When the
* pattern is unset/empty, no CLI detection happens and the source stays 'web'.
*/

const CLI_UA_RE = /@trawlme\/cli\/(\S+)/;
import config from '../../config/index.js';

/**
* Build the CLI User-Agent matcher from config, or `null` when unconfigured.
*
* @returns {RegExp|null} compiled regex, or `null` if no/invalid pattern is set
*/
const getCliUaRe = () => {
const pattern = config.analytics?.cliUserAgentPattern;
if (!pattern) return null;
try {
return new RegExp(pattern);
} catch {
return null;
}
};
Comment on lines +15 to +28

/**
* Attach PostHog context to every request based on the User-Agent header.
*
* Sets `req.posthogContext` with:
* - `source`: `'cli'` when `@trawlme/cli/<version>` is detected, `'web'` otherwise
* - `source`: `'cli'` when the configured CLI user-agent is detected, `'web'` otherwise
* - `cli_version`: CLI version string (only present when source is `'cli'`)
*
* @param {import('express').Request} req - Express request
Expand All @@ -25,7 +41,8 @@ const CLI_UA_RE = /@trawlme\/cli\/(\S+)/;
*/
export const posthogContextMiddleware = (req, _res, next) => {
const ua = req.get('User-Agent') || '';
const match = ua.match(CLI_UA_RE);
const cliUaRe = getCliUaRe();
const match = cliUaRe ? ua.match(cliUaRe) : null;
req.posthogContext = match
? { source: 'cli', cli_version: match[1] }
: { source: 'web' };
Expand Down
Loading
Loading