Repository navigation
Conversation
…dNames Wraps ShadowRepo.getStatus, commitChanges, and detectMovedFiles, plus populateTypesAndNames, in Effect spans with event-loop-delay percentiles. Spans are no-op unless STL_OTEL_SPANS=1, in which case runtime.ts attaches a NodeSdk Layer that writes JSONL to ~/.sf/source-tracking-spans (override via STL_OTEL_DIR). scripts/aggregateSpans.ts aggregates the JSONL into a markdown summary for before/after comparisons. Pins six previously-transitive @effect/platform and @opentelemetry deps explicitly so yarn dedup can't prune them. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pin to exact version (no caret) for predictability since this affects hot paths in source tracking. Span-level perf comparison from local NUT suite (test:nuts:local) across 124 spans on an idle machine: | span | metric | 1.37.5 | 1.38.3 | delta | | -------------------------- | ------ | --------- | --------- | ----- | | getStatus (71 calls) | total | 30,230 ms | 25,275 ms | -16% | | | max | 16,809 ms | 12,200 ms | -27% | | detectMovedFiles (38) | total | 17,378 ms | 12,702 ms | -27% | | | max | 16,685 ms | 11,977 ms | -28% | | commitChanges (15) | total | 68,273 ms | 64,827 ms | -5% | Net win on 2k-file workloads (statusMatrix and move detection ~27% faster); 200k-file extremes are roughly flat (+3-9%). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ManagedRuntime.runPromise rejects with FiberFailure for both typed failures and defects, breaking callers that branch on `e instanceof SfError` or read `e.name`. Throwing inside `Effect.sync` (which ShadowRepo's `Effect.catchAll(e => Effect.sync(() => redirectToCliRepoError(e)))` relies on) is captured as a defect, so the SfError-with-actions text that runGitAdd builds for ulimit failures was reaching CLI consumers wrapped in FiberFailure and falling through to the generic error path. Adds a `runPromise` helper in runtime.ts that runs via runPromiseExit, extracts the original failure or defect from the Cause, and rethrows it directly. Switches all five existing call sites (ShadowRepo getStatus/commitChanges/detectMovedFiles, conflicts.ts, sourceTracking.ts) to the new helper. Adds test/unit/shared/runtime.test.ts pinning the contract for typed failure, defect, tryPromise rejection, and the catchAll(Effect.sync(throw)) shape used by ShadowRepo. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces `Stream.fromIterable + runFoldEffect + Effect.reduce` over
persistent `HashMap`/`HashSet` (with `Data.struct` keying) with a plain
`Array.reduce` over native `Map`/`Set`. The fold is purely synchronous
(no Effect step ever yielded), so the Stream + persistent-collection
machinery was pure allocation overhead — every `HashSet.add` returned a
new HAMT node, paying GC for every claimed file.
Also drops the `applyAndDedupe` HashSet round-trip (Data.struct/Data.array
wrapping just so HashSet could hash structurally) since elementMap
already keys by relative path; reference-identity dedup over the source
ChangeResult preserves the test-pinned "collapse multi-filename bundle
to one entry" behavior without HashSet.fromIterable + HashSet.toValues.
The outer `Effect.fn('populateTypesAndNames')` span is preserved; the
behavioral test in populateTypesAndNames.nut.ts (including the
multi-filename-bundle dedup pin) still passes.
Median wall-time on the existing perf NUT (4 runs each):
| fixture | before | after | delta |
| ----------------------- | ------- | ------- | ----- |
| EDA (6.5k files) | ~186 ms | ~152 ms | -18% |
| synthetic (35k files) | ~333 ms | ~241 ms | -28% |
The 35k case is very stable (±1.5%); the 28% reduction = ~92 ms saved
per call on a worst-case status refresh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First step of the src/git/ replacement for isomorphic-git. Captures the inputs every later phase will assert against: - six fixture .git/ snapshots under test/git/fixtures/, regenerable via generate.sh with pinned author/email/date so commit oids are stable - iso-git baseline numbers in test/git/baselines.json (cold statusMatrix 5.2s, max event-loop delay 572ms, applyChanges-50k 13.4s on 37k files Apple M2 Max). Targets per the migration spec: <=50%, <50ms p99, parity respectively - UNTR-FORMAT.md spike: byte layout lifted from real-git index-format.txt - STATUS-COLLAPSE.md: (head, workdir, stage) -> public StatusEntry union No src/git/ code yet; phase 1 lands the service scaffolding.
Empty Repo service shell with all 9 methods returning Effect.die("not
implemented") or RepoNotConfiguredError pre-switchTo. Plumbs schemas,
tagged errors, and the switchTo handle-swap lifecycle so phases 2+ are
pure additions inside src/git/.
- src/git/schemas.ts: Oid, RepoPath, RefName, StatusEntry, Author,
SwitchCfg with Schema.brand validation
- src/git/errors.ts: 8 Schema.TaggedError types covering the closed set
source-tracking will catchTags against
- src/git/capabilities.ts: Capabilities tag + Node/memfs default layers,
no platform detection inside lite
- src/git/repo.ts: Effect.Service shell with handleRef + swapSemaphore +
per-handle child scope; switchTo atomically swaps handles and closes
the prior scope
- @effect/platform-node added as a devDependency for tests; runtime
consumers will gain it via the integration phase
26 unit tests cover schema brand rejection and the switchTo lifecycle.
hashBlob, writeLooseObject, readLooseObject. Pure web-platform crypto
+ compression — no node:zlib, no node:crypto, no Buffer — so the same
code path lights up in the future web bundle.
- crypto.subtle.digest('SHA-1') for hashing (Node 22+, browser)
- CompressionStream / DecompressionStream for zlib deflate/inflate
- writeLooseObject is idempotent (skip if oid path exists), git is
content-addressed
- readLooseObject verifies sha1 after inflate; truncated bytes raise
ObjectCorruptError; missing path raises ObjectNotFoundError
Repo.readBlob now wires through to readLooseObject (gates on tracked
handle, returns ObjectCorruptError if oid is not a blob). Repo.hashBlob
is the public hash entrypoint.
11 unit tests including byte-fidelity assertions against the empty/,
single-file/, and nested-dirs/ fixtures. Verified the symlink-blob
oid against `git hash-object --stdin`.
Adds DOM lib to tsconfig + test/tsconfig so CompressionStream and
crypto.subtle are typed in both compilations.
readHead, readDirectRef, resolveRef, writeSymbolicHead, writeDirectRef. Atomic writes via temp + rename. resolveRef does single-hop deref: HEAD -> (symbolic branch leaf) -> oid; bare leaf refs return their oid directly. Lite has no other ref types so multi-hop deref is unneeded. Repo.resolveRef wired through. RefNotFoundError on missing files; WorkdirIoError on malformed contents. 8 unit tests cover symbolic HEAD parsing, leaf-ref reads against the empty/ and single-file/ fixtures, RefNotFoundError on missing refs, and a write+readback round-trip that asserts byte-equality of HEAD's "ref: refs/heads/main\n" form and the leaf ref's "<oid>\n" form.
Real-git compatible lock primitive: open <gitdir>/index.lock with
{flag:'wx'} (atomic create), hold during the effect, release on success
via rename-to-index (the new index bytes become the new index in one
atomic step) or remove (for callers who don't write an index).
- withIndexLock(gitdir, releaseStrategy)(effect): runs effect with the
lock held; releases per strategy; failure path always removes lock
- Age-based auto-clear: on EEXIST, stat the lockfile; if older than
SF_SOURCE_TRACKING_LOCK_AUTOCLEAR_SECONDS (default 300s), remove and
retry. Replaces real-git's pid-liveness check (lite has no
process.kill, no platform branches; works in every Layer)
- Exponential backoff with jitter, capped at ~1s, until
SF_SOURCE_TRACKING_LOCK_TIMEOUT_SECONDS (default 600s) elapses
- RepoLockedError carries ageMs + ageHumanReadable for the manual-
delete CTA in the eventual SfError mapping
5 unit tests cover: rename-to-index byte-equality, failure-path orphan
prevention, in-process serialization, stale-lock auto-clear, timeout
on a young pre-existing lock.
init writes the canonical empty shadow:
- .git/HEAD "ref: refs/heads/main\n"
- .git/objects/4b/825dc6... empty-tree object (canonical 4b825dc6...)
- .git/objects/<a..b>/<rest> initial commit object
- .git/refs/heads/main "<initial-commit-oid>\n"
- .git/info/exclude empty (caller mutates via setInfoExclude)
- .git/config [core] repositoryformatversion=0,
untrackedCache=true if Capabilities
reports supportsUntr
Idempotency rules per the migration spec:
- valid shadow (HEAD + refs/heads/main + objects all present): no-op,
returns existing commit oid (NB the timestamp in the commit body does
NOT change on re-init)
- missing .git/: fresh write
- partial state (e.g. HEAD without refs/heads/main): IndexCorruptError,
never overwrites
Initial commit body is real-git-format-compatible: trailing LF after
message is required (e.g. "init\n" not bare "init"), confirmed against
the empty/ fixture (commit oid a69e437d3322f...). Without that LF, lite
hashed bbf63786..., which is wrong.
Repo.init now uses the real impl + folds into a switchTo of the same
cfg so callers don't need a follow-up.
5 unit tests cover: byte-fidelity vs the empty/ fixture (HEAD + refs +
oid path-on-disk), idempotency over re-init with a different timestamp,
partial-shadow IndexCorruptError, post-init resolveRef round-trip, and
config-flag toggling via Node vs Memfs Capabilities layers.
Loose-object byte-equality (the zlib bytes themselves) is intentionally
NOT asserted: real-git accepts any deflate level for the same content,
and CompressionStream uses level 6 vs git's traditional level 1. The
oid is the assertion; if the file lives at the right oid path, the
content sha matches by construction.
parseIndexV2 + readIndex. Header validation, entry parsing with stat fields + oid + flags + path + 8-byte alignment, extension capture (TREE/UNTR retained as raw payloads for round-trip and phase 11), 20- byte trailer captured but not verified on read (verification is the writer's mirror; index is hot-path). Rejects v3/v4 with IndexCorruptError. 7 unit tests cover: single-file/ + nested-dirs/ entry parsing with mixed file modes (100644 + 100755 + 120000), TREE extension capture, UNTR extension capture (with-untr/), v3 rejection, malformed buffer rejection, missing-file failure path.
parseTreeObject decodes real-git's tree format (<mode-octal-ascii> <name>\0<20-byte-oid> repeated). readTree recursively walks a tree oid; submodule entries (mode 160000) are emitted but not recursed. streamHeadTree resolves HEAD -> commit -> tree, then walks. Repo.streamHeadTree wired through. moveDetection's per-file readBlob loop in phase 12 will swap to one streamHeadTree iteration. 3 unit tests cover empty/, single-file/, and nested-dirs/ traversal. The single-file/ blob oid (ce01362...) round-trips against the fixture; nested-dirs emits all four leaf paths regardless of mode (plain, executable, symlink).
Three-way merge (HEAD-tree | index | workdir) collapsed to the public StatusEntry union per test/git/STATUS-COLLAPSE.md. The walker: - loads ignore matcher from .git/info/exclude (empty if absent) - parses the index v2 (phase 6) - collects HEAD-tree paths via streamHeadTree (phase 7) - recursively scans cfg.roots via FileSystem.readDirectory(recursive:true) with stat-based file/dir discrimination - hashes workdir bytes via hashBlob (phase 2) for present files - emits one StatusEntry per path, filtered to roots, in path-sorted order ENOENT mid-walk is silently dropped. ENOENT for the root itself is silent (returns empty). Other PlatformErrors raise WorkdirIoError. `ignore` added as a direct dep (was transitive via SDR; lite must not rely on the transitive). Stat-trust optimization (skip rehashing files whose stat matches the index entry) is deferred to phase 11 since it intertwines with UNTR. Phase 8 hashes every workdir-present file unconditionally — correct, but slower than the eventual warm path. Repo.statusMatrix and Repo.collectStatus wired through. 3 unit tests against the with-untracked/, with-info-exclude/, and empty/ fixtures. with-info-exclude verifies negation patterns (`!**/.keep`) work via the ignore lib's full-spec support. The phase-1 "switchTo installs a handle" lifecycle test is updated since collectStatus is no longer a defect.
writers.ts: - buildTreeMap: flat (path, mode, oid) iterable -> nested TreeMapNode - writeTreeFromMap: bottom-up recursion, real-git's canonical entry ordering (byte-wise sort with directory suffix `/`) - hashTreeFromMap: same as write but no fs side effects, useful for the empty-tree shortcut - writeIndexV2: header + sorted entries + extensions + sha1 trailer. Each entry encoded with 10 stat u32s + 20 oid bytes + flags + path + NUL pad to 8-byte alignment - writeCommit: real-git format, parent line included iff non-null, trailing LF after message applyChanges.ts: - HashSet<RepoPath> dedup of adds + removes - empty-streams short-circuit: returns existing HEAD oid, no commit - under withIndexLock so concurrent writers serialize - read current index, apply removes, hash + write blobs for adds, build tree from new entries, write commit, update refs/heads/main, write index atomically Repo.applyChanges wired through to applyChangesImpl. 3 unit tests cover empty-streams short-circuit (no advance), single- file add (HEAD advances, index has the entry, commit decodes with correct parent + tree + message), and add-then-remove (index ends empty, tree references the canonical empty-tree oid). Phase 9 keeps stat fields zeroed for new entries — phase 11 will populate them via UNTR. Tests don't depend on the stat fields.
iso-git ↔ lite round-trip: - iso-git init + commit → lite resolveRef + readIndex sees the result - lite init + applyChanges → iso-git resolveRef + readBlob sees lite's bytes - lite hashBlob and iso-git hashBlob produce the same oid for the same content 3 unit tests. Lives in test/unit/git/migration.oracle.test.ts; deleted in phase 14 with the iso-git removal.
Lands the conservative half of phase 11. The headline UNTR-driven warm
scan is descoped per the migration spec's risk register §"UNTR format
misread" (mitigation: descope warm scan from v1, ship cold-only). The
warm path becomes its own follow-up once the byte format is exercised
on real workloads.
What lands:
- src/git/untrProbe.ts: one-time probe that checks ino + mtime_nsec +
ctime advance after a directory mutation. Returns 'ok' or 'failed'
with a reason ('unstable_ino' | 'coarse_mtime' | 'ctime_static').
Invoked by phase 12 on first switchTo per gitdir.
- src/git/statusMatrix.ts: stat-trust optimization. If a workdir entry's
size + mtime match the index entry's recorded stat fields, skip
rehashing the bytes — trust the recorded oid. Real-git's racy-stat
behavior. Cuts the workdir-hash cost on warm runs to a single stat
per file. Phase 9's writer leaves stat fields zeroed for new entries,
so the optimization currently activates only after at least one
applyChanges that records stat (out of phase 11 scope to fix).
What's NOT yet wired:
- UNTR write/read into the index. Phase 6's reader already captures
UNTR as opaque payload, so the bytes-on-disk path is forward-compat.
- Warm-scan walker keyed on UNTR invariants.
- Capabilities-driven probe-on-switchTo (phase 12 wires it).
1 unit test on the probe.
Lands the integration boundary; defers the deep ShadowRepo rewrite to
follow-up commits. With this in place, future PRs can swap
ShadowRepo's iso-git call sites one at a time, gated by the flag,
without touching the lite implementation.
- src/git/integration.ts:
- useLiteGit(): reads SF_SOURCE_TRACKING_USE_LITE_GIT once;
strict 'true' (no '1', no truthy-coercion). Default off.
- repoErrorToSfError(): maps the closed RepoError set to SfError with
stable error names (repoLocked, indexCorrupt, objectNotFound,
objectCorrupt, repoNotConfigured, workdirIo, invalidPath,
refNotFound). Replaces the iso-git redirectToCliRepoError catch.
- provideRepoErrorMapping(): catchAll wrapper for callers.
- messages/sourceTracking.md: human messages for the 8 RepoError tags.
- src/git/repo.ts: switchTo now invokes the UNTR probe (phase 11) on
every transition. Result is logged at trace; no failure surfaces to
the caller. capabilities.supportsUntr=false short-circuits.
What's intentionally NOT in this commit (each is its own follow-up):
- Replacing iso-git calls in localShadowRepo.ts. The 9 iso-git APIs
are listed in the migration spec table; flipping them per-call
inside ShadowRepo is a separate PR.
- Removing MAX_FILE_ADD batching + SF_SOURCE_TRACKING_BATCH_SIZE.
- moveDetection swap from per-file readBlob to streamHeadTree.
- fileFilter rules → setInfoExclude on init.
- Adding `ignore` to package.json (already done by phase 8).
5 unit tests: useLiteGit toggle behavior, repoErrorToSfError type
coverage including WorkdirIoError.
Optional startup integrity check, gated by SF_SOURCE_TRACKING_INTEGRITY_CHECK=true. Verifies that an existing shadow's HEAD points at refs/heads/main, refs/heads/main resolves to a 40-hex oid, and .git/index parses as v2. Source-tracking calls runIntegrityCheck on first ShadowRepo construction when the flag is set. On failure, the IndexCorruptError surfaces (via repoErrorToSfError from phase 12) with a delete-and- re-init message. What phase 13 does NOT include (these are process steps, not code): - Flipping SF_SOURCE_TRACKING_USE_LITE_GIT on internally for one release cycle. That's a CI/dev-machine env-var flip; no code change. - Telemetry-dashboard review of repo.lock.*, repo.status.entries_count, repo.untr.disabled. Those metrics will be plumbed once phase 12's follow-ups land the actual call sites. - Re-running the acceptance gate against the dogfood build. That uses test/git/baselines.json as the iso-git baseline (already captured in phase 0). 4 unit tests: isEnabled toggle, integrity-check passes on a healthy shadow, fails closed when refs/heads/main is missing.
Speculative scaffolding from earlier phases that no caller picked up: - REPO_ERROR_TAGS, repoPathDescription (errors.ts) — dead constants - provideRepoErrorMapping (integration.ts) — duplicates direct catchAll - lockBackoffSchedule (lock.ts) — withIndexLock has its own loop - oidEquivalence/oidHash, repoPathEquivalence/repoPathHash (schemas.ts) — Set/Map.has() works on string brands directly - CommitOid runtime const (kept the type alias) — Oid is the only oid - parseTreeObject, readTree, TreeEntry, RepoHandle — internal helpers that should not be exported
Feedback round:
- src/git/lock.ts: withIndexLock now returns a pipe-able combinator
(`.pipe(withIndexLock(gitdir))`) instead of taking the body as a
callback. Drops the awkward "effect inside an effect" reading. The
callback form lives on as withIndexLockCtx for callers that need
the lockPath (rename-to-index atomic-swap pattern). Also: humanReadable
uses Duration.format from Effect.
- src/git/applyChanges.ts: applyChanges and helpers (readWorkdirBytes,
stageAdd, buildAndCommit) are Effect.fn instead of Effect.gen,
per the effect-best-practices skill. The critical section pipes
through withIndexLock at the end.
- src/git/indexV2.ts: parser is now functional. oidFromBytes uses
Array.from + map + join (no for/let). The entry-fold uses
Array.from({length: N}).reduce, threading an immutable
{entries, offset} accumulator. The extension parser is recursive,
not while-loop. No mutation in the parse path.
- src/git/init.ts: the three exists() checks (HEAD, refs/heads/main,
objects/) run in parallel via Effect.all(..., { concurrency:
'unbounded' }) instead of sequentially.
- .vscode/settings.json: cSpell.words extended with the git-internals
vocabulary (gitdir, oid, untr, ctime, mtime, posix, ...) so the
blue squiggles stop flagging legitimate domain terms.
5 lock tests pass after the API rewrite. 186 unit tests passing.
Sweeping feedback round. effect.fn replaces effect.gen everywhere except shared-wrapping pipes (statusMatrix.cold, trees.streamHeadTree, repo.switchTo's inner semaphore body, writers' recursive helpers) and the Repo service-definition body. Per the effect-best-practices skill. Drop explicit Effect.Effect<...> return-type annotations, let TS infer. Two recursive helpers (writeTreeRecursive, hashTreeRecursive in writers.ts) keep an explicit return type with eslint-disable because TS can't break the self-reference cycle without one. repoErrorToSfError now has a pipe-able sibling mapToSfError built on Effect.catchTags. No _tag reads at the integration boundary. lock.ts replaces the while loop with native Effect retry (Schedule.exponential.either(spaced).jittered.upTo). Only RepoLockedError is retriable; WorkdirIoError surfaces. statusMatrix.ts extracts walkOneRoot, statTrustOid, hashWorkdirOid, workdirOidFor, cellFor to module scope (cold was huge); union of paths now built functionally. collapse returns Option<StatusEntry>. indexV2.ts and trees.ts: parser short-circuits via throw inside Effect.try's sync body, with a file-level eslint-disable for functional/no-throw-statements (the throws never escape the Effect boundary). writers: extract buildTreeBody helper, parallel recursion via Effect.forEach. Drop null from src/git/ entirely. Use Option<T> at domain boundaries (readUtf8, hashWorkdirOid, collapse, CommitArgs.parent). init.ts parallelizes the three exists() probes via Effect.all. eslint config: src/git override enables @effect plugin, functional rules, the local no-explicit-effect-return-type rule; disables the no-unsafe-* family (Effect.fn-inferred any noise). test/unit/git override disables the no-unsafe-* family. .vscode/settings.json: cspell vocabulary extended (gitdir, oid, untr, ctime, mtime, posix, ...). 186 unit tests passing.
- applyChanges replaces the mutable Map<string, IndexEntry> with Effect.HashMap. Removes apply via HashSet.reduce + HashMap.remove; adds merge via reduce + HashMap.set. The whole flow is immutable through to the sorted entry-list materialization. - objects.ts: LooseObject is now a Schema.Struct (type + Uint8Array content). LooseObjectType is Schema.Literal for the three valid kinds. Pairs naturally with the framing helpers; future use sites can decode/encode through Schema if they want. 186 unit tests passing.
- mapToSfError (integration.ts): pipe-able combinator built but not consumed; phase 12 follow-up will wire it inside source-tracking via Effect.catchTags directly. Keep the synchronous repoErrorToSfError for the existing redirectToCliRepoError site. - LooseObject type alias (objects.ts): demote to internal Schema struct (LooseObjectSchema). Future call sites can import it if they want validation; the type alias was never picked up. - HeadValue (refs.ts): demote to module-private. - UntrProbeResult (untrProbe.ts): the only consumer was the function body itself — drop the alias, the discriminated union literal stays inline.
- untr-warm-plan.md: design for the lite shadow's warm getStatus (sidecar JSON cache, Effect Schema codec, per-directory fingerprints, long-lived-process invalidation). - adr/0004-untr-cache-format-and-location.md: locks in the format (Schema-driven JSON), location (sidecar), fingerprint shape (no ino), and ignore scope (per-directory .gitignore). - CONTEXT.md: glossary entries for cold/warm status, UNTR cache, sidecar; updated to match the format/fingerprint decisions in ADR-4. - .gitignore: explain why authored docs live outside docs/. - .vscode/settings.json: spell-check additions for terms in the plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures iso-vs-lite numbers on the existing perf NUTs (commitPerf, localTrackingScale) so we have a citeable baseline before the warm UNTR work lands. - test/perf-utils/measure.ts: shared wallMs + event-loop-delay harness; appends to ~/.sf/source-tracking-perf/iso-vs-lite.jsonl per call. - test/perf-utils/renderIsoVsLite.ts: reads the JSONL and emits docs/perf/getStatus-iso-vs-lite.md, grouped by (workload, op). - commitPerf / localTrackingScale: instrumented with measure() around getInstance / getChangedFilenames.cold / commitChanges. Existing assertions retained. - package.json: yarn perf:iso-vs-lite[:iso|:lite|:render|:reset]. - docs/perf: also picks up the two prior isogit baselines that were on disk but not tracked. .gitignore narrowed to docs/* with a carve-out for docs/perf/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UntrCache schema (Effect Schema), JSON-via-parseJson codec, and read/write/delete helpers backed by atomic temp+rename. Sidecar lives at .git/sftracking/untr.json. Includes the trackedNames-per-dir field needed for warm to reconstruct the workdir-set without a re-walk. Plan + ADR scope-call locked in.
- ignoreChain.ts: anchor-relative chain so a path can be evaluated against ancestors' .gitignores at every depth. - dirWalk.ts: per-directory non-recursive snapshot returning (dirMtime, totalEntryCount, gitignoreMtime, fileNames, subdirNames), plus walkAllRoots that recurses with chain accumulation. - statusMatrix cold rewritten on top of walkAllRoots: gains nested .gitignore evaluation by sharing the same per-dir matcher builder. - statusMatrixWarm.ts: validates per-dir fingerprints, walks stale dirs inline, returns 'invalidated' on info/exclude drift. - untrBuild.ts: assembles the cache from walkAllRoots + index. Not yet wired into Repo — that's commit 3.
- RepoHandle.internals carries untrEnabled (probe + capability) and a Ref<Option<LoadedUntrCache>> populated lazily by the first warm getStatus and reloaded whenever the on-disk sidecar's mtime advances. - statusMatrix() / collectStatus() route through warmOrCold: excludeMtime drift / decode failure / probe failure all fall back to cold and clear the slice. - applyChanges rebuilds + writes the cache after the commit lands and refreshes the in-process Ref to the on-disk mtime so the next getStatus on a long-lived consumer sees fresh data. - Repo unit test exercises the round-trip: applyChanges writes the sidecar at .git/sftracking/untr.json; collectStatus stays consistent. - New scale NUT (test/nuts/local/localTrackingScaleWarm.nut.ts) drives ShadowRepo through commit + warm getStatus + mutate + warm getStatus on a 2k-file synthetic project. - nonTopLevelIgnore.nut.ts updated for nested-.gitignore parity. - CONTEXT.md gains a note on the shared per-directory ignore-chain.
- Remove isomorphic-git dependency - Remove localShadowRepoIso.ts (isogit backend) - Remove moveDetection.ts (isogit move detection) - Simplify localShadowRepo.ts to use only ShadowRepoLite - Remove SF_SOURCE_TRACKING_USE_LITE_GIT flag and related test infrastructure - Remove migration docs (isogit-migration.md, isogit-migration-plan.md) - Remove perf comparison scripts and docs - Update tests to use only lite backend - Regenerate test fixtures for statusMatrix tests - Fix pre-existing lint issues in test files
mshanemc
force-pushed
the
sm/perf-improvements
branch
from
October 5, 2026 20:39
2c9d5d6 to
c2f4f41
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Performance and resilience improvements to
@salesforce/source-trackingto reduce CPU/latency in the hot paths driving the Salesforce Metadata VS Code extension. Compared againstmain:populateTypesAndNames(src/shared/populateTypesAndNames.ts)Replace
Stream.fromIterable + runFoldEffect + Effect.reduceover persistentHashMap/HashSet(withData.structkeying) withArray.reduceover nativeMap/Set. The fold is purely synchronous, so the Stream + persistent-collection machinery was pure allocation overhead — everyHashSet.addreturned a new HAMT node. DropsapplyAndDedupe's HashSet round-trip in favor of reference-identity dedup. Legacy implementation kept atpopulateTypesAndNamesLegacy.tsfor comparison. Median wall-time on the new perf NUT (4 runs each):ShadowRepo(src/shared/local/localShadowRepo.ts)Convert
getStatus/commitChanges/detectMovedFilesto Effect withrunPromise. Adds OTel span instrumentation andeventLoopDelayCaptureso we can measure on real projects without modifying NUTs.Effect runtime (
src/shared/runtime.ts, new)Shared module-level
ManagedRuntimeso everyrunPromiseinsrc/executes on the same fiber/runtime instead of paying construction cost per call. NewrunPromisehelper unwrapsCauseand rethrows the original failure or defect —ManagedRuntime.runPromisewas rejecting withFiberFailurefor both, breaking callers that branch one instanceof SfError/e.name(notably the ulimit-actions error fromrunGitAddreaching CLI consumers asFiberFailure). Optional file-based JSONL span exporter behindSTL_OTEL_SPANS=1for capturing baselines.ChangeResultschema (src/shared/types.ts)Re-derive
ChangeResultfrom aneffect/SchemaStructso internal code can use structuralEqual/HashviaData.structwhile consumers see the same shape.Conflicts + sourceTracking (
src/shared/conflicts.ts,src/sourceTracking.ts)getDedupedConflictsFromChangesis now async (callspopulateTypesAndNamesviarunPromise).getChangesruns the threepopulateTypesAndNamescalls concurrently under a singlerunPromise/Effect.allso they share one fiber.Misc
isomorphic-git1.37.5 → 1.38.3perf_hooksnoExplicitEffectReturnTypelocal ESLint rule +eslint-plugin-functional/eslint-plugin-local-ruleswired uptest/nuts/local/populateTypesAndNamesPerf.nut.ts) and unit tests forruntimescripts/aggregateSpans.ts) + node/web exporter helpersremoteSourceTrackingService.tsWhat issues does this PR fix or reference?
forcedotcom/salesforcedx-vscode#7174, forcedotcom/salesforcedx-vscode#7278, @W-22711959@