[feat]: Command to trigger a full re-crawl after the initial index on demand

[bygadd]

Observation (context_chat main, 5.4.0-beta0)

While reviewing the file-indexing job lifecycle on main I noticed the crawl chain is bootstrapped only on fresh install and self-removes, and I'd like to understand how a full re-crawl is meant to be (re)triggered afterwards.

What I see in the code:

• SchedulerJob (lib/BackgroundJobs/SchedulerJob.php) enumerates the mounts, adds one StorageCrawlJob per mount, then removes itself ($this->jobList->remove(self::class), ~L48). It is a QueuedJob, seeded only by the <install> repair step AppInstallStep (appinfo/info.xml declares only FileSystemListenerJob + RotateLogsJob under <background-jobs>; there is no <post-migration> step).
• StorageCrawlJob self-perpetuates while a mount still has files (scheduleAfter(self::class, …), ~L85) and otherwise removes itself.

So on a healthy fresh install the initial crawl runs to completion and then both jobs are gone. My question is about what happens after that:

1. A new external storage is mounted after the initial crawl has finished — SchedulerJob (which is what enumerates mounts) is no longer scheduled, so is there anything that discovers and crawls the new mount? (FileSystemListenerJob handles live filesystem events, but those don't fire for the pre-existing contents of a freshly-mounted storage.)
2. An app upgrade — since the seed lives under <install> (not <post-migration>), occ upgrade won't re-run it. If the initial crawl had been interrupted/incomplete before the upgrade, is there a path that resumes a full crawl, or does it rely solely on FileSystemListenerJob from then on?

Question

Is a <post-migration> re-seed of SchedulerJob (mirroring the existing <install> AppInstallStep, similar to how Recognize wires its InstallDeps under both <install> and <post-migration>) intended/desirable — or is mount discovery / re-crawl already handled by a mechanism I've missed? Happy to open a small PR for the <post-migration> re-seed if it'd be useful.

_{Context: we hit a related "initial indexing never completes" symptom on the 5.3.x line where the crawl chain had self-deleted with a backlog still queued; on main the backend-pull rearchitecture changes this substantially, so I want to confirm the intended behaviour before proposing anything.}

[kyteinsky]

hello,
this functionality is not available right now so I'll convert this into a feature request.
your observations are correct and it is done like that by design so app disable+re-enable does not populate the queue again and the indexed does not have to go through the whole long list again.

A new external storage is mounted after the initial crawl has finished

I would expect the newly discovered files to emit an event CacheEntryInsertedEvent which would be captured and processed by context chat.
did you notice a change in the queued items when a new external mount was added to the instance and scanned/visited from the web UI?
this command can help with the stats: occ context_chat:stats, or it can be seen in the Admin settings -> Assistant -> Context Chat section.

[kyteinsky]

some work regarding re-queueing files is done here: https://github.com/nextcloud/context_chat/pull/227/changes#diff-b8991b7385a0d21fe721ea8b2aa2c1eee0be3d9d2e904b41088de8db0ea4581b

[bygadd]

Thanks @kyteinsky — that clears up the design intent (re-enable deliberately not re-seeding the queue, so the indexer doesn't re-walk the whole list), and #227's re-queueing work looks like it covers the live-discovery side.

I went back and read the listener wiring to answer your question precisely. CacheEntryInsertedEvent is registered → FileListener (lib/AppInfo/Application.php), so a newly-mounted external storage that then gets scanned (occ files:scan / first web-UI visit) inserts oc_filecache rows, fires the event, and does get queued. So that sub-case is handled — thanks, that corrects my original assumption.

The gap that remains — and what the title feature is really for — is files that are already in oc_filecache before context_chat enumerates them: no insert event fires for those, so they rely solely on the one-shot <install> StorageCrawlJob seed. Concretely:

• context_chat installed on an existing instance (files already cached) → the only full enumeration is the install-time crawl.
• Interrupted/incomplete initial crawl, then occ upgrade → the seed is <install>, not <post-migration>, so there's no path to resume a full enumeration; event capture only covers files touched from then on.

So the request reduces to an idempotent, on-demand full re-enumeration — a CLI command and/or a <post-migration> re-seed of SchedulerJob — so operators can recover index completeness without an app reinstall. If #227 already exposes that, great; happy to test against it.

---

Separately: while stabilising context_chat on a production NC 33.0.5 (app 5.3.1 / backend) we diagnosed and fixed several backend/indexing issues that may be useful upstream:

• multipart-CR indexing freeze — loadSources rejected with Did not find LF character at end of header (found 13) after a provider change (CR/LF mangling); CR-tolerant multipart parsing.
• exec_in_proc fork-deadlock — parent cconn left open → recv() hangs on a child that dies without a result (no join-timeout).
• recv-leak / orphaned _indexing lock — the leak left the lock held → persistent 503 "already being processed"; fixed with a lease + poll.
• child-log relay — forkserver children lost their logging config; relayed embed logs to a backend log.
• SMB non-seekable stream — IndexerJob fopen('rb') on SMB external storage yields a non-seekable stream that reports isSeekable()=true → multipart send fails; wrapped unconditionally in CachingStream in LangRopeService.
• HNSW selective-context 0 results — for a user holding only a small fraction of the collection, once id IN (...) exceeds ~5.5k Postgres flips to an HNSW index scan that takes the global-nearest then post-filters by id → 0 documents; reworked the pgvector query.

Patches + per-issue write-ups are consolidated here: https://github.com/bygadd/context_chat-community-fixes. Happy to split any of them into focused PRs if useful.

(Disclosure: this investigation were done with AI assistance; every fix above was diagnosed against the source and verified on our live deployment.)

[bygadd]

Follow-up correction on the HNSW selective-context item above:

1. I implied the build lacked the relevant tuning knobs — that was a measurement artefact on my side: pgvector registers hnsw.ef_search / hnsw.iterative_scan only after its library is loaded into the session, so a fresh-session SHOW reports them as unknown. This deployment runs pgvector 0.8.0, so SET hnsw.iterative_scan = relaxed_order (off by default) is available as the native, config-level remedy for the filtered-HNSW under-return — likely simpler than an app-side query rewrite for most cases.

2. On a fresh re-test against the current DB I couldn't re-trigger the original 0-results plan via a plain id = ANY(...) filter — the planner used the btree id index and returned correct rows — so that pathological HNSW plan was planner-cost-dependent, not guaranteed. I've kept the CTE in the write-up as a deterministic fallback for very sparse corpora.

I've updated the write-up accordingly.

(Disclosure: this analysis was done with AI assistance; verified against the source and on a live deployment.)

[kyteinsky]

is files that are already in oc_filecache before context_chat enumerates them: no insert event fires for those, so they rely solely on the one-shot StorageCrawlJob seed

which is fine since the crawl job would list all the existing files in all the mounts.
if the indexing is interrupted due to downtime, the indexing queue remains in the db so it can continue with the remaining items the next time the instance is up.

If #227 already exposes that, great; happy to test against it.

that is not merged yet, will be done sometime in the future but we can accept a PR that only adds this part

multipart-CR indexing freeze

this I suppose would not be required now since starting from v5.4.0 we don't use multipart requests to send files from context_chat to the backend but the backend pulls the files.

exec_in_proc fork-deadlock

are you sure that doesn't throw a BrokenPipeError in that case? It would be interesting to have a reproducible example to confirm that before going about fixing it, preferably in a separate issue.

recv-leak / orphaned _indexing lock

these are not used now

child-log relay — forkserver children lost their logging config; relayed embed logs to a backend log.

please open a bug report in the context_chat_backend repo, seems like an issue that could still exist.

SMB non-seekable stream

this may not be an issue starting from v5.4.0 but if it is, please open a separate issue.

HNSW selective-context 0 results

didn't quite understand what's the problem here, would you mind creating a new issue in the backend repo to keep track of it?

also, please write the issue texts in your own words so it's easily understandable.
see the "AI-assisted contributions" section here: https://github.com/nextcloud/context_chat/pull/245/changes#diff-98e64bc1cd2db9333c6effe87bbe0d6dfe8714aba4c6bde45aa037fe0796e44c

[bygadd]

On the fork-deadlock point: I re-checked main and it's already covered there — exec_in_proc (context_chat_backend/utils.py) reads with pconn.poll(...) in a loop and with suppress(EOFError, OSError, BrokenPipeError): pconn.recv(), then p.join(). So a child that dies without sending a result doesn't block the parent, and BrokenPipeError/EOFError are suppressed. No separate issue needed — thanks for the pointer.