Feat/synth scholar public reads - #82
Merged
tekrajchhetri merged 3 commits intoAug 13, 2026
Merged
Conversation
ml_service dies at boot on EC2 — gunicorn exit 3 (WORKER_BOOT_ERROR) after 2.7s,
four times, then supervisor gives up and 8007 stays dead while the container
reports healthy on 8000/8004/8010.
Exit 3 with no bind means the workers raised before the app object existed, and
`from structsense import kickoff` is the only unguarded heavy import on that path
(core/main.py already guards the SynthScholar one for exactly this reason). It was
imported in two places:
* core/routers/structsense.py — never used. This module reaches structsense only
through core.shared.run_kickoff_with_config. Removed.
* core/shared.py — the real one. Now guarded, with kickoff = None on failure and
a 503 raised per-request from run_kickoff_with_config.
So a broken structsense install now costs the extraction endpoints only. Everything
that never touches kickoff keeps serving: GET /api/ner and the saved-annotation
surface behind /knowledge-base/ner, and the whole /api/synth-scholar tree.
Why the install can be broken while the image builds green (Dockerfile.unified):
the structsense step falls back to `--no-deps` when the legacy resolver fails, so
crewai/litellm never arrive — and ml_service's requirements.txt lists neither, so
nothing fills the gap. The RUN still exits 0. Added an import check to that step so
this fails the build. (Not a precedence bug: `A || B && C` already groups as
`(A || B) && C`; the parentheses added are only for the reader.)
Diagnosis note for next time: gunicorn's traceback goes to
/var/log/supervisor/ml_service.err.log, not the container's stdout, so
`docker logs brainkb-unified` shows only supervisord's exit codes.
…tion The traceback names the cause exactly: structsense -> crewai -> litellm -> openai openai/_vendor/httpx_aiohttp/transport.py:17 aiohttp.SocketTimeoutError AttributeError: module aiohttp has no attribute SocketTimeoutError requirements.txt pinned aiohttp==3.8.6, and Dockerfile.unified installs that file AFTER structsense — so the pin downgraded the aiohttp that crewai/litellm/openai had just pulled in. Recent openai vendors httpx_aiohttp, which needs aiohttp.SocketTimeoutError (added in 3.10). Import raised, all 6 workers died before the app object existed, supervisor gave up, 8007 stayed dead while the container looked healthy on 8000/8004/8010. Changed to `aiohttp>=3.10,<4` — a floor, not a pin, so pip can reconcile with whatever openai/litellm require. chat_service and usermanagement_service keep 3.9.1; neither imports openai. Also: both import guards caught the wrong exception. This failure is an AttributeError, so `except ImportError` does not catch it — the guard added in c970966 would not have prevented this outage, and main.py's SynthScholar guard had the same hole since it was written. Both now catch Exception: a four-package import chain can fail in any number of ways, and each must cost one feature rather than the process. Verified the widening matters rather than assuming it: `except ImportError` lets an AttributeError through, `except Exception` does not.
…override Each service kept its own hardcoded list and they had drifted. chat_service was actually broken: it was missing https://brainkb.org and https://www.brainkb.org entirely, so the production UI was blocked from it on the main domain while beta and sandbox worked. It also carried "http://127.0.0.1:300" — a typo for :3000. Dropped the schemeless "localhost:3000" entries from usermanagement and chat: the browser's Origin header always carries a scheme, so they could never match. Dropped ml_service's "http://localhost" and its two :3001 entries — nothing in the repo serves the UI on 3001, and CORS_ALLOWED_ORIGINS covers anyone who does. All four now share one 6-entry default, plus CORS_ALLOWED_ORIGINS (comma-separated) so a new domain does not mean editing four files. Documented in env.template. Not a fix for the reported failure. The synth-scholar public route reporting "No Access-Control-Allow-Origin" from brainkb.org is ml_service being down: the ALB answers 502 (server: awselb/2.0) and its error page has no CORS headers, so an outage is indistinguishable from a CORS misconfiguration in the browser console. https://brainkb.org was already in ml_service's list. Noted in both the code and env.template, since this will mislead again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed mlservice container crashing due to aiohttp version issue and updated cors.