Switch to event based/callback based approach for dream web - #129
Closed
TesseractCat wants to merge 5 commits into
Closed
Switch to event based/callback based approach for dream web#129TesseractCat wants to merge 5 commits into
TesseractCat wants to merge 5 commits into
Conversation
tildebyte
reviewed
Aug 27, 2022
Merged
Contributor
|
Here is a commit on top of this branch which also streams the intermediate images, so you can watch it denoise. Feel free to pull this in to this branch. (This really makes me want to have the ability to cancel requests, because you can often see it's not going to be what you want before it actually finishes.)
|
This was referenced Aug 28, 2022
Collaborator
|
I'm fixing issues with the dream_web having to do with upscaling and will work on this next. |
lstein
self-requested a review
August 29, 2022 03:16
lstein
reviewed
Aug 29, 2022
Collaborator
There was a problem hiding this comment.
Unfortunately in the intervening time between submission of the PR, dream_web.py has been made into a module and moved into ldm/dream/server.py. scripts/dream.py --web now invokes this module. I think this makes better sense than maintaining two different launch scripts that share identical arguments.
Would it be possible to redo the PR using the new structure?
Contributor
|
I took the liberty of doing that rebase in #180. Feel free to ignore it if you'd prefer to keep this PR as the source of truth, though. |
kappacommit
pushed a commit
to kappacommit/InvokeAI
that referenced
this pull request
May 30, 2026
chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR invoke-ai#129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @
lstein
added a commit
that referenced
this pull request
Jun 29, 2026
* Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * chore(uv): update uv.lock --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
lstein
added a commit
that referenced
this pull request
Jul 27, 2026
* Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This uses HTTP fetch streaming to push images to the web interface as they are generated, and adds a progress bar by exposing the img_callback parameter to
prompt2image.As a result of using a callback based approach, when generating multiple images, it will add them to the result list immediately, rather than waiting until all images have completed.