Skip to content

Inbox mobile: attachments and annotation in the iPhone app (M2) - #1785

Merged
backnotprop merged 17 commits into
mainfrom
feat/mobile-m2-attachments
Oct 9, 2026
Merged

backnotprop merged 17 commits into
mainfrom
feat/mobile-m2-attachments

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

The story

An agent's files now open on the iPhone the way they do on the desktop: full screen in Plannotator's own surface, with comments by touch (a long press and Comment in the system edit menu on markdown, a pin on an HTML page with Parent and Child, a tap on a diagram node), each written in a native sheet and saved through the device door. The comments wait in a "3 annotations" sheet and ride the next Send as Plannotator's feedback text, so the agent reads the same thing the desktop window would have sent. The page an agent wrote can neither reach the shell nor open anything on its own: bridge messages count only from the main frame, page-reported pins only right after the person's touch, and links open only as https or mailto, only after a tap.

Stacked on #1778 (M1, feat/mobile-m1-app), with main merged in for S1 (#1777) and its follow-up (#1782). The overseer retargets this to main when M1 merges. CI has not run on this PR: ios.yml and inbox-e2e.yml trigger on pull requests to main, so they first run once this PR is retargeted; the local runs below stand in until then.

What changed

  • The surface host apps/ios/Plannotator/Attachments/SurfaceHost.swift: one warm WKWebView loading the bundled surface.html from plannotator-surface://app/surface.html; plannotator-asset://inbox/api/html-assets/<token>/<path> served from the door's GET html-assets (7.16) for the open page's own token only, anything else fails; the WKScriptMessageHandler drops any message whose frameInfo.isMainFrame is false; pin, selection and draft for an HTML page are taken only within 1.2 s of the person's own touch on the web view (a UIGestureRecognizer that observes and never cancels), as Inbox mobile S1 follow-up: the review's items #1782 wrote into section 5; decidePolicyFor cancels every navigation away from the surface except the page's own about:srcdoc frame and frames from its folder; createWebViewWith never makes a web view. SurfaceWebView puts Comment first in the edit menu (buildMenu(with:)).
  • The screens Attachments/AttachmentCover.swift (4.1 to 4.4: the bars, the changed line, Annotate and Interact with Fit, Parent and Child, the comment panels, the saved comment with Edit and Remove, Share and the version switch in the "..." menu), AnnotationsSheet.swift (3.6, and the Edit sheet), AttachmentTiles.swift (the files at the foot of a message, 3.4A), LinkViews.swift (Safari View Controller, the mail sheet). Thread/ReplyBar.swift gains the "3 annotations" chip; Thread/ThreadScreen.swift the tiles, the cover and the sheet.
  • The model App/SourceSession.swift: each open thread's files and waiting annotations (cached like the thread), save and remove, and Send carrying feedback and annotation_ids on the last reply. PlannotatorKit/Attachments.swift: the attachment and annotation wire shapes with explicit keys and an opaque JSONValue, so Plannotator's Annotation crosses the app untouched (a round-trip test pins it), and the door calls 7.14 to 7.18.
  • The surface (packages/inbox/surface, packages/core/inbox-surface-bridge.ts): export_feedback / feedback and comment_selection (contract section 5, below); hostNavigates on the HTML viewer; on iOS the document's text follows Dynamic Type through -webkit-text-size-adjust (surface.css, below).
  • ui seams, additive, opt-in, Plannotator's render unchanged, noted in packages/ui/HANDOFF.md "unreleased": HtmlViewer hostNavigates; ViewerHandle.takeSelection with no touch selection painted while a host composes.
  • The build: a post-build phase in apps/ios/project.yml runs build:surface when its sources are newer than apps/inbox/dist/surface.html and copies it into the app (ENABLE_USER_SCRIPT_SANDBOXING: NO on the app target: the phase runs Vite over the repo's packages). Warning and Accent join the generated colours.
  • The proof: apps/ios/PlannotatorUITests/AttachmentProofTests.swift, the shared helpers moved into ProofSupport.swift (M1's test unchanged in what it does), scripts/proof.ts gains /attach and /edit-plan, an --only flag and the data-dir assertion (row 5109), scripts/fixtures/ the three files. .github/workflows/ios.yml stays one job; its paths add the surface's files.
  • Docs: CLAUDE.md "Plannotator Inbox" (on a phone, the surface) and Project Structure, apps/ios/README.md, the contract.

The review's fixes (REVIEW-PLANNOTATOR-MOBILE-M2-2026-10-08, one push)

  • Blocker 1, 4.3: an HTML page now ends above the switch and any open panel (the screen measures them), and the surface scrolls the pinned draft into the middle of what is left whenever the web view resizes or Parent or Child moves it (ui's new ViewerHandle.revealPin, bridge message plannotator-bridge-reveal-pin, HANDOFF "unreleased"). The proof asserts the pinned button is on screen and wholly above the switch after the pin, after Parent and after Child (4.3-pin, 4.3-pin-parent).
  • Blocker 2: the asset scheme decoded nothing and the door URL encoded again, so my pic.png went out as my%2520pic.png. The path is now decoded once (url.path(percentEncoded: false)) and encoded once when the door URL is built. The ticket page's three "going" photos come from its folder as going/mira.png, going/dj tern.png and going/joão.png; the page writes how many drew and the proof asserts "Photos: 3 of 3".
  • The black first paint: the web view stays hidden until the surface has drawn the file (two animation frames after open_attachment); the recording shows white, then the plan.
  • Send with a commented file deleted on the computer: the file's text goes empty and the Send goes, as the window does.
  • The fixture's .btnx rule: font-size: 18px; font-weight: 900 as in the record; the 4.3 frames are retaken.
  • Contract section 5's link paragraph says what the code does; the link doors are one rule (createWebViewWith, within the touch window; the unreachable .linkActivated arms are gone). Also from the items: the folder token is read from the door's <base href> only, and an edit made in the 3.6 sheet over an open file reaches the surface.
  • Left as follow-ups: the CI paths for packages/ui and packages/guide-viewer, the 4.4 node ring and zoom control under the panel, heading tracking at the largest type size.

plannotator-ops's review: agent HTML cannot phone home (one push)

A page the agent's page embeds from its own folder (<iframe src="extra.html">) loaded through plannotator-asset: with no policy of its own, so its script could send an image beacon. Now every asset answer carries the surface's own CSP (no network; scripts, styles, images, fonts, media and frames from the asset scheme, data: and blob: only), and the web view carries a WKContentRuleList that blocks every http and https load. One exception in that list, named: the request to open a new window, which is how a tapped link reaches createWebViewWith; no web view is ever made for it, and the URL goes to Safari View Controller. Proof: the ticket page embeds venue-notes.html from its folder; its script runs (the test reads the text it writes) and sends an image beacon and a fetch to the proof script's own loopback server; the server records none. Contract section 5 and CLAUDE.md say so.

plannotator-ops's delta review (one push)

  • The rule list blocks ws and wss too (a data: or blob: frame may not inherit the CSP); the nested embed's proof now also opens a WebSocket to the proof's loopback server, which records nothing.
  • Fail closed: if the rule list cannot be compiled, the surface is never loaded and an attachment shows "Files can't be shown on this iPhone right now."; Send with annotations then says it could not write them in. Seen for real once: a first rule list with an alternation in its url-filter (which content rules do not support) failed to compile, and the run showed "Files can't be shown on this iPhone right now." where the plan would open (filed as proof-run-rule-list-refused.log); ws and wss are now a rule of their own. No standing test forces the failure, since that needs a hook in the app.
  • The viewer's sandbox is spelled "allow-scripts" + (hostNavigates ? " allow-popups" : ""), so the isolation test's pin on sandbox: "allow-scripts" holds (it failed CI at 4823f4c); the proof scrolls the commented line clear of the toolstrip before the long press (it hit the toolstrip on the slower CI runner).

The CI fix (run 37826449162)

The attachments proof failed in CI on the first open of the plan: the spinner for 100 s, the surface never reported ready; not reproduced locally (on a warm build folder or a fresh one); the trace names the cause next time. What changed:

  • The surface starts loading when a thread's files show, not when one is opened.

  • If the surface page itself fails to load, the screen says "Files can't be shown on this iPhone right now." instead of spinning.

  • With -PlannotatorProof the screen carries a trace: loads, web content restarts, the rule list's state, ready, the load error. A failing expect prints it in the screen dump.

  • The proof's first wait is main's ProofWait.opening (120 s, from M3's cold-runner measurements) instead of 30 s.

  • Every proof class of this kind (ProofCase) starts from the app reinstalled from the run's own build and the computer's paired phones revoked (/reset-app), and resets again when it fails, so no class starts paired because the one before it failed partway. The first version only deleted a preferences domain outside the app's container, which left the app paired: in run on dc03ea9 Decisions and ProofTests failed at their first lines after the attachments class failed. Proved locally by forcing the attachments class to fail right after pairing: ProofTests then passed.

  • The plan's comment: on the CI simulator a synthesized long press raised no edit menu in 120 s (run on dc03ea9; the trace showed the surface ready). The proof presses again once, and if the menu still does not come, comments through the toolstrip's pinpoint (a tap on the line opens the same sheet); the log and the results name the path that ran ("Comment on the plan through ..."). Both paths pass locally (the pinpoint path forced once). The edit menu itself is still proved where the simulator raises it, and on the device.

  • /derived*/ is ignored at the repo root: a build folder kept in a worktree was committed once and taken out of the history.

  • On the CI simulator the surface twice loaded and never said ready (trace on c0c4c60: loads=1 ended=0 rules=ok ready=false error=none after 120 s; not reproduced locally). The shell now loads the surface again when someone has waited 30 s on a silent load, twice at most; the trace counts the loads.

Record sections made real

3.4A (files and the annotations chip), 3.6, 4.1, 4.2, 4.3, 4.4, light and dark; contract section 5 (both schemes, main frame only, the touch window, links), exchanges 7.14 to 7.18 and 7.11 with feedback; BRIEF section 7 "An attachment"; row 5120 (no URL from content reaches the app's handler).

Proofs and how to run them

bun apps/ios/scripts/proof.ts --binary .local/plannotator (after bun run --cwd apps/review build && bun run build:hook && bun build apps/hook/server/index.ts --compile --no-compile-autoload-bunfig --define '__CLI_VERSION__="0.0.0-dev"' --outfile .local/plannotator), iPhone 17 simulator on iOS 26.5, Xcode 26.6, on this branch rebased onto M1's head (7f714b7) with main merged in, after the review's fixes: both XCUITest flows pass in one run (M2's in 213 s, M1's in 241 s). --only PlannotatorUITests/AttachmentProofTests runs M2's alone. What M2's asserts, against the compiled Inbox and an agent through inbox mcp:

  • The agent sends retry-plan.md, the record's Low Tide ticket-page.html (rebuilt from the iPhone record's partyPage() in index.html, with a footer of two links and a script that tries the forgeries) and install-flow.mmd.
  • Markdown: the commented line starts below the fold; a long press selects, Comment is in the edit menu, the 4.2 panel saves; the count reads 1.
  • The agent edits the plan: the changed line names the agent and both times; "Open the version it sent" shows the sent text.
  • Links in markdown: https opens Safari View Controller on docs.stripe.com; a plannotator://pair link opens nothing (no pairing, no Safari, the file still on screen).
  • Share presents the system share sheet with the file.
  • The ticket page posts all five surface-to-shell types straight to the handler from its own frame, forges a pin and a link through the viewer's frame protocol and sets location.href, 1.5 s after load with nobody touching: no panel opens, the shell counted 5 frame messages and 1 untouched pin dropped, the page stays.
  • The page's photos from its own folder, named with a space and an accent, all draw.
  • A pin by touch on "Get tickets", in sight above the panel after the pin, after Parent and after Child; saved; a tap on its badge shows the saved comment with Edit and Remove (4.3).
  • Interact: "Venue map" opens Safari View Controller on example.com; "Open in the app" (plannotator://pair) opens nothing.
  • The diagram: a tap on "Pick a host" opens "On Pick a host (node E)", saved.
  • The chip reads "3 annotations"; the 3.6 sheet; its file name opens the plan at the mark (the line on screen without scrolling).
  • Send: the agent's wait_for_reply body names retry-plan.md, ticket-page.html and install-flow.mmd and carries all three comments; the chip goes.
  • Dynamic Type at the largest size on the tiles and on the plan.

Light and dark frames of every screen beside their renders, and the recording of the comment flow, are in research/operational/mobile-proofs/M2-2026-10-08/ in the Workspaces meta repo (built/, compare/, M2-comment-flow.mp4).

Also run after the fixes: tests/e2e/surface.spec.ts (9 of 9, Playwright WebKit against the rebuilt surface and the compiled Inbox); swift test in PlannotatorKit (the record round trip with the rest).

Gates

bun run typecheck; bun test packages/ui/components/html-viewer packages/ui/hooks packages/core packages/inbox (393 pass); DOM_TESTS=1 bun test --isolate on every html-viewer, Viewer and highlighter test file (376 tests, 0 fail); bun test tests/entry-assets.test.ts tests/favicon-surfaces.test.ts; bun run --cwd packages/ui smoke:package; bun run check:release-version; bun apps/ios/scripts/gen-colors.ts --check; Gitleaks on the range.

Contract changes (section 5, edited in this PR)

  • export_feedback / feedback: Send's feedback text is Plannotator's export, which needs Plannotator's parser for line numbers; the phone has it only inside the surface, so the surface builds it with the window's own attachmentFeedback instead of a second implementation in Swift.
  • comment_selection: on the iPhone the viewer painted a markdown selection as a pending highlight when it settled, and the library painted it at touchend; either replaced the system selection and closed the edit menu before the person reached Comment (seen in the simulator). On a touch screen the selection now stays the system's until Comment asks for it.
  • Links: the section said the shell would see a link in the agent's frame as a .linkActivated frame navigation. The surface's CSP (frame-src plannotator-asset: about: data: blob:) refuses that load before WebKit asks the shell, so a tapped link could never open. With hostNavigates a person's tap (isTrusted) opens the link as a new window instead, which createWebViewWith opens only within the touch window and never makes a web view; the frame never navigates. Written into the section.

Deviations from the record, each with its reason

  • The comment composers (4.2, 4.3, 4.4) are panels inside the attachment screen with the record's grabber, corners and layout, not system sheets: the Annotate and Interact switch has to sit above the 4.3 panel and the page stays live behind it. 3.6 and Edit are system sheets.
  • 4.3 draws Parent and Child on a saved comment; here they are on the pin before it is saved, where a step can still move it. A saved comment shows Edit and Remove, as drawn.
  • The pin's label is Plannotator's pinpoint name ("Button"), not the render's "div.btnx" (the contract's S1 ruling).
  • An HTML page ends above the switch and any open panel instead of running under them, so its last lines and links can be reached and a pin stays in sight; the render floats the switch over the page.
  • The keyboard rises at once for a selection (4.2 draws it) and waits for a tap on the field for a pin or a diagram part (4.3 and 4.4 draw the page in view).
  • Not drawn, from the desktop window's words: the line for the sent version ("The version Claude Code sent at 1:56 AM. Open the file as it is now"), the line for a file gone from disk, the version switch in the "..." menu. Not drawn: no mail account (a sheet with the address and Copy Address), the count hidden at zero, the bar's text stopping at the largest non-accessibility size (as system bars do).
  • Fit returns the page to its fitted zoom. HtmlViewer draws a page at the phone's width; a desktop-width page is not laid out at its own width here.
  • A saved pin is drawn as Plannotator's HTML viewer draws one (a mark on the element's text with a numbered badge and a "Comment 1" tag); 4.3 draws a ring around the element. The surface's drawing, from S1.
  • The diagram's ground is the surface's --background (light grey) where 4.4 draws white.
  • 4.2 draws the selection still marked behind the sheet; here the system selection ends when the field takes the keyboard (the quote is in the panel). A rough edge, reported.
  • The proof runs on a fresh simulator per run (M1's proof.ts), not a named one.
  • Proof only: with the launch argument -PlannotatorProof the attachment screen carries an invisible element with the shell's dropped and cancelled counts, which the test reads.
  • On iOS, WebKit does not apply CSS zoom to font sizes (the spacing grew, the letters did not: measured in the simulator, computed 22 px at both sizes), so S1's Dynamic Type did nothing to the document's text on a phone. The surface now also sets -webkit-text-size-adjust by the same factor on the document; desktop WebKit ignores it (the S1 spec still passes). The guide reader's card (.sf-gcard) has the same zoom and is left for M4.

What it leaves out

Multi-block selections, multi-target HTML comments, Pencil, guides (M4). Fetching view again when an asset answers 404 after the Inbox restarted while a page is open (reopening the file does it). Remote images and web fonts in an agent's page (no network by design). What needs the device: the edit menu's feel under a finger, a finger pin and Parent and Child on the Low Tide page, a node tap on a dense diagram, the mail sheet with an account, haptics, the 20 MB surface's first open on the iPhone 15.

Limits

One number, from the contract: a page-reported pin, selection or draft, or a link, counts within 1.2 s of the person's touch on the web view. No others.

@backnotprop
backnotprop force-pushed the feat/mobile-m2-attachments branch from 459479e to 203c131 Compare October 8, 2026 12:28
@backnotprop
backnotprop force-pushed the feat/mobile-m2-attachments branch from 203c131 to af0ee68 Compare October 8, 2026 13:07
@backnotprop
backnotprop changed the base branch from feat/mobile-m1-app to main October 8, 2026 13:37
@backnotprop
backnotprop force-pushed the feat/mobile-m2-attachments branch from af0ee68 to 59da3d9 Compare October 8, 2026 13:37
Attachments full screen in Plannotator's bundled surface (4.1 to 4.4): a long
press and Comment in the edit menu on markdown, a pin with Parent and Child on
HTML, a diagram node, each saved through the device door from a native panel;
the "3 annotations" sheet (3.6); the changed line and the version it sent;
Share; annotations riding the next Send as Plannotator's feedback text.

The shell takes bridge messages from the main frame only, page-reported pins
only right after a touch, and opens links only as https or mailto after a tap.
Contract section 5 gains comment_selection, export_feedback and feedback; ui
gains HtmlViewer hostNavigates and ViewerHandle.takeSelection.
…aces and accents, first paint, Send with a deleted file, fixture, contract links)
…urface's CSP on every asset answer, http and https blocked in the web view)
…er; the viewer's sandbox spelled as the isolation test pins it
…ments

# Conflicts:
#	AGENTS.md
#	apps/ios/Plannotator.xcodeproj/project.pbxproj
…ments

# Conflicts:
#	AGENTS.md
#	apps/ios/Plannotator.xcodeproj/project.pbxproj
#	apps/ios/Plannotator/App/AppModel.swift
#	apps/ios/Plannotator/App/SourceSession.swift
#	apps/ios/project.yml
…oad says so instead of spinning, the proof traces the surface and leaves the app fresh after every class
@backnotprop
backnotprop force-pushed the feat/mobile-m2-attachments branch from b14005c to d1c4bab Compare October 8, 2026 19:34
…ments

# Conflicts:
#	.github/workflows/ios.yml
#	AGENTS.md
#	apps/ios/Plannotator.xcodeproj/project.pbxproj
#	apps/ios/Plannotator/Thread/ReplyBar.swift
#	apps/ios/README.md
#	apps/ios/scripts/proof.ts
…computer's phones revoked); the plan's comment falls back to the toolstrip's pinpoint when the simulator raises no edit menu, naming the path
… every proof class starts from a cold install
…ments

# Conflicts:
#	.github/workflows/ios.yml
#	AGENTS.md
#	apps/ios/Plannotator.xcodeproj/project.pbxproj
#	apps/ios/PlannotatorKit/Sources/PlannotatorKit/InboxClient.swift
#	apps/ios/project.yml
#	apps/ios/scripts/proof.ts
@backnotprop
backnotprop merged commit f1446e4 into main Oct 9, 2026
34 checks passed
@backnotprop
backnotprop deleted the feat/mobile-m2-attachments branch October 9, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant