Skip to content

feat(preview): open heic, heif and avif images - #4009

Merged
kulmann merged 6 commits into
mainfrom
feat/preview-heic-avif
Oct 26, 2026
Merged

kulmann merged 6 commits into
mainfrom
feat/preview-heic-avif

Conversation

@dschmidt

@dschmidt dschmidt commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

Stacked on #3997, which makes the dev stack render previews for heic, heif, avif and the two HEIF image sequence types through imagor and announce them. This PR makes the web client open those files.

The viewer never hands the original bytes of a raster image to the browser, it shows a preview the server rendered. So whether a heic opens is not a question of the browser's decoders or of a format list compiled into the app, but of whether the server has a preview for that file, and the server already says so per file via oc:has-preview and the graph thumbnails relationship.

  • The preview app claims image/avif, image/heic, image/heic-sequence, image/heif and image/heif-sequence like any other image. All five are in reva's extension-to-type table (avif, heic, heics, heif, heifs), so the server reports them for uploaded files. .hif, the extension Canon and Sony cameras use for HEIF, is added to that table in Map .hif files to image/heif reva#911.
  • Two filters, kept apart on purpose. canOpen decides whether the viewer takes a file at all, from the file and the server only, so the "open with" action and the carousel come out the same in every browser: audio, video, svg and vault images are taken as they are, every other image only if hasPreview() says yes. canRender decides in the viewer whether this browser can show a file it took.
  • ApplicationFileExtension gets a canOpen hook, evaluated in useFileActions, so an app can narrow down the files of a claimed type it is offered for. The preview app passes its canOpen. On a server whose thumbnailer does not read a format the editor stays away and download takes over, instead of opening into an empty stage. A jpeg the server has no preview for is treated the same way, it had nothing to show before either.
  • Only vault images depend on the browser: the server holds just their ciphertext, so the browser gets the decrypted original, and whether it reads heic, avif or tiff differs between browsers (Safari does, Chromium and Firefox don't read heic and tiff). useBrowserImageSupport probes that once per viewer, by decoding a 2x2 image of each of the three formats. The probe images live in src/assets/probes and are imported with ?inline, so they ship inside the bundle and no request leaves the browser; assetsInclude in the shared vite config teaches vite that .heic and .tif are assets. A vault image in a format the browser doesn't read is not fetched at all, the viewer says "cannot be displayed in this browser" instead.
  • Vault files get their mime type from the cleartext extension on the client (vaultMimeType.ts), that table now knows avif, heic, heics, heif, heifs, hif, tif and tiff. Without it a heic in a vault was application/octet-stream and did not open at all.
  • Two more things are reported as errors now rather than left blank: a resolved load without a url, and an original the browser fails to decode after all, via an error event on MediaImage and MediaMotionPhoto.
  • avif, heic and heif get the image icon in the file list instead of the preview app's own icon.

Supersedes #3187, which gated the same formats with a compiled-in requiresServerPreview flag.

Needs a server image newer than the merge of opencloud-eu/opencloud#3397, see #3997.

Docs: the new canOpen hook on ApplicationFileExtension is public extension API and needs a mention in the developer docs (https://docs.opencloud.eu/docs/dev/web/, where the file extensions of an app are described), next to mimeType, extension, routeName and secureView: signature (resource: Resource) => boolean, called after the type matched, false keeps the app out of the "open with" actions for that one file.

Follow-up, not part of this PR: every vault upload scenario in rclone-crypt/ fails in Playwright WebKit, the new ones included, in the pre-existing upload and save steps. They prove the encryption by reading the request body through page.route() and looking for the RCLONE magic, and Playwright's WebKit driver returns null from postDataBuffer() for Blob bodies (fetch and XHR alike, checked with a probe in both browsers), which is what Uppy/tus sends. That is the driver, not the OS, so it holds on a mac too, and it keeps the @heic-decoder scenario below from ever reaching the viewer in WebKit. CI runs rclone-crypt in Chromium only, so it does not show there. The fix is isolated: one API helper that lists the vault folder over WebDAV (ciphertext names), fetches each blob and checks it starts with RCLONE and holds no cleartext, replacing the three page.route() blocks in resource/upload.ts (two) and resource/viewer.ts (one), plus passing the user into those page object calls. Roughly 60 lines in the e2e support code, no product change.

Related Issue

How Has This Been Tested?

  • test environment: unit tests; e2e in Chromium and Linux WebKit against opencloudeu/opencloud-rolling:daily with shumc/imagor:1.9.11 and the announced types from feat(dev): make previews with imagor and announce heic, heif and avif #3997; the probe images decoded in both browsers via Playwright (png and avif load, heic and tiff fire error, in Linux WebKit too)
  • fixtures samplePhoto.heic and samplePhoto.avif: a 128x96 test pattern from ffmpeg, encoded with heif-enc, so there is nothing to license
  • e2e preview/serverRenderedImages.feature: a heic and an avif uploaded to a folder show a thumbnail in the list and a preview in the sidebar, and open in the media viewer. New CI suite preview with an imagor service and the announced types, like the dev stack. Passes in Chromium and WebKit
  • e2e rclone-crypt/vault.feature, three scenarios: an avif in a vault opens from the original in every browser; a heic in a vault opens from the original in a browser that decodes heic (@heic-decoder); a heic in a vault is listed, opens, and the viewer says "cannot be displayed in this browser" without a single GET for the file (@no-heic-decoder). The playwright config picks the tagged scenario per project: Chromium and Firefox never decode heic, WebKit does on macOS through the system image frameworks and not on Linux, so the suite can run on a developer's mac as well as in CI. Verified here in Chromium; in WebKit they hit the follow-up above before reaching the viewer, and the @heic-decoder scenario additionally needs a mac
  • unit: a heic with has-preview is in the carousel and gets the preview action, one without is left out and falls back to download; svg, vault images, audio and video stay in the carousel without a server preview, a vault heic included; a vault image in a format the browser doesn't read is not fetched and shows the unsupported state; the browser probe takes jpeg, png, gif and webp for granted, reports what the browser decoded for avif, heic and tiff, lets heif and the sequence types follow heic; a vault IMG_0001.HEIC gets image/heic
  • a .hif is typed application/octet-stream by the server and gets has-preview 0 without Map .hif files to image/heif reva#911, so it stays download-only here
  • noted on the way: the server does not publish files.thumbnail in its capabilities, so the announced types cannot be read from there
  • open: shares and public links deliver has-preview / thumbnails for heic; a heic live photo in the viewer; Firefox was not run locally

Types of changes

  • Bugfix
  • Enhancement (a change that doesn't break existing code or deployments)
  • Breaking change (a modification that affects current functionality)
  • Technical debt (addressing code that needs refactoring or improvements)
  • Tests (adding or improving tests)
  • Documentation (updates or additions to documentation)
  • Maintenance (like dependency updates or tooling adjustments)

@dschmidt dschmidt self-assigned this Oct 24, 2026
@dschmidt
dschmidt force-pushed the feat/preview-heic-avif branch from dc7b350 to f3c46ca Compare October 24, 2026 06:11
@dschmidt
dschmidt force-pushed the feat/preview-heic-avif branch 2 times, most recently from 48e671d to 1cb37eb Compare October 24, 2026 06:50
@dschmidt
dschmidt added this pull request to stack #4011 October 24, 2026 06:55
@dschmidt
dschmidt force-pushed the feat/preview-heic-avif branch 4 times, most recently from af6f04f to 54100f9 Compare October 24, 2026 08:04
@dschmidt
dschmidt marked this pull request as ready for review October 24, 2026 09:05
@dschmidt
dschmidt force-pushed the feat/preview-heic-avif branch from 6e12172 to 0a04d36 Compare October 26, 2026 08:53
Base automatically changed from feat/dev-stack-imagor to main October 26, 2026 09:06
@dschmidt
dschmidt force-pushed the feat/preview-heic-avif branch from 0a04d36 to 2c7ebd6 Compare October 26, 2026 09:15
The viewer never hands the original bytes of a raster image to the
browser, it shows a preview the server rendered. So whether a heic opens
is not a question of the browser's decoders or of a format list compiled
into the app, but of whether the server has a preview for that file, and
the server already says so per file via `oc:has-preview` and the graph
thumbnails relationship.

The preview app therefore claims heic, heif and avif like any other image
and decides per resource. `canOpen` decides whether the viewer takes a
file at all, from the file and the server only, so the action and the
carousel come out the same in every browser: audio, video, svg and vault
images are taken as they are, every other image only if `hasPreview()`
says yes. A new `canOpen` hook on `ApplicationFileExtension` carries that
into the file actions, so on a server whose thumbnailer does not read a
format the editor stays away and download takes over, instead of opening
into an empty stage. A jpeg the server has no preview for is treated the
same way, it had nothing to show before either.

`canRender` then decides in the viewer whether this browser can show a
file it took. Only vault images depend on it: the browser gets their
decrypted original, and whether it reads heic, avif or tiff differs
between browsers. That is probed once when the app is set up, by decoding
a 2x2 image of each format that ships inlined with the app, and read
synchronously from then on. A vault image in a format the browser doesn't
read is not fetched at all, the viewer says so instead.

Two more things are reported as errors now rather than left blank: a
resolved load without a url, and an original the browser fails to decode
after all, via an `error` event on the image components. heic, heif and
avif also get the image icon in the file list instead of the preview
app's own icon.
The fixtures are a 128x96 test pattern encoded with heif-enc, so there
is nothing to license. In a folder both get a thumbnail and open in the
viewer, which needs the server to render previews for them: the CI
stack gets an imagor service for the new preview suite, announced the
same way as in the dev stack.

In a vault the browser gets the original, so what happens depends on
its decoders. An avif opens in every browser. A heic opens only in a
browser that decodes heic, and is otherwise listed but not fetched, the
viewer says so. Both are their own scenario, tagged @heic-decoder and
@no-heic-decoder, and the playwright config picks the one that fits the
project: chromium and firefox never decode heic, WebKit does on macOS
through the system image frameworks and not elsewhere. So the suite runs
on a developer's mac as well as in CI.
The pinned commit predates opencloud-eu/opencloud#3397, so the server
built for CI ignored WEBDAV_THUMBNAILS_GENERATOR_URL and
OC_THUMBNAILS_ADDITIONAL_SOURCE_TYPES and the preview suite found no
thumbnails. Pinned to the current main.
canRender answers false for a probed format until the probes finished,
so a vault avif, heic, heif or tiff that loaded before that was marked
unsupported and never fetched. performLoad now awaits the probes and
sets isUnsupported from the result, so a file that recovered no longer
shows the browser message for an unrelated decode error.
@kulmann
kulmann enabled stacked auto-merge October 26, 2026 11:23
@kulmann
kulmann merged commit d70dcb2 into main Oct 26, 2026
28 checks passed
@kulmann
kulmann deleted the feat/preview-heic-avif branch October 26, 2026 11:38
@openclouders openclouders mentioned this pull request Oct 26, 2026
1 task
@JammingBen JammingBen mentioned this pull request Oct 26, 2026
1 of 7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FR] Thumbnailer for HEIC

2 participants