Switch all rendering fully to Vulkan and update filament to latest. by ssheorey · Pull Request #7550 · isl-org/Open3D

ssheorey · 2026-09-07T23:54:00Z

The previous 3DGS rendering design with hybrid OpenGL - Vulkan rendering for 3DGS was buggy in the case of multi-GPU machines. Windows provides a way to query the GPU in use and create an independent context on the same device (# GPU selection PR ) , but Linux does not have a portable way to do that. This led to crashes if opengl and vulkan were using different GPUs, or if there were attempts / miitigations to force that (such as env vars).

This PR simplifies the whole design by switching completely to Vulkan - so that a single context can be used for both filament and 3DGS shaders.

Type

Bug fix (non-breaking change which fixes an issue): Fixes #
New feature (non-breaking change which adds functionality). Resolves #
Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

Checklist:

I have run python util/check_style.py --apply to apply Open3D code style
to my code.
[ x This PR changes Open3D behavior or adds new functionality.
- Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
  updated accordingly.
- I have added or updated C++ and / or Python unit tests OR included test
  results (e.g. screenshots or numbers) here.
I will follow up and update the code if CI fails.
For fork PRs, I have selected Allow edits from maintainers.

Description

…ackaging Enable BUNDLE_OPEN3D_ML automatically when PyTorch or TensorFlow ops are built, ship Open3D-ML pip deps as extras_require[ml], and validate imports via require_ml_extra. Bundle Filament LLVM libc++ on Linux GUI wheels with a small ldd helper; refresh CI/Docker for Open3D-ML main, CUDA 12.6+ arch defaults, and local style --changed-only.

Make BUNDLE_OPEN3D_ML an option again, defaulting to ON only when ML ops are built and OPEN3D_ML_ROOT resolves, so local builds can opt out and CI jobs without an Open3D-ML checkout configure instead of erroring out. Windows wheels now bundle Open3D-ML too. Import the Open3D-ML backed submodules of open3d.ml, open3d.ml.torch and open3d.ml.tf lazily and gate them on require_ml_extra(), so the ops and layers stay usable with a core-only install and a missing dependency reports how to install the ml extra. Wheel tests install that extra. Keep Turing usable after dropping its SASS by emitting compute_75 PTX, which also needs a -virtual case when translating architectures for PyTorch. Fall back to the pybind extension when probing for libc++ in static builds, where libOpen3D is not packaged. Co-authored-by: Cursor <cursoragent@cursor.com>

NVIDIA's static CUDA libraries carry device code back to sm_50, ~47MB of which no GPU that Open3D supports can run. nvprune only accepts static libraries, not the linked Open3D library, so the imported CUDA targets are pruned and re-pointed before linking, cached per architecture list. Also fix the Windows path separator assertion in the TensorBoard plugin test, which started running there now that Windows wheels bundle Open3D-ML, and skip the module cleanly when the `ml` extra is not installed. Co-authored-by: Cursor <cursoragent@cursor.com>

…ask written from a parallel loop Fixes #7538

…v0.20

…hing PyTorch/NumPy (#7363)

…7497) Replace AssertTensorDevice(CPU) with .To(CPU) in InverseTransformation and TransformIndexer so callers with CUDA/SYCL tensors do not get device mismatch errors. Both functions read small float64 tensors (3x3 and 4x4) and copy the data into host-side arrays, so a device transfer is cheap and semantically correct. This is a general fix that covers all call sites including VoxelBlockGrid, PointCloud, and Image kernel paths.

Fix compilation error when building with USE_SYSTEM_EIGEN3=ON on macOS with clang++. The AddPoint template functions used Eigen::ArrayBase<Derived3> as an unconstrained parameter type, which causes substitution failures when the compiler instantiates Eigen::ArrayBase for types that don't derive from it. This manifests under stricter C++17 template resolution when Eigen is included from a system path (-isystem) rather than as a project-local header. Fix: - Add std::enable_if constraint requiring Derived3 to be derived from Eigen::ArrayBase<Derived3> - Change parameter from 'const Eigen::ArrayBase<Derived3>&' to 'const Derived3&' (semantically identical since Derived3 IS-A ArrayBase) Applied to both Accumulator::AddPoint and AccumulatorBackprop::AddPoint.

Previously BUNDLE_OPEN3D_ML=ON hardcoded GIT_TAG origin/main in the ExternalProject and ci_utils.sh forced a main branch checkout. This prevented reproducible builds using matching release tags. Add OPEN3D_ML_VERSION CMake cache variable (default: main) that controls the git ref checked out from the Open3D-ML repo. Users can set -DOPEN3D_ML_VERSION=v0.19.0 for reproducible builds or any other tag/branch. Also exposed as a Docker build arg in Dockerfile.ci, forwarded to cmake, and added to ci_build() in docker_build.sh (defaults to main). Removed the checkout -b main workaround in ci_utils.sh and Dockerfile.ci; replaced with fetch --tags so version tags are available when OPEN3D_ML_VERSION is set to one.

Return an unpacking status for non-standard exceptions in the receiver loop, preventing malformed RPC requests from hanging the worker. Make the Ubuntu 22.04 LLVM key download fail-fast and retryable so ARM64 build images can configure the LLVM 17 repository.

Export the installed GUI resource directory so installed-library wheel builds copy it into the wheel. Install the C++ agent skill in the devel component only, and build the Windows viewer archive from the runtime component.

filament needs iterator debug level 2 mdd build ninja needs cmake_build_type set to debug. remove C++ skills from windows viewer.

Copilot

🟡 Changes recommended

Vulkan import failure handling, Metal invalidation, GUI startup, and Windows Debug Filament configuration contain blocking correctness issues.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

cpp/open3d/visualization/rendering/filament/FilamentScene.cpp:1125

Non-Gaussian point-position updates no longer advance the splat revision. On Metal, the composite pass is only rerun when this revision/view state is dirty, so moving an occluding point cloud can leave the Gaussian overlay using the previous depth result. Mark the splat composite dirty when point positions are updated.
docs/tutorial/visualization/cpu_rendering.rst:19
VK_DRIVER_FILES selects Mesa's Lavapipe Vulkan ICD, not llvmpipe. Using the correct driver name avoids directing users toward the OpenGL software renderer.

Files reviewed: 88/89 changed files
Comments generated: 8
Review effort level: Balanced

Remove obsolete frame hooks and OpenGL-era abstractions, centralize per-view GPU resource teardown, avoid radix-pass label allocations, and consolidate availability checks for Vulkan and Metal rendering.

Use Debug-compatible prebuilt Filament variants for /MDd and /MTd, preserving iterator debug level 2 instead of selecting RelWithDebInfo archives. Preserve the Debug configuration when building Filament from source with MSVC, and propagate Debug into Windows Ninja ExternalProject children so their postfix libraries match the parent ABI. Link Debug matdbg when the selected Filament archive requires its DebugServer symbols. Remove ineffective Lavapipe setup from Windows workflows and skip Gaussian splat rendering only on Windows CI, where the hosted Vulkan loader cannot find a usable ICD. Make a partial Vulkan imported-target setup fatal after releasing the created wrappers and shared images, because compute must sample the same external depth image Filament renders. Address review feedback by starting dense SLAM construction only from Start, checking winget.exe exit status, correcting Lavapipe documentation and a prompt typo, retrying transient LLVM key connection refusals, and invalidating Gaussian composite output after non-Gaussian point-position updates.

Integrate the remote Gaussian-splat backend simplification while retaining the reviewed fatal handling for partial Vulkan output texture imports.

Remove the obsolete LLVM and libc++ development setup now that patched prebuilt Filament packages its GNU-ABI libc++ runtime. Link filamat after matdbg for Windows Debug archives, and skip the known macOS Tensor initializer-list test anomaly.

make windows cuda. apreview wheel.

Declare the Windows Debug Filament archives as download byproducts so Ninja can build their consumers. Detect the static GNU-ABI Filament runtime in installed packages and omit obsolete shared LLVM libc++ runtime discovery and wheel payloads.

Detect the static Filament C++ runtime through the public installed Open3D target so wheel builds do not require legacy shared LLVM libraries.\n\nUse oneMKL's Debug TBB threading archive for Windows Debug builds to match the /MDd runtime and avoid the static archive linker limit.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ssheorey · 2026-09-12T17:37:27Z

Reproduced the Metal RGB readback abort with the official unmodified Open3D development wheel 0.19.0+1a9eb99 on macOS26.6.2 / Apple M4 Max / Python3.12.11.

Minimal reproduction, executed with python -u -X faulthandler upstream-metal-repro.py:
import open3d as o3d
print('Open3D:', o3d.__version__, flush=True)
r = o3d.visualization.rendering.OffscreenRenderer(160, 90)
r.scene.set_background([1.0, 1.0, 1.0, 1.0])
print('Capturing RGB image', flush=True)
image = r.render_to_image()
print('Capture completed', image, flush=True)
Captured output:
Open3D: 0.19.0+1a9eb99
FEngine (64 bits) created at 0x9ba2b4000 (threading is enabled)
FEngine resolved backend: Metal
Selected physical device 'Apple M4 Max'
Backend feature level: 3
FEngine feature level: 1
Selected physical device 'Apple M4 Max'
Capturing RGB image
Precondition
in readPixels:1487
reason: The chosen combination of PixelDataFormat (4) and PixelDataType (0) is not supported for readPixels.


libc++abi: terminating due to uncaught exception of type utils::PreconditionPanic
Fatal Python error: Aborted

Thread 0x00000001f6596180 (most recent call first):
  File "/tmp/upstream-metal-repro.py", line 6 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, markupsafe._speedups, simplejson._speedups, charset_normalizer.md, charset_normalizer.cd (total: 6)
Exit status: 134
The failure identifies PixelDataFormat(4) / PixelDataType(0) in readPixels. The RGBA readback path in this PR addresses that format requirement while allowing the public RGB image API to be retained. Could the RGBA correction be merged as a focused fix ahead of the broader renderer changes?

We expect this to PR to be merged in a day or two.

Disable the unstable Windows XPU Debug matrix entry and skip only Gaussian-splat rendering tests on Linux ARM64. Configure installed-wheel CI to use Filament's paired prebuilt static C++ runtime instead of falling back to shared system libc++ discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use fixture-level Gaussian-splat skips for unsupported CI renderers, align external dependency Debug configuration, simplify Filament and Windows setup tooling, and refresh Gaussian-splat validation documentation.

review-notebook-app · 2026-09-14T05:34:17Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

ssheorey and others added 30 commits August 21, 2026 02:55

fix remove_radius_outlier is not thread-safe: std::vector<bool> m…

83cbd31

…ask written from a parallel loop Fixes #7538

remove poisson recon n_threads=1 constraint on c++ test.

51fdab2

Fix depth truncation for float depth images (issue #6347)

568670f

Stream large binary PCD writes #6607

8bbab4e

Make seeded RANSAC deterministic #7475

6fe3f01

dix tensorboard summary test errors.

23c70b4

issue fixer agent

ca83b1f

simplify agent instructions

3cda9b9

fix links for main-devel

9091117

Close TensorBoard writer in PyTorch test

6a74f7e

Revert CUDA static library pruning

de3b8d2

remove requirements install from cmake stub generation.

eabd589

Merge branch 'ss/v0.20' of github.com:isl-org/Open3D into ss/v0.20

f389b87

Fix tensor ICP convergence criteria #7367

d00b663

Clarify Jupyter visualizer setup #6979

e1a966f

Avoid redundant tensor ICP metric evaluation #7367

6447f0d

Enable Jupyter visualizer on macOS ARM64 #6979

e0a44f1

Update copyright years to 2026

cbbc9bd

Merge branch 'ss/v0.20' of https://github.com/isl-org/Open3D into ss/…

d68a70a

…v0.20

Fix PointCloud include ordering

0005ecb

remove warnings from __init__.py

84790f3

agents: prefer quick tests

2917469

Clarify IsContiguous/Contiguous docs as C-contiguous (row-major) matc…

8940ffc

…hing PyTorch/NumPy (#7363)

separate dev_wheels.html page for wheels to simplify install command

961cc98

ssheorey added 5 commits September 10, 2026 21:46

Fix resources and C++ skill packaging

7cb1d4f

Export the installed GUI resource directory so installed-library wheel builds copy it into the wheel. Install the C++ agent skill in the devel component only, and build the Windows viewer archive from the runtime component.

windows CI fix (SW lavapipe install, crl debug ninja)

3db9bec

Merge branch 'ss/gs-vk-fm' of github.com:isl-org/Open3D into ss/gs-vk-fm

524d5c8

Merge branch 'main' of github.com:isl-org/Open3D into ss/gs-vk-fm

a993955

ssheorey marked this pull request as ready for review September 11, 2026 05:19

ssheorey added 3 commits September 11, 2026 12:48

minor example fixes.

a86ae66

fix for:

4bee430

filament needs iterator debug level 2 mdd build ninja needs cmake_build_type set to debug. remove C++ skills from windows viewer.

Merge branch 'ss/gs-vk-fm' of github.com:isl-org/Open3D into ss/gs-vk-fm

b4375e1

ssheorey added this to the v0.20 milestone Sep 11, 2026

ssheorey requested a balanced review from Copilot September 11, 2026 23:40

Copilot started reviewing on behalf of ssheorey September 11, 2026 23:40

View session

Copilot AI reviewed Sep 11, 2026

View reviewed changes

ssheorey added 7 commits September 11, 2026 17:12

Simplify Gaussian splat renderer backends

743fb57

Remove obsolete frame hooks and OpenGL-era abstractions, centralize per-view GPU resource teardown, avoid radix-pass label allocations, and consolidate availability checks for Vulkan and Metal rendering.

Merge origin/ss/gs-vk-fm

f8872bf

Integrate the remote Gaussian-splat backend simplification while retaining the reviewed fatal handling for partial Vulkan output texture imports.

Fix prebuilt Filament CI dependencies

97b2a62

Remove the obsolete LLVM and libc++ development setup now that patched prebuilt Filament packages its GNU-ABI libc++ runtime. Link filamat after matdbg for Windows Debug archives, and skip the known macOS Tensor initializer-list test anomaly.

consolidate ci fix agent instructions.

8482d90

make windows cuda. apreview wheel.

Fix Filament runtime wheel packaging

4232ca2

Declare the Windows Debug Filament archives as download byproducts so Ninja can build their consumers. Detect the static GNU-ABI Filament runtime in installed packages and omit obsolete shared LLVM libc++ runtime discovery and wheel payloads.

ssheorey and others added 5 commits September 12, 2026 14:41

Harden Vulkan rendering CI setup

cc7b9d3

Use fixture-level Gaussian-splat skips for unsupported CI renderers, align external dependency Debug configuration, simplify Filament and Windows setup tooling, and refresh Gaussian-splat validation documentation.

web_visualizer.draw() now forwards args to draw()

cdf1d4d

Add loopback interfaces to webrtc for localhost use

8b26cd2

Merge branch 'ss/gs-vk-fm' of github.com:isl-org/Open3D into ss/gs-vk-fm

ad68529

ssheorey merged commit 099051e into main Sep 14, 2026
35 of 37 checks passed

ssheorey deleted the ss/gs-vk-fm branch September 14, 2026 18:34

mstoelzle mentioned this pull request Sep 17, 2026

Uninitialized UV0 corrupts legacy TriangleMesh rendering on Mesa Lavapipe #7565

Open

Navigation Menu

Repository navigation

Switch all rendering fully to Vulkan and update filament to latest. - #7550

Switch all rendering fully to Vulkan and update filament to latest.#7550
ssheorey merged 103 commits into
mainfrom
ss/gs-vk-fm

ssheorey commented Sep 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ssheorey commented Sep 12, 2026

Uh oh!

review-notebook-app Bot commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ssheorey commented Sep 7, 2026

Type

Motivation and Context

Checklist:

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ssheorey commented Sep 12, 2026

Uh oh!

review-notebook-app Bot commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants