Repository navigation
HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI - #8844
Open
GauthamBanasandra wants to merge 3 commits into
Open
GauthamBanasandra wants to merge 3 commits into
GauthamBanasandra wants to merge 3 commits into
Conversation
The Windows container image and the libhdfspp CMake files had drifted out of sync with trunk, leaving `mvn clean install` unbuildable on Windows: * Dockerfile_windows_10 still installed JDK 8, which the enforcer rejects now that Hadoop requires JDK 17. Switch to the Microsoft Build of OpenJDK 17. * vcpkg.json requested protobuf 3.25.5, a version that has never existed in the vcpkg version database, so dependency installation failed outright. Pin to 4.25.1, the nearest version vcpkg actually publishes. * vcpkg tag 2025.03.19 referenced MSYS2 packages that have since been purged from the upstream mirror. Move to 2026.07.29 and refresh the libzstd and rsync pins to versions that are still published. Refresh CMake to 3.30.2 alongside the toolchain bump. * Boost floated past 1.86, which removed boost/asio/io_service.hpp that libhdfspp includes. Pin all 55 Boost ports to 1.86.0 -- the same version install-boost.sh already pins for the Linux build. Narrow the Boost dependency to the four ports libhdfspp actually includes. * The protobuf/protoc compatibility check leaves CMAKE_REQUIRED_LIBRARIES pointing at the Abseil targets, whose C++ feature requirements a later C-language check_symbol_exists() cannot satisfy under the Visual Studio generator. Reset the variables once the check is done. * The legacy PROTOBUF_LIBRARY and PROTOBUF_PROTOC_LIBRARY variables resolve to import DLLs under vcpkg, which MSVC rejects with LNK1107. Use the protobuf::libprotobuf and protobuf::libprotoc imported targets instead. * hadoop-yarn-capacity-scheduler-ui pointed packageManagerScript at node_modules/.bin/react-router, an sh shim that node cannot execute on Windows. Point it at the real entry point instead. * Switch the base image to windows/servercore, which carries everything the build needs at a fraction of the size of the full Windows image. Server Core enforces password complexity by default, so relax that policy before creating the passwordless HadoopBuilder account. Verified with a single clean `mvn clean install` in the Windows container against a fresh trunk clone: 120/120 modules SUCCESS in 48:59 min, producing winutils.exe, hadoop.dll and hdfs.dll in the distribution tarball. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six corrections, all found while reproducing the Windows native build:
* Docker Desktop has to be switched to Windows containers first. In the
default Linux container mode every command in this section fails straight
away with "no matching manifest for linux/amd64", which the instructions
never mentioned.
* The Maven repository mount pointed at ContainerAdministrator's home.
Dockerfile_windows_10 ends with USER HadoopBuilder, so that mount is
silently ignored and the host repository is never actually reused.
* docker build and docker run both need -m 8g. The default container memory
is too small for the vcpkg source builds and for the Maven reactor.
* "docker cp" is suggested for retrieving the tarball, but it fails against a
running Hyper-V isolated container -- the default isolation on Windows
client hosts. Document that, and the -v alternative.
* The manual (non-Docker) vcpkg route still checked out tag 2025.03.19, whose
MSYS2 packages have since been purged from the upstream mirror.
* -Dcmake.prefix.path is read by nothing. The only reader in the tree is
${windows.cmake.prefix.path} in hadoop-hdfs-native-client/pom.xml, and
neither property is defined in any pom, so the documented flag left
CMAKE_PREFIX_PATH holding an unexpanded literal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
9 open issues
protobuf::libprotobuf/protobuf::libprotocimported targets, but… · Newboost-*overrides, which is hard to maintain… · New4.25.1, while the PR description notes Linux pins Protobuf3.25.5… · Newboost-*overrides, which is hard to maintain… · NewCMAKE_REQUIRED_*after the compatibility check changes global state for the rest of… · New
What changed in this PR
Fixes Windows native build breakages by updating the Windows Docker build environment, pinning vcpkg dependencies, modernizing libhdfspp’s Protobuf linking in CMake, and correcting the YARN capacity-scheduler UI build script path.
Changes:
- Update Windows Docker image/tooling (Server Core base, JDK 17, newer vcpkg tag, newer CMake, refreshed MSYS2 pins).
- Pin Windows vcpkg dependency versions (narrow Boost ports; pin Boost to 1.86.0 and Protobuf to a resolvable version).
- Migrate libhdfspp CMake from legacy
PROTOBUF_*variables toprotobuf::imported targets; fix a CMake required-libs/includes leak. - Fix Windows YARN UI build by pointing
packageManagerScriptto a real JS entry point.
| File | Description |
|---|---|
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/pom.xml | Adjusts UI build script path to be Windows-executable. |
| hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt | Switches test linkage to protobuf::libprotobuf imported target. |
| hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto/CMakeLists.txt | Switches protoc plugin linkage to protobuf::libprotoc / protobuf::libprotobuf. |
| hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt | Uses protobuf:: targets; resets CMake “required” vars after checks. |
| dev-support/docker/vcpkg/vcpkg.json | Pins Boost ports and Protobuf version; narrows Boost dependencies. |
| dev-support/docker/Dockerfile_windows_10 | Refreshes Windows build image (base image, JDK 17, vcpkg tag, CMake, MSYS2 package pins, password policy tweak). |
| BUILDING.txt | Updates Windows Docker/vcpkg build instructions and flags. |
Review effort: Lite
Comment on lines
152
to
156
| set (CMAKE_REQUIRED_INCLUDES ${PROTOBUF_INCLUDE_DIRS}) | ||
| set (CMAKE_REQUIRED_LIBRARIES | ||
| ${PROTOBUF_LIBRARY} | ||
| ${PROTOBUF_PROTOC_LIBRARY} | ||
| protobuf::libprotobuf | ||
| protobuf::libprotoc | ||
| ${protobuf_ABSL_USED_TARGETS}) |
Comment on lines
+61
to
+62
| RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi | ||
| RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait |
Contributor
There was a problem hiding this comment.
Yes, Please can we add a sha checksum stage here.
| RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile $Env:TEMP\cmake-3.19.0-win64-x64.zip | ||
| RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip -DestinationPath "C:\CMake" | ||
| # Install CMake 3.30.2. | ||
| RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile $Env:TEMP\cmake-3.30.2-windows-x86_64.zip |
Comment on lines
+130
to
137
| # Windows Server Core enables password complexity by default (unlike the | ||
| # Windows client image), which rejects the passwordless build account below. | ||
| RUN powershell secedit /export /cfg "C:\pwpol.cfg" | ||
| RUN powershell "(Get-Content C:\pwpol.cfg).Replace('PasswordComplexity = 1', 'PasswordComplexity = 0') | Out-File C:\pwpol.cfg" | ||
| RUN powershell secedit /configure /db "C:\windows\security\local.sdb" /cfg "C:\pwpol.cfg" | ||
| RUN powershell Remove-Item -Force "C:\pwpol.cfg" -Confirm:$false | ||
|
|
||
| RUN powershell New-LocalUser -Name 'HadoopBuilder' -Description 'User account for building Apache Hadoop' -Password ([securestring]::new()) -AccountNeverExpires -PasswordNeverExpires |
Comment on lines
+4
to
+7
| "boost-asio", | ||
| "boost-date-time", | ||
| "boost-system", | ||
| "boost-program-options", |
| { | ||
| "name": "protobuf", | ||
| "version": "3.25.5" | ||
| "version": "4.25.1" |
Comment on lines
+12
to
+21
| "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d", | ||
| "overrides": [ | ||
| { | ||
| "name": "protobuf", | ||
| "version": "3.25.5" | ||
| "version": "4.25.1" | ||
| }, | ||
| { | ||
| "name": "boost-algorithm", | ||
| "version": "1.86.0" | ||
| }, |
Comment on lines
+178
to
+179
| unset (CMAKE_REQUIRED_INCLUDES) | ||
| unset (CMAKE_REQUIRED_LIBRARIES) |
Comment on lines
+619
to
+620
| and the resulting image to be around 23 GB. Note that -m and --isolation are options of the classic | ||
| builder, so leave DOCKER_BUILDKIT unset; BuildKit does not accept them. |
Copilot stopped reviewing on behalf of
GauthamBanasandra due to an error
October 19, 2026 17:42
|
💔 -1 overall
This message was automatically generated. |
steveloughran
requested changes
Oct 22, 2026
steveloughran
left a comment
steveloughran
left a comment
Contributor
There was a problem hiding this comment.
Looks good, though I would like that sha 512 checksum validation in, assuming have it present in the target VMs.
One point: text is all about Windows 10. Does this work in windows 11?
Comment on lines
+61
to
+62
| RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi | ||
| RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait |
Contributor
There was a problem hiding this comment.
Yes, Please can we add a sha checksum stage here.
Member
Author
Thanks for the review @steveloughran. Yes, it works on Windows 11. I'll update the text to show Windows 10/11. |
Contributor
|
That's good. I have a windows 11 laptop for my college work, see. My PR #8811 updates some of docs related to windows, specifically states how to configure git to check out the repo. I don't look at the rest of the build (which I haven't tried). |
pan3793
reviewed
Oct 23, 2026
| RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.5-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst | ||
| RUN powershell zstd -d $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst -o $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar | ||
| # Install libzstd 1.5.7 needed for rsync 3.5.1. | ||
| RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.7-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.7-1-x86_64.pkg.tar.zst |
Member
There was a problem hiding this comment.
we don't need zstd native lib since HADOOP-19855 (#8399)
Review comments from @steveloughran, @pan3793 and the Copilot reviewer: * Verify the integrity of every artifact the image downloads. All nine are now pinned by SHA-512 and checked with Get-FileHash before use, following the pattern install-pip.ps1 already uses for get-pip.py. Each value was cross-checked against the publisher's own published digest where one exists: Microsoft publishes SHA-256 for the JDK, cmake.org for CMake and the ASF publishes SHA-512 for Maven, and all three matched. * Pin boost-coroutine. It was missing from the override list, so it alone resolved to the baseline and installed at 1.91.0 while the other 55 Boost ports were pinned at 1.86.0. Confirmed by reading vcpkg's own package inventory in the built image. * Restore the password complexity policy once HadoopBuilder exists, instead of leaving it relaxed for the life of the image. The restore folds into the secedit block that already runs immediately afterwards, so no extra layer is added and the account stays passwordless, which is what lets USER work. * Save and restore CMAKE_REQUIRED_INCLUDES and CMAKE_REQUIRED_LIBRARIES around the protoc compatibility check rather than unsetting them. An earlier check in the same file sets CMAKE_REQUIRED_LIBRARIES for the thread_local probe, so unsetting discarded state rather than restoring it. * Give the explicit commands for clearing DOCKER_BUILDKIT in BUILDING.txt, for both cmd.exe and PowerShell, rather than only saying to leave it unset. * Note in BUILDING.txt that the Docker route is verified on a Windows 11 host. The build runs inside a Windows container, so the host version matters only for Docker's own requirements. Also documents that the zstd CLI the image installs is only used to unpack the MSYS2 archives, and is unrelated to the native zstd codec that HADOOP-19855 replaced with zstd-jni. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
steveloughran
approved these changes
Oct 23, 2026
steveloughran
left a comment
steveloughran
left a comment
Contributor
There was a problem hiding this comment.
+1 with my requests, leaving on @pan3793's requested change.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
mvn clean installwith-Pnative-wincannot complete on Windows. The Docker image described inBUILDING.txt no longer builds at all, and even with a working image the native build fails in
libhdfspp and again in the YARN capacity-scheduler UI. There is no single cause — eight separate
defects have accumulated, each of which independently stops the build.
The build image no longer builds
Dockerfile_windows_10installs Azul JDK 8. Hadoop requires JDK 17, so the enforcer rejects thebuild immediately. This line has not been touched since 2022.
vcpkg installfails because the pinnedprotobuf 3.25.5cannot be resolved from vcpkg'sversion database. The pin was added by HADOOP-19605 on 2025-11-10.
2025.03.19references MSYS2 packages (msys2-runtime-3.5.4-2) that havesince been purged from the upstream mirror. MSYS2 keeps only current versions.
libzstdandrsyncMSYS2 pins in the Dockerfile have been purged for the same reason.The native build fails
builtin-baseline, so it floats as thebaseline moves. Boost 1.87 removed
boost/asio/io_service.hpp, which five libhdfspp files stillinclude (
ioservice.h,ioservice_impl.h,mock_connection.{h,cc},remote_block_reader_test.cc).libhdfspp/CMakeLists.txtleavesCMAKE_REQUIRED_LIBRARIESpointing at the Abseil targets. Abseil carries C++ featurerequirements that a later C-language
check_symbol_exists()cannot satisfy under the VisualStudio generator, so that check fails spuriously.
PROTOBUF_LIBRARY,PROTOBUF_PROTOC_LIBRARYandPROTOBUF_LIBRARIESvariablesresolve to import DLLs under vcpkg, which MSVC rejects with
LNK1107.hadoop-yarn-capacity-scheduler-uisetspackageManagerScripttonode_modules/.bin/react-router. On Windows that path is anshshim, whichnodecannotexecute.
Why Linux is unaffected
The Linux build pins its native dependencies inside the repository —
dev-support/docker/install-boost.shhard-codes Boost 1.86.0 andinstall-protobuf.shhard-codesProtobuf 3.25.5, both built from source. The Windows build instead delegates to vcpkg and lets
versions float with the baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3:
the Windows build has no equivalent of the version lock the Linux build has had all along.
Fix
2026.07.29; refreshthe
libzstdandrsyncpins; refresh CMake to 3.30.2. Switch the base image towindows/servercore, which carries everything the build needs at a fraction of the size, andrelax the password policy that Server Core enables by default so the existing passwordless
HadoopBuilderaccount can still be created.boostmetapackage to the four portslibhdfspp actually includes (asio, date-time, program-options, system), and pin all 55
transitively required Boost ports to 1.86.0 — the same version
install-boost.shpins for Linux.Pin protobuf to a version vcpkg publishes.
CMAKE_REQUIRED_INCLUDESandCMAKE_REQUIRED_LIBRARIESafter the protoccompatibility check, and migrate from the legacy
PROTOBUF_*variables to theprotobuf::libprotobufandprotobuf::libprotocimported targets.packageManagerScriptat the real entry point,node_modules/@react-router/dev/bin.js.switched to Windows containers; it documented the Maven repository mount under
ContainerAdministratoralthough the Dockerfile ends withUSER HadoopBuilder, so the documentedmount is silently ignored; it recommended
docker cp, which fails against a running Hyper-Visolated container; it omitted the container memory the build needs; and the manual vcpkg route
still pinned the purged
2025.03.19tag.How was this patch tested?
Tested in the Windows container built from this PR's own
Dockerfile_windows_10(
windows/servercore:ltsc2019, Hyper-V isolation, 8 GB), against a fresh clone of trunk with thesechanges applied.
Full build — a single clean
mvn clean installwith-Pnative-win,dist, no resume:winutils.exe,hadoop.dllandhdfs.dll, present both inhadoop-dist/target/hadoop-3.6.0-SNAPSHOT/binand inside the distribution tarballwinutils smoke test —
systeminfo,ls,chmodandhardlinkall run correctly against theproduced binary.
Unit tests —
hadoop-common, 5564 tests in 39:02 min: 5063 passed, 38 failures, 78 errors, 385skipped. None of the failures are attributable to this PR: no file under
hadoop-commonorwinutilsis modified here, so that module builds and tests as pristine trunk. The failures are thepre-existing Windows set (
TestSSLHttpServer,TestHttpServer,TestShellBasedUnixGroupsMapping,TestGroupFallbackand similar), plus some forked-JVMOutOfMemoryErrors caused by the 8 GBcontainer limit rather than by any code defect.
Two honest caveats, both pre-existing and outside this PR's scope:
hadoop checknative -areportswinutils: trueandhadoop: true, butzlib,bzip2,openssl,ISA-LandPMDKallfalse. This PR restores the ability to build; it does not addcodec support.
-Drequire.openssldoes not appear to be enforced on the Windows native path: vcpkg installsopenssl 3.6.3 and
libssl.lib/libcrypto.libare present, yet the build reports "build does notsupport openssl" and still succeeds. Worth a separate issue.
Separately,
-Dcmake.prefix.pathas documented in BUILDING.txt is read by nothing — the only readerin the tree is
${windows.cmake.prefix.path}inhadoop-hdfs-native-client/pom.xml, and neitherproperty is defined in any pom, so the documented flag left
CMAKE_PREFIX_PATHholding anunexpanded literal. BUILDING.txt is corrected here. The same misspelling also appears in
dev-support/bin/hadoop.shand may deserve its own issue. Note that the verified build above ranwith the misspelled flag, i.e. exactly as BUILDING.txt documented it before this change.
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required. — not applicable, no object storage code is touched.
LICENSE,LICENSE-binary,NOTICE-binaryfiles? — not applicable, nothing new is bundled into Hadoop's artifacts.AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
🤖 Generated with Claude Code