Repository navigation
Support for FreeBSD-arm64 #71338
Description
Hi,
Not to spam #14537 (as it's already quite big) :)
As full native build of either v6 and v7 is possible under FreeBSD-amd64, I went ahead and tried to make the same process work for ARM64.
After getting configure to success, I've tried:
- Using Linux arm64 host to cross compile to FreeBSD (with custom rootfs created with libs, etc.)
As it was used before, to cross compile from Linux-amd64 to Freebsd-x64, this fail at very early stage with:
[ 1%] Built target System.Globalization.Native-Static
Scanning dependencies of target coreclrpal
[ 1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/dec/state.c.o
[ 1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
In file included from /root/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
/root/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
#include <type_traits>
^~~~~~~~~~~~~
[ 1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/enc/backward_references.c.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
1 error generated.
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_time.c.o
make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
Command I've used ROOTFS_DIR=/home/ubuntu/arm64 ./src/coreclr/build-runtime.sh -arm64 -debug clang14 -cross -os FreeBSD.
This does look like something missing in configure step, as those headers are present on rootfs_dir and host system:
root@instance-20220624-1221:/# find . -name type_traits
./home/ubuntu/arm64/usr/include/c++/v1/tr1/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/experimental/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/type_traits
./usr/include/c++/11/tr2/type_traits
./usr/include/c++/11/tr1/type_traits
./usr/include/c++/11/experimental/type_traits
./usr/include/c++/11/type_traits
./root/runtime/src/coreclr/inc/clr_std/type_traits
so I guess something during configure stage should be added to look in those directories?
- Using FreeBSD arm64 host directly to compile
Trying with both-crossand without, it fails with the same error:
[ 4%] Building C object libs-native/System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_evp_pkey_ecdh.c.o
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:450:1: error: unknown type name 'fpsimd_context'
fpsimd_context* GetNativeSigSimdContext(native_context_t *mc)
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:9: error: unknown type name 'fpsimd_context'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:47: error: unknown type name 'fpsimd_context'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:82: error: no member named '__reserved' in '__mcontext'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:458:30: error: use of undeclared identifier 'FPSIMD_MAGIC'
if(fp->head.magic == FPSIMD_MAGIC)
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:460:46: error: use of undeclared identifier 'fpsimd_context'
_ASSERTE(fp->head.size >= sizeof(fpsimd_context));
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:461:69: error: no member named '__reserved' in '__mcontext'
_ASSERTE(size + fp->head.size <= sizeof(mc->uc_mcontext.__reserved));
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/dbgmsg.h:362:35: note: expanded from macro '_ASSERTE'
#define _ASSERTE(expr) do { if (!(expr)) { ASSERT("Expression: " #expr "\n"); } } while(0)
^~~~
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:28: error: use of undeclared identifier 'fpsimd_context'
} while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:70: error: no member named '__reserved' in '__mcontext'
} while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:480:7: error: unknown type name 'fpsimd_context'
const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
^
10 errors generated.
[ 4%] Linking C static library libSystem.Native.a
--- pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o] Error code 1
make[2]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
*** [all] Error code 6
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
1 error
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
Command used was the same as with ad. 1. This looks more complicated, as it have something to do with libunwind being used? No idea how to get over that :)
Including patches I've come up with.
If anyone expierenced with cross build could comment on that, to even confirm it's possible in the current state to make that work or would it require far more work?
If someone is interested in helping/testing/making this work I can provide access to either FreeBSD or Linux ARM64 box that I use.
Activity
Thefrank commentedon Jun 28, 2022
OSX/BSD builds use the target system's libunwind:
from src/coreclr/pal/src/CMakeLists.txt
which makes it add only
include_directories(SYSTEM $ENV{ROOTFS_DIR}/usr/local/include)am11 commentedon Jun 28, 2022
Yup, I think we just need to update the list of compatible context registers for freebsd-arm64 to make it work.
sec commentedon Jun 28, 2022
If that would only make sense to me I would maybe help more, still learing on the fly. I found #40149 and doing quick change:
native compile fail with:
Is this the context registers that needs to be updated (looked at the code and there's no ARM64 for any of the targets)?
janvorli commentedon Jun 28, 2022
@sec, what are the detected features (the HAVE_xxx) for the FreeBSD arm64 build? When you do a clean build, it lists them to the console. From those, I should be able to tell where to update the context registers extraction.
sec commentedon Jun 28, 2022
@janvorli There you go:
edit: also attached full log from build attempt.
build_log.txt
janvorli commentedon Jun 28, 2022
It seems to me that your last change was correct. But you also need to add #elif (defined(HOST_UNIX) && defined(HOST_ARM64)) before the #else with #error that fires. In that #elif, you'd add:
I hope I got it right.
sec commentedon Jun 28, 2022
@janvorli Tried that, it failed with bunch of errors for every register:
Leaving this empty, will go on but then fail again with similar errors. If I'm looking at this correctly, I think it's missing some #def for HOST_ARM64 and TARGET_FREEBSD inside
src/coreclr/pal/src/include/pal/context.hwith those registers?And also there's again
error: unknown type name 'fpsimd_context'- which FreeBSD doesn't have, but if I add|| defined(TARGET_FREEBSD)next to TARGET_OSX those errors seems to go away (don't know if that will work in the end though, as I going blind in here. Also the undefined registers errors are there, so my bet is that fixing those first would be real fix).I wodner, if there's libunwind used (which state FreeBSD is supported under x64 and arm64) - shouldn't this be used (if I assume this code here is about that).
As I saw @sdmaclea did the OSX part of changes - maybe some hints? :)
Just to make sure, this is native build under ARM64 FreeBSD host.
janvorli commentedon Jun 28, 2022
Ah, I haven't noticed that your change around line 490 was like this:
That way, you have eliminated both definitions of the MCREG_xxx for arm64. What does the ucontext_t struct look like on FreeBSD arm64? We'll either need to define the MCREG_xxx for the FreeBSD separately or reuse the one for Linux / OSX if it happens to be the same.
janvorli commentedon Jun 28, 2022
Ah, found it in FreeBSD sources. So it is different and we will need to have another
#elif TARGET_FREEBSDthere. It seems that the definitions should look like:As for the floating point access we will also need an extra code here. Take the following with a grain of salt, it might not be exactly right or might need extra change at another place too:
janvorli commentedon Jun 28, 2022
Btw, this is based on https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/arm64/include/ucontext.h#L53
sec commentedon Jun 28, 2022
@janvorli Thanks, I was looking at the same files and was about to post :)
Can't find
PcandFpinucontextin structs btw.Managed to get past some errors, but now a lot of:
I haven't saw any existing
BSDREG_Xfor defined (even for OSX)I'm starting to think does this road make sense. Shouldn't this reuse existing ARM64 code and/or use machine specific libunwind for the rest?
115 remaining items
sec commentedon Mar 14, 2024
@janvorli Thanks (again) for looking into this - anything more I could provide/extract (and how) to maybe help?
janvorli commentedon Mar 14, 2024
You can use
x/32a $spto dump 32 addresses at the current sp. That will dump addresses and also symbols for ones that are native code addresses.sec commentedon Mar 14, 2024
Here you go
@janvorli Isn't this connected with dotnet/source-build#4007 - just saw this...
edit: using crossgen2 binary from 8.0.100 SDK makes the build complete without errors... so the issue is still in 8.0.3 (as I assume this crossgen2 was built during the vmr build) ?
edit 2: also hitting the issue with cross-compiled 9.0.100-preview.2.24103.2 used to compile 9-preview.2 from vmr :(
Thefrank commentedon May 4, 2024
Has this been resolved in net9p3?
sec commentedon May 5, 2024
The bug is in
libunwind, fix for .net was to revert some recent changes in internal lib they use, but for FreeBSD we use the ports version which have this bug for some time now :(janvorli commentedon May 6, 2024
You can possibly temporarily switch to using the internal libunwind copy. It is essentially the same code as upstream with a couple of patches that we remove as soon as a fix is made upstream.
Thefrank commentedon May 7, 2024
AFAIK there should only be two places that need changes:
runtime/src/coreclr/pal/src/CMakeLists.txt
Lines 1 to 5 in 7dc3669
runtime/src/native/corehost/apphost/static/CMakeLists.txt
Lines 225 to 227 in 7dc3669
That should change the behavior back to using the internal version
janvorli commentedon May 7, 2024
Right, that should be all that's needed.
sec commentedon Aug 21, 2024
Small update for arm64 interested. Doing crossbuild for 9-preview.7 is possible (there are 2 patches needed for ILC, more here #104497). Using that as bootstrap for building natively runtime and aspnet mostly work - produced ILC segfaults becuase of missing symbols export. sdk/installer build and produce working SDK and the rest. Should be possible to perform VMR build, when ILC thing is resolved.
edit: ilc bug was resolved, it was my fault not to replace old nuget, so building all repos is a go.
sec commentedon Dec 4, 2024
Tried to build 9.0.0 using internal libunwind (CLR_CMAKE_USE_SYSTEM_LIBUNWIND 0 in both places), (with libunwind installed system-wide, as needed by bootstrap SDK), but it fails with:
can this be related to fact, that we have
/usr/local/includein other places maybe or this is something different?am11 commentedon Dec 4, 2024
Based on libunwind/libunwind@032abaa, we will need to add aarch64/setcontext.S next to
runtime/src/native/external/libunwind.cmake
Line 92 in 1fc8222
dotnet-policy-service commentedon Apr 9, 2026
Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.