Document supported OS, architectures, and compilers #482

Activity

1. daanx commented on Oct 27, 2021

   

   Daan (daanx)

   on Oct 27, 2021

   Collaborator

   More actions

   Hi Christian -- yes, it would be good to create a list of platforms.
   Generally though, mimalloc is written to be as portable as possible and should support a wide range of platforms; and generally supporting new/untested platforms should not require much effort.

   Currently, there are 2 tiers: the "official" supported ones that are always tested and used widely (e.g. Windows x64, Linux x64/arm64, and macOS x64/M1), and the rest (thanks to all the contributors to make mimalloc run on these! in particular, David CARLIER (@devnexen) :-) ).

   Now, there are 2 modes of using mimalloc (and other allocators for that matter): use the mimalloc API's directly and call mi_malloc, mi_strdup etc, or, override the standard allocator where calls to malloc are redirected to mi_malloc etc. When using mimalloc in static way, it usually just works (and this is how CPython would integrate)

   Overriding however is tricky on many platforms (but often unavoidable in many large code bases) (and fortunately CPython does not need this). Mimalloc has robust support on Windows, and recently the macOS zone based overriding is also getting more robust. Linux/FreeBSD works well with LD_PRELOAD.

   On other platforms this may not always be the case though: the main issue is that sometimes the exe loader will already call malloc itself to reserve memory for thread-local storage which means that mimalloc cannot use thread local variables (or we recurse). This is the case on macOS and most BSD variants. When using overriding we need to initialize very carefully in those situations. This is the situation that is not well-tested on OpenBSD/DragonFly etc but I am sure it can be addressed though.

   However, this is only the case for dynamic overriding and not an issue when using mimalloc API's directly. Since CPython uses the mimalloc API's directly the above issues should not apply.
   Similarly, on some platforms we use assembly to get the thread id efficiently but there is always a portable fallback that usually works just as well.

   So, if we make a supported platforms table, we should probably have 2 columns for dynamic overriding and static usage.
2. devnexen commented on Oct 28, 2021

   

   David CARLIER (devnexen)

   on Oct 28, 2021

   Contributor

   More actions

   If that helps basic functionality works fine on solaris based systems (with base compiler which build 32 bits binaries by default), api usage (unit tests pass) and overriding minimal usage work too even tough not used extensively.
   PowerPC archs should work fine, it's harder to tell with less used architectures like the MIPS family (also for example freebsd folks are actually discussing its entire removal support system-wise).

   We would be lucky if one HP-UX/AIX folk test one of these days :-)
3. thehans commented on Nov 17, 2021

   

   Hans Loeblich (thehans)

   on Nov 17, 2021

   Contributor

   More actions

   I'd like to know if mingw builds are officially supported or tested for, and is it possible to statically link a cmake-built mimalloc on mingw/windows?

   From the mimalloc README, there is only mention of building for windows with visual studio, and I'm finding it difficult to understand which combinations of cmake build options are valid for the various platforms.

   I ask about this since am attempting to implement mimalloc for an existing cross-platform application, and our windows build in particular relies on mingw. The overall build succeeds on our windows CI servers, but 99%+ of our tests are simply crashing after adding mimalloc. I have not yet been able to get a debug stacktrace and determine the exact cause of the crash, but I figured I'd check if any particular configuration like this is known to work before going much further.

   I'm also attempting this by having mimalloc as a git submodule of the parent project, and using cmake add_subdirectory to build it. Are there any examples of this approach, or pitfalls to watch out for when doing that?
4. daanx commented on Nov 17, 2021

   

   Daan (daanx)

   on Nov 17, 2021

   Collaborator

   More actions

   Hi Hans Loeblich (@thehans) -- with regard to Windows/mingw/cygwin,

   static or dynamic link, use mi_malloc api:     msvc: yes, clang-cl: yes, mingw: yes, cygwin: probably (unconfirmed)
   static override malloc api:                    msvc: yes, clang-cl: yes, mingw: ?, cygwin: ?    
   dynamic override malloc api.                   msvc: yes, clang-cl: yes, mingw: ?, cygwin: ?
   

   (where yes is confirmed working).
   Note that static override is dangerous on Windows as it won't override in imported DLL's. The dynamic override eventually depends on ucrtbase.dll so should be fairly compiler/language agnostic. That is why I think it should probably work on mingw but clearly you are seeing crashes for this scenario (right? with dynamically overriding malloc?). -- perhaps there is an easy fix; I can take a look at this early next week.

   Using mimalloc by using cmake add_subdirectory is excellent. Also, many people that use the mi_malloc api's directly (without overriding) include the src/static.c directly in a parent project (and add the include directory of mimalloc). This works well too and can be used if not using cmake for example.
5. thehans commented on Nov 17, 2021

   

   Hans Loeblich (thehans)

   on Nov 17, 2021

   Contributor

   More actions

   Thanks for the quick reply. The current approach I have which crashes on Windows is:

   Statically built library from MI_BUILD_STATIC=ON, and linked to mimalloc-static
   And MI_OVERRIDE=OFF, so not "statically overriden"

   Most of the heavy allocations in the application come from our usage of CGAL, and specifically its dependency on GMP.

   • For GMP I provide mi_malloc's functions by using their mp_set_memory_functions interface. (Note: their function signatures are apparently nonstandard so I use small wrappers which I've shown below * )
   • For CGAL itself I redefine their provided stl allocator macro as: #define CGAL_ALLOCATOR(t) mi_stl_allocator<t>
   • Finally, I have #include <mimalloc-new-delete.h> in a single file to override c++ new/delete

   Interestingly, I had previously tried the statically linked build, except with MI_OVERRIDE=ON, (and without any "manual" GMP or CGAL overrides shown above, nor the mimallloc-new-delete include) and that method passed all of our Windows CI server tests. I did not confirm if mimalloc was actually being applied in that case, so that is still a possible explanation of why that worked at all.
   However that previous approach failed all the MacOS tests instead, while the current approach only fails on Windows.
   (Either approach works fine on Linux).

   function wrappers for gmp:

   // gmp requires function signature with extra oldsize parameters for some reason.
   inline void *gmp_realloc (void *ptr, size_t /*oldsize*/, size_t newsize) { return mi_realloc(ptr, newsize); }
   inline void gmp_free (void *ptr, size_t /*oldsize*/) { mi_free(ptr); }
   #include <gmp.h>
   // init_mimalloc() called at beginning of "main" c++ function.
   inline void init_mimalloc() { mp_set_memory_functions(mi_malloc, gmp_realloc, gmp_free); } 
   
6. daanx commented on Nov 23, 2021

   

   Daan (daanx)

   on Nov 23, 2021

   Collaborator

   More actions

   Hi Hans Loeblich (@thehans), it seems you do everything correctly for a static Windows build; For Koka language, we use this with mingw all the time and it works fine. My guess is that there is some call to realloc/ strdup / realpath etc. that is not using mimalloc and later freed using mimalloc. On Linux the standard MI_OVERRIDE=ON will declare those using mimalloc and override all when loaded (while on Windows and macOS shared object files are independent/namespace'd).

   Best is probably build mimalloc in debug mode, get a backtrace and see if it was mixing of pointers.
   Otherwise, on Windows I recommend dynamic overriding which is very robust -- but I have not tested this with mingw; I'll try this later this week if I get time :-).

   (ps. clang-cl has come a long way on Windows, you may want to give that a try instead of mingw?)
   (ps. try again on macOS with the 1.7.3/2.0.3 versions as I recently improved the overriding on macOS)
7. thehans commented on Jan 15, 2022

   

   Hans Loeblich (thehans)

   on Jan 15, 2022

   Contributor

   More actions

   (ps. try again on macOS with the 1.7.3/2.0.3 versions as I recently improved the overriding on macOS)

   I've just tried this and still the same problem. I don't have direct access to mac hardware, but another user got this result:

   OpenSCAD(49739,0x208d35600) malloc: *** error for object 0x2e4c3093240: pointer being freed was not allocated
   OpenSCAD(49739,0x208d35600) malloc: *** set a breakpoint in malloc_error_break to debug
   

   Definitiely looks like a mixing of pointers as you suggested, would it still help to request a full backtrace from them?
8. tiran commented on Feb 10, 2022

   

   Christian Heimes (tiran)

   on Feb 10, 2022

   ContributorAuthor

   More actions

   I have updated the first document. I got Python with mimalloc 2.0.3 compiled and working with Intel ICC 2021.5. AIX xlc does not feature modern C11 stdatomic.h. It has GCC legacy __sync functions like __sync_val_compare_and_swap.