[C++] Many tests are not being run on Windows

[WillAyd]

Describe the bug, including details regarding any error messages, version, and platform.

With a clean build of Arrow:

$ cmake -S . -B build -G Ninja -DARROW_COMPUTE=ON -DARROW_BUILD_TESTS=ON
$ cmake --build build

The default ctest invocation seems to show everything as passing:

$ cd build
$ ctest

However, with verbose inspection, you will see that many tests are not actually running:

$ ctest --verbose
UpdateCTestConfiguration  from :C:/Users/Will/clones/arrow/cpp/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :C:/Users/Will/clones/arrow/cpp/build/DartConfiguration.tcl
Test project C:/Users/Will/clones/arrow/cpp/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
      Start  1: arrow-array-test

1: Test command: C:\Users\Will\clones\arrow\cpp\build\debug\arrow-array-test.exe
1: Working Directory: C:/Users/Will/clones/arrow/cpp/build/src/arrow
1: Test timeout computed to be: 10000000
1: Running main() from C:\Users\Will\clones\arrow\cpp\build\_deps\googletest-src\googletest\src\gtest_main.cc
1: [==========] Running 0 tests from 0 test suites.
1: [==========] 0 tests from 0 test suites ran. (0 ms total)
1: [  PASSED  ] 0 tests.
 1/49 Test  #1: arrow-array-test ......................   Passed    0.02 sec
test 2
      Start  2: arrow-buffer-test

2: Test command: C:\Users\Will\clones\arrow\cpp\build\debug\arrow-buffer-test.exe
2: Working Directory: C:/Users/Will/clones/arrow/cpp/build/src/arrow
2: Test timeout computed to be: 10000000
2: Running main() from C:\Users\Will\clones\arrow\cpp\build\_deps\googletest-src\googletest\src\gtest_main.cc
2: [==========] Running 0 tests from 0 test suites.
2: [==========] 0 tests from 0 test suites ran. (0 ms total)
2: [  PASSED  ] 0 tests.
 2/49 Test  #2: arrow-buffer-test .....................   Passed    0.01 sec
test 3
      Start  3: arrow-extension-type-test

3: Test command: C:\Users\Will\clones\arrow\cpp\build\debug\arrow-extension-type-test.exe
3: Working Directory: C:/Users/Will/clones/arrow/cpp/build/src/arrow
3: Test timeout computed to be: 10000000
3: Running main() from C:\Users\Will\clones\arrow\cpp\build\_deps\googletest-src\googletest\src\gtest_main.cc
3: [==========] Running 0 tests from 0 test suites.
3: [==========] 0 tests from 0 test suites ran. (0 ms total)
3: [  PASSED  ] 0 tests.
 3/49 Test  #3: arrow-extension-type-test .............   Passed    0.01 sec
test 4
      Start  4: arrow-misc-test

4: Test command: C:\Users\Will\clones\arrow\cpp\build\debug\arrow-misc-test.exe
4: Working Directory: C:/Users/Will/clones/arrow/cpp/build/src/arrow
4: Test timeout computed to be: 10000000
4: Running main() from C:\Users\Will\clones\arrow\cpp\build\_deps\googletest-src\googletest\src\gtest_main.cc
4: [==========] Running 0 tests from 0 test suites.
4: [==========] 0 tests from 0 test suites ran. (0 ms total)
4: [  PASSED  ] 0 tests.

...

From a quick glance, the only test executables actually running are arrow-compute-internals-test, arrow-json-integration-test, arrow-utility-test, and arrow-compute-scalar-cast-test. The common pattern for these is that they define their own main function. I also noticed that adding the following main function:

int main(int argc, char **argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

To gtest_util_test.cc allows you to run the tests in the arrow-gtest-util-test executable

Component(s)

C++

[lidavidm]

Hmm, linking with gtest_main is supposed to provide the main function; is that not being linked on Windows?

[WillAyd]

It does link gtest main - here is the verbose output of a compilation showing the linkage:

[3/3] cmake -E vs_link_exe --msvc-ver=1944 --intdir=src\arrow\testing\CMakeFiles\arrow-gtest-util-test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1444~1.352\bin\Hostx64\x64\link.exe /nologo src\arrow\testing\CMakeFiles\arrow-gtest-util-test.dir\gtest_util_test.cc.obj  /out:release\arrow-gtest-util-test.exe /implib:release\arrow-gtest-util-test.lib /pdb:release\arrow-gtest-util-test.pdb /version:0.0 /machine:x64  /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO /subsystem:console  release\arrow_testing.lib  release\arrow_gmock.lib  release\arrow_gtest_main.lib  release\arrow.lib  ws2_32.lib  release\arrow_gtest.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib

From comparing to the Meson build that was working, I think I've pinpointed the issue down to the fact that CMake is linking all ofgtest_main, gtest, and gmock. If you combine those into one linkage back to gmock_main then things work as expected:

cmake -E vs_link_exe --msvc-ver=1944 --intdir=src\arrow\testing\CMakeFiles\arrow-gtest-util-test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1444~1.352\bin\Hostx64\x64\link.exe /nologo src\arrow\testing\CMakeFiles\arrow-gtest-util-test.dir\gtest_util_test.cc.obj  /out:release\arrow-gtest-util-test.exe /implib:release\arrow-gtest-util-test.lib /pdb:release\arrow-gtest-util-test.pdb /version:0.0 /machine:x64  /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO /subsystem:console  release\arrow_testing.lib release\arrow_gmock_main.lib  release\arrow.lib  ws2_32.lib   kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib

[WillAyd]

There's also an upstream issue that may be related, although it doesn't come to the same conclusion that I have. See google/googletest#2157

[lidavidm]

Yikes. Maybe it's safest if we just have our own main.cpp and link that in everywhere on Windows? (Maybe via an object library?)

[WillAyd]

Possibly, although maybe this comment supports the behavior I found above:

google/googletest#2157 (comment)

Cliff notes: if your setup is compiled/installed as they are similar to vcpkg as of this writing, then link using only one of the following cases:

1. BOTH gtestd.lib and gtest_maind.lib; or
2. ONLY gmock_maind.lib

[lidavidm]

google/googletest#4360 is unfortunate.

From https://stackoverflow.com/questions/73300768/get-number-of-tests-ran-by-googletest-fail-on-0-tests-run it sounds like it may be good to have our own main.cpp to catch this case.

[WillAyd]

Yea that's possible, although I recall we recently had a change where we had to not use a custom main specifically with MSVC because it was causing an SEH exception...

#47049

This is super strange

[lidavidm]

Somehow I feel like I must not be using MSVC right because it feels like everything is always slightly broken (e.g. the gRPC issue that's dogged conda-forge for a while) 😅