Update README.md - #3
Closed
altinity-robot wants to merge 1 commit into
Closed
Conversation
Enmk
pushed a commit
that referenced
this pull request
Sep 13, 2022
…yFunc()
clang-15 reports [1]:
<details>
<summary>ASan report</summary>
```
==1==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f1d04c4eb20 at pc 0x000031c4803c bp 0x7f1d05e19a00 sp 0x7f1d05e199f8
READ of size 8 at 0x7f1d04c4eb20 thread T200 (QueryPullPipeEx)
#0 0x31c4803b in DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3::operator()(unsigned long) const build_docker/../src/Common/GetPriorityForLoadBalancing.cpp:42:40
#1 0x31c4803b in decltype(static_cast<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&>(fp)(static_cast<unsigned long>(fp0))) std::__1::__invoke<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/type_traits:3640:23
#2 0x31c4803b in unsigned long std::__1::__invoke_void_return_wrapper<unsigned long, false>::__call<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/__functional/invoke.h:30:16
#3 0x31c4803b in std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>::operator()(unsigned long&&) build_docker/../contrib/libcxx/include/__functional/function.h:230:12
#4 0x31c4803b in unsigned long std::__1::__function::__policy_invoker<unsigned long (unsigned long)>::__call_impl<std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>>(std::__1::__function::__policy_storage const*, unsigned long) build_docker/../contrib/libcxx/include/__functional/function.h:711:16
#5 0x31c38b07 in std::__1::__function::__policy_func<unsigned long (unsigned long)>::operator()(unsigned long&&) const build_docker/../contrib/libcxx/include/__functional/function.h:843:16
#6 0x31c38b07 in std::__1::function<unsigned long (unsigned long)>::operator()(unsigned long) const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12
#7 0x31c38b07 in PoolWithFailoverBase<DB::IConnectionPool>::getShuffledPools(unsigned long, std::__1::function<unsigned long (unsigned long)> const&) build_docker/../src/Common/PoolWithFailoverBase.h:174:39
This frame has 2 object(s):
[32, 40) 'pool_size.addr' <== Memory access at offset 32 is inside this variable
[64, 88) 'ref.tmp' (line 18)
```
</details>
[1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/integration_tests__asan__[1/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Enmk
pushed a commit
that referenced
this pull request
Sep 21, 2022
…s and parallel KILL
Right now it is possible to call QueryStatus::addPipelineExecutor() when
the executors_mutex already acquired, it is possible when the query was
cancelled via KILL QUERY.
Here I will show some traces from debugger from a real example, where
tons of ProcessList::insert() got deadlocked.
Let's look at the lock owner for one of the threads that was deadlocked
in ProcessList::insert():
(gdb) p *mutex
$2 = {
__data = {
__owner = 46899,
},
}
And now let's see the stack trace of the 46899:
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1 0x00007fb65569b714 in __GI___pthread_mutex_lock (mutex=0x7fb4a9d15298) at ../nptl/pthread_mutex_lock.c:80
#2 0x000000001b6edd91 in pthread_mutex_lock (arg=0x7fb4a9d15298) at ../src/Common/ThreadFuzzer.cpp:317
#3 std::__1::__libcpp_mutex_lock (__m=0x7fb4a9d15298) at ../contrib/libcxx/include/__threading_support:303
#4 std::__1::mutex::lock (this=0x7fb4a9d15298) at ../contrib/libcxx/src/mutex.cpp:33
#5 0x0000000014c7ae63 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
#6 DB::QueryStatus::addPipelineExecutor (this=0x7fb4a9d14f90, e=0x80) at ../src/Interpreters/ProcessList.cpp:372
#7 0x0000000015bee4a7 in DB::PipelineExecutor::PipelineExecutor (this=0x7fb4b1e53618, processors=..., elem=<optimized out>) at ../src/Processors/Executors/PipelineExecutor.cpp:54
#12 std::__1::make_shared<DB::PipelineExecutor, std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >&, DB::QueryStatus*&, void> (__args=@0x7fb63095b9b0: 0x7fb4a9d14f90, __args=@0x7fb63095b9b0: 0x7fb4a9d14f90) at ../contrib/libcxx/include/__memory/shared_ptr.h:963
#13 DB::QueryPipelineBuilder::execute (this=0x7fb63095b8b0) at ../src/QueryPipeline/QueryPipelineBuilder.cpp:552
#14 0x00000000158c6c27 in DB::Connection::sendExternalTablesData (this=0x7fb6545e9d98, data=...) at ../src/Client/Connection.cpp:797
#27 0x0000000014043a81 in DB::RemoteQueryExecutorRoutine::operator() (this=0x7fb63095bf20, sink=...) at ../src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:46
#32 0x000000000a16dd4f in make_fcontext () at ../contrib/boost/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:71
And also in the logs you can see very strange things for this thread:
2022.09.13 14:14:51.228979 [ 51145 ] {1712D4E914EC7C99} <Debug> Connection (localhost:9000): Sent data for 1 external tables, total 11 rows in 0.00046389 sec., 23688 rows/sec., 3.84 KiB (8.07 MiB/sec.), compressed 1.1070121092649958 times to 3.47 KiB (7.29 MiB/sec.)
...
2022.09.13 14:14:51.719402 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (from 10.101.15.181:42478) KILL QUERY WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
2022.09.13 14:14:51.719488 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (internal) SELECT query_id, user, query FROM system.processes WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
2022.09.13 14:14:51.719754 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> ContextAccess (default): Access granted: SELECT(user, query_id, query) ON system.processes
2022.09.13 14:14:51.720544 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2022.09.13 14:14:53.228964 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> Connection (localhost:9000): Sent data for 2 scalars, total 2 rows in 2.6838e-05 sec., 73461 rows/sec., 68.00 B (2.38 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (5.16 MiB/sec.)
How is this possible? The answer is fibers and query cancellation
routine. During cancellation of async queries it going into fibers again
and try to do this gracefully. However because of this during canceling
query it may call QueryStatus::addPipelineExecutor() from
QueryStatus::cancelQuery().
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Enmk
pushed a commit
that referenced
this pull request
Oct 26, 2022
…verflow
UBSAN report:
$ UBSAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-14 UBSAN_OPTIONS=print_stacktrace=1 ./unit_tests_dbms --gtest_filter=*Gorilla*
../src/Compression/tests/gtest_compressionCodec.cpp:1216:47: runtime error: signed integer overflow: 23 * 100000000 cannot be represented in type 'int'
#0 0x14f67fd1 in auto (anonymous namespace)::$_6::operator()(int) const::'lambda'(auto)::operator()<int>(auto) const build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1216:47
#1 0x14f67fd1 in (anonymous namespace)::CodecTestSequence (anonymous namespace)::generateSeq<long, (anonymous namespace)::$_6::operator()(int) const::'lambda'(auto), int, int>((anonymous namespace)::$_6::operator()(int) const::'lambda'(auto), char const*, int, int) build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:394:36
#2 0x14f67fd1 in auto (anonymous namespace)::GCompatibilityTestSequence<long>() build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1224:12
#3 0x14f3c7f3 in (anonymous namespace)::gtest_GorillaCodecTestCompatibility_EvalGenerator_() build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1227:1
#4 0x14f6bdb5 in testing::internal::ParameterizedTestSuiteInfo<(anonymous namespace)::CodecTestCompatibility>::RegisterTests() build_docker/../contrib/googletest/googletest/include/gtest/internal/gtest-param-util.h:553:45
#5 0x27d87988 in testing::internal::ParameterizedTestSuiteRegistry::RegisterTests() build_docker/../contrib/googletest/googletest/include/gtest/internal/gtest-param-util.h:726:24
#6 0x27d87988 in testing::internal::UnitTestImpl::RegisterParameterizedTests() build_docker/../contrib/googletest/googletest/src/gtest.cc:2805:34
#7 0x27d87988 in testing::internal::UnitTestImpl::PostFlagParsingInit() build_docker/../contrib/googletest/googletest/src/gtest.cc:5492:5
#8 0x27d9d002 in void testing::internal::InitGoogleTestImpl<char>(int*, char**) build_docker/../contrib/googletest/googletest/src/gtest.cc:6499:22
#9 0x14fd5495 in main build_docker/../src/Coordination/tests/gtest_coordination.cpp:2189:5
#10 0x7f8c29005209 (/lib/x86_64-linux-gnu/libc.so.6+0x29209) (BuildId: 71a7c7b97bc0b3e349a3d8640252655552082bf5)
#11 0x7f8c290052bb in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x292bb) (BuildId: 71a7c7b97bc0b3e349a3d8640252655552082bf5)
#12 0x14ce356d in _start (/work1/azat/tmp/42190/unit_tests_dbms+0x14ce356d) (BuildId: 482550e3f8d45f06e8c7f8147f427ee798c1f645)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../src/Compression/tests/gtest_compressionCodec.cpp:1216:47 in
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Enmk
pushed a commit
that referenced
this pull request
Apr 29, 2024
Fixes integration test_reload_certificate/test.py::test_first_than_second_cert --- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E #9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E #10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E #11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E #12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E #16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E #17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E #18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E #20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E #21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E #23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E #24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E #25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E #26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E #27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E #29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E #30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E #10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E #12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E #15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E #17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E #18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E
Enmk
pushed a commit
that referenced
this pull request
Apr 29, 2024
This commit adresses below leaksan find. I tried to reproduce this locally but 02802_clickhouse_disks_s3_copy.sh but couldn't. clickhouse-disks did not do anything useful (it would not even print logging), neither with a standard build nor with a leaksan build. Could not find further documentation of it or even what this tool is supposed to do, perhaps it is just for internal use. Also, - line numbers in the leaksan report were partially missing, - I am not really sure how Sha256HMACOpenSSLImpl::Calculate is calling into hmac_init (there must be some sort of static initialization somewhere but I did not find it), and - my fix is in a weird place due to other restrictions (see the commit in the aws-sdk-cpp contrib repo). The chance that this fix fixes the leak are low. If it doesn't work, add "# Tag no-asan" + a comment to 02802_clickhouse_disks_s3_copy.sh and don't worry further. EDIT: The commit fixes the issue, everything is good. https://s3.amazonaws.com/clickhouse-test-reports/59870/b452e3d1ab87b8cc5810693aeea28f69ad28d671/stateless_tests__asan__[3_4].html 2024-03-19 08:34:03 ================================================================= 2024-03-19 08:34:03 ==13149==ERROR: LeakSanitizer: detected memory leaks 2024-03-19 08:34:03 2024-03-19 08:34:03 Direct leak of 904 byte(s) in 1 object(s) allocated from: 2024-03-19 08:34:03 #0 0x55f9cb5a18ee in malloc (/usr/bin/clickhouse+0xa9a48ee) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #1 0x55fa01e34070 in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 2024-03-19 08:34:03 #2 0x55fa01e34070 in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 2024-03-19 08:34:03 #3 0x55fa01d6dcca in ossl_err_get_state_int build_docker/./contrib/openssl/crypto/err/err.c:691:17 2024-03-19 08:34:03 #4 0x55fa01d71748 in ERR_set_mark build_docker/./contrib/openssl/crypto/err/err_mark.c:19:10 2024-03-19 08:34:03 #5 0x55fa01f4735b in ossl_prov_digest_load_from_params build_docker/./contrib/openssl/providers/common/provider_util.c:194:5 2024-03-19 08:34:03 #6 0x55fa01ff467a in hmac_set_ctx_params build_docker/./contrib/openssl/providers/implementations/macs/hmac_prov.c:307:10 2024-03-19 08:34:03 #7 0x55fa01ff3ef2 in hmac_init build_docker/./contrib/openssl/providers/implementations/macs/hmac_prov.c:169:37 2024-03-19 08:34:03 #8 0x55f9fb83c75b in Aws::Utils::Crypto::Sha256HMACOpenSSLImpl::Calculate(Aws::Utils::Array<unsigned char> const&, Aws::Utils::Array<unsigned char> const&) (/usr/bin/clickhouse+0x3ac3f75b) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #9 0x55f9fb82ebeb in Aws::Utils::Crypto::Sha256HMAC::Calculate(Aws::Utils::Array<unsigned char> const&, Aws::Utils::Array<unsigned char> const&) (/usr/bin/clickhouse+0x3ac31beb) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #10 0x55f9fb6d5afd in Aws::Client::AWSAuthV4Signer::ComputeHash(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const (/usr/bin/clickhouse+0x3aad8afd) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #11 0x55f9fb6e0bad in Aws::Client::AWSAuthV4Signer::SignRequest(Aws::Http::HttpRequest&, char const*, char const*, bool) const (/usr/bin/clickhouse+0x3aae3bad) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #12 0x55f9fb72651d in bool smithy::components::tracing::TracingUtils::MakeCallWithTiming<bool>(std::__1::function<bool ()>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, smithy::components::tracing::Meter const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) (/usr/bin/clickhouse+0x3ab2951d) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #13 0x55f9fb70adcb in Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const (/usr/bin/clickhouse+0x3ab0ddcb) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #14 0x55f9fb6fdb17 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char cons
zvonand
pushed a commit
that referenced
this pull request
Nov 29, 2024
Add 'date_time_utc' format to tests
zvonand
pushed a commit
that referenced
this pull request
Nov 29, 2024
…working-backup-sync-3 Corrections after reworking backup/restore synchronization #3
Enmk
pushed a commit
that referenced
this pull request
Mar 31, 2025
CI: https://s3.amazonaws.com/clickhouse-test-reports/75848/479ca9b0a40fa32074501bea7d9eb7d369f3ddd4/integration_tests__tsan__2_6_.html <details> <summary>Race</summary> ``` E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=8) E Write of size 8 at 0x726000005c78 by thread T585 (mutexes: write M0, write M1): E #0 std::__1::enable_if<is_move_constructible<DB::SnapshotFileInfo*>::value && is_move_assignable<DB::SnapshotFileInfo*>::value, void>::type std::__1::swap[abi:ne180100]<DB::SnapshotFileInfo*>(DB::SnapshotFileInfo*&, DB::SnapshotFileInfo*&) build_docker/./contrib/llvm-project/libcxx/include/__utility/swap.h:44:7 (clickhouse+0x1dad0fb6) (BuildId: eb398fb9d76a60d418ba7222d10f730fae2208dc) E #1 std::__1::shared_ptr<DB::SnapshotFileInfo>::swap[abi:ne180100](std::__1::shared_ptr<DB::SnapshotFileInfo>&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:694:5 (clickhouse+0x1dad0fb6) E #2 std::__1::shared_ptr<DB::SnapshotFileInfo>::operator=[abi:ne180100](std::__1::shared_ptr<DB::SnapshotFileInfo>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:663:32 (clickhouse+0x1dad0fb6) E #3 DB::KeeperStateMachine<DB::KeeperStorage<DB::SnapshotableHashTable<DB::KeeperMemNode>>>::save_logical_snp_obj(nuraft::snapshot&, unsigned long&, nuraft::buffer&, bool, bool) build_docker/./src/Coordination/KeeperStateMachine.cpp:846:30 (clickhouse+0x1dad0fb6) E Previous read of size 8 at 0x726000005c78 by thread T572: E #0 std::__1::shared_ptr<DB::SnapshotFileInfo>::shared_ptr[abi:ne180100](std::__1::shared_ptr<DB::SnapshotFileInfo> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:565:82 (clickhouse+0x1daea9b2) (BuildId: eb398fb9d76a60d418ba7222d10f730fae2208dc) E #1 DB::KeeperStateMachine<DB::KeeperStorage<DB::SnapshotableHashTable<DB::KeeperMemNode>>>::create_snapshot(nuraft::snapshot&, std::__1::function<void (bool&, std::__1::shared_ptr<std::exception>&)>&)::'lambda'(std::__1::variant<std::__1::shared_ptr<DB::KeeperStorageSnapshot<DB::KeeperStorage<DB::SnapshotableHashTable<DB::KeeperMemNode>>>>, std::__1::shared_ptr<DB::KeeperStorageSnapshot<DB::KeeperStorage<DB::RocksDBContainer<DB::KeeperRocksNode>>>>>&&, bool)::operator()(std::__1::variant<std::__1::shared_ptr<DB::KeeperStorageSnapshot<DB::KeeperStorage<DB::SnapshotableHashTable<DB::KeeperMemNode>>>>, std::__1::shared_ptr<DB::KeeperStorageSnapshot<DB::KeeperStorage<DB::RocksDBContainer<DB::KeeperRocksNode>>>>>&&, bool) const build_docker/./src/Coordination/KeeperStateMachine.cpp:803:22 (clickhouse+0x1daea9b2) ``` </details> v2: guard latest_snapshot_meta and latest_snapshot_buf as well (thanks to @davenger)
strtgbb
pushed a commit
that referenced
this pull request
Jul 1, 2025
strtgbb
pushed a commit
that referenced
this pull request
Jul 1, 2025
Backport ClickHouse#82422 to 25.3: Read AWS ECS token from file, attempt #3
25 tasks
25 tasks
27 tasks
25 tasks
27 tasks
27 tasks
6 tasks
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
…iers + inline traces) Full triage of docs/superpowers/reports/20260717_codex_review.md. Headline outcomes: reviewer's blocker (#1) downgraded to real-but-narrow (multipart ACCESS_DENIED fallback drops the precondition); #4 (swallowed condemn-marker vs same-token adopt) confirmed as the top data-loss class; #5 (pre-CAS generation prune) confirmed as a permanent GC wedge (~2-line fix); one NEW finding (19c) discovered during verification: emu tokens are a per-process counter (not etag-seeded as their comments claim) -> persisted condemn tokens can collide across a restart = latent local-CA data loss. Refuted at harm level: #2, #3, #20b, #27. Design decisions (#14/#15/#17/#29/#30/#31) collected for user review. No code changed. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
…#6,#11,#26 deferred) Comment-only wave from the 2026-07-17 codex-review triage: for every finding the triage refuted or classified BY-DESIGN, state the invariant in place so future reviewers stop re-reporting it. Covers resurrectStaged (content-addressed byte-identity + tokenless-on-ref promote + fresh-tag defeats stale deletes, finding #2), ObjectStorageBackend::get (HEAD/GET ordering argument, finding #3), removeDirectory/moveDirectory (call-time durability + compensation contract, finding #12), casPutObject and its append-path caller (single-appender invariant, finding #26), and the moveFile re-drive branch (why an unrelated pre-existing destination can never reach it, finding #27). Also records the three still-open findings the triage deferred rather than fixed in this wave: #6 (wire the specced fetch-handoff retention pin), #11 (GC backstop for empty ownerless Live namespaces), and #26 (implement casAppendObject before any concurrent appender exists). No behavior change: every hunk is a comment or BACKLOG.md entry. Verified the three touched translation units still compile clean, and the full CAS gtest battery (Cas*:CA*:ContentAddressedLog*:CountingBackendShape*:RefSnapshotCodec*: RefTableCacheEviction*:RefWriter*) passes 919/919. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LN7m5KvCqwnfutnPDNUncF Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
28 tasks
filimonov
added a commit
that referenced
this pull request
Jul 23, 2026
…t throw-on-not-live Narrow the design after codex found 3 P0 blockers in rev.1: - Core fix is now identity-validated recovery: self-remount must exact-HEAD _pool_meta and require the original pool ID/format before re-arming, else fail-loud "data root erased" — never bootstrap an erased pool as empty-healthy (rev.1 P0 #1). The fresh-incarnation recovery already exists (tryRemountOnce) and is kept; rev.1 misdescribed it (P0 #2). - Drop runtime STOP/START (unsound without a process-wide disk-use barrier, P0 #3); keep only GC STOP/START. - Roll back Part 6 -> throw everywhere ("let it break"), with ONE typed, non-lying per-disk isolation in the DROP finalizer so a lingering dead CA disk cannot wedge DROP of unrelated tables server-wide. - Access gate on Pool::mayMutate (local, TTL-hysteretic; no availability regression); read-only CA disks are not lease-gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
filimonov
added a commit
that referenced
this pull request
Jul 26, 2026
…eriment needs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012UJqPB1gNiKk5AKi91zZyY
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…thority, four latent test bugs fixed
test_cas_gc_sharded has been permanently skipped since it was written (MinIO OSS lacks the
conditional-DELETE support the CA mount capability probe requires), so this module has never
actually executed until now. Enabling it surfaced four independent, pre-existing, latent bugs
in the test itself -- none in production code:
1. Wrong S3 backend: swapped with_minio=True + a MinIO endpoint for with_rustfs=True + the
standard rustfs1:11121 endpoint/credentials, matching every other CAS integration test.
2. A Python str.format/ClickHouse-macro escaping bug: the live_table CREATE queries wrote
'{replica}' (single braces) instead of '{{replica}}' (double braces), so .format(STORAGE_POLICY)
raised KeyError('replica') on its very first query. The correctly-escaped form already existed
two functions below in the same file.
3. A post-restart race querying system.text_log right after node2.restart_clickhouse(kill=True):
the underlying table can take a moment to become queryable. Fixed with a keyword-free
readiness probe via the existing query_with_retry helper, run once before the real keyword
checks (not by retrying the real checks themselves -- retrying THOSE is self-defeating: a
failed attempt's own <Error> log line echoes the failing query text, which contains the search
keyword, and that line then lands in system.text_log itself, so a later successful attempt of
the SAME keyword query counts its own failed predecessors as matches. Verified directly: a red
run's "6 == 0" failure was exactly 3 retry attempts times 2 log lines each, all
"Unknown table 'system.text_log'" echoing the '%DANGLE%' query text; zero real dangle-shaped
log content anywhere in the run).
4. The completion-detection and shard-coverage assertions polled entirely wrong S3 keys:
"completion_seal" never existed anywhere in production (grep returns zero hits; the real name
is "fold_seal"), and the assumed path shape omitted the "attempt/<attempt>/" segment production
actually uses. Rewritten to resolve the ADOPTED (generation, attempt) pair from gc/state's own
text wire format (fields "sg"/"sa") -- the same two-hop lookup production's own
Gc/CasOrphanManifestSweep.cpp readAdoptedFoldSeal uses -- so a retry-created, never-adopted
attempt's own seal can never be mistaken for a second completion signal. Assertion C (shard
coverage) scans the whole gc/gen/ subtree rather than one (generation, attempt): a seal's
blob_target_runs carry a PARENT's runs forward as references, but the physical objects stay
under the (generation, attempt) prefix where they were originally written, so a late-soak
adopted attempt with no new deltas of its own can have an empty prefix even though earlier
attempts wrote plenty.
Six-run arc: #1 (minio->rustfs swap only) red on the brace bug; #2 red on the text_log race;
#3 red went past both into the real key-layout bug (branch confirmed: wrong key name AND wrong
path shape, not a GC defect); #4 (gc/state authority for B, adopted-attempt blob_target listing
for C) reached real assertions and failed on 6 self-inflicted "DANGLE" log matches (RCA above);
#5 (readiness-probe split) passed B, failed C on the adopted-attempt-scoped listing (the
carry-forward-by-reference-not-physical-location gap); #6 (whole-subtree scan for C) green:
1 passed in 78.01s.
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…ty-universe floor); consult launched Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159e4eaPFKDFgxG83d5VL73
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…iverse GC gate fix checkCompatibility rejects only versions ABOVE G_BUILD; decodeRefCatalog discards the parsed header, so a version below a type's birth generation decodes as if legal. Not load-bearing for the empty-universe proof (token + structural decode + zero entries already accepts the well-formed-illicit-catalog residual under the trusted-store model); shared format-layer blast radius makes it its own task. Found while ruling on FINDING #3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
… comms dead; worktree rule) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159e4eaPFKDFgxG83d5VL73
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…#3 arc state) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159e4eaPFKDFgxG83d5VL73
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
FINDING #3: after a pool's last namespace is dropped, the catalog reads genuinely empty forever, so R11's frontier_namespaces > 0 floor never opens the destructive gate again -- an emptied pool stops reclaiming permanently. Codex ruling (tmp/codex_emptyuniverse_answer.md) endorsed a narrow fix: the non-vacuity term of frontier_complete can also be satisfied by the round's own hot-scan catalog cut positively proving the universe empty (present, token-bearing, decoded, zero rows of every lifecycle state including Creating -- never by the frontier counters alone, which a Creating-only catalog also reads as 0/0). Ruling requirement -> implementation: - FoldResult::catalog_cut_proved_empty, computed once at the catalog cut already in the walk plan (no second GET) -> CasGc.h/.cpp. - frontier_complete = universe_authoritative && frontier_proven == frontier_namespaces && (frontier_namespaces > 0 || catalog_cut_proved_empty) -> CasGc.cpp gate site. - StageA_Suppressed still suppresses a proved-empty catalog -> unchanged anomaly/hold terms; pinned by AProvedEmptyCatalogUnderStageASuppressedStaysSuppressed. - catalog_proved_empty / catalog_entries added to fold_ref_intake; the old "authoritative-but-empty is always a WARNING" clause narrowed to fire only when the catalog did NOT prove the emptiness (e.g. a Creating-only cut), naming the real row count. - R11 comment and UniversePolicy::kDefault doc updated to state the new rule. - Test replacement per the ruling's list: the positive/negative proved- empty pair, absent-catalog, a malformed-catalog table (wrong-type, future-version, trailer-count-mismatch, missing-trailer, truncated-mid-line), StageA_Suppressed, and the birth-after-empty-cut blob race (a new Gc::setPostHotScanCatalogReadHookForTest one-shot hook pauses right after the hot-scan catalog GET; a real namespace is born through the production createNamespace/precommitAdd/putBlob path and resurrects the already-condemned blob before the round's own pre-CAS delete runs against the stale token). Scope cut (team decision): the decoder-hardening half of the ruling (reject a RefCatalog header version below its birth generation) is NOT included here -- not load-bearing for this proof (token + full structural decode + zero entries already accepts a well-formed out-of-protocol empty catalog as a residual under the trusted-store model) and its blast radius is the shared format layer used by every CAS object type. Tracked as [cas-format-version-floor] in docs/superpowers/cas/BACKLOG.md (master worktree, a1e9678's child commit). Known-red at this commit (being fixed): AZeroWalkableFrontierWith ACreatingCatalogRowIsNotProvedEmpty, AProvedEmptyCatalogUnderStageA SuppressedStaysSuppressed, ANamespaceBornAfterTheEmptyCutResurrects TheCondemnedBlobInstead -- test-harness bugs in the new tests themselves (casUpdate cannot add entries; beginPartWrite needs intended_namespace), not gate defects; a saw_fold=false root cause on the positive drain test is still under investigation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…-existing tests the widened gate broke Per team review: bound the positive drain test's drive loop at the MEASURED cadence (4 rounds: lease-arm, confirm-marker carry, graduate, delete -- via phase-sink instrumentation, not a guess) plus one, and assert rounds_run <= that bound afterward, so a future regression in the confirm/retry pacing fails loudly instead of silently absorbing into a generous loop. Classified the libc++ hardening abort from the full-suite run (gdb backtrace, ulimit -c unlimited): the disengaged optional::operator-> lives directly in TestBody, not production code -- an unchecked `backend->get(key)->bytes` in two pre-existing tests that assumed a removed namespace's _ckpt would survive the round. It no longer does: the widened gate is designed to also unlock runNamespaceJanitorPage (gated by the same suppress_destructive), and reclaiming a removed namespace's checkpoint once the pool empties is exactly the FINDING #3 standstill this gate exists to fix -- confirmed correct per team ruling, not scoped out. Fixed both crash sites plus the invariant each test pinned: - CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanitor and LostCatalogCasResponseIsResolvedBeforeListing: capture the namespace_cleanup phase's own janitor_deleted counter via a phase sink and assert the delete against IT, not against end-state absence (which can't tell "the janitor did it" from "something else did") or an unchecked dereference (which can't tell "deleted" from "never existed"). - StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListing: the fold's own walk-plan single-cut invariant already holds by construction (fresh_catalog_cut is defined as the first catalog get after the ref-object LIST); the OLD, broader "no second catalog get before the seal PUT" claim is no longer true once other destructive families (orphan-manifest sweep, namespace janitor) also take their own separate cuts in the same, now-unsuppressed round -- confirmed from the journal (list p/cas/manifests/ then a second get p/cas/ref_catalog, before the seal, from the orphan sweep). Dropped the obsolete claim, kept the one that's still true, same fix applied to the crash site. - CompletedRemovalDrainUsesNPlusOneCatalogReads: the orphan-manifest sweep's own list+cut is a THIRD catalog-read contributor this test's fine-grained journal assertions didn't yet name; located it explicitly between the walk-plan cut and the janitor's page and updated the total from deletes+3 to deletes+4. Re-ran the full CASGCFrontierGate*/CASGCFrontierGateCleanupRange*/ CASCatalogLifecycleReconciler*/CASWinnerShape* suite (59 tests, the whole file) after each fix per the "abort kills everything queued after it" lesson: two more instances of the identical unchecked- dereference class were hiding behind the first crash (LostCatalogCasResponseIsResolvedBeforeListing was masked entirely; CompletedRemovalDrainUsesNPlusOneCatalogReads's count drifted silently once the crash was fixed). All 59 green now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
…tale standstill assertions rewritten) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159e4eaPFKDFgxG83d5VL73
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Detailed description / Documentation draft:
...
By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in docs folder.
If you are doing this for the first time, it's recommended to read the lightweight Contributing to ClickHouse Documentation guide first.