Repository navigation
CI: run the self-hosted log collector's two end-to-end tests on pull requests - #4504
Merged
Merged
Conversation
…requests Adds a fifth throwaway log-format cluster (port 5545, logging_collector on, log_connections/log_disconnections/log_lock_waits on, a %u@%d-bearing prefix, UTC timezone) to build.yml's darling-pg job, plus a deterministic workload step that produces one real error, connection, and lock-wait event before the tests run. Sets DARLING_TEST_PG_LOG_TARGET and DARLING_TEST_PG_UTC_LOG_TARGET on the test step so both previously-skipped live end-to-end tests in PgLogEventsPipelineTests run on every pull request. nightly.yml starts no log-format clusters today, so it is left unchanged.
psql only accepts libpq keywords or -h/-p/-U/-d flags, not an Npgsql-style connection string; the step's calls used the latter and failed with invalid connection option 'Host'. The tests' own env vars (Npgsql form, read by Npgsql) are unchanged.
Outside a transaction block, PostgreSQL rejects LOCK TABLE immediately, so the waiter session never actually waits for the lock. Wrap it in BEGIN/COMMIT to match the holder session. Also add -v ON_ERROR_STOP=1 to the psql calls that are meant to succeed, so a SQL error in them is no longer silently swallowed.
…e 5545 workload step
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Darling/Darling.Tests/PgLogEventsPipelineTests.cshas two live end-to-end tests gated onDARLING_TEST_PG_LOG_TARGETandDARLING_TEST_PG_UTC_LOG_TARGET. Neither variable was set in anyworkflow, so both tests skipped in every CI run since they were written.
What changes
build.yml'sdarling-pgjob gets a fifth throwaway PostgreSQL cluster (port 5545), alongside theexisting store (5541) and three log-format clusters (5542/5543/5544):
logging_collector = on,log_destination = 'stderr',log_timezone = 'UTC'log_line_prefix = '%m %u@%d [%p] '(the%u@%dshape the second test'sreader.GetName(1) == "log_timezone"assertion needs)log_connections = on,log_disconnections = on,log_lock_waits = ondeadlock_timeout = '100ms'(inherited fromStart-LogFormatCluster's shared defaults)One cluster serves both tests: both need
logging_collectoron and UTC timezone, and the firsttest's extra preconditions (
log_connections,log_lock_waits) don't conflict with the second's.A workload step immediately after the cluster starts puts one real event of each family the first
test asserts on into that cluster's own log, via
psql.exefrom the job's own pg-runtime bin:SELECT 1) — produces the connection/disconnection LOG pair;SELECT 1/0— a real ERROR the server writes itself;LOCK TABLEinside a transaction and sleeps 3s, while asecond
psqlrequests the same table and blocks pastdeadlock_timeout, producing thelog_lock_waits"still waiting for" / "acquired" pair;Every workload call checks its own exit code and throws on an unmet precondition (a psql failure
that should have succeeded, or a success where a failure was expected), so a silently-quiet log
fails the step instead of failing the test with no explanation.
DARLING_TEST_PG_LOG_TARGETandDARLING_TEST_PG_UTC_LOG_TARGETare set on "Run Darling PG tests"(the connection-string shape of the existing 5542-5544 vars), both comment blocks are updated to
name the new cluster and both tests, and the "Stop PostgreSQL" / failure-artifact steps are
extended to cover the new data directory and log file.
nightly.ymlstarts no log-format clusters today (only the 5541 store), so there is nothing tomirror there; left unchanged.
Test plan
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" .github/workflows/build.yml .github/workflows/nightly.yml— clean.actionlint .github/workflows/build.yml— clean.dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Release -p:EnableWindowsTargeting=true— 0 Warnings, 0 Errors.Darling.Tests.dll -class Darling.Tests.PgLogEventsPipelineTestsagainst a two-container docker rig with the 5545 settings: Total: 61, Failed: 0 (pre-mutation) / Total: 61, Failed: 13 (mutated, including both new E2Es).0e99b5f3a): the 5545 workload step passes, and all threeDarling PG testsshards pass (5,674 / 4,962 / 5,103 run, 0 failed). The shard that hostsPgLogEventsLivePostgresTestsnow skips 0; before this change it skipped exactly these two tests. So both run and pass.LOGdropped fromPgLogEntryAssembler's label alternation: the workload step passes, andTheSelfHostedCollector_ReadsTheTargetsOwnLog_EndToEndgoes RED, along withThePipeline_StoresReadsFiltersAndDedupes_AgainstDevPostgresand the parser pins.AgainstAUtcTarget_…stays green under that mutation, because its line is anERROR, not aLOG.DateTimeStyles.AdjustToUniversaldropped fromPgLogEntryAssembler's timestamp parse, so parsed times come back as local rather than UTC:AgainstAUtcTarget_TheLogQueriesReadTheSettingWithTheTail_AndStoreWhatTheServerWroteandTheSelfHostedCollector_ReadsTheTargetsOwnLog_EndToEndboth go RED on theirOccurredAtUtc.Kind == Utcassertion.CHANGELOG
None: CI-only change.