Add possibility to use different passwords for Sentinel and Redis host (#1698) - #3140
Conversation
mgravell
left a comment
There was a problem hiding this comment.
Concept is great, +1; a few nits in the ConfigurationOptions piece just because that type is super weird (not your fault), and I'm not a huge fan of the clone usage - I welcome your thoughts there.
Thanks! |
…nOptions - fixed typos, so fields, not properties, are used on the correct places
|
@mgravell is there a nice/correct way to clear the CodeQL security failure or do I need to add an inline suppression message? |
|
I have dismissed the CodeQL alert; the alert is "right" in that this should be a considered decision, but this is an entirely consistent and appropriateusage. |
mgravell
left a comment
There was a problem hiding this comment.
super close now; sorry to be pedantic on some of these - I'm not trying to frustrate you
|
(note Windows CI may stall due to a packaging bug in 8.10 - server folks are looking into it; I've added a workaround in |
Does not feel like it :P
Don't worry about that! Better safe than sorry ... |
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
…) in ConnectionMultiplexer.Sentinel
|
I moved back to overwriting the user/pw with the Sentinel credentials in the I tried to implement the usage of the Sentinel credentials in the Maybe this can be accepted as a stable and working solution (tested this build thorough in our corporate OTA environment), as we really need this solution in the short term. |
|
OK; we do need the |
|
(sorry for force-push; repairing after a git identity failure) |
|
Thank you for the quick responses, feedback and release. |
…bscription poisoning on downgrade (#3154) (#3176) * Never send both HELLO AUTH and a standalone AUTH Fixes #3172, and the RESP3 credential-failure hang more generally. Redis has a protocol bug (reproduced on 7.4.10, 8.0.6 and an 8.9 preview; valkey 8.1.9 is unaffected): inside a pipelined batch, a *failing* AUTH only gets a reply if it is the first command in that batch. Otherwise the error is silently dropped, so every subsequent reply on that connection lands on the wrong message: AUTH bad, PING -> -WRONGPASS, +PONG (correct) PING, AUTH bad, PING -> +PONG, +PONG (error dropped) HELLO 3, AUTH bad, PING -> <hello map>, +PONG (error dropped) AUTH bad, AUTH bad, PING -> -WRONGPASS, +PONG (second dropped) The handshake wrote `HELLO <ver> AUTH user pass` *and* a standalone `AUTH user pass` ("we auth EVEN IF we have used HELLO to AUTH"), so with credentials the server rejects, the second AUTH's error vanished and the whole handshake shifted by one reply: the tie-breaker GET consumed the tracer's reply, the tracer never completed, and the connection never became usable. Instead of a clean AuthException, callers saw "no connection became available" timeouts, and topology detection read the wrong replies - which is #3172: a sentinel reporting "The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone" under v3's RESP3 default, cured by forcing protocol=resp2. v2 never hit this because it assumed server version 3.0 and so never issued HELLO. #3140 (separate sentinel credentials) makes "credentials the sentinel rejects" an easy case to land on. So: AUTH goes first in the batch (where its errors are reported correctly), and HELLO follows it, bare. HELLO only carries credentials when AUTH is unavailable in the command map - which the ConnectionMultiplexer constructor already only permits when RESP3 is being attempted. Verified against the live sentinel topology: before, RESP3 + rejected credentials gave "Detected as: Standalone" and no primary connection; after, all four protocol/credential combinations resolve the primary. ConfigTests. MutableOptions no longer needs to skip RESP3 - that asymmetry was this bug - so it now covers both protocols as a regression test. * Reroute subscriptions off the interactive connection when we downgrade to RESP2 Fixes #3154. Under RESP3 subscriptions share the interactive connection; under RESP2 they need their own. Which bridge a subscription goes to is therefore decided by what we know-or-expect the protocol to be at the time it is queued - and that can change underneath a queued message: a connection negotiates RESP3, later reconnects, the new HELLO fails (timeout, expired token, failover to a down-level node), and the tracer settles the connection as RESP2. Anything queued in the interactive bridge's backlog while we still expected RESP3 - notably the SUBSCRIBEs that EnsureSubscriptions pushes during the disconnect - is then written to the interactive connection, which puts it into subscriber mode: from that point it rejects every ordinary command with ERR only [P|S][UN]SUBSCRIBE / PING / QUIT allowed in this context so the connection is permanently poisoned, and the multiplexer has no idea. The fix is at the write, not at the downgrade: when the interactive bridge is about to write a subscription-bridge message on a connection it now knows to be RESP2, it hands the message to the subscription bridge instead. That covers every drain path (direct write, sync and async backlog processors) without needing to catch the moment the protocol resolved, and it is robust to any other way a queue can outlive the expectation it was filled under. Only that direction is handled: we never upgrade an existing expectation from RESP2 to RESP3 (HELLO is only issued when RESP3 is expected in the first place), so the mirror case cannot arise from a protocol change. Resp3DowngradeTests reproduces the whole sequence in-process: it negotiates RESP3, makes the toy server stop understanding HELLO, holds the accept gate shut so there is a real window in which we are disconnected but still expecting RESP3, queues a subscribe into that window, then releases the gate. Before this change the SET after reconnect fails with the subscriber-mode error above. It also asserts the underlying invariant directly - under RESP2 no single connection may carry both subscription and ordinary commands - which needed a per-connection command record in the test server rather than inferring it from symptoms.
Adding the possibility to use different passwords for Sentinel and Redis host.
Fixing the long standing issue #1698
Checklist