Skip to content

Add possibility to use different passwords for Sentinel and Redis host (#1698) - #3140

Merged
mgravell merged 14 commits into
StackExchange:mainfrom
doosterkamp:main
Aug 6, 2026
Merged

Add possibility to use different passwords for Sentinel and Redis host (#1698)#3140
mgravell merged 14 commits into
StackExchange:mainfrom
doosterkamp:main

Conversation

@doosterkamp

@doosterkamp doosterkamp commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Adding the possibility to use different passwords for Sentinel and Redis host.
Fixing the long standing issue #1698

Checklist

  • I fully and freely contribute this code in accordance with the project license (and am legally able to do so)
  • I take responsibility for this contribution's quality and correctness, including any portions produced with AI assistance (see CONTRIBUTING.md).

Comment thread src/StackExchange.Redis/ConfigurationOptions.cs
Comment thread src/StackExchange.Redis/ConfigurationOptions.cs Outdated
Comment thread src/StackExchange.Redis/ConfigurationOptions.cs Outdated
Comment thread src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs

@mgravell mgravell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept is great, +1; a few nits in the ConfigurationOptions piece just because that type is super weird (not your fault), and I'm not a huge fan of the clone usage - I welcome your thoughts there.

@doosterkamp

Copy link
Copy Markdown
Contributor Author

Concept is great, +1; a few nits in the ConfigurationOptions piece just because that type is super weird (not your fault), and I'm not a huge fan of the clone usage - I welcome your thoughts there.

Thanks!
I will try to fix the 'nits' today! We really need this fix and are testing with these changes as we speak.
I'm not a huge fan of the clone usage either, but it does the job for now, without a 'big' rewrite of the under-laying code ( I started trying to change that, but it got a lot bigger, very fast, so I stepped back for a moment).

@doosterkamp

doosterkamp commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

@mgravell is there a nice/correct way to clear the CodeQL security failure or do I need to add an inline suppression message?

@mgravell

Copy link
Copy Markdown
Collaborator

I have dismissed the CodeQL alert; the alert is "right" in that this should be a considered decision, but this is an entirely consistent and appropriateusage.

Comment thread src/StackExchange.Redis/PublicAPI/PublicAPI.SentinelUnshipped.txt Outdated
Comment thread src/StackExchange.Redis/ConfigurationOptions.cs Outdated
Comment thread src/StackExchange.Redis/ConfigurationOptions.cs Outdated
Comment thread src/StackExchange.Redis/ServerEndPoint.cs Outdated

@mgravell mgravell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super close now; sorry to be pedantic on some of these - I'm not trying to frustrate you

@mgravell

mgravell commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

(note Windows CI may stall due to a packaging bug in 8.10 - server folks are looking into it; I've added a workaround in main (#3145), but I don't think this branch includes that yet; basically: don't panic if Windows CI fails in truly epic ways)

@doosterkamp

Copy link
Copy Markdown
Contributor Author

super close now;

Does not feel like it :P
Testing in our setup fails during handshake (sees serverType == ServerType.Standalone), trying to fix this now

sorry to be pedantic on some of these - I'm not trying to frustrate you

Don't worry about that! Better safe than sorry ...

doosterkamp and others added 4 commits July 30, 2026 12:37
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
@doosterkamp

Copy link
Copy Markdown
Contributor Author

I moved back to overwriting the user/pw with the Sentinel credentials in the SentinelPrimaryConnect(Async) method in the ConnectionMultiplexer(.Sentinel). Even though I'm not a big fan of the use of Clone(), this is simply working well in my environment.

I tried to implement the usage of the Sentinel credentials in the HandshakeAsync method (in ServerEndPoint), but I lose track of the origin of the call in spaghetti chain of calls leading to the handshake.
Yes, I can see if the configuration is a Sentinel setup (using the created IsSentinel property), but not if it's originated from sentinelPrimaryConnect (which needs the Sentinel credentials in the handshake) of GetSentinelMasterConnection (which needs the Redis credentials in the handshake). After a few days of trying and debugging I gave up and reverted back to my first solution.

Maybe this can be accepted as a stable and working solution (tested this build thorough in our corporate OTA environment), as we really need this solution in the short term.

Comment thread src/StackExchange.Redis/ConfigurationOptions.cs
@mgravell

mgravell commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

OK; we do need the using directive that got culled, for Experimental - let me see if I can push to the branch...

Comment thread src/StackExchange.Redis/PublicAPI/PublicAPI.Shipped.txt Outdated
@mgravell

mgravell commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

(sorry for force-push; repairing after a git identity failure)

@mgravell
mgravell merged commit 211cf06 into StackExchange:main Aug 6, 2026
7 of 10 checks passed
@mgravell

mgravell commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

https://github.com/StackExchange/StackExchange.Redis/releases/tag/3.1.13

@doosterkamp

Copy link
Copy Markdown
Contributor Author

Thank you for the quick responses, feedback and release.

mgravell added a commit that referenced this pull request Aug 14, 2026
…bscription poisoning on downgrade (#3154) (#3176)

* Never send both HELLO AUTH and a standalone AUTH

Fixes #3172, and the RESP3 credential-failure hang more generally.

Redis has a protocol bug (reproduced on 7.4.10, 8.0.6 and an 8.9 preview;
valkey 8.1.9 is unaffected): inside a pipelined batch, a *failing* AUTH only
gets a reply if it is the first command in that batch. Otherwise the error is
silently dropped, so every subsequent reply on that connection lands on the
wrong message:

    AUTH bad, PING           -> -WRONGPASS, +PONG      (correct)
    PING, AUTH bad, PING     -> +PONG, +PONG           (error dropped)
    HELLO 3, AUTH bad, PING  -> <hello map>, +PONG     (error dropped)
    AUTH bad, AUTH bad, PING -> -WRONGPASS, +PONG      (second dropped)

The handshake wrote `HELLO <ver> AUTH user pass` *and* a standalone `AUTH user
pass` ("we auth EVEN IF we have used HELLO to AUTH"), so with credentials the
server rejects, the second AUTH's error vanished and the whole handshake shifted
by one reply: the tie-breaker GET consumed the tracer's reply, the tracer never
completed, and the connection never became usable. Instead of a clean
AuthException, callers saw "no connection became available" timeouts, and
topology detection read the wrong replies - which is #3172: a sentinel reporting
"The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone"
under v3's RESP3 default, cured by forcing protocol=resp2. v2 never hit this
because it assumed server version 3.0 and so never issued HELLO. #3140 (separate
sentinel credentials) makes "credentials the sentinel rejects" an easy case to
land on.

So: AUTH goes first in the batch (where its errors are reported correctly), and
HELLO follows it, bare. HELLO only carries credentials when AUTH is unavailable
in the command map - which the ConnectionMultiplexer constructor already only
permits when RESP3 is being attempted.

Verified against the live sentinel topology: before, RESP3 + rejected
credentials gave "Detected as: Standalone" and no primary connection; after, all
four protocol/credential combinations resolve the primary. ConfigTests.
MutableOptions no longer needs to skip RESP3 - that asymmetry was this bug - so
it now covers both protocols as a regression test.

* Reroute subscriptions off the interactive connection when we downgrade to RESP2

Fixes #3154.

Under RESP3 subscriptions share the interactive connection; under RESP2 they need
their own. Which bridge a subscription goes to is therefore decided by what we
know-or-expect the protocol to be at the time it is queued - and that can change
underneath a queued message: a connection negotiates RESP3, later reconnects, the
new HELLO fails (timeout, expired token, failover to a down-level node), and the
tracer settles the connection as RESP2. Anything queued in the interactive
bridge's backlog while we still expected RESP3 - notably the SUBSCRIBEs that
EnsureSubscriptions pushes during the disconnect - is then written to the
interactive connection, which puts it into subscriber mode: from that point it
rejects every ordinary command with

    ERR only [P|S][UN]SUBSCRIBE / PING / QUIT allowed in this context

so the connection is permanently poisoned, and the multiplexer has no idea.

The fix is at the write, not at the downgrade: when the interactive bridge is
about to write a subscription-bridge message on a connection it now knows to be
RESP2, it hands the message to the subscription bridge instead. That covers every
drain path (direct write, sync and async backlog processors) without needing to
catch the moment the protocol resolved, and it is robust to any other way a queue
can outlive the expectation it was filled under.

Only that direction is handled: we never upgrade an existing expectation from
RESP2 to RESP3 (HELLO is only issued when RESP3 is expected in the first place),
so the mirror case cannot arise from a protocol change.

Resp3DowngradeTests reproduces the whole sequence in-process: it negotiates
RESP3, makes the toy server stop understanding HELLO, holds the accept gate shut
so there is a real window in which we are disconnected but still expecting RESP3,
queues a subscribe into that window, then releases the gate. Before this change
the SET after reconnect fails with the subscriber-mode error above. It also
asserts the underlying invariant directly - under RESP2 no single connection may
carry both subscription and ordinary commands - which needed a per-connection
command record in the test server rather than inferring it from symptoms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants