Regression: IndexOutOfRangeException on a 5-byte zero-length handshake record #134784

Description

A TLS server that receives 16 03 01 00 00 (a handshake record with no payload) throws IndexOutOfRangeException from SslStream.ReceiveHandshakeFrameAsync, whether the certificate is set directly or chosen by a callback, on both the new and the legacy handshake path.

"Fix SslStream detection of exactly-5-byte TLS frames" (dotnet/runtime#132694, Aug 25) made a 5-byte record count as a complete frame. The server's first-frame checks in SslStream.IO.cs still read EncryptedReadOnlySpan[HandshakeTypeOffsetTls], index 5, of a 5-byte span. That PR's test only covers a 5-byte record after the handshake. On 11.0 RC1 the same input ends in an IOException.

Any unauthenticated client can trigger this, and callers of AuthenticateAsServerAsync don't expect IndexOutOfRangeException.

Reproduction Steps

// Finding (regression on main, not in 11.0 RC1): an SslStream server receiving the 5-byte zero-length TLS handshake record
// 16 03 01 00 00 throws IndexOutOfRangeException from ReceiveHandshakeFrameAsync. Since "Fix SslStream detection of
// exactly-5-byte TLS frames" (#132694) a 5-byte record counts as a complete frame, but the ClientHello checks still
// read the handshake type at offset 5.
// Needs a runtime built from main: see ../FINDINGS.md ("Findings on main only"). On 11.0 RC1 it prints NOT REPRODUCED.
// Run: dotnet run 01-SslStream-ZeroLengthRecord.cs

using System.Net.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

using var rsa = RSA.Create(2048);
using X509Certificate2 certificate = new CertificateRequest("CN=localhost", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1)
    .CreateSelfSigned(DateTimeOffset.Now.AddDays(-1), DateTimeOffset.Now.AddDays(1));

using var server = new SslStream(new MemoryStream([0x16, 0x03, 0x01, 0x00, 0x00]));
string outcome;
try
{
    await server.AuthenticateAsServerAsync(new SslServerAuthenticationOptions { ServerCertificate = certificate });
    outcome = "no exception";
}
catch (Exception ex) { outcome = ex.GetType().Name + ": " + ex.Message; }

Console.WriteLine($"{Environment.Version}: AuthenticateAsServerAsync(16 03 01 00 00) -> {outcome}");
Console.WriteLine(outcome.StartsWith(nameof(IndexOutOfRangeException), StringComparison.Ordinal) ? "REPRODUCED: IndexOutOfRangeException from untrusted input." : "NOT REPRODUCED (expected IOException/AuthenticationException)");

Expected behavior

Either same behaviour as .NET 11 RC1 or no exception

Actual behavior

IndexOutOfRangeException

Regression?

Kind of

Known Workarounds

None

Configuration

Main, compared to .NET 11 RC1.
Windows
x64
Should not be tied to my config
No

Other information