HDDS-16722. Serve ReadBlock over Ratis read-only data streams on the datanode by peterxcli · Pull Request #11412 · apache/ozone

peterxcli · 2026-10-05T16:34:02Z

What changes were proposed in this pull request?

HDDS-9904 adds block reads over Ratis data streams. Ratis 3.3.1 supports read-only data streams (RATIS-1240): the client sends one request, and the server sends the data back in a sequence of replies. This PR adds the datanode side for ReadBlock. The client side comes in later sub-tasks of HDDS-9904.

Changes:

ContainerStateMachine#transferTo serves ReadBlock and rejects other commands. It passes the request to the dispatcher's streaming read (streamDataReadOnly, the same one the gRPC streaming read uses) and writes each response to the stream as one reply, with layout: [int: metadata length][the response without its data][data].
- When the read is done, it closes the stream, which sends the terminal reply.
- When the read fails (for example, the offset is past the end of the block, or the connection breaks), it throws without closing the stream, and Ratis sends the failure as the terminal reply.
- An error response from the dispatcher, such as CONTAINER_NOT_FOUND, is sent as a reply with no data, so the client still gets the result code.
- When a reply cannot be written, for example because the client went away, the read is cancelled with a CANCELLED status. KeyValueHandler.readBlock passes a CANCELLED status on instead of returning CONTAINER_INTERNAL_ERROR, so the datanode does not scan the container. This also covers the gRPC streaming read, whose onNext throws CANCELLED once the client is gone.
ClosedContainerReadResolver is a Ratis DataStreamApi.Resolver (RATIS-2603), registered in XceiverServerRatis. Ratis asks it before it looks up the Raft group. It serves the read by container ID when the request is a ReadBlock, the container is not open or closing, and the request names this datanode. SCM gives every such container a read pipeline with a random ID. For any other request it returns null, and Ratis uses the Raft group as before. Such a container no longer changes through Raft, so its read needs neither the group nor the leader.

Block tokens are checked as in the gRPC read path: the dispatcher validates the token when the DispatcherContext is null.

Each reply is built in one heap buffer, which copies the data once more. Removing that copy is left to a later sub-task, because it requires:

streamDataReadOnly and Handler.readBlock take a new observer type that keeps data outside the protobuf and supplies the buffer to read into.
A change to readBlockImpl so that it reads into that buffer. Now that HDDS-16258. Refactor streaming block reads and fix checksum verification for variable-sized chunks #11302 is merged, this is small: BlockReadCursor knows the length and the chunks of each read before the read.
Pooled direct buffers.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16722

How was this patch tested?

New tests in ContainerStateMachineTests (run by TestContainerStateMachineLeader and TestContainerStateMachineFollower):
- each response becomes one reply with the response without its data, then the data;
- an error response becomes a reply without data;
- a failed read or a failed write throws and leaves the stream open, and nothing more is written after a failed write;
- commands other than ReadBlock are rejected.
New TestClosedContainerReadResolver: the resolver serves ReadBlock only for a container on this datanode that is not open or closing.
New TestHddsDispatcher#testReadBlockScansContainerOnlyIfTheStreamIsNotCancelled: a cancelled ReadBlock stream does not make the datanode scan the container; any other failure still does.
New TestStreamRead#testRatisStreamReadBlock: on a mini cluster, a plain Ratis client reads a block on a read-only stream, first from the open container through its Raft group, then, after the container is closed, through the read pipeline from SCM, which has a random ID and no Raft group. Both reads match the block file.

…datanode

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

# Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java

amaliujia · 2026-10-07T03:53:26Z

+   *
+   * @return the number of bytes written to the stream
+   */
+  static long streamReadBlock(ContainerDispatcher dispatcher, ContainerCommandRequestProto request,


Curious why we make this as static? Is it for easy testing?

just because it doesnt depend on ContainerStateMachine.

amaliujia · 2026-10-07T04:13:20Z

      CertificateClient caClient, StateContext context) throws IOException {
    Parameters parameters = createTlsParameters(
        new SecurityConfig(ozoneConf), caClient);
+    if (parameters == null) {


In this case, createTlsParameters could just return new Parameters() than null?

rich7420 · 2026-10-07T08:45:02Z

+        } catch (IOException e) {
+          error.set(e);
+          // Throw to stop reading the block
+          throw new UncheckedIOException(e);


Could we let UncheckedIOException propagate through KeyValueHandler.readBlock? It currently becomes CONTAINER_INTERNAL_ERROR and triggers a container scan on client disconnect. Please add a test asserting no scan is triggered.

rich7420 · 2026-10-07T08:45:02Z

+    }
+
+    final Container<?> container = containerController.getContainer(requestProto.getContainerID());
+    if (container == null || container.getContainerState() != CLOSED) {


Should this also accept QUASI_CLOSED? SCM returns a read pipeline without an existing Raft group for that state too.

rich7420 · 2026-10-07T08:45:31Z

@peterxcli thanks for the patch!

Adapt the tests to HDDS-16258: ReadBlockResponseProto no longer has checksumData, and buildReadBlockCommandProto takes includeChecksums.

… off

…elled When a reply cannot be written, the Ratis read stream now cancels the read with a CANCELLED status. KeyValueHandler.readBlock passes a CANCELLED status on instead of turning it into CONTAINER_INTERNAL_ERROR, which made the dispatcher scan the container. This also covers the gRPC streaming read, whose onNext throws CANCELLED once the client has gone away.

… is not open or closing SCM gives every container that is not open or closing a read pipeline with a random ID, so the resolver now serves those containers instead of only CLOSED ones.

# Conflicts: # hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachineTests.java

…ir own Also say in the resolver why it checks isClosing, and in the Ratis read stream why it throws CANCELLED.

HDDS-16722. Serve ReadBlock over Ratis read-only data streams on the …

c688981

…datanode

Copilot AI balanced review requested due to automatic review settings October 5, 2026 16:34

Copilot AI reviewed Oct 5, 2026

github-actions Bot added the datanode label Oct 5, 2026

HDDS-16722. Do not add a DatanodeVersion before the client side exists

394ce69

peterxcli marked this pull request as ready for review October 6, 2026 03:19

peterxcli requested review from adoroszlai, amaliujia, ivandika3, sodonnel and szetszwo October 6, 2026 03:19

Merge remote-tracking branch 'upstream/master' into HDDS-16722

6c60bd5

# Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java

amaliujia reviewed Oct 7, 2026

View reviewed changes

rich7420 reviewed Oct 7, 2026

View reviewed changes

peterxcli added 7 commits October 8, 2026 10:44

Merge remote-tracking branch 'upstream/master' into HDDS-16722

81c2e7d

Adapt the tests to HDDS-16258: ReadBlockResponseProto no longer has checksumData, and buildReadBlockCommandProto takes includeChecksums.

HDDS-16722. Return empty Ratis parameters instead of null when TLS is…

a4d95fa

… off

HDDS-16722. Serve reads without a Raft group for every container that…

7fb1932

… is not open or closing SCM gives every container that is not open or closing a read pipeline with a random ID, so the resolver now serves those containers instead of only CLOSED ones.

Merge remote-tracking branch 'upstream/master' into HDDS-16722

95fdf12

# Conflicts: # hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachineTests.java

HDDS-16722. Pass stream status exceptions on in a catch branch of the…

f6a6591

…ir own Also say in the resolver why it checks isClosing, and in the Ratis read stream why it throws CANCELLED.

HDDS-16722. Remove three code comments

a273495

Navigation Menu

Repository navigation

HDDS-16722. Serve ReadBlock over Ratis read-only data streams on the datanode - #11412

HDDS-16722. Serve ReadBlock over Ratis read-only data streams on the datanode#11412
peterxcli wants to merge 10 commits into
apache:masterfrom
peterxcli:HDDS-16722

peterxcli commented Oct 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

amaliujia Oct 7, 2026

Uh oh!

peterxcli Oct 9, 2026

Uh oh!

amaliujia Oct 7, 2026

Uh oh!

rich7420 Oct 7, 2026

Uh oh!

rich7420 Oct 7, 2026

Uh oh!

rich7420 commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

peterxcli commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

amaliujia Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

peterxcli Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

amaliujia Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

rich7420 Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

rich7420 Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

rich7420 commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

peterxcli commented Oct 5, 2026 •

edited

Loading