TEZ-4466: Measure exact stream read time while fetching by Aggarwal-Raghav · Pull Request #530 · apache/tez

Aggarwal-Raghav · 2026-08-05T17:52:57Z

new MeasuredDataInputStream( 
    new MeasuredInputStream(            //  STOPWATCH WRAPPER (the actual logic)
        new DataInputStream(          
            new BufferedInputStream(    
                connection.getInputStream()
            )
        )
    )
)

Aggarwal-Raghav · 2026-08-06T17:46:28Z

Ran basic groupby query, SET tez.runtime.optimize.local.fetch=false; ran with this on local cluster.

--
SELECT 
  group_key, 
  COUNT(id), 
  SUM(rand_val1), 
  MAX(padding)
FROM large_data
GROUP BY group_key;

tez-yetus · 2026-08-06T17:50:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 54s		master passed
+1 💚	compile	2m 33s		master passed
+1 💚	checkstyle	0m 43s		master passed
+1 💚	javadoc	0m 53s		master passed
+0 🆗	spotbugs	0m 50s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 12s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 32s		the patch passed
+1 💚	javac	2m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	javadoc	0m 48s		the patch passed
+1 💚	spotbugs	1m 50s		the patch passed
			_ Other Tests _
+1 💚	unit	59m 59s		root in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		80m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 13e32d590552 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/testReport/
Max. process+thread count	1380 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav · 2026-08-06T18:01:38Z

This is ready @abstractdog from my end, requesting your eagle eye review.

Aggarwal-Raghav · 2026-08-06T18:13:10Z

    this.badIdErrs = badIdErrsCounter;
    this.connectionErrs = connectionErrsCounter;
    this.wrongReduceErrs = wrongReduceErrsCounter;
+    this.ioTimeCounter = inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_IO_TIME_MILLISECONDS);


1 thing to call out is even tez.runtime.shuffle.measure.io.time=false the counter will have SHUFFLE_IO_TIME_MILLISECONDS: 0 to prevent this the following is required or we can have it but might be misleading looking at the counter stats..

this.ioTimeCounter = conf.getBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_MEASURE_IO_TIME, TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_MEASURE_IO_TIME_DEFAULT) ? inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_IO_TIME_MILLISECONDS) : null;

Addressed in b1dd37b

Aggarwal-Raghav · 2026-08-06T18:37:11Z

New test:
With set tez.runtime.shuffle.measure.io.time=false;

With set tez.runtime.shuffle.measure.io.time=true;

tez-yetus · 2026-08-06T19:57:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 52s		master passed
+1 💚	compile	2m 31s		master passed
+1 💚	checkstyle	0m 43s		master passed
+1 💚	javadoc	0m 52s		master passed
+0 🆗	spotbugs	0m 49s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 11s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 30s		the patch passed
+1 💚	javac	2m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed
+1 💚	spotbugs	1m 46s		the patch passed
			_ Other Tests _
+1 💚	unit	61m 24s		root in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		81m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux cb72d4f2d834 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/testReport/
Max. process+thread count	2124 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog

looks very good so far @Aggarwal-Raghav , please find a few comments

abstractdog · 2026-08-17T09:42:40Z

+    // Check if io time counter is updated
+    TezCounter ioTimeCounter = inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_IO_TIME_MILLISECONDS);
+    long ioTime = ioTimeCounter.getValue();
+    assertTrue(ioTime >= 10, "IO Time should be at least 10ms, but was " + ioTime);


10ms is a wild guess, even if it's working for 99% of the cases: I would use ioTime >= 0, because it already proves that the IO time was indeed measure

tez-yetus · 2026-08-17T13:16:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 50s		master passed
+1 💚	compile	3m 57s		master passed
+1 💚	checkstyle	1m 3s		master passed
+1 💚	javadoc	1m 17s		master passed
+0 🆗	spotbugs	1m 23s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	1m 12s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 2s		the patch passed
+1 💚	codespell	1m 46s		No new issues.
+1 💚	compile	3m 59s		the patch passed
+1 💚	javac	3m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	javadoc	1m 16s		the patch passed
+1 💚	spotbugs	3m 1s		the patch passed
			_ Other Tests _
-1 ❌	unit	74m 26s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		115m 54s

Reason	Tests
Failed junit tests	tez.dag.app.TestSpeculation

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/8/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 83eff14f3ccb 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/8/testReport/
Max. process+thread count	1474 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/8/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav · 2026-08-17T15:15:34Z

test is flaky, will re-trigger the UT

tez-yetus · 2026-08-17T17:00:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for branch
-1 ❌	mvninstall	4m 17s	/branch-mvninstall-root.txt	root in master failed.
-1 ❌	compile	2m 58s	/branch-compile-root.txt	root in master failed.
+1 💚	checkstyle	1m 3s		master passed
+1 💚	javadoc	1m 15s		master passed
+0 🆗	spotbugs	1m 24s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	1m 12s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s		Maven dependency ordering for patch
-1 ❌	mvninstall	3m 39s	/patch-mvninstall-root.txt	root in the patch failed.
+1 💚	codespell	1m 20s		No new issues.
-1 ❌	compile	2m 55s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	2m 55s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 58s		the patch passed
+1 💚	javadoc	1m 15s		the patch passed
+1 💚	spotbugs	2m 58s		the patch passed
			_ Other Tests _
+1 💚	unit	74m 2s		root in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		103m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/9/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux df1e15e05860 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/9/testReport/
Max. process+thread count	1652 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/9/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2026-08-17T18:49:35Z

    synchronized (cleanupLock) {
      try {
        if (httpConnection != null) {
+          if (input instanceof MeasuredDataInputStream && ioTimeCounter != null) {


nit: maybe evaluate "ioTimeCounter != null" first, looks more efficient than an instanceof check
also, is the instanceof check needed at all? by design, ioTimeCounter is non-null, if and only if measurement is enabled

sure. the if statement will be like

if (ioTimeCounter != null && input != null) {

tez-yetus · 2026-08-17T21:04:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 13s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 32s		master passed
+1 💚	compile	4m 0s		master passed
+1 💚	checkstyle	1m 6s		master passed
+1 💚	javadoc	1m 19s		master passed
+0 🆗	spotbugs	1m 23s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	1m 11s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 47s		the patch passed
+1 💚	codespell	1m 36s		No new issues.
+1 💚	compile	3m 59s		the patch passed
+1 💚	javac	3m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 2s		the patch passed
+1 💚	javadoc	1m 18s		the patch passed
+1 💚	spotbugs	2m 54s		the patch passed
			_ Other Tests _
+1 💚	unit	75m 13s		root in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		107m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/11/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux b55e06115cbd 5.15.0-186-generic #196-Ubuntu SMP Sat Jun 20 16:09:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/11/testReport/
Max. process+thread count	1499 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/11/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog

nice, looks good to me, +1

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 0120635 to e459166 Compare August 5, 2026 17:57

Aggarwal-Raghav mentioned this pull request Aug 5, 2026

TEZ-4748: Expose SHUFFLE_IO_TIME_MILLISECONDS via TaskCounter #531

Closed

This comment was marked as outdated.

Sign in to view

Aggarwal-Raghav force-pushed the TEZ-4466 branch from e459166 to 4ad0480 Compare August 5, 2026 19:23

This comment was marked as outdated.

Sign in to view

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 4ad0480 to 3bd8eaa Compare August 6, 2026 06:26

This comment was marked as outdated.

Sign in to view

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 3bd8eaa to d99959a Compare August 6, 2026 10:34

TEZ-4466: Measure exact stream read time while fetching

65ef7ac

Aggarwal-Raghav force-pushed the TEZ-4466 branch from d99959a to 65ef7ac Compare August 6, 2026 16:29

Aggarwal-Raghav commented Aug 6, 2026

View reviewed changes

Fix SHUFFLE_IO_TIME off counter stat behaviour

b1dd37b

abstractdog requested changes Aug 17, 2026

View reviewed changes

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 6b1f1f0 to ecbfad8 Compare August 17, 2026 15:15

Address review comments

6b05939

Aggarwal-Raghav force-pushed the TEZ-4466 branch from ecbfad8 to 6b05939 Compare August 17, 2026 17:22

abstractdog reviewed Aug 17, 2026

View reviewed changes

review comments

1caccbc

abstractdog approved these changes Aug 18, 2026

View reviewed changes

abstractdog merged commit 64e79ee into apache:master Aug 18, 2026
6 checks passed

Uh oh!

Conversation

Aggarwal-Raghav commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

abstractdog Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tez-yetus commented Aug 17, 2026

Uh oh!

Aggarwal-Raghav commented Aug 17, 2026

Uh oh!

tez-yetus commented Aug 17, 2026

Uh oh!

abstractdog Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

tez-yetus commented Aug 17, 2026

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Aggarwal-Raghav commented Aug 5, 2026 •

edited

Loading

abstractdog Aug 17, 2026 •

edited

Loading