The HTTP2 streams sometimes don't emit the `end` event

[szmarczak]

• Version: v13.13.0
• Platform: Linux solus 5.5.11-151.current deps: update openssl to 1.0.1j #1 SMP PREEMPT Tue Mar 24 18:06:46 UTC 2020 x86_64 GNU/Linux
• Subsystem: http2

What steps will reproduce the bug?

const {connect} = require('http2');
const session = connect('https://www.facebook.com');
session.once('remoteSettings', () => {
	console.log('got settings');
	session.request({
		'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36',
		'accept-encoding': 'gzip, deflate, br'
	}).once('response', headers => {
		console.log('got headers', headers);
	}).on('data', chunk => {
		console.log(chunk.toString().length);
	}).on('end', () => {
		console.log('got end');
	}).resume();
});

How often does it reproduce? Is there a required condition?

It reproduces 80% of the time. Sometimes it emits the end event as expected.

What is the expected behavior?

got settings
got headers [Object: null prototype] { ... }
[numbers here]
+got end

What do you see instead?

got settings
got headers [Object: null prototype] { ... }
[numbers here]

Additional information

curl works as expected:

curl --http2 -H 'accept-encoding: gzip, deflate, br' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36' https://www.facebook.com>/dev/null

First discovered by @kaatt szmarczak/http2-wrapper#38

[rexagod]

@szmarczak In events where 'end' is not emitted, is there an unusual delay before it's emitted, or is it not emitted at all (and the session closes abruptly with/without throwing)?

[szmarczak]

Not emitted at all. It doesn't throw, just hangs.

[rexagod]

Given the flaky nature of this issue, I can narrow this down to the fact that either Facebook sent you a "falsely" chunk (improbable) or the chunk size you received was over the highwatermark (which paused the stream and since you're waiting for the 'end' event before resuming, it hung up).

[szmarczak]

But there is a data event handler, so it shouldn't be paused...

[rexagod]

Refer this (L1023). ret can be a falsey value if either the stream has ended or the highwatermark is exceeded.

node/lib/_stream_readable.js

Lines 1009 to 1025 in a4e273b

	stream.on('data', (chunk) => {
	debug('wrapped data');
	if (state.decoder)
	chunk = state.decoder.write(chunk);
	// Don't skip over falsy values in objectMode.
	if (state.objectMode && (chunk === null || chunk === undefined))
	return;
	else if (!state.objectMode && (!chunk || !chunk.length))
	return;
	const ret = this.push(chunk);
	if (!ret) {
	paused = true;
	stream.pause();
	}
	});

[kaatt]

A more consistent repro would be to create a http2 server that does what @rexagod mentioned or inspect the network traffic of facebook.com on wireshark

[duongvanba]

Hi everyone, any fix or solution for this ?

[duongvanba]

HI everyone
Any fix on this ?

[szmarczak]

Fixed in #33875