Error in process._tickCallback

[hoschid]

After upgrading from node v0.12.7 to v4.2.3 our application crashes every few hours or after 1 or 2 days with the following uncaught exception:

TypeError: Cannot read property 'callback' of undefined
    at process._tickCallback (node.js:341:26)

Could this be a bug in node.js?

The code in src/node.js doesn't look like tock is expected to be undefined:

    // Run callbacks that have no domain.
    // Using domains will cause this to be overridden.
    function _tickCallback() {
      var callback, args, tock;

      do {
        while (tickInfo[kIndex] < tickInfo[kLength]) {
          tock = nextTickQueue[tickInfo[kIndex]++];
          callback = tock.callback;       // <------- line 341. tock is undefined
          args = tock.args;

[Fishrock123]

I'm pretty sure this will never happen unless something is modifying nextTickQueue...

cc @trevnorris?

EDIT: read it wrong and thought you meant .callback was undefined, checking again for tock.
EDIT2: Yeah I think this means you have something modifying nextTickQueue or tickInfo incorrectly.

[fa137]

Can you tell us about the other modules you are working with?

[hoschid]

I did grep -Rl "_tickCallback" * in my whole project to find something which could possibly modify nextTickQueue. The only thing I found was: weak/src/weakref.cc.

I'm using weak@1.0.0 (as a dependency of dnode@1.2.2). Could this module be the source of the problem?

Can nextTickQueue or tickInfo only be modified by c++ modules or by every module?

[Fishrock123]

@hoschid I don't think so: https://github.com/TooTallNate/node-weak/blob/master/src/weakref.cc#L142-L146

I think it would be if anything touches https://github.com/nodejs/node/blob/master/src/node.cc#L1086-L1093 but I really don't know for sure.

[hoschid]

The error didn't occur again for about 3 weeks now, although i didn't change anything on my app. I think this issue can be closed.

[kevinroisin]

I'm not sure if this is related or if I should open a new issue, so apologies in advance if I've done the incorrect thing.

I'm seeing what I think is the same error occur in seemingly random places. An example:

TypeError: Cannot read property 'callback' of undefined
     at process._tickDomainCallback (node.js:379:26)
     at Redis.exports.returnReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:148:100)
     at Redis.exports.returnReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:147:18)
     at exports.returnReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:145:30)
     at Deque$shift (/home/vcap/app/node_modules/ioredis/node_modules/double-ended-queue/js/deque.js:103:17)
     at exports.returnReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:145:30)
     at Deque$shift (/home/vcap/app/node_modules/ioredis/node_modules/double-ended-queue/js/deque.js:106:1)
     at exports.returnReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:145:30)
     at ReplyParser.replyParser.sendReply (/home/vcap/app/node_modules/ioredis/lib/redis/parser.js:25:10)
     at ReplyParser.execute (/home/vcap/app/node_modules/ioredis/lib/parsers/javascript.js:167:14)
     at Socket.<anonymous> (/home/vcap/app/node_modules/ioredis/lib/redis/event_handler.js:90:22)
     at emitOne (events.js:77:13)
     at Socket.emit (events.js:169:7)
     at readableAddChunk (_stream_readable.js:146:16)
     at Socket.Readable.push (_stream_readable.js:110:10)

I've grepped through the files for things that look obvious (anything with _tickCallback as @hoschid did) but have not had any luck. Any suggestions on where to look or things to check would be very appreciated.

Edit: left out that this on happening on node 4.2.5.

Edit 2: is it possible that the nextTickQueue being over 10,000 (1e4) is causing this issue?

[trevnorris]

@kevinroisin If you have a reproducable test case I'd say go ahead and open another issue. Being over 1e4 isn't a problem. That's simply a value to force processing of the nextTickQueue to the next loop as to not starve I/O.

[kevinroisin]

@trevnorris Thank you very much for your response. I've spent a few days trying to replicate it without our large code base (so I have something sharable) but it's proven challenging.

I've been looking a bit at the node.js and am wondering where _tickCallback and tickDomainCallback are called from. Is there any possibility that _tickDomainCallback is operating on the nextTickQueue without becoming aware of any splicing caused by _tickCallback via nextTick?

I'll keep trying to replicate the issue in a sharable way, but right now it is occurring consistently at load.

[trevnorris]

@kevinroisin When you require('domain') it actually overwrites _tickCallback(). Making it impossible to have collision between the two. That callback is called from MakeCallback located in both src/node.cc and src/async-wrap.cc.

[vitaliylag]

@hoschid I had some problems when tried to run dnode server with it's weak on node 4.x.x. But dnode clients work on 4.x.x greate. So I run server on 0.12.12 and clients on 4.4.1 now.

More info:

1. I had memory leaks on dnode clients when I tried to run dnode server on 4.x.x.
2. I had problems with process._tickCallback error and problems with setTimeout but I don't really know if that was dnode's weakmap fault.