Node.js tests are failing

[pboothe]

It looks like node changed some of their internals, and now the end-to-end tests that use src/node_tests/ndt_client.js are failing.

[nkinkade]

Additional background:

[11:38] Nathan @pboothe: It seems that ndt_client.js doesn't work on newer versions of nodejs. The tests fail with:
[11:38] 
0 'DIED: ' { Error: read ECONNRESET
   at exports._errnoException (util.js:1020:11)
   at TCP.onread (net.js:568:26) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
[11:38] Peter Boothe !!!
[11:39] Nathan I've narrowed it down to the end of the C2S test.
[11:39] Could it possibly have something to do with this?
[11:39] https://github.com/nodejs/node/wiki/API-changes-between-v0.10-and-v4#child_process
[11:39] Peter Boothe Could be
[11:39] I would really like those unit tests to work
[11:40] Nathan I don't know whether the `ws.send()` is in any way related to `ChildProcess.prototype.send()`
[11:40] Peter Boothe So I am willing to fix them
[11:40] Where is this problem happening?
[11:40] Nathan C2S never exits `keep_sending_data()`
[11:41] That is, it makes it to here: https://github.com/m-lab/ndt/blob/master/src/node_tests/ndt_client.js#L256
[11:42] But putting a debug log statement after here never prints: https://github.com/m-lab/ndt/blob/master/src/node_tests/ndt_client.js#L283
[11:42] @pboothe ^^
[11:42] Peter Boothe weird
[11:43] But also: on what machine?
[11:43] georgiabullen @iros mentioned the other day that there have been some nodejs changes...
[11:43] just as a general comment
[11:43] Peter Boothe Are there web100 libraries on that machine?
[11:43] Nathan So it would seems that it makes it into the `keep_sending_data()` loop, and correctly notices that it's time to stop, but doesn't make it back to the point where `keep_sending_data()` was called.
[11:43] No, I'm running the client on my local machine.
[11:43] But there are certainly web100 libs on the server (mlab1.iad1t).
[11:44] I'm running a separate instance of ndtd on port 3002 to isolate the logs (on stdout/err).
[11:44] Peter Boothe Cool! So you are testing from your local machine to a server in the fleet!

[pboothe]

Okay! I think I may have a lead on this one, after some frustrating times. Nathan, what happens if you delete the node_modules directory in ndt/src/node_tests and then (in that same directory) run npm install ws minimist ?

Do things work then? If they do, the the problem was version skew between ws and node, that is we were using old ws code with a new node version that wasn't compatible with the old one.

[nkinkade]

@pboothe: I'm not sure that that is it. See the following. I'm using pretty much the latest versions of everything, and there is no node_modules directory in the same location as ndt_client.js. Am I doing something else wrong?

root@e1df094301f4:/# nodejs --version
v9.4.0
root@e1df094301f4:/# npm list --global minimist ws
/usr/lib
+-- minimist@1.2.0 
+-- npm@5.6.0
| +-- mkdirp@0.5.1
| | `-- minimist@0.0.8 
| `-- update-notifier@2.3.0
|   `-- latest-version@3.1.0
|     `-- package-json@4.0.1
|       +-- registry-auth-token@3.3.1
|       | `-- rc@1.2.1
|       |   `-- minimist@1.2.0 
|       `-- registry-url@3.1.0
|         `-- rc@1.2.1
|           `-- minimist@1.2.0 
`-- ws@4.0.0 

root@e1df094301f4:/# NODE_PATH=/usr/lib/node_modules nodejs /opt/mlab/ndt/src/node_tests/ndt_client.js --server ndt.iupui.mlab1.iad1t.m
easurement-lab.org
Running NDT test to ws://ndt.iupui.mlab1.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
root@e1df094301f4:/# ls /opt/mlab/ndt/src/node_tests/
README  ndt_client.js

[nkinkade]

Also, for what it's worth, it seems to succeed once in a while, and fail others. In my experience over the past few days, it fails more often than it succeeds. Here is runs 10 times, fails 8 times and succeeds twice.

root@ff590f85e48c:/# for i in $(seq 1 10); do NODE_PATH=/usr/lib/node_modules nodejs /opt/mlab/ndt/src/node_tests/ndt_client.js --server ndt.iupui.mlab3.iad1t.measurement-lab.org; done
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
Measured upload rate of 450318.36342753086 Kbps
Measured download rate of 96.27264746893145 Kbps
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
Measured upload rate of 447217.664 Kbps
Measured download rate of 98.05273980165866 Kbps
Running NDT test to ws://ndt.iupui.mlab3.iad1t.measurement-lab.org:3001/ndt_protocol
0 'DIED: ' { Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TCP.onread (net.js:620:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
root@ff590f85e48c:/#

[nkinkade]

More "FWIW" information. The following combination causes the for loop from my previous comment to succeed 10/10 times:

root@5b6c50eb17f1:/# nodejs --version
v0.12.18
root@5b6c50eb17f1:/# npm list -g minimist ws
/usr/lib
├── minimist@1.2.0 
├─┬ npm@2.15.11
│ └─┬ mkdirp@0.5.1
│   └── minimist@0.0.8 
└── ws@1.0.1