Requests to the stats endpoints are timing out/hanging

[kashav]

Requests to the stats endpoints seem to be timing out on specific servers. Tested on a Digital Ocean instance, request does resolve, but takes significantly longer (as expected, kind of at least), can see why this is a problem for slower connections.

Would love some input on how to fix this, current course of action is to investigate/research this a little bit more, got has a timeout option that might help with this.

Originally reported on Twitter by Justin Conklin.

[eyev]

Thanks for recording the issue, I originally discovered this last night when attempting to host an express app that uses nba.js on an ec2 instance. My application was loading all the data endpoints fine but, none of the stats endpoints resolved. I tried heroku as well and had the same results.

The original endpoints failing for me were scoreboard and allPlayers but I soon discovered that all of them were failing. So i picked one with a very small return object to eliminate any chance of it being the size of the returns.

2017-03-18T04:17:53.225127+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/test" host=murmuring-beyond-xxxxx.herokuapp.com request_id=38a4d20d-1515-4b95-aa19-fa10efe9bf85 fwd="24.165.xxx.xxx" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=https

here is the /test endpoint

app.get('/test', (req, res) => { 
    nba.stats.franchiseHistory({LeagueID: 00 })
        .then(response => res.json(response))
        .catch(err => console.log(err));
})

[colbywhite]

I believe this is with the API in general, right? Not just slow connections?

Doing the following command hangs forever:

curl -X GET \
  -H "Referer: http://stats.nba.com/scores/" \
  -H "User-Agent: curl/7.43.0" \
  "http://stats.nba.com/stats/scoreboardV2?GameDate=2017-03-18&LeagueID=00&DayOffset=7"

That may not be the exact equivalent of what the JS lib is doing under the covers. But close? If so, something's going on with the NBA's servers unfortunately. From what I can tell, this has been happening with the NBA's servers since Mar. 15th.

EDIT: My User-Agent is much diff than yours. That could be it.

[eyev]

I did further testing today and I was able to get my express app working on Azure just fine. The stats endpoints do take a bit longer than data, but I believe that is normal. Regardless, it's still completely failing on AWS and Heroku for me.

[kashav]

@colbywhite Good catch! I tested your script with the user-agent used in nba.js and the request also seems to be hanging. The odd thing is, it doesn't hang when used with nba.js itself. I'm convinced that the API is looking for one of the other request headers, but will need to do a little more testing for that.

Accessing the URL directly from the browsers results in a 403 (I'm fairly sure this wasn't always the case, but I'm not certain). The page that makes the request to /scoreboardV2 is still up and running (and still retrieving data from that endpoint).

@eyev I'm glad it's working on Azure!

Do you think you could test the following on AWS/Heroku as well? I doubt Express is the problem here, but just in case --

const nba = require('nba.js').default;

nba.stats.franchiseHistory({ LeagueID: "00" })
  .then(response => console.log(response))
  .catch(err => console.log(err));

If that doesn't work, can you try editing (in the project root) node_modules/nba.js/lib/api/stats/utils/fetch.js by adding the following (I don't think you'll be able to test this on Heroku):

opts = Object.assign({
  headers: {
    host: 'stats.nba.com',
    connection: 'keep-alive',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
    referer: 'http://stats.nba.com/'
  },
  json: false,
+ timeout: 100000,
}, opts || {});

[eyev]

@kshvmdn not getting a response or error on heroku. I don't have the time right now but I will try to fork add the timeout later... (i think i can do this on heroku if i npm install a git repo rather than the npm package)

[Dolevco]

Adding this to the header in node_modules/nba.js/lib/api/stats/utils/fetch.js file worked for me:
opts = Object.assign({
headers: {
host: 'stats.nba.com',
"cache-control":"max-age=0",
connection: 'keep-alive',
"accept-encoding" : "Accepflate, sdch",
'accept-language':'he-IL,he;q=0.8,en-US;q=0.6,en;q=0.4',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
},
json: false
}, opts || {});

[kashav]

@Dolevco were you running this locally or on some server? If locally, was it not working before? Which endpoint(s)?

[Dolevco]

Im running this locally.
the teamLineups at nba stats endpoint didnt repond.
It worked fine just month ago, but when I tried to run the same code last week it waited forever.

changing the header like I mentioned did the work for me.

[kashav]

@Dolevco just tested, looks like local requests are broken for me too!

Tried with your changes to the header fields and requests are resolving again, will push a change for this!

Edit: looks like its only the accept-language field that's required to make the requests work, but I think I'll add all 3 anyways.

[kashav]

Published in v0.4.0!

Looks like we're still having the problem with running this on servers though. Build is failing on Travis, but running completely fine locally. Will have to look into this a bit more.

There are similar problems going on at bttmly/nba#41.