[Bug]: Same domain requests are timing out

[aidik]

Environment

Docker (self-hosted)

Version

0.1.4

Describe the Issue

I noticed that the Domain Updater keeps failing for the same domains. I suspect it's caused by the rate limits of the WHOIS services that Domain Locker uses. The first five domains are served correctly, and the rest time out.

A possible but hacky fix could be to use ORDER BY RANDOM(); in the domain-updater.
https://github.com/Lissy93/domain-locker/blob/8abded457fac300a742b683cc1594200cd2d26e3/src/server/routes/domain-updater/index.ts#L52-L57
That way, different domains would be served each time, and all would eventually get updated.

Logs

   ...
   ...
   ... 
   {
      "domain": "domain1.cz",
      "changes": [],
      "note": "ℹ️ No changes for domain1.cz, all data is up-to-date"
    },
    {
      "domain": "domain2.cz",
      "changes": [],
      "note": "ℹ️ No changes for domain2.cz, all data is up-to-date"
    },
    {
      "domain": "domain3.cz",
      "error": "Request timed out after 5 seconds for \"domain3.cz\""
    },
    {
      "domain": "domain4.cz",
      "error": "Request timed out after 5 seconds for \"domain4.cz\""
    },
    {
      "domain": "domain5.cz",
      "error": "Request timed out after 5 seconds for \"domain5.cz\""
    },
   ...
   ...
   ... 

System

===================================
APP INFO
===================================
Version:    0.1.4
Env Type:   selfHosted
Database:    Postgres
App ID:     domain-locker
Platform ID browser

Before submitting

• I have clearly described the issue and included all relevant information
• I have checked the documentation for a solution
• I have confirmed that this issue has not already been reported
• I have redacted any sensitive information from logs or debug data
• I agree to follow the Code of Conduct

[hvisage]

Hitting the exact same issue with ZARC requests.

Random is one option/method. but what about rather a more deterministic approach, add a "last valid check" value, and perhaps rate limit options, like, only do 5, then ignore the rest (that way the negative from the site that might block if too many negative requests is asked for eg. .za zones etc.)

then we sort and query only those that was the least recently succesfully checked first.

[aidik]

@hvisage random is just quick and dirty. Your "last valid check" is better in every aspect.

[liss-bot]

Hey @aidik, this has now been implemented by @jamalkamaladdin in #123, and will be released shortly in v0.2.6 😇

We're sorry this one took so long 😔