Better support for throttling/rate limiting in `std.net.http.server.Server`

[yorickpeterse]

Description

An HTTP server is created using std.net.http.server.Server.new in combination with a closure that returns a uni type that implements std.net.http.server.Handle.

Currently this closure isn't passed any arguments, making it difficult to apply some form of rate limiting before accepting the first request.

For example, when throttling by IP some form of shared state is required. Using a process is possible but because of the above you have to wait for the first request to come in before you can determine the IP of the request. This means a client could still try to exhaust the server's resources by establishing many connections but never sending a request.

To prevent this from happening we need to:

1. Pass the IP address to the handler, so it can be used before establishing the connection state
2. A way to signal back to the Server to abort setting up the connection, ideally without being forced to always return a Option[uni H: Handle]

Using this approach you could for example stick the IP into a Map along with the connection count, then reject the connection if the number is too large.

There are however some additional challenges with this:

1. Whatever dataset you store the IP in should have an upper bound, so you don't end up with 200 000 entries consuming more memory than the connections would otherwise consume
2. It will likely make the API more clunky, which I want to avoid
3. It doesn't account for residential proxies where some scraper uses 200 different IPs each establishing one connection, instead of one IP establishing 200 connections

Related work

No response

[yorickpeterse]

The motivation here is that some random scraper bot decided to start indexing the v0.21.1 manual, establishing what seems to be around 20 concurrent connections per second over the course of a few minutes.

The server handled this just fine and the memory increase was only about 20 MiB, but it made me realize we should at least consider making it a little easier to throttle/rate limit connections before they are set up.

[yorickpeterse]

Just to add to this: a Map[IpAddress, Int] with 10 000 randomly generated IPv6 entries takes 1 MiB of memory when using Map.with_capacity(10_000), so for 500 000 IPs you'd need 50 MiB which isn't all that bad.

[yorickpeterse]

A challenge with simply throttling concurrent connection counts is that a user opening multiple tabs to visit your site can easily hit such a limit.

Throttling based on the last request time requires shared concurrent state between different connections, and would inherently introduce some sort of delay before a request is denied.

For shost one approach we could take is that if the user-agent is not a browser-like agent (i.e. it doesn't start with Mozilla/), we restrict the number of requests to N per X seconds. In those cases the cost of accessing data synchronized through a process would only affect bots, and bots can in general go pound sand. This wouldn't require any changes in Inko.