Support for robots.txt

Obey robots.txt. Minimum functonality:

**cancel all requests which globally disallow the huntsman `User-Agent`**

``` text
User-agent: huntsman
Disallow: /
```

**cancel all requests for urls which match `Disallow` statements**

``` text
User-agent: huntsman
Disallow: /private/area
```
