Tighten robots.txt to cut duplicate bot crawl - #3654

Tighten robots.txt to cut duplicate bot crawl#3654

ChristopherChudzicki wants to merge 3 commits into

mainmitodl/mit-learn:mainfrom

robots-txt-crawl-rulesmitodl/mit-learn:robots-txt-crawl-rulesCopy head branch name to clipboard