It seems like AI would be incredible at building these crawlers. We should build a system where we give it the name of the company/store, and it goes and does the research necessary to gather the information to build a successful crawler.
We need to add safe guards and protections to ensure that it is sandboxed properly. We only want to give it access to view the internet and modify 1 file (the new crawler). It shouldn't have any other access to the file system or anything.
We probably also need to write up some instructions for how to build a crawler (API, etc).
It seems like AI would be incredible at building these crawlers. We should build a system where we give it the name of the company/store, and it goes and does the research necessary to gather the information to build a successful crawler.
We need to add safe guards and protections to ensure that it is sandboxed properly. We only want to give it access to view the internet and modify 1 file (the new crawler). It shouldn't have any other access to the file system or anything.
We probably also need to write up some instructions for how to build a crawler (API, etc).