Asap 225 add comparison to crawler

[lkacenja]

When we crawl a site multiple times, we want to store whether a document has been removed or was not present in a previous crawl. We also want the UI to show this information.

This PR updates the crawler script to perform a comparison step that joins the current crawled data to an optional previous crawl. The data from this comparison is handled via updated to the schema and document import rake task. Lastly the UI now shows badges for "Removed" and "New" documents. The latter only appear for a week. This PR adds test coverage for the import task and UI changes.

This PR also adds a shells script that streamlines the crawling and categorizing process.

• What additional steps are required to test this branch locally?

Try out the crawler shell script on a small site like oca.georgia.gov. Should be able to run something like:

 bin/crawl https://oca.georgia.gov db/seeds/site_documents_2025_09 db/seeds/site_documents 

If it works you should find a csv file in db/seeds/site_documents_2025_09/oca_georgia.csv.

Then you could create a site and try imported the documents via the rake command. Something like:

 bin/rake documents:import_documents"[<your site id>, db/seeds/site_documents_2025_09/oca_georgia.csv, false]"

• Are there any areas you would like extra review?

Just the above.

• Are there any rake tasks to run on production?

Not until we are ready to import a new crawl.

[lkacenja]

Some SLC documents are returning timestamps in various formats for their created or modified dates now. I added a helper to try and anticipate that.

[lkacenja]

It seems like there is something going on with the modification dates being collected by the crawler. I started working on a simple due diligence hex app for the crawling data and noticed a 8900+ row jump in null modification dates. Going to move this back into draft and figure out what's going on.

[lkacenja]

It seems like there is something going on with the modification dates being collected by the crawler. I started working on a simple due diligence hex app for the crawling data and noticed a 8900+ row jump in null modification dates. Going to move this back into draft and figure out what's going on.

False alarm, the hex notebook file upload to data frame functionality was doing implicit date conversion on the date columns with coercion enabled. I replaced those inputs with manual file uploads and select lists, so we can handle the data types. Now, things look much better. We actually lost two null modification dates.