Improve feeders

[AlCutter]

This PR improves the feeder implementation, making it easier to use and more friendly to feeding much larger numbers of logs without needing a correspondingly large number of goroutines.

• Extract a RunFeeders func, and share between omniwitness and feedwitness binaries.
• Only retry submissions to the witness if we've detected we're stale, other errors will be retried at a higher level
• Expose the current size of log according to the witnesses to higher levels, allowing that size to be cached and used as a hint for the next submission
• Don't allow one stalled witness to interfere with submissions to other witnesses.
• Use a more sensible HTTP client in feedwitness

[codecov-commenter]

Codecov Report

❌ Patch coverage is 10.69182% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.17%. Comparing base (3c58af4) to head (1b25d4c).
⚠️ Report is 218 commits behind head on main.

Files with missing lines	Patch %	Lines
omniwitness/run_feeders.go	0.00%	69 Missing ⚠️
cmd/feedwitness/main.go	0.00%	31 Missing ⚠️
internal/feeder/feeder.go	62.96%	8 Missing and 2 partials ⚠️
omniwitness/omniwitness.go	0.00%	6 Missing ⚠️
cmd/gcp/omniwitness/main.go	0.00%	5 Missing ⚠️
internal/feeder/rekor_v1/rekor_feeder.go	0.00%	5 Missing ⚠️
internal/feeder/tiles/tiles_feeder.go	0.00%	5 Missing ⚠️
internal/feeder/pixelbt/pixel_feeder.go	0.00%	4 Missing ⚠️
internal/feeder/serverless/serverless_feeder.go	0.00%	4 Missing ⚠️
internal/feeder/sumdb/sumdb_feeder.go	0.00%	3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (3c58af4) and HEAD (1b25d4c). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (3c58af4)	HEAD (1b25d4c)
	4	2

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #406       +/-   ##
===========================================
- Coverage   51.05%   20.17%   -30.88%     
===========================================
  Files          11       28       +17     
  Lines         903     2052     +1149     
===========================================
- Hits          461      414       -47     
- Misses        374     1577     +1203     
+ Partials       68       61        -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.