Draft: Introduce HTML Fallback for Lucene Index

[finnwilken]

MARIN currently relies on repositories providing a Lucene index to iterate artifacts. However, some Maven repositories do not expose a Lucene index. One concrete example is: https://maven.artifacts.atlassian.com/

To support their repositories, this PR introduces a first draft of an HTML-based fallback iterator that discovers artifacts by parsing repository directory listings when a Lucene index is unavailable.

This first draft implementation (HTMLBasedIndexIterator) traverses directory pages and extracts artifact coordinates/paths.

Open questions:

1. Integration into MARIN
   I have not decided yet on the best integration point. Would you prefer this to be a separate configuration argument, or would you prefer it to automatically try the Lucene index first?

2. Missing metadata compared to Lucene
   The Lucene index provides metadata that, to my knowledge, cannot be reconstructed reliably from an HTML directory listing. One example is the “lastModified” argument.

This value is currently required to create IndexInformation objects. Therefore, HTMLBasedIndexIterator cannot be plugged in cleanly without adjusting the interface/data model (e.g., making some fields optional).

3. Modification of the RepositoryURL:
   It seems that currently only Maven Central is supported by MavenCentralArtifactAnalysis and MavenCentralLibraryAnalysis. The IndexIterator, on the other hand, allows you to set a different repository URL.

To allow users to benefit from analysis features (e.g., dependency resolution) for other repositories as well, it would be beneficial to make the repository base URL configurable beyond the iterator layer.

[johannesduesing]

Thanks for your contribution! I think using this type of discovery as a fallback when the Lucene index is not available is sensible - it should be very slow and resource intensive, so i think users would always prefer to rely on the Lucene index if possible.
Regarding your question 2, that's a difficult one. I agree that this HTML discovery likely cannot replace full index information. I know that the lastModified field can be extracted from the lastmodified headers of the HTTP response when performing a GET on the pom.xml file, but event if that would be possible, its not replacing other information about packaing and so on. I think it only makes sense to use HTML discovery to generate ArtifactIdent objects, so IndexInformation will not be available if this fallback is used.
As for question 3, i'd agree it would be nice to add the repository base URL to the regular configuration for all analyses, that should not be too difficult.

I'd like to point out that we also need Library-Discovery for Library Analyses - in those cases, the HTML discovery can stop whenever it finds a directory that only contains version numbers (and maybe a maven-metadata.xml).