Missing sha1 digests from nexus-maven-repository-index.gz

[ariel-miculas]

Affected version

latest

Bug description

There are some sha1 digests missing from nexus-maven-repository-index.gz
One such example is: 92b6631659ba35ca09e44874d3eb936edfeee532 related to jakarta.validation-api

$ curl https://search.maven.org/solrsearch/select\?q\="1:92b6631659ba35ca09e44874d3eb936edfeee532"

{"responseHeader":{"status":0,"QTime":111,"params":{"q":"1:92b6631659ba35ca09e44874d3eb936edfeee532","core":"","indent":"off","fl":"id,g,a,v,p,ec,timestamp,tags","start":"","sort":"score desc,timestamp desc,g asc,a asc,v desc","rows":"20","wt":"json","version":"2.2"}},"response":{"numFound":1,"start":0,"docs":[{"id":"jakarta.validation:jakarta.validation-api:3.0.2","g":"jakarta.validation","a":"jakarta.validation-api","v":"3.0.2","p":"jar","timestamp":1653588099000,"ec":["-sources.jar",".pom","-javadoc.jar",".jar"],"tags":["jakarta","bean","validation"]}]}}%

Looking at the index with Luke I get the following results when searching for g:jakarta.validation v:3.0.2:

17136615	8.715791	u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar.asc.sha512;
17136616	8.715791	1=b1581d1ef516be94c473c0f4d97e59e394c70150; u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar;
17136617	8.715791	1=eb8bb58bbce6761e2e8095f48f383da117fb109c; u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar;
62019267	8.715791	u=jakarta.validation|jakarta.validation-api|3.0.2|NA|pom.asc.sha512;
62019268	8.715791	u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar.asc.sha256;
62019269	8.715791	u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar.asc.sha512;
62019270	8.715791	u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar.asc.sha256;

Here, I can only see the sha1sums for sources.jar and javadoc.jar, but not for the main jar with digest 92b6631659ba35ca09e44874d3eb936edfeee532

It may be related to this other IndexSearcher issue.

What I also tried was running the latest version of the indexer on the following directory structure:

$ tree jakarta
jakarta
└── validation
    └── jakarta.validation-api
        └── 3.0.2
            ├── jakarta.validation-api-3.0.2-javadoc.jar
            ├── jakarta.validation-api-3.0.2-javadoc.jar.asc
            ├── jakarta.validation-api-3.0.2-javadoc.jar.asc.md5
            ├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha1
            ├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha256
            ├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha512
            ├── jakarta.validation-api-3.0.2-javadoc.jar.md5
            ├── jakarta.validation-api-3.0.2-javadoc.jar.sha1
            ├── jakarta.validation-api-3.0.2-sources.jar
            ├── jakarta.validation-api-3.0.2-sources.jar.asc
            ├── jakarta.validation-api-3.0.2-sources.jar.asc.md5
            ├── jakarta.validation-api-3.0.2-sources.jar.asc.sha1
            ├── jakarta.validation-api-3.0.2-sources.jar.asc.sha256
            ├── jakarta.validation-api-3.0.2-sources.jar.asc.sha512
            ├── jakarta.validation-api-3.0.2-sources.jar.md5
            ├── jakarta.validation-api-3.0.2-sources.jar.sha1
            ├── jakarta.validation-api-3.0.2.jar
            ├── jakarta.validation-api-3.0.2.jar.asc
            ├── jakarta.validation-api-3.0.2.jar.asc.md5
            ├── jakarta.validation-api-3.0.2.jar.asc.sha1
            ├── jakarta.validation-api-3.0.2.jar.asc.sha256
            ├── jakarta.validation-api-3.0.2.jar.asc.sha512
            ├── jakarta.validation-api-3.0.2.jar.md5
            ├── jakarta.validation-api-3.0.2.jar.sha1
            ├── jakarta.validation-api-3.0.2.pom
            ├── jakarta.validation-api-3.0.2.pom.asc
            ├── jakarta.validation-api-3.0.2.pom.asc.md5
            ├── jakarta.validation-api-3.0.2.pom.asc.sha1
            ├── jakarta.validation-api-3.0.2.pom.asc.sha256
            ├── jakarta.validation-api-3.0.2.pom.asc.sha512
            ├── jakarta.validation-api-3.0.2.pom.md5
            └── jakarta.validation-api-3.0.2.pom.sha1

4 directories, 32 files

$ java -jar indexer-cli/target/indexer-cli-7.1.7-SNAPSHOT-cli.jar -X -i /tmp/jakarta-validation-download/.index -d /tmp/jakarta-validation-download/.index -r /tmp/jakarta-validation-download
Version: 7.1.7-SNAPSHOT
Repository Folder: /tmp/jakarta-validation-download
Index Folder:      /tmp/jakarta-validation-download/.index
Output Folder:     /tmp/jakarta-validation-download/.index
Repository name:   .index
Indexers: [min, jarContent]
Will not create checksum files.
Will create baseline file.
Scanning started
       1 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
       2 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
       3 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
       4 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
Artifacts added:   4
Artifacts deleted: 0
[main] INFO org.apache.maven.index.packer.DefaultIndexPacker - Unable to read properties file, will force index regeneration
Total time:   0 sec
Final memory: 29M/260M

And I got the following error when unpacking the index:

$ java -jar ~/work/maven-indexer/indexer-cli/target/indexer-cli-7.1.7-SNAPSHOT-cli.jar -u
Index Folder:      /private/tmp/jakarta-validation-download/.index
Output Folder:     /private/tmp/jakarta-validation-download/.index
java.lang.IllegalArgumentException: cannot change field "m" from index options=NONE to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS
        at org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:273)
        at org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:567)
        at org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:476)
        at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3074)
        at org.apache.maven.index.updater.IndexDataReader.readIndexMT(IndexDataReader.java:268)
        at org.apache.maven.index.updater.IndexDataReader.readIndex(IndexDataReader.java:132)
        at org.apache.maven.index.updater.DefaultIndexUpdater.unpackIndexData(DefaultIndexUpdater.java:344)
        at org.apache.maven.index.updater.DefaultIndexUpdater.unpackIndexData(DefaultIndexUpdater.java:324)
        at org.apache.maven.index.cli.NexusIndexerCli.unpack(NexusIndexerCli.java:500)
        at org.apache.maven.index.cli.NexusIndexerCli.execute(NexusIndexerCli.java:157)
        at org.apache.maven.index.cli.NexusIndexerCli.main(NexusIndexerCli.java:116)

I'm curious to know what the search API does in order to find this digest that isn't present in the central maven index and what I can do in order to accurately get all the sha1sums identifying the java packages.

[cstamas]

Yes, and sadly index publishing is totally out of our control. It is done by Sonatype with unknown version (to us. probably ancient) of Maven Indexer. @mbien @brianf

[ariel-miculas]

So what would be the way forward? I guess the first step is to get my PR merged, then wait for a new release of the maven indexer.
Do you have any contacts at Sonatype that could clarify what version of the indexer they're using? And then convince them to use the latest one?

[cstamas]

Ideally, to have @brianf chime in and see what options we have here

[ariel-miculas]

@brianf Any chance you could clarify the index publishing process and how we could update the maven-indexer that's doing the publishing?