Skip to content

Bypass jena when hefquin can directly execute query - #755

Draft
KentThang wants to merge 1 commit into
mainfrom
bypass-jena-query-processor-if-possible
Draft

KentThang wants to merge 1 commit into
mainfrom
bypass-jena-query-processor-if-possible

Conversation

@KentThang

Copy link
Copy Markdown
Collaborator

Bypass Jena execution when HeFQUIN can directly execute the query.

WIP at the moment, notably ExampleQuery1.rq consistently returns 429 with source assignment:

--------- Source Assignment ---------
filter (4)
│  - expression: contains(str(?cc), "wikidata")
│  - may reduce duplicates: false
│  - certain variables: ?cc, ?c, ?o
│  - possible variables: none
│  - query planning info: none
│
└── mj (3)
    │  - may reduce duplicates: false
    │  - certain variables: ?cc, ?c, ?o
    │  - possible variables: none
    │  - query planning info: none
    │
    ├── req (1)
    │      - fm (3) SPARQL endpoint at https://qlever.dev/api/dbpedia
    │      - pattern (-1202458124): { <http://dbpedia.org/resource/Berlin> <[...]tology/country> ?c . ?c owl:sameAs ?cc }
    │      - projection variables: *
    │      - may reduce duplicates: false
    │      - certain variables: ?c, ?cc
    │      - possible variables: none
    │      - query planning info: none
    │
    └── req (2)
           - fm (13) SPARQL endpoint at https://qlever.dev/api/wikidata
           - pattern: { ?cc rdfs:label ?o }
           - projection variables: *
           - may reduce duplicates: false
           - certain variables: ?cc, ?o
           - possible variables: none
           - query planning info: none

--- pattern (-1202458124) se.liu.ida.hefquin.base.query.impl.GenericSPARQLGraphPatternImpl2
{ <http://dbpedia.org/resource/Berlin> <http://dbpedia.org/ontology/country> ?c . ?c owl:sameAs ?cc }

and logical plan:

--------- Final Logical Plan ---------
mj (11)
│  - may reduce duplicates: false
│  - certain variables: ?cc, ?c, ?o
│  - possible variables: none
│  - query planning info:  max cardinality = 2147483647 (min or max possible)
│  -                       min cardinality = 0 (min or max possible)
│  -                       cardinality = 2147483647 (pure guess)
│
├── req (8)
│      - fm (3) SPARQL endpoint at https://qlever.dev/api/dbpedia
│      - pattern (586751518): { <http://dbpedia.org/resource/Berlin> <[...] FILTER contains(str(?cc), "wikidata") }
│      - projection variables: *
│      - may reduce duplicates: false
│      - certain variables: ?c, ?cc
│      - possible variables: none
│      - query planning info:  max cardinality = 3 (accurate)
│      -                       min cardinality = 3 (accurate)
│      -                       cardinality = 3 (accurate)
│
└── req (10)
       - fm (13) SPARQL endpoint at https://qlever.dev/api/wikidata
       - pattern: { ?cc rdfs:label ?o FILTER contains(str(?cc), "wikidata") }
       - projection variables: *
       - may reduce duplicates: false
       - certain variables: ?cc, ?o
       - possible variables: none
       - query planning info:  max cardinality = 2147483647 (min or max possible)
       -                       min cardinality = 0 (min or max possible)
       -                       cardinality = 2147483647 (pure guess)

--- pattern (586751518) se.liu.ida.hefquin.base.query.impl.GenericSPARQLGraphPatternImpl1
{ <http://dbpedia.org/resource/Berlin> <http://dbpedia.org/ontology/country> ?c . ?c owl:sameAs ?cc FILTER contains(str(?cc), "wikidata") }

Notes before making it PR:

  • HeFQUINQuerySupportChecker and ResultSetSolutionMapping need to be moved to a more appropriate place
  • Remove the "DIRECT HEFQUIN" debug printout in HeFQUINEngine

@KentThang
KentThang requested a review from hartig October 8, 2026 09:16
@KentThang

Copy link
Copy Markdown
Collaborator Author

For comparison's sake, here are the plans for executing via Jena:

--------- Source Assignment ---------
mj (4)
│  - may reduce duplicates: false
│  - certain variables: ?cc, ?c, ?o
│  - possible variables: none
│  - query planning info: none
│
├── filter (2)
│   │  - expression: contains(str(?cc), "wikidata")
│   │  - may reduce duplicates: false
│   │  - certain variables: ?c, ?cc
│   │  - possible variables: none
│   │  - query planning info: none
│   │
│   └── req (1)
│          - fm (3) SPARQL endpoint at https://qlever.dev/api/dbpedia
│          - pattern (-1202458124): { <http://dbpedia.org/resource/Berlin> <[...]tology/country> ?c . ?c owl:sameAs ?cc }
│          - projection variables: *
│          - may reduce duplicates: false
│          - certain variables: ?c, ?cc
│          - possible variables: none
│          - query planning info: none
│
└── req (3)
       - fm (13) SPARQL endpoint at https://qlever.dev/api/wikidata
       - pattern: { ?cc rdfs:label ?o }
       - projection variables: *
       - may reduce duplicates: false
       - certain variables: ?cc, ?o
       - possible variables: none
       - query planning info: none

--- pattern (-1202458124) se.liu.ida.hefquin.base.query.impl.GenericSPARQLGraphPatternImpl2
{ <http://dbpedia.org/resource/Berlin> <http://dbpedia.org/ontology/country> ?c . ?c owl:sameAs ?cc }
SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
    FILTER contains(str(?cc), "wikidata")
  }

--------- Final Logical Plan ---------
mj (8)
│  - may reduce duplicates: false
│  - certain variables: ?cc, ?c, ?o
│  - possible variables: none
│  - query planning info:  min cardinality = 0 (min or max possible)
│  -                       cardinality = 759456611 (estimate based on accurates)
│  -                       max cardinality = 2147483647 (accurate)
│
├── req (7)
│      - fm (3) SPARQL endpoint at https://qlever.dev/api/dbpedia
│      - pattern (586751518): { <http://dbpedia.org/resource/Berlin> <[...] FILTER contains(str(?cc), "wikidata") }
│      - projection variables: *
│      - may reduce duplicates: false
│      - certain variables: ?c, ?cc
│      - possible variables: none
│      - query planning info:  min cardinality = 3 (accurate)
│      -                       cardinality = 3 (accurate)
│      -                       max cardinality = 3 (accurate)
│
└── req (3)
       - fm (13) SPARQL endpoint at https://qlever.dev/api/wikidata
       - pattern: { ?cc rdfs:label ?o }
       - projection variables: *
       - may reduce duplicates: false
       - certain variables: ?cc, ?o
       - possible variables: none
       - query planning info:  min cardinality = 759456611 (accurate)
       -                       cardinality = 759456611 (accurate)
       -                       max cardinality = 759456611 (accurate)

--- pattern (586751518) se.liu.ida.hefquin.base.query.impl.GenericSPARQLGraphPatternImpl1
{ <http://dbpedia.org/resource/Berlin> <http://dbpedia.org/ontology/country> ?c . ?c owl:sameAs ?cc FILTER contains(str(?cc), "wikidata") }

Notice how the filter operator is under the multijoin vs. it being over it in direct HeFQUIN.

@KentThang

KentThang commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

The requests printed out for direct HeFQUIN execution:

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
    FILTER contains(str(?cc), "wikidata")
  }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
    FILTER contains(str(?cc), "wikidata")
  }

SPARQLRequest with pattern: { <http://dbpedia.org/resource/Berlin>
            <http://dbpedia.org/ontology/country>  ?c .
  ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  FILTER contains(str(?cc), "wikidata")
}, projection variables: null, distinct required: false
SPARQLRequest with pattern: { VALUES ?cc { <http://www.wikidata.org/entity/Q183> <http://www.wikidata.org/entity/Q43287> <http://www.wikidata.org/entity/Q41304> }
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
    FILTER contains(str(?cc), "wikidata")
  }
}, projection variables: null, distinct required: false

Jena execution:

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o }

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { <http://dbpedia.org/resource/Berlin>
              <http://dbpedia.org/ontology/country>  ?c .
    ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
    FILTER contains(str(?cc), "wikidata")
  }

SPARQLRequest with pattern: { <http://dbpedia.org/resource/Berlin>
            <http://dbpedia.org/ontology/country>  ?c .
  ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  FILTER contains(str(?cc), "wikidata")
}, projection variables: null, distinct required: false
SPARQLRequest with pattern: { VALUES ?cc { <http://www.wikidata.org/entity/Q183> <http://www.wikidata.org/entity/Q43287> <http://www.wikidata.org/entity/Q41304> }
  ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
}, projection variables: null, distinct required: false

Notice this extra request in the direct HeFQUIN execution:

SPARQLRequest with query: SELECT  (count(*) AS ?__hefquinCountVar)
WHERE
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
    FILTER contains(str(?cc), "wikidata")
  }

@hartig

hartig commented Oct 9, 2026

Copy link
Copy Markdown
Member

Thanks for the detailed information about the issue that you have run into with this change.

Two more questions:
1/ What does the physical plan look like? (for the pure HeFQUIN case)
2/ When exactly does the error with status code 429 occur? In other words, which request is the one for which you get the 429 response? Is it during query planning (i.e., one of the cardinality requests with the COUNT queries) or during query execution (i.e., one of the data retrieval requests)?

@KentThang

Copy link
Copy Markdown
Collaborator Author
--------- Physical Plan ---------
bind join for SPARQL (14)
│  - type: VALUES_BASED
│  - parallel version: true
│  - batch size: 30
│  - lop: gpAdd
│  - fm (13) SPARQL endpoint at https://qlever.dev/api/wikidata
│  - parameter variables: none
│  - pattern: { ?cc rdfs:label ?o FILTER contains(str(?cc), "wikidata") }
│  - may reduce duplicates: false
│  - certain variables: ?cc, ?c, ?o
│  - possible variables: none
│  - query planning info:  min cardinality = 0 (min or max possible)
│  -                       cardinality = 2147483647 (pure guess)
│  -                       max cardinality = 2147483647 (min or max possible)
│
└── req (12)
       - lop: req
       - fm (3) SPARQL endpoint at https://qlever.dev/api/dbpedia
       - pattern (586751518): { <http://dbpedia.org/resource/Berlin> <[...] FILTER contains(str(?cc), "wikidata") }
       - projection variables: *
       - may reduce duplicates: false
       - certain variables: ?c, ?cc
       - possible variables: none
       - query planning info:  min cardinality = 3 (accurate)
       -                       cardinality = 3 (accurate)
       -                       max cardinality = 3 (accurate)

--- pattern (586751518) se.liu.ida.hefquin.base.query.impl.GenericSPARQLGraphPatternImpl1
{ <http://dbpedia.org/resource/Berlin> <http://dbpedia.org/ontology/country> ?c . ?c owl:sameAs ?cc FILTER contains(str(?cc), "wikidata") }

It should be the last query performed right? In that case, it should be this request:

SPARQLRequest with pattern: { <http://dbpedia.org/resource/Berlin>
            <http://dbpedia.org/ontology/country>  ?c .
  ?c        <http://www.w3.org/2002/07/owl#sameAs>  ?cc
  FILTER contains(str(?cc), "wikidata")
}, projection variables: null, distinct required: false
SPARQLRequest with pattern: { VALUES ?cc { <http://www.wikidata.org/entity/Q183> <http://www.wikidata.org/entity/Q43287> <http://www.wikidata.org/entity/Q41304> }
  { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
    FILTER contains(str(?cc), "wikidata")
  }
}, projection variables: null, distinct required: false

If I run the query with --skipExecution, the cardinality requests are made without causing a Too Many Requests so I want to say the error occurs during query execution.

@hartig

hartig commented Oct 9, 2026

Copy link
Copy Markdown
Member

Regarding 2, notice that your print out mentions two requests. Assuming it is the second one of these two that causes the error, can you try to do just this query with curl (here is a comment with some curl commands that you can adapt: #737 (comment))

If that works without problems, can you put the pattern of that query into a single SERVICE clause and execute that as a query with HeFQUIN.

Also, assuming you get the Too Many Requests error as an exception printed after execution, how does the complete stack trace of this exception look like?

@KentThang

Copy link
Copy Markdown
Collaborator Author

Regarding 2, notice that your print out mentions two requests. Assuming it is the second one of these two that causes the error, can you try to do just this query with curl (here is a comment with some curl commands that you can adapt: #737 (comment))

It seems to work and return results when using curl.

If that works without problems, can you put the pattern of that query into a single SERVICE clause and execute that as a query with HeFQUIN.

Running this query:

SELECT ?cc ?o WHERE {
	SERVICE <https://qlever.dev/api/wikidata> {
		VALUES ?cc {
		<http://www.wikidata.org/entity/Q183>
		<http://www.wikidata.org/entity/Q43287>
		<http://www.wikidata.org/entity/Q41304>
		}
		?cc <http://www.w3.org/2000/01/rdf-schema#label> ?o .

		FILTER(CONTAINS(STR(?cc), "wikidata"))
	}
}

also worked without problems.

The full stack trace is this:

Exception 1: Performing the remote execution of a query at the SPARQL endpoint with service URI https://qlever.dev/api/wikidata caused an exception (type: org.apache.jena.sparql.engine.http.QueryExceptionHTTP) with the following message: Too Many Requests
StackTrace:
se.liu.ida.hefquin.engine.queryplan.executable.ExecOpExecutionException: Performing the remote execution of a query at the SPARQL endpoint with service URI https://qlever.dev/api/wikidata caused an exception (type: org.apache.jena.sparql.engine.http.QueryExceptionHTTP) with the following message: Too Many Requests
        at se.liu.ida.hefquin.engine.queryplan.executable.impl.ops.BaseForExecOpParallelBindJoin.recordException(BaseForExecOpParallelBindJoin.java:602)
        at se.liu.ida.hefquin.engine.queryplan.executable.impl.ops.BaseForExecOpParallelBindJoin$MyResponseProcessor.accept(BaseForExecOpParallelBindJoin.java:545)
        at se.liu.ida.hefquin.engine.queryplan.executable.impl.ops.BaseForExecOpParallelBindJoin$MyResponseProcessor.accept(BaseForExecOpParallelBindJoin.java:531)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: se.liu.ida.hefquin.federation.access.FederationAccessException: Performing the remote execution of a query at the SPARQL endpoint with service URI https://qlever.dev/api/wikidata caused an exception (type: org.apache.jena.sparql.engine.http.QueryExceptionHTTP) with the following message: Too Many Requests
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequestWithQueryExecutionHTTP(SPARQLRequestProcessorImpl.java:91)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequest(SPARQLRequestProcessorImpl.java:48)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequest(SPARQLRequestProcessorImpl.java:28)
        at se.liu.ida.hefquin.federation.access.impl.AsyncFederationAccessManagerImpl$1.get(AsyncFederationAccessManagerImpl.java:58)
        at se.liu.ida.hefquin.federation.access.impl.AsyncFederationAccessManagerImpl$1.get(AsyncFederationAccessManagerImpl.java:56)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        ... 3 more
Caused by: HttpException: 429 Too Many Requests
        at org.apache.jena.sparql.engine.http.QueryExceptionHTTP.rewrap(QueryExceptionHTTP.java:49)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.executeQuery(QueryExecHTTP.java:552)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.execRowSet(QueryExecHTTP.java:173)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.select(QueryExecHTTP.java:164)
        at org.apache.jena.sparql.exec.QueryExecutionAdapter.execSelect(QueryExecutionAdapter.java:108)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequestWithQueryExecutionHTTP(SPARQLRequestProcessorImpl.java:84)
        ... 8 more

@KentThang

Copy link
Copy Markdown
Collaborator Author

Could it be that the extra request that occurs when executing the query directly via HeFQUIN is enough to cause the Too Many Requests?

@hartig

hartig commented Oct 9, 2026

Copy link
Copy Markdown
Member

Could it be that the extra request that occurs when executing the query directly via HeFQUIN is enough to cause the Too Many Requests?

It seems so. I mean, it is the only possible reason that I can see at the moment.

The strange thing is that the other endpoint does not complain about too many requests, even if HeFQUIN sends three requests to that one as well. And another thing that is strange is that, if you run the query again, the complaint doesn't come already for the first of the three request. Or, perhaps it does!?

If you look at the "query planning info" of the Wikidata request (req (10) in the final logical plan of the pure HeFQUIN case), you will see that the cardinality estimate is "2147483647 (pure guess)". Given that this is not an accurate number (as there is for req (8) in the same plan), this means that the cardinality request (i.e., the COUNT query) for the Wikidata request must have failed in some way. The relevant place in the source code is the if-block at line 245 of RequestBasedCardinalityEstimator. As you can see there, we are not currently surfacing these errors/exceptions into the list of exceptions that is printed in the end (because they are not actually exceptions during query execution and, thus, do not have an effect on the completeness of the query result).

Can you please check what exactly is happening with the cardinality requests for this query by putting a debug break point into that if-block (or temporarily putting a println into the if-block).

@KentThang

KentThang commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

I printed out the following in the aforementioned if-block:

Request operator request: SPARQLRequest with pattern: { ?cc  <http://www.w3.org/2000/01/rdf-schema#label>  ?o
  FILTER contains(str(?cc), "wikidata")
}, projection variables: null, distinct required: false
Request operator fm service URI: https://qlever.dev/api/wikidata
Query planning info: se.liu.ida.hefquin.engine.queryplan.info.QueryPlanningInfo@5bcde458
Cardinality response error description: null
isError: false
isDefective: true

Some fields are evidently more helpful than others, can't say this has clarified anything for me though.

@hartig

hartig commented Oct 10, 2026

Copy link
Copy Markdown
Member

If isDefective() is true, then the response object should contain an exception. What is the stack trace of this exception?

@KentThang

Copy link
Copy Markdown
Collaborator Author

It seems to be this:

se.liu.ida.hefquin.federation.access.FederationAccessException: Performing the remote execution of a query at the SPARQL endpoint with service URI https://qlever.dev/api/wikidata caused an exception (type: org.apache.jena.sparql.engine.http.QueryExceptionHTTP) with the following message: Too Many Requests
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequestWithQueryExecutionHTTP(SPARQLRequestProcessorImpl.java:91)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequest(SPARQLRequestProcessorImpl.java:48)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequest(SPARQLRequestProcessorImpl.java:28)
        at se.liu.ida.hefquin.federation.access.impl.AsyncFederationAccessManagerImpl$1.get(AsyncFederationAccessManagerImpl.java:58)
        at se.liu.ida.hefquin.federation.access.impl.AsyncFederationAccessManagerImpl$1.get(AsyncFederationAccessManagerImpl.java:56)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: HttpException: 429 Too Many Requests
        at org.apache.jena.sparql.engine.http.QueryExceptionHTTP.rewrap(QueryExceptionHTTP.java:49)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.executeQuery(QueryExecHTTP.java:552)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.execRowSet(QueryExecHTTP.java:173)
        at org.apache.jena.sparql.exec.http.QueryExecHTTP.select(QueryExecHTTP.java:164)
        at org.apache.jena.sparql.exec.QueryExecutionAdapter.execSelect(QueryExecutionAdapter.java:108)
        at se.liu.ida.hefquin.federation.access.impl.reqproc.SPARQLRequestProcessorImpl.performRequestWithQueryExecutionHTTP(SPARQLRequestProcessorImpl.java:84)
        ... 8 more

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants