QRiS Report

[philipbaileynar]

As QRiS gains traction, users will start uploading the projects to the Data Exchange and we need a way to analyse QRiS data across multiple QRiS projects.

This task is more about figuring out the data pipeline to make this possible, rather than the specific report.

Questions

1. Should the report gen Python query the Data Exchange API for QRiS projects, retrieve the QRiS GeoPackages and then iterate over them to generate the report? Or do we want some kind of data stream that watches for QRiS project updates and scrapes these data in real time into an Athena table that is used for reporting? I prefer the former. It is simpler and has less infrasture to maintain/break while we figure out even if this process will get widespread adoption.
2. Assume that we don't need imagery in the QRiS report. These rasters are typically extremely bulky. The single QRiS GeoPackage for each project is typically extremely small (a few Mb) and should be fast to download.
3. QRiS and the QRiS GeoPackage schema is constantly evolving. For the report to be stable, we must minimize the risks of the report breaking if QRiS projects have different versions of the schema. That said, the things we want to report on are fairly established and the dust has settled on their design. Can we keep the report simple enough to be robust?
4. Most temporal QRiS data exists in DCE layers that are specific to [protocols](https://qris.riverscapes.net/technical-reference/protocols/). QRiS stores these in extremely generic feature classes (one for each geometry type) with attributes stored as JSON. (There are views to make these data more easily accessible.) For example, something as simple as "LTPBR structures" might be stored in several different DCE layers depending on which protocol was used. How is the report supposed to know this and query the correct data? (This presumes DCE data are needed in the report. Check first.)
5. Is this report only for public projects or does the report gen need to know the user's ID so that it can incorporate private projects? See PBR report where the same question is relevant.

Notes

• Do not include a GeoFence. QRiS projects already exist in Scotland and we anticipate them being adopted in France and NZ.

FYI @KellyMWhitehead @MattReimer

[MattReimer]

1. Should the report gen Python query the Data Exchange API for QRiS projects, retrieve the QRiS GeoPackages and then iterate over them to generate the report? Or do we want some kind of data stream that watches for QRiS project updates and scrapes these data in real time into an Athena table that is used for reporting? I prefer the former. It is simpler and has less infrasture to maintain/break while we figure out even if this process will get widespread adoption.

I think we need to scrape the projects manually and not involve Athena. If we load every project into Athena automatically then that is Data Exchange CDK infrastructure we need to invent for one project type which is kind of an anti-pattern.

Also it means decoupling the data from the permissions grid and will need to re-couple it before any data gets transferred back. That's really easy to get wrong and leak data you shouldn't.

I also really would prefer not to have to implement the data exchange map UI inside the report UI.

An idea that might work is to let the user make a Collection or SavedSearch inside the data exchange with everything and then we can just make an API call to retrieve that Collection / SS with the explicit understand that only PUBLIC projects will be used.

There are a million ways the user might want to curate projects so it's much better to use the tools we already have instead of duplicating functionality inside another system if possible.

5. Is this report only for public projects or does the report gen need to know the user's ID so that it can incorporate private projects? See PBR report where the same question is relevant.

PUBLIC-only projects would be far preferable. This lets us ignore the permissions grid entirely. If we do allow PRIVATE projects then we need to pass the user's auth token to cybercastor which has it's own caveats (like it might expire before the report is done and transferring it is not a super secure thing to do.)

Currently the search API endpoint on the graphql does not allow a delegated userID (And by this I mean perform the search with admin credentials but as if you were user with the egentId)

Expense

This is the first report where the length of generation could get quite long. Should we really allow them to pull every QRIS project in the world?

[weblorin]

Okay, we can try 1:1 scrape to report and see how it goes. It feels very heavy and inefficient but what I'm hearing is the volume is quite small (of source projects, and of report users) and the churn in the source is high. Certainly I will build a local cache for development and testing - I do that with Athena anyway. But we should keep a close eye on the performance - as Matt points out longer computation is not only an inconvenience to users, it's a cost to us.

[philipbaileynar]

@joewheaton pointed out today, that when we get to the point that we want to incorporate temporal QRiS data (that is problematic because it is scattered across lots of different protocol layers that might or might not exist in every QRiS project), we should focus our efforts on the "as built" layers in the LTPBR V2. This should be the most ubiquitous/consistent layer in use (although note that Anabranch has their own experimental as-built protocol that will not show up if we use the LTPBR V2 layers).