Testing Big GML Files

[bermud]

Seems that testing Big GML files is a problem 100 MBs or larger.

Discussed in OGC CITE SC Meeting - Washington TC 2016

Plan:

• set up a GitHub project to only tackle this problem
• it should:
  • validate the schema
  • validate the file using a schematron file
  • do some kind of geometry validation

We will test a 300 MB file, which is a file that normal validators (e.g. Oxygen, XML Spy, etc.) cannot handle. The files are here:

• CDDA_SITES_10000.zip: A 300 MB xml file, with 10,000 features of a pan-European dataset, relevant to INSPIRE Protected Sites data theme
• ProtectedSites_v4.0.sch.zip: a schematron file relevant to the INSPIRE Protected Sites theme

[bermud]

Forum discussions related to this topic:

• http://stackoverflow.com/questions/40663/validating-a-huge-xml-file
• http://stackoverflow.com/questions/9788868/how-to-validate-big-xml-against-xsd-schema
• https://dzone.com/articles/splitting-large-xml-files-java

[bermud]

From Richard's ideas, next steps might be:

• implement a utility class to split up very large GML documents (GmlDocumentSplitter).
• experiment with various chunk sizes to determine what works best (e.g. if doc size > 100 MB, split it into 50 MB chunks)
• if necessary, do Schematron validation of the resulting chunks in parallel

For example:

It should be possible to split any GML document into a desired chunk size at feature member boundaries.
For example, consider a utility class GmlDocumentSplitter. It might have a utility method that accepts the desired chunk size as an input parameter:

public File splitDocument(File gmlFile, int chunkSizeMB) {
// return first file in set: dataFile-01 (dataFile-02, dataFile-03, …)
}