-
Notifications
You must be signed in to change notification settings - Fork 0
Buzzbang crawl data format
This is a page to capture thoughts about the Buzzbang MongoDB Common format (see Overall Buzzbang Framework Architecture for more details).
The structure has yet to be decided. One possibility (currently done by Buzzbang-NG crawler, is to add fields directly to the metadata. For instance, from a Biosamples dataset page
{
"@context": "http://schema.org",
"@type": "Dataset",
"lastCrawled": "2018-06-07T10:39:12.445037038+01:00",
"description": "BioSamples stores and supplies descriptions and metadata about biological samples used in research and development by academia and industry. Samples are either 'reference' samples (e.g. from 1000 Genomes, HipSci, FAANG) or have been used in an assay database such as the European Nucleotide Archive (ENA) or ArrayExpress.",
"includedInDataCatalog": {
"@id": "https://www.ebi.ac.uk/biosamples",
"@type": "DataCatalog"
},
"name": "Sample collection",
"url": "https://www.ebi.ac.uk/biosamples/samples"
}
where datetime in inserted by the crawler itself.
The pros of taking this approach is that it keeps MongoDB querying simple (e.g. if we wanted to pull all structures for Datasets we can simply query without needing to know any Buzzbang specific structure, providing all the markup is normalized). The cons is that we may overwrite information in the markup. If we decide this is the best approach, we could use some marker for added metadata pairs (e.g. "__datettime__": "2018-06-07T10:39:12.445037038+01:00") which is extremely unlikely to clash with any markup. This is similar to the approach taken by the JSON-LD spec where anything starting with @ is regarded as a reserved word, though they have the advantage of being able to recommend this via their spec.
Another approach, proposed by Ricardo, is to put the crawled markup into a schema field and have the crawler metadata outside it, like so
{
"schema": {
"@context": "http://schema.org",
"@type": "Dataset",
"description": "BioSamples stores and supplies descriptions and metadata about biological samples used in research and development by academia and industry. Samples are either 'reference' samples (e.g. from 1000 Genomes, HipSci, FAANG) or have been used in an assay database such as the European Nucleotide Archive (ENA) or ArrayExpress.",
"includedInDataCatalog": {
"@id": "https://www.ebi.ac.uk/biosamples",
"@type": "DataCatalog"
},
"name": "Sample collection",
"url": "http://wp-p1m-41.ebi.ac.uk:8081/biosamples/samples",
},
"last_crawled": "2018-06-07T10:39:12.445037038+01:00"
}
where schema contains the schema data and url, datetime are crawler added metadata. This is probably the cleaner way to go and does not overly complicated MongoDB queries (you just need to know that the markup is within the schema field).
However, we have yet to make a definitive decision. We also still need to agree on the metadata fields and their format.
This is an example of the proposed fields
{
"schema": {
"@context": "http://schema.org",
"@type": "Dataset",
"description": "BioSamples stores and supplies descriptions and metadata about biological samples used in research and development by academia and industry. Samples are either 'reference' samples (e.g. from 1000 Genomes, HipSci, FAANG) or have been used in an assay database such as the European Nucleotide Archive (ENA) or ArrayExpress.",
"includedInDataCatalog": {
"@id": "https://www.ebi.ac.uk/biosamples",
"@type": "DataCatalog"
},
"name": "Sample collection",
"page": "https://www.ebi.ac.uk/biosamples/samples/?start=19380"
},
"url": "http://wp-p1m-41.ebi.ac.uk:8081/biosamples/samples",
"last_crawled": "2018-06-07T10:39:12.445037038+01:00",
"schema_source_type": "Microdata",
"ip_address": "xxx.xxx.xxx.xxx",
"content_type": "html",
"source_link_type": "sitemap.xml",
"crawler_id": "leCrawler"
}
schema - mandatory, the crawled structure, probably canonicalized (see below)
url - mandatory, the url from which this data came (Ankit - in Buzzbang-ng you are removing the URL scheme. I know you told us the reason, could you remind me (justincc) why?)
last_crawled - mandatory, ISO-8601. Although MongoDB may always add initial date time under its _id field, this is probably still useful in case the structure gets updated (if we are not keeping all historical data)
crawled_id - mandatory, the ID of the crawler that scraped this data
schema_source_type - optional, the original type of structure for the scraped data. How could this be useful?
ip_address - optional, the IP address of the host for this data. How could this be useful?
content_type - optional, the mime type of the originating page. How could this be useful?
source_link_type - optional, whether this data came from navigating a sitemap.xml or by single site link following. How could this be useful?
There are many ways of representing the same thing in JSON-LD. For instance all these forms of Person data are equivalent (taken from the JSON-LD Playground).
{
"@context": "http://schema.org/",
"@type": "Person",
"name": "Jane Doe",
"jobTitle": "Professor",
"telephone": "(425) 123-4567",
"url": "http://www.janedoe.com"
}
Where:
- No @context is given so all URLs are written out in full.
- http://schema.org is labelled as an @id since we are now not getting this information from the @context fetched from http://schema.org
- @value is used to explicitly specify the value (this is usually used where other information needs to be included, such as @type of the property.
[
{
"@type": [
"http://schema.org/Person"
],
"http://schema.org/jobTitle": [
{
"@value": "Professor"
}
],
"http://schema.org/name": [
{
"@value": "Jane Doe"
}
],
"http://schema.org/telephone": [
{
"@value": "(425) 123-4567"
}
],
"http://schema.org/url": [
{
"@id": "http://www.janedoe.com"
}
]
}
]
Where the data is placed explicitly in a @graph that one can make statements about (here just with an explicit ID)
{
"@context": "http://schema.org/",
"@graph": [
{
"id": "_:b0",
"type": "Person",
"jobTitle": "Professor",
"name": "Jane Doe",
"telephone": "(425) 123-4567",
"url": "http://www.janedoe.com"
}
]
}
This describes how a graph of nodes can actually have many different tree structures (these are apparently often expected by JSON programmers). See the JSON-LD Framing Spec for examples.
Although Bioschemas only explicitly supports JSON-LD, there might be older formats used on certain sites that could be converted to JSON-LD. We probably won't require parsers to support this, though they can if they choose.
Because JSON-LD can have many different structures for the same data, I believe canonicalization is necessary somewhere along the line. This could take place a) as scraped markup is inserted into MongoDB b) by the tools that extract data from MongoDB.
My (justincc) current position is that we should canonicalize before we insert the data into MongoDB. This is because a) it will reduce the work for consumers, since each will not need to separately canonicalize and more importantly b) only having canonicalized data in MongoDB will allow us to efficiently query it. For example, if we wanted to find all the data with type Samples, we could execute something like
db.samples.find({ 'schema.mainEntity.@type': 'Samples'})
if the data in MongoDB is not canonicalized then this is not possible.
I (justincc) suggest that we canonicalize everything to the compact form, as this is the one which takes the least amount of space and is the easiest to query.
JSON-LD libraries are thin on the ground. For Python, the most complete looks to be pyld (https://github.com/digitalbazaar/pyld). It is sensitive to malformed data (e.g. will throw an exception if a @context is referenced that does not yet exist, like "@context": "http://bioschemas.org. If we want to support such malformed data we may have to a) write our own canonicalization code (not at all recommended since we would then have to maintain it), b) hack pyld to accept some malformed JSON-LD (I (justincc) did look at this once but it did not seem easy. I feel such hacks are unlikely to be accepted by the maintainers and so we would end up needing to maintain a forked pyld). So I feel we simply do not want to support data that pyld cannot process (unless it is due to an obvious pyld bug that we can fix), though this might miss out some fairly high proportion of not well-formed markup since Bioschemas is still very much evolving.