Helpful hints

Lessons learned

• Variant annotations change often

It is important to de-couple variant annotations from the samples they were found in. Variant annotations in this context refer to information in the INFO field of VCF files. The specification of VCF files can be found here. This category of annotation can be variable from project to project, and may only make sense in certain contexts. It is not a good idea to replicate all the INFO field data into every sampleFormat field because there will be so much data duplication that the cost of building and maintaining a server to host it becomes impractical, not to mention the problems that would arise when you would need to update the contents every month. These types of annotations are not specific to any sample, i.e. they mean the same thing if they were found in sampleA and sampleB.

• Scale matters

We have previously examined document stores that have the annotation stored once, with an array of sample-level data embedded as sub-documents. We found this strategy maxes out when you approach 100,000 samples. We define 'maxing out' as hitting the 16MB document size limit in MongoDB.