CVE data and services requirements for an at-scale SADP future

[zmanion]

The CVE Program is starting to broadly consider scaling issues. This is driven by at least two factors:

1. The number and rate of change of Records
2. The number and rate of change of containers (particularly SADP containers)

So, this is a Program-wide discussion, and it may live in the AWG. But this GitHub discussion is about the SADP aspects. Here are some starting notes.

What are some future estimates of number of records, number of (S)ADP containers, and rates of change?

Data storage and presentation probably need to be separate (i.e., storing a JSON file and delivering a JSON file won't scale).

Probably want coordinates/search/filtering capabilities, as CVE Services API calls, to support things like:

1. Show me SADP containers
   a. For a given SADP
   b. For a date (published, updated) range
   c. For a given CVE ID
   d. For a given CVE ID and SADP
2. Product search would be nice to have, even if it is string matching on elements in the affected array
3. How do we support bulk download? How does a consumer maintain a current copy of the CVE corpus? Is this even supported? There is an expectation that GitHub wil not remain the solution.
   a. One model is "download it all once then use these search features to make periodic updates." This generally works but is subject to abuse (e.g., a poorly behaving consumer just downloading the entire corpus every 15 minutes) unless there are technical and policy rate limiting measures in place.