There are concerns about the current design of the pipeline, specifically regarding the storage of large or unbounded data—such as test data—within single documents in mapping packages. While this approach is functional for now, it is likely to encounter scalability issues in the future.
Key points:
-
Test data can grow without bound and is therefore not well-suited for inclusion in a single document.
-
Resource sizes are unpredictable and can be significant, especially for shared or common assets.
Although including common rules within each package aligns with the intended design, duplicating large shared resources across all packages leads to inefficiencies and unnecessary package bloat.
Proposed solution:
Test data should not be stored directly in mapping packages. Instead, it should be separated to avoid size limitations and improve maintainability and scalability.
This change would help ensure that the system remains robust as data volumes increase over time.
There are concerns about the current design of the pipeline, specifically regarding the storage of large or unbounded data—such as test data—within single documents in mapping packages. While this approach is functional for now, it is likely to encounter scalability issues in the future.
Key points:
Test data can grow without bound and is therefore not well-suited for inclusion in a single document.
Resource sizes are unpredictable and can be significant, especially for shared or common assets.
Although including common rules within each package aligns with the intended design, duplicating large shared resources across all packages leads to inefficiencies and unnecessary package bloat.
Proposed solution:
Test data should not be stored directly in mapping packages. Instead, it should be separated to avoid size limitations and improve maintainability and scalability.
This change would help ensure that the system remains robust as data volumes increase over time.