Associate Software Engineer, Optimizer Team · Enterprise MPP Database Company · July 2022 – June 2024
A high-level overview of my work building indexing and query-optimization infrastructure for cloud-object-store-backed tables in a massively parallel processing (MPP) database. This document describes the problem space, my engineering approach, and the skills demonstrated at a conceptual level. It intentionally contains no proprietary source code, internal identifiers, or confidential design details.
Modern cloud data warehouses are increasingly decoupling storage from compute. Instead of keeping table data on local disks attached to each processing node, data is stored as objects in cloud object storage (e.g., Amazon S3). This is cheaper and scales elastically — but it introduces a performance challenge.
To read data from an object-store-backed table, the engine typically has to traverse a multi-level metadata structure (a root index, then leaf-level metadata) before it can locate and fetch the actual data object from the object store. Each of these steps is a network round-trip to cloud storage, and the metadata traversal can involve every node in the cluster.
This is acceptable for large analytical scans. But it is a serious problem for tactical queries — short, highly-tuned, point-lookup queries that return a handful of rows and must meet strict, predictable response-time SLAs (think: "fetch this one customer record in under 200ms"). The metadata-traversal overhead, and the fact that it ties up the whole cluster, made these queries slow and unpredictable on object-store-backed tables compared to traditional local-disk tables.
I contributed to a feature that introduced a local, compute-side index designed to short-circuit the expensive metadata traversal for tactical queries against object-store tables.
The core idea: materialize the object-location metadata into a compact index that lives on fast local storage (on the compute cluster), keyed by a stable logical identifier. When a tactical query runs, instead of walking the full multi-level metadata chain in the object store, the engine performs a small number of fast local lookups and then a single object-store fetch for the exact data needed.
This collapsed a multi-step, all-node query plan into a tightly bounded few-node operation — dramatically reducing both latency and cluster-wide resource contention for tactical workloads.
Index design & query planning
- Helped design and implement the new index structure and the query-optimizer changes needed to recognize when it could be used as a fast access path.
- Worked on the access-path selection logic and the generation of the optimized execution plan that leverages the new index.
Index maintenance (DML consistency)
- Designed and implemented end-to-end maintenance workflows so the index
stays consistent when the base table changes through
INSERT,UPDATE, andDELETEoperations. - This required reasoning carefully about how object-store data is rewritten on modification (objects are immutable, so updates create new objects and retire old ones) and ensuring the index correctly reflects every change.
- Handled the differences between physical deletes (row-format tables) and logical deletes (columnar tables, which mark rows deleted via metadata rather than rewriting data immediately).
Code architecture
- Consolidated the feature's maintenance logic into a single, well-structured module using a composable design pattern, so that the distinct DML operations could reuse a common set of building-block steps.
- This made the system extensible — additional maintenance scenarios (e.g., storage reorganization) could be added by composing existing building blocks rather than rewriting logic.
Integration across the query-processing pipeline
- Identified and implemented the integration points across the parsing, resolution, optimization, and node-level execution stages of the query engine required to make the feature work end to end.
Quality & debugging
- Wrote unit and functional tests and ran regression suites to ensure the feature was correct and didn't affect existing query behavior.
- Diagnosed and fixed defects using low-level debugging tools including core dump analysis, including a subtle bug where index maintenance was being triggered incorrectly on certain delete operations.
On a 2 TB industry-style benchmark dataset running on a public-cloud deployment (with object-store caching disabled to measure true performance), the feature delivered a ~2.7× throughput improvement for tactical query workloads.
The improvement came primarily from eliminating the cluster-wide metadata traversal: by reducing each tactical query from an all-node operation to a bounded few-node operation, the system freed up capacity to process far more concurrent queries in the same time window.
| Area | Detail |
|---|---|
| Systems programming | Large-scale C++ codebase, performance-critical query engine internals |
| Database internals | Query optimizer, access-path selection, execution-plan generation, index design, DML maintenance |
| Distributed systems | Massively parallel processing (MPP) architecture, storage/compute separation, cloud object storage |
| Software design | Applying design patterns to build extensible, maintainable feature modules |
| Debugging | Core dump analysis, root-causing subtle correctness bugs in a complex pipeline |
| Testing | Unit, functional, and regression testing for a production database engine |
This overview describes my work at a conceptual level using publicly-known database and cloud-computing concepts. It deliberately omits proprietary source code, internal component and identifier names, specific design documents, and any other confidential or trade-secret material belonging to my former employer (a major enterprise MPP database company). All implementation specifics remain confidential.
© 2026 Karan Verma. This document is a personal description of professional experience. All proprietary implementation details remain the property of their respective owner.