Skip to content

README.md

Latest commit

 

History

History
57 lines (48 loc) · 4.05 KB

README.md

File metadata and controls

57 lines (48 loc) · 4.05 KB
Raw

Pixels Retina

Retina (real-time analytics) is the real-time data synchronization framework in Pixels. It supports replaying data-change operations from a log-based CDC (change-data-capture) source as mirror transactions on the columnar table data. It proposes a light-weight MVCC mechanism and corresponding version storage to support parallel mirror transaction replay and concurrent analytical query processing, and a vectorized-filter-on-read (VFoR) approach for analytical queries to read consistent data snapshots.

Compared to the merge-on-read (MoR) based on catalog snapshots in existing lakehouse systems, such as Apache Iceberg and Apache Paimon, Retina supports real row-granular (instead of batch-granular) transactional data change replay without the expensive version merging and data compaction mechanisms. Evaluations show that Retina simultaneously provides 10-ms-level data freshness and over 3.2M row/s scalable data-change replay throughput, without compromising query performance or resource cost-efficiency, significantly outperforming state-of-the-art lakehouses, Iceberg and Paimon, which provides minute-level data freshness and one order of magnitude lower data-change throughput.

Visualization

We provide an interactive animation in your browser to present how Retina works.

In the default view of this animation, we show the replay progress of an example mirror transaction that inserts a tuple, delete it, and update another tuple. You can also click the buttons on the left bar to see the animations of detailed scenarios or stages, such as the replay of insert-only, delete-only, update-only mirror transactions, the asynchronous garbage collection behind transaction replay, and the VFoR during query processing.

Components

The components related to Retina are:

  • Sink: It connects to CDC streams from Debezium, reconstructs the data-change messages in the CDC stream into mirror transactions, and send the data-change operations in mirror transaction through stream RPC to Pixels-Retina. Source code;
  • Replayer: It receives the data-change operations from Pixels-Sink and replays them on the columnar data tables. Source code: core data structures and operations, top-level replay and garbage collection, client, and server. RPC handling are in this directory and .
  • Transaction Service: It allocates transaction timestamps for mirror transactions and analytical queries, and manages the timestamp watermarks for the MVCC protocol of Retina. Source code: client and server.
  • Index Service: It is a multi-version index that mapping the index key (e.g., primary key or secondary key) to row location. Replayer looks up and updates the primary index during data-change replay. Source code: framework and interfaces, pluggable implementations, clients server
  • Catalog Service (i.e., metadata service): It manages the schema, statistics, and data catalog of tables. Source code: client and server.
  • Columnar file format: It provides the file format definition, reader, writer of the Pixels file format. Source code.
  • Trino connector: It runs inside the Trino cluster to access the services of Retina/Pixels, and calls the file reader to read data. Source code.

Usage