From 23f67473f38cb6979c75ab65d0a04462d23e6cfc Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Wed, 30 Sep 2020 17:10:10 +0800 Subject: [PATCH 1/4] Add document about 1PC Signed-off-by: MyonKeminta --- design/async-commit/single-region-1pc.md | 46 ++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 design/async-commit/single-region-1pc.md diff --git a/design/async-commit/single-region-1pc.md b/design/async-commit/single-region-1pc.md new file mode 100644 index 0000000..c4c0d1b --- /dev/null +++ b/design/async-commit/single-region-1pc.md @@ -0,0 +1,46 @@ +# Single Region 1PC + +In the old days we did some attempt to implement single region 1PC for single region transactions, but due to incompatibility and some technical problem it was delayed. [There is some document left at that time (note that it's very outdated)](https://docs.google.com/document/d/1Vkk8LpYbXaQ0ualdFFH35V6mv9c9RsWJu2s6nhJz9E4/edit). But now since async commit, which meets and solved the same problem as 1PC, is implemented, we can continue supporting 1PC with much less effort to make. + +It's expected that by supporting single region 1PC, scenarios such as update_non_index will have better performance, lower latency and higher throughput. + +* Problems already solved by supporting async commit: + * Commit ts calculation + * Non-unique commit ts and rollback record overlapping + * Memory lock + * Replica read problem +* Problems that we do not care anymore: + * Binlog incompatibility + +## Basic Design + +Since async commit is implemented, it's not difficult to implement a working 1PC. However, it's still hard to make it perfectly correct and compatible with other components. + +* When committing a transaction, if TiDB finds that the prewrite phase can be done with only one single request, the transaction is allowed to be committed with 1PC protocol. A field named `try_one_pc` in the prewrite request will be set to let TiKV know that 1PC is available for this transaction. +* When TiKV receives a request with `try_one_pc` set, it first handle it just like how it handles normal prewrite requests.But after generating write buffer and before writing them down to RocksDB, it will additionally check if the prewrite is fully successful, and convert the locks into commit records if so. And finally write them down to RocksDB. The `commit_ts` is `max(max_ts, start_ts, for_update_ts) + 1`. It fetches the `max_ts` while acquiring the memory lock, and the memory lock is released after applying, just like how async commit does. The final `commit_ts` will be sent back to TiDB via prewrite response. +* 1PC and async commit can be independent. When TiKV rejects to commit a transaction with 1PC, the transaction can then fallback to normal transactions, and it may become a normal 2PC transaction or an async commit transaction, according to if the async commit flag is set. + +## Problems need to solve + +### Schema version checking problem + +[This problem exists in async commit too](https://github.com/tikv/sig-transaction/blob/master/design/async-commit/parallel-commit-known-issues-and-solutions.md#schema-version-checking). But it's even harder to solve for 1PC, because if it's committed in TiKV, it will have no chance to check if the schema version, while for async commit it can be checked after prewrite finishing. + +Possible solution: When trying committing a transaction with 1PC, find a ts `one_pc_max_commit_ts` before which we can guarantee that the schema version can't change, and send it to TiKV. TiKV will reject committing if the calculated `commit_ts` exceeds the `one_pc_max_commit_ts`. + +### CDC compatibility problem + +CDC syncs data from TiKV by observing applying events. Prewrites and commits are distinguished, and CDC will use these events to compose complete transactions and then send them to the downstream. So when 1PC is enabled, there need to be some way for CDC to distinguish if a apply event is caused by 1PC committing, otherwise CDC will expect that a commit event must has a corresponding prewrite event to compose a complete transaction. + +Possible solutions: +1. Passing a `is_1pc` flag from txn layer to apply, just like how `TxnExtra` (which is used to support CDC outputting old value) was written previously. It would be ugly. +2. When an apply batch puts write records but without deleting lockcf, we conclude that it must be a 1PC committing. This sounds like a simple way, but this may leave troubles to future. For example, if we want to introduce some new mechanisms that may amend persisted write records, we will then also need a new mechanism to distinguish the amending and 1PC committing. There's even one more problem: if the transaction being committed is a pessimistic transaction, the pessimistic lock need to be removed while writing the commit record +3. Add a field to write record that marks it's committed by 1PC. This approach makes the write record more complicated may waste some disk space. + +## POC Test + +WIP + +## Plan + +TBD From 109dc162afcc97dd9c05bc858e57ae0ae61d3d50 Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Mon, 12 Oct 2020 12:57:04 +0800 Subject: [PATCH 2/4] Fix typo Signed-off-by: MyonKeminta --- design/async-commit/single-region-1pc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design/async-commit/single-region-1pc.md b/design/async-commit/single-region-1pc.md index c4c0d1b..a0f261c 100644 --- a/design/async-commit/single-region-1pc.md +++ b/design/async-commit/single-region-1pc.md @@ -20,11 +20,11 @@ Since async commit is implemented, it's not difficult to implement a working 1PC * When TiKV receives a request with `try_one_pc` set, it first handle it just like how it handles normal prewrite requests.But after generating write buffer and before writing them down to RocksDB, it will additionally check if the prewrite is fully successful, and convert the locks into commit records if so. And finally write them down to RocksDB. The `commit_ts` is `max(max_ts, start_ts, for_update_ts) + 1`. It fetches the `max_ts` while acquiring the memory lock, and the memory lock is released after applying, just like how async commit does. The final `commit_ts` will be sent back to TiDB via prewrite response. * 1PC and async commit can be independent. When TiKV rejects to commit a transaction with 1PC, the transaction can then fallback to normal transactions, and it may become a normal 2PC transaction or an async commit transaction, according to if the async commit flag is set. -## Problems need to solve +## Problems need to be solved ### Schema version checking problem -[This problem exists in async commit too](https://github.com/tikv/sig-transaction/blob/master/design/async-commit/parallel-commit-known-issues-and-solutions.md#schema-version-checking). But it's even harder to solve for 1PC, because if it's committed in TiKV, it will have no chance to check if the schema version, while for async commit it can be checked after prewrite finishing. +[This problem exists in async commit too](https://github.com/tikv/sig-transaction/blob/master/design/async-commit/parallel-commit-known-issues-and-solutions.md#schema-version-checking). But it's even harder to solve for 1PC, because if it's committed in TiKV, it will have no chance to check if the schema version has changed between the `start_ts` and `commit_ts`, while for async commit it can be checked after prewrite finishing. Possible solution: When trying committing a transaction with 1PC, find a ts `one_pc_max_commit_ts` before which we can guarantee that the schema version can't change, and send it to TiKV. TiKV will reject committing if the calculated `commit_ts` exceeds the `one_pc_max_commit_ts`. From 76d978a5198ab67e857f148df4c118d55c7cba23 Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Tue, 13 Oct 2020 14:19:40 +0800 Subject: [PATCH 3/4] Add more descriptions and some test results Signed-off-by: MyonKeminta --- design/async-commit/single-region-1pc.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/design/async-commit/single-region-1pc.md b/design/async-commit/single-region-1pc.md index a0f261c..2b9d5f0 100644 --- a/design/async-commit/single-region-1pc.md +++ b/design/async-commit/single-region-1pc.md @@ -1,6 +1,8 @@ # Single Region 1PC -In the old days we did some attempt to implement single region 1PC for single region transactions, but due to incompatibility and some technical problem it was delayed. [There is some document left at that time (note that it's very outdated)](https://docs.google.com/document/d/1Vkk8LpYbXaQ0ualdFFH35V6mv9c9RsWJu2s6nhJz9E4/edit). But now since async commit, which meets and solved the same problem as 1PC, is implemented, we can continue supporting 1PC with much less effort to make. +For transactions that affect only one region, or more strictly speaking, that can be prewritten with only one prewrite request, can be committed directly while the prewrite request is being handled, so the commit phase can be totally removed. Therefore we can get less latency and more throughput. For TiDB, indices and rows are unsally not in a same region, **so this optimization only works under a limited amount of scenarios, like sysbench oltp_update_non_index**. But in a suitable scenario, it gains significantly better performance. + +In the old days we did some attempt to implement single region 1PC for single-region transactions, but due to incompatibility and some technical problem it was delayed. [There is some document left at that time (note that it's very outdated)](https://docs.google.com/document/d/1Vkk8LpYbXaQ0ualdFFH35V6mv9c9RsWJu2s6nhJz9E4/edit). But now since async commit, which meets and solved mostly the same problem as 1PC, is implemented, we can continue supporting 1PC with much less effort to make. It's expected that by supporting single region 1PC, scenarios such as update_non_index will have better performance, lower latency and higher throughput. @@ -39,7 +41,20 @@ Possible solutions: ## POC Test -WIP +We verified the improvement on a draft implementation with a brief test with following configurations, on a cluster with 3 TiKV nodes(16c 90G each) and 1 TiDB node(16c 64G): + +``` +sysbench oltp_update_non_index run --rand-type=uniform --db-driver=mysql --tables=8 --table-size=10000000 --time=270 --percentile=99 --report-interval=10 --threads=128 +``` + +Here's the result: + +| test | qps | lat avg (ms) | lat .99 (ms) | lat max (ms)| +|----------------------------|---------|--------------|--------------|-------------| +|update_non_index
1pc | 16603.37| 7.71| 15.00| 81.50| +|update_non_index
async_commit|13278.29| 9.64| 18.61| 110.50| +|update_non_index
2pc | 8892.85| 14.39| 24.83| 118.57| + ## Plan From 2e84434af935e793d7379f1ce310f93b9357a673 Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Tue, 13 Oct 2020 17:09:37 +0800 Subject: [PATCH 4/4] Remove inappropriate doc link Signed-off-by: MyonKeminta --- design/async-commit/single-region-1pc.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/design/async-commit/single-region-1pc.md b/design/async-commit/single-region-1pc.md index 2b9d5f0..f78fd9d 100644 --- a/design/async-commit/single-region-1pc.md +++ b/design/async-commit/single-region-1pc.md @@ -2,9 +2,7 @@ For transactions that affect only one region, or more strictly speaking, that can be prewritten with only one prewrite request, can be committed directly while the prewrite request is being handled, so the commit phase can be totally removed. Therefore we can get less latency and more throughput. For TiDB, indices and rows are unsally not in a same region, **so this optimization only works under a limited amount of scenarios, like sysbench oltp_update_non_index**. But in a suitable scenario, it gains significantly better performance. -In the old days we did some attempt to implement single region 1PC for single-region transactions, but due to incompatibility and some technical problem it was delayed. [There is some document left at that time (note that it's very outdated)](https://docs.google.com/document/d/1Vkk8LpYbXaQ0ualdFFH35V6mv9c9RsWJu2s6nhJz9E4/edit). But now since async commit, which meets and solved mostly the same problem as 1PC, is implemented, we can continue supporting 1PC with much less effort to make. - -It's expected that by supporting single region 1PC, scenarios such as update_non_index will have better performance, lower latency and higher throughput. +We used to think about supporting 1PC before but stopped after meeting many difficulties. Now since async commit, which meets and solved mostly the same problem as 1PC, is implemented, we can continue supporting 1PC with much less effort to make. * Problems already solved by supporting async commit: * Commit ts calculation