build a transaction server

[dpetran]

Manages adding new commits to a ledger.

• Consensus
• Initiate commits
• Notify listeners (one of which is a listener)

[bplatz]

A version of this is what we need for Nexus, and that at least minimal use case should be pushed into Alpha 1.

We currently use Raft consensus across a cluster in Nexus - and this is what I think a general 'on-prem' deployment will want/need for redundancy and failover.

But, I think a discussion could be had to instead use a Kafka-like Queue for Nexus with set of "topics" that transactions for a given ledger reliable go onto the same topic. This will allow a cluster of transaction processors to fairly reliably scale - with only needing to deal with the 'at least once' problem, which Fluree I think makes it easy to deal with (as tx-ids, being the hash of the tx, are recorded to the ledger and can be looked up @ transaction processor startup to see if it sucessfully made it into the ledger.

If focused on AWS, Kinesis might be a good choice - but not sure if supports the same type of topic scaling and server management that Kafka does.

[bplatz]

If having more ephemeral transaction processors as per the queue idea above, we still need a reliable name service to record the latest commit for a ledger once persistent storage guarantees are met. Perhaps an easy way to handle this for the moment is Dynamo DB if on AWS.

We could look to use Fluree itself in a custom internal ledger, or some other DB as well.

[mpoffald]

Blocked on fluree/db#289

[dpetran]

I spoke with @jakep36 about transaction server functionality that would be beneficial for Nexus:

• A way to be able to run individual ledgers on different performance tiers
• Minimize the number of long-running processes
• A more stable method of providing fault tolerance, as managing raft requires a decent amount of attention

I spoke with Eliud and Abdul about Fluree Sense's requirements, and they primarily require performance as they deal with large amounts of data.

I spoke with Brian about the general design for alpha-1:

• We can use the existing raft implementation and run a raft cluster for consensus

Open questions:

• What are the precise perf requirements for Fluree Sense?
  • average transaction size, number of transactions per unit of time
• How do we integrate Fluree into the consensus layer?