General performance improvements on data ingestion - #67
Conversation
`tracestats` reads cosmos-sdk traces written to a file by the`--trace-store` command flag, and retrieves - block height - key length - value length - total trace length off each and every trace line, then creates a CSV file with this data. Useful for data analysis purposes.
Make it work with the new no-middleman type.
DeshErBojhaa
left a comment
There was a problem hiding this comment.
Please clarify the race case comment!!
Since tracewatcher now holds a sync.Pool, we can't just copy it around anymore, needs pointers.
| func (to *TraceOperation) Copy() TraceOperation { | ||
| ret := *to | ||
| return ret | ||
| } |
There was a problem hiding this comment.
Can we consider a deep copy here.
I know we're assigning new slice in the Reset, but doing a shallow copy of a struct containing slice is risky in a sense later someone else can abuse the data integrity.
If we don't introduce deep copy, at least leave some comments on that.
There was a problem hiding this comment.
This is a deep copy: https://go.dev/play/p/zEQGXtnVsHa.
Since we're dealing with a simple struct, assigning a dereferenced pointer to a variable effectively copies the whole object, including slices.
| func (to TraceOperation) String() string { | ||
| return fmt.Sprintf(`[%s] "%v" -> "%v"`, to.Operation, string(to.Key), string(to.Value)) |
There was a problem hiding this comment.
Please consider to *TraceOperator as receiver. Not a fan of mixing value and pointer receiver when not strictly necessary.
There was a problem hiding this comment.
Since nobody uses this method I was actually thinking of getting rid of it. WDYT?
| func (to *TraceOperation) Copy() TraceOperation { | ||
| ret := *to | ||
| return ret | ||
| } |
There was a problem hiding this comment.
Can we consider a deep copy here.
I know we're assigning new slice in the Reset, but doing a shallow copy of a struct containing slice is risky in a sense later someone else can abuse the data integrity.
If we don't introduce deep copy, at least leave some comments on that.
Also added missing godoc strings, removed `TraceOperation.String()` method, it wasn't needed anymore.
This PR attempts to optimize some rough corners on the ingestion side.
This is a deviation from @akhilkumarpilli proposed benchmarks, which were insanely helpful during the profiling and execution the code contained in this pull request.
A breakdown of what I did follows:
Got rid of the intermediate
traceOperationInterstructThat struct was used to obtain a """cuter"""
TraceOperationstruct, withBlockHeightandTxHashtop-level fields. In the end, I figured we could just have a top-levelMetadatastruct which enhances type safety by specifyingBlockHeightandTxHashtypes directly, without requiring implementing thejson.UnmarshalJSONinterface implementation.Operationtype is now a string, not a byte sliceHaving
Operationas a byte slice was useful but it's incredibly resource-intensive when having it in a tight loop, because it requires converting it tostringback and forth.Usage of a data allocation pool
Since
TraceOperation's are used a lot during the ingestion process, using async.Poolhelps with memory pressure by re-using structs already allocated previously. Each struct gets carefully wiped before usage. When data is selected to be parsed (e.g. theOperationmatches what we want) we copy theTraceOperationcontent, otherwise it's discarded.Zero-copy
stringto[]byteconversionThis is sorta controversial, I personally don't like this method at all but sometimes you gotta be flexible and bend to the Go runtime. The row that we read off the FIFO is potentially very big:tm:, and since each
stringto[]byteconversion involves an allocation + copy we risk wasting precious time just doing that. This PR introduces theunsafeGetBytes()function which does a zero-copy, zero-allocation conversion to[]byte, alleviating the tight loop penalty we were paying before. Keep in mind that in Go strings are immutable and that slices are passed by reference, so the output of this function effectively creates a mutable string. I ended up using this method anyway because the source string is discarded after use.I did not came up with this function, Ian Lance Taylor wrote it.
TL;DR: on my M1 Pro ARM machine, this branch produces the following numbers
So in a nutshell, it's about 21% quicker in reading and parsing traces while allocating less total memory, with a slight trade-off in terms of total allocations.
Source data for those results is here.