I want to do something like round(val*scale)/scale but in a way that avoids the computation involved with actually rescaling the value (and it can be ceil, floor, etc. instead of round - whatever is the fastest). The value of scale can be significantly restricted (e.g. only powers of 2 or 10 or something along those lines would be perfectly fine). For fixed point numbers, this can be done by simply masking off LSBs, which is very fast. Is there any "nice" way to do this sort of controlled truncation in fastnum? I did notice that you're using powers of 10 in the exponent instead of powers of 2, so it looks like a simple bit mask would not work. Looks like quantize does this, but it looks like it does so by rescaling the value which seems to involve lots of repeated division by 10, which is decidedly not optimal from a performance perspective.
The use case here is that I have some traces that are potentially quite long (millions to billions of points) and have both a large time range (months to years) and fine time resolution (picoseconds, or potentially even finer). Think unix time, but with a heck of a lot of fractional bits. And the X points are not regularly spaced as they're coming from high-resolution timestampers. When plotting such a large trace, it is necessary to down-sample it in some way to reduce the number of points that have to be plotted on screen. One way to do this is to bin the points into time bins, keeping only a few points per bin (first, last, min, max). The exact size of the bins isn't all that important, so long as they're narrower than the pixels on the screen, so being limited to scale factors that are powers of 2 or powers of 10 is fine. Being able to do an efficient round/floor/ceil offset from the actual decimal point would be a nice way to determine which bin a given point falls into. The X points are monotonically increasing, so I guess I could also rescale the first point and then compare and increment, but a more general ability to truncate the value might have other uses. Although if a compare operation requires multiplication/division for realignment, that could still be a problem.
Also, in relation to the exponent being powers of 10 instead of powers of 2, have you looked at all into the performance implications of that design decision? Seems like powers of 10 requires a lot of multiplying and dividing by 10 to align things, vs. powers of 2 where that can be handled with very fast bit shifts. I know IEEE floating point uses base 2 so it can use barrel-shifters in hardware. Presumably there are also some good reasons for using powers of 10. I checked the documentation, but didn't see any discussion of trade-offs of base 10 vs. base 2. Have you done much benchmarking that does a lot of rescaling?
I want to do something like
round(val*scale)/scalebut in a way that avoids the computation involved with actually rescaling the value (and it can beceil,floor, etc. instead ofround- whatever is the fastest). The value ofscalecan be significantly restricted (e.g. only powers of 2 or 10 or something along those lines would be perfectly fine). For fixed point numbers, this can be done by simply masking off LSBs, which is very fast. Is there any "nice" way to do this sort of controlled truncation in fastnum? I did notice that you're using powers of 10 in the exponent instead of powers of 2, so it looks like a simple bit mask would not work. Looks likequantizedoes this, but it looks like it does so by rescaling the value which seems to involve lots of repeated division by 10, which is decidedly not optimal from a performance perspective.The use case here is that I have some traces that are potentially quite long (millions to billions of points) and have both a large time range (months to years) and fine time resolution (picoseconds, or potentially even finer). Think unix time, but with a heck of a lot of fractional bits. And the X points are not regularly spaced as they're coming from high-resolution timestampers. When plotting such a large trace, it is necessary to down-sample it in some way to reduce the number of points that have to be plotted on screen. One way to do this is to bin the points into time bins, keeping only a few points per bin (first, last, min, max). The exact size of the bins isn't all that important, so long as they're narrower than the pixels on the screen, so being limited to scale factors that are powers of 2 or powers of 10 is fine. Being able to do an efficient round/floor/ceil offset from the actual decimal point would be a nice way to determine which bin a given point falls into. The X points are monotonically increasing, so I guess I could also rescale the first point and then compare and increment, but a more general ability to truncate the value might have other uses. Although if a compare operation requires multiplication/division for realignment, that could still be a problem.
Also, in relation to the exponent being powers of 10 instead of powers of 2, have you looked at all into the performance implications of that design decision? Seems like powers of 10 requires a lot of multiplying and dividing by 10 to align things, vs. powers of 2 where that can be handled with very fast bit shifts. I know IEEE floating point uses base 2 so it can use barrel-shifters in hardware. Presumably there are also some good reasons for using powers of 10. I checked the documentation, but didn't see any discussion of trade-offs of base 10 vs. base 2. Have you done much benchmarking that does a lot of rescaling?