RISC-V Implementation for TensTorrent

[sempervictus]

Thank you for publishing this @williamofai - we work in healthcare/lifesci/energy/critical path industries and this class of computational library for inference/ML work has come up multiple times in those spaces but oddly outside the finserv space the others tend to be extremely under-funded for foundational work like this even if they have cash pouring in for sexy chatbots they can't use (its a madhouse out there).

I've asked @brixen, one of the consultants on our team and a man "somewhat familiar with architectural semantics" to take a look at implementing this atop the tenstorrent interfaces to their RISC-V kit as the static allocation mechanics should permit granular targeting of host, device, and even SRAM in those things to make use of larger models feasible in smaller timeframes (sometimes clinically precise decisions have to come quick while being complex and conventional architectures like ARM/x86 aren't well suited for that presently). We have a BH QuietBox with 4xp150 cards in it to qualify the effort and can provide access through our private cloud edge for R&D if this thread is of interest.

Thanks again, hopefully we can help kick this effort up a notch toward production utilization in more use-cases.

[williamofai]

@sempervictus @brixen

Thank you for reaching out - this is exactly the kind of collaboration that moves certifiable inference from proof-of-concept toward production reality.

The static allocation model in certifiable-inference was designed with precisely this kind of hardware targeting in mind. Being able to map buffers explicitly to host, device, and SRAM rather than hoping a runtime makes sensible choices is fundamental to both timing guarantees and certification arguments.

A few thoughts:

RISC-V alignment: The fixed-point integer arithmetic is already architecture-agnostic - no floating-point variance to worry about. RISC-V's clean ISA makes WCET analysis more tractable than x86's complexity.

Tenstorrent specifics: I'd be very interested to understand the memory hierarchy on the p150s and how explicit placement maps to their programming model. The documentation I've seen suggests Metalium gives fairly direct control, which suits this approach.

Healthcare/clinical timing: The "fast and certifiable" requirement is the hard problem most people underestimate. Happy to discuss how the bit-identical replay and deterministic scheduling properties could support the regulatory arguments you'll eventually need.

I'd welcome access to the QuietBox for R&D. A working Tenstorrent implementation would be a significant proof point for the approach.

What's the best way to coordinate - continue here, or shall I reach out directly?

Gratefully,

William

[sempervictus]

Pinged by email re access but on the public-facing side of this:

RISC-V alignment: The fixed-point integer arithmetic is already architecture-agnostic - no floating-point variance to worry about. RISC-V's clean ISA makes WCET analysis more tractable than x86's complexity.

Precisely our thinking - formal proofs are a lot more feasible when you can order operations and the microcode doesn't change the actual RISC execution beneath what looks like CISC to the ISA to your code.

Tenstorrent specifics: I'd be very interested to understand the memory hierarchy on the p150s and how explicit placement maps to their programming model. The documentation I've seen suggests Metalium gives fairly direct control, which suits this approach.

As i understand it "its not hierarchical" in that we need to manage device, host, and inter-chip SRAM explicitly for better and worse.

Healthcare/clinical timing: The "fast and certifiable" requirement is the hard problem most people underestimate. Happy to discuss how the bit-identical replay and deterministic scheduling properties could support the regulatory arguments you'll eventually need.

We won't need them but the hospitals, pharmas, med device companies, CROs, etc with which we work will. Ditto the finserv world and my guess is the money to perform initial certification will come from there as there's a certain versioning policy on Wall St required for adoption which is usually the space wherein certification (and relevant code freeze, which means "can implement new features until" more or less) occurs. Thankfully my job is to break into and harden or design/implement systems - far more intelligent people on the team deal with regulatory 😄

[williamofai]

@sempervictus

Got your email - replied there re: access. Scotland-based, so export compliance should be straightforward.
On the memory model:

"its not hierarchical" in that we need to manage device, host, and inter-chip SRAM explicitly

This is actually ideal. The certifiable-inference architecture assumes explicit buffer placement - caller provides memory, no hidden allocations, no runtime decisions about where data lives. A non-hierarchical model where we control placement directly maps cleanly to the static allocation patterns needed for WCET guarantees.

The "worse" part of explicit management becomes a feature when certification requires you to prove exactly where every byte lives and for how long.

Next steps:
Happy to start with a single VFIO-passed card. The initial work would be:

Verify the integer primitives behave identically on Tenstorrent silicon
Map the static buffer model to device/host/SRAM placement
Run the existing test suite and confirm bit-identical outputs

Once that's solid, we can look at what a full Tenstorrent backend would involve.

Looking forward to getting hands on the hardware.

[brixen]

Not directly related to Tenstorrent, but here is a PR to get the build working on vanilla macOS aarch64.

If you'd like, I can also send a PR to get GitHub Actions set up for building on Ubuntu and macOS.