Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions ptodsl/lib/a5/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ Entry points:
- [`kernels.py`](./kernels.py): translated example kernels, including the
no-section `build_hivm_vadd_demo()` flow that lowers through PTOAS VPTO into
`llvm.hivm.*` intrinsics
- [`tile_op_kernels.py`](./tile_op_kernels.py): one builder per public tile op
that already has a micro-backed implementation
- [`generated`](./generated): emitted `.pto` artifacts from `scripts/generate_a5_pto.py`

Recommended usage:
Expand All @@ -56,6 +58,14 @@ PYTHONPATH=/Users/zhoubot/github/.llvm-19.1.7/build-mlir-py312/tools/mlir/python
/Users/zhoubot/github/.venv-ptoas-src312/bin/python scripts/generate_a5_pto.py --emit-hivm-llvm
```

To emit one `.pto` file per public tile op that already has a micro-backed
implementation:

```bash
PYTHONPATH=/Users/zhoubot/github/.llvm-19.1.7/build-mlir-py312/tools/mlir/python_packages/mlir_core:/Users/zhoubot/github/pto-org/PTOAS/install-src312:/Users/zhoubot/github/pto-org/PTOAS/build-src312/python \
/Users/zhoubot/github/.venv-ptoas-src312/bin/python scripts/generate_tile_op_pto.py
```

`--emit-cpp` and `--emit-hivm-llvm` are intentionally asymmetric:
- pure micro kernels such as `a5_hivm_vadd_demo` now lower end-to-end through
PTOAS VPTO into `llvm.hivm.*`
Expand Down
8 changes: 8 additions & 0 deletions ptodsl/lib/a5/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@
build_templated_elementwise_add,
build_vector_copy,
)
from .tile_op_kernels import (
TILE_OP_KERNEL_BUILDERS,
TILE_OP_KERNEL_SPECS,
tile_op_generation_index_markdown,
)
from .ops import *
from .tile_micro_coverage import (
TILE_MICRO_COVERAGE,
Expand All @@ -21,6 +26,8 @@
"A5_HEADER_COVERAGE",
"HIVM_LLVM_KERNELS",
"KERNEL_BUILDERS",
"TILE_OP_KERNEL_BUILDERS",
"TILE_OP_KERNEL_SPECS",
"TILE_MICRO_COVERAGE",
"a5_header_coverage_markdown",
"build_cube_matmul",
Expand All @@ -37,5 +44,6 @@
"treduce",
"tscalar",
"tsort",
"tile_op_generation_index_markdown",
"tunary",
]
38 changes: 38 additions & 0 deletions ptodsl/lib/a5/generated/tile_ops/TILE_OP_GENERATION_INDEX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Tile Op PTO Generation

| tile op | status | artifact | note |
| --- | --- | --- | --- |
| `mov` | `generated` | `tile_ops/mov.pto` | UB stage + vlds/vsts copy loop. |
| `add` | `generated` | `tile_ops/add.pto` | UB stage + constexpr-specialized TBinOp-style vlds/vadd/vsts lowering. |
| `sub` | `generated` | `tile_ops/sub.pto` | UB stage + constexpr-specialized TBinOp-style vlds/vsub/vsts lowering. |
| `div` | `generated` | `tile_ops/div.pto` | UB stage + constexpr-specialized TBinOp-style vlds/vdiv/vsts lowering. |
| `mul` | `generated` | `tile_ops/mul.pto` | UB stage + constexpr-specialized TBinOp-style vlds/vmul/vsts lowering. |
| `or_` | `generated` | `tile_ops/or_.pto` | UB stage + constexpr-specialized TBinOp-style vlds/vor/vsts lowering. |
| `gather` | `generated` | `tile_ops/gather.pto` | Indexed gather is implemented via vgather2 for same-width source/index pairs; mask-pattern gather still needs unsupported vsqz-style micro support. |
| `exp` | `generated` | `tile_ops/exp.pto` | UB stage + vlds/vexp/vsts loop. |
| `log` | `generated` | `tile_ops/log.pto` | UB stage + vlds/vln/vsts loop. |
| `relu` | `generated` | `tile_ops/relu.pto` | UB stage + vlds/vrelu/vsts loop. |
| `abs` | `generated` | `tile_ops/abs.pto` | UB stage + vlds/vabs/vsts loop. |
| `sqrt` | `generated` | `tile_ops/sqrt.pto` | UB stage + vlds/vsqrt/vsts loop. |
| `rsqrt` | `generated` | `tile_ops/rsqrt.pto` | UB stage + vsqrt/vrec sequence. |
| `reciprocal` | `generated` | `tile_ops/reciprocal.pto` | UB stage + vlds/vrec/vsts loop. |
| `matmul` | `blocked` | - | Cube/L0 path is not a pure vector-micro rewrite target. |
| `matmul_bias` | `blocked` | - | Cube/L0 path is not a pure vector-micro rewrite target. |
| `matmul_acc` | `blocked` | - | Cube/L0 path is not a pure vector-micro rewrite target. |
| `extract` | `blocked` | - | Layout/L0 extraction op, not a vector-micro compute rewrite. |
| `row_sum` | `generated` | `tile_ops/row_sum.pto` | Static-shape row reduction via vcadd + point-store. |
| `row_min` | `generated` | `tile_ops/row_min.pto` | Static-shape row reduction via vcmin + point-store. |
| `row_max` | `generated` | `tile_ops/row_max.pto` | Static-shape row reduction via vcmax + point-store. |
| `row_prod` | `blocked` | - | No row-product micro lowering is wired yet. |
| `row_expand` | `generated` | `tile_ops/row_expand.pto` | Static-shape canonical broadcast via vldas/vldus/vdup/vsts. |
| `row_expand_sub` | `generated` | `tile_ops/row_expand_sub.pto` | Static-shape canonical broadcast via vldas/vldus/vdup/vsub/vsts. |
| `row_expand_div` | `generated` | `tile_ops/row_expand_div.pto` | Static-shape canonical broadcast via vldas/vldus/vdup/vdiv/vsts. |
| `row_expand_mul` | `generated` | `tile_ops/row_expand_mul.pto` | Static-shape canonical broadcast via vldas/vldus/vdup/vmul/vsts. |
| `col_sum` | `generated` | `tile_ops/col_sum.pto` | Static-shape TColReduceOps-style column reduction via vadd. |
| `col_min` | `generated` | `tile_ops/col_min.pto` | Static-shape TColReduceOps-style column reduction via vmin. |
| `col_max` | `generated` | `tile_ops/col_max.pto` | Static-shape TColReduceOps-style column reduction via vmax. |
| `col_prod` | `blocked` | - | No column-product micro lowering is wired yet. |
| `col_expand` | `generated` | `tile_ops/col_expand.pto` | Static-shape canonical broadcast via vlds/vsts replication. |
| `mrgsort` | `generated` | `tile_ops/mrgsort.pto` | Single-list row-major merge sort via vmrgsort4. |
| `sort32` | `generated` | `tile_ops/sort32.pto` | Static-shape block sort via vbitsort. |
| `subset` | `not_applicable` | - | View helper only, not a tile compute op. |
75 changes: 75 additions & 0 deletions ptodsl/lib/a5/generated/tile_ops/abs.pto
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
module {
func.func @tile_op_abs(%arg0: !pto.ptr<f32, gm>, %arg1: !pto.ptr<f32, gm>) {
%c64 = arith.constant 64 : index
%c1 = arith.constant 1 : index
%c8 = arith.constant 8 : index
%c64_0 = arith.constant 64 : index
%0 = pto.make_tensor_view %arg0, shape = [%c8, %c64_0], strides = [%c64, %c1] : !pto.tensor_view<?x?xf32>
%c64_1 = arith.constant 64 : index
%c1_2 = arith.constant 1 : index
%c8_3 = arith.constant 8 : index
%c64_4 = arith.constant 64 : index
%1 = pto.make_tensor_view %arg1, shape = [%c8_3, %c64_4], strides = [%c64_1, %c1_2] : !pto.tensor_view<?x?xf32>
pto.section.vector {
%c0 = arith.constant 0 : index
%c0_5 = arith.constant 0 : index
%c8_6 = arith.constant 8 : index
%c64_7 = arith.constant 64 : index
%2 = pto.partition_view %0, offsets = [%c0, %c0_5], sizes = [%c8_6, %c64_7] : !pto.tensor_view<?x?xf32> -> !pto.partition_tensor_view<8x64xf32>
%c0_8 = arith.constant 0 : index
%c0_9 = arith.constant 0 : index
%c8_10 = arith.constant 8 : index
%c64_11 = arith.constant 64 : index
%3 = pto.partition_view %1, offsets = [%c0_8, %c0_9], sizes = [%c8_10, %c64_11] : !pto.tensor_view<?x?xf32> -> !pto.partition_tensor_view<8x64xf32>
%c0_i64 = arith.constant 0 : i64
%c2048_i64 = arith.constant 2048 : i64
%4 = pto.alloc_tile addr = %c0_i64 : !pto.tile_buf<vec, 8x64xf32>
%5 = pto.alloc_tile addr = %c2048_i64 : !pto.tile_buf<vec, 8x64xf32>
pto.tload ins(%2 : !pto.partition_tensor_view<8x64xf32>) outs(%4 : !pto.tile_buf<vec, 8x64xf32>)
%6 = pto.castptr %c0_i64 : i64 -> !pto.ptr<f32, ub>
%7 = pto.castptr %c2048_i64 : i64 -> !pto.ptr<f32, ub>
%8 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c0_12 = arith.constant 0 : index
%9 = pto.vlds %6[%c0_12] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%10 = pto.vabs %9, %8 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %10, %7[%c0_12], %8 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%11 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c64_13 = arith.constant 64 : index
%12 = pto.vlds %6[%c64_13] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%13 = pto.vabs %12, %11 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %13, %7[%c64_13], %11 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%14 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c128 = arith.constant 128 : index
%15 = pto.vlds %6[%c128] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%16 = pto.vabs %15, %14 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %16, %7[%c128], %14 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%17 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c192 = arith.constant 192 : index
%18 = pto.vlds %6[%c192] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%19 = pto.vabs %18, %17 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %19, %7[%c192], %17 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%20 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c256 = arith.constant 256 : index
%21 = pto.vlds %6[%c256] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%22 = pto.vabs %21, %20 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %22, %7[%c256], %20 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%23 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c320 = arith.constant 320 : index
%24 = pto.vlds %6[%c320] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%25 = pto.vabs %24, %23 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %25, %7[%c320], %23 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%26 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c384 = arith.constant 384 : index
%27 = pto.vlds %6[%c384] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%28 = pto.vabs %27, %26 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %28, %7[%c384], %26 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
%29 = pto.pset_b32 "PAT_ALL" : !pto.mask
%c448 = arith.constant 448 : index
%30 = pto.vlds %6[%c448] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>
%31 = pto.vabs %30, %29 : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
pto.vsts %31, %7[%c448], %29 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask
pto.tstore ins(%5 : !pto.tile_buf<vec, 8x64xf32>) outs(%3 : !pto.partition_tensor_view<8x64xf32>)
}
return
}
}
90 changes: 90 additions & 0 deletions ptodsl/lib/a5/generated/tile_ops/add.pto
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
module {
func.func @tile_op_add(%arg0: !pto.ptr<f32, gm>, %arg1: !pto.ptr<f32, gm>, %arg2: !pto.ptr<f32, gm>) {
%c64 = arith.constant 64 : index
%c1 = arith.constant 1 : index
%c8 = arith.constant 8 : index
%c64_0 = arith.constant 64 : index
%0 = pto.make_tensor_view %arg0, shape = [%c8, %c64_0], strides = [%c64, %c1] : !pto.tensor_view<?x?xf32>
%c64_1 = arith.constant 64 : index
%c1_2 = arith.constant 1 : index
%c8_3 = arith.constant 8 : index
%c64_4 = arith.constant 64 : index
%1 = pto.make_tensor_view %arg1, shape = [%c8_3, %c64_4], strides = [%c64_1, %c1_2] : !pto.tensor_view<?x?xf32>
%c64_5 = arith.constant 64 : index
%c1_6 = arith.constant 1 : index
%c8_7 = arith.constant 8 : index
%c64_8 = arith.constant 64 : index
%2 = pto.make_tensor_view %arg2, shape = [%c8_7, %c64_8], strides = [%c64_5, %c1_6] : !pto.tensor_view<?x?xf32>
pto.section.vector {
%c0 = arith.constant 0 : index
%c0_9 = arith.constant 0 : index
%c8_10 = arith.constant 8 : index
%c64_11 = arith.constant 64 : index
%3 = pto.partition_view %0, offsets = [%c0, %c0_9], sizes = [%c8_10, %c64_11] : !pto.tensor_view<?x?xf32> -> !pto.partition_tensor_view<8x64xf32>
%c0_12 = arith.constant 0 : index
%c0_13 = arith.constant 0 : index
%c8_14 = arith.constant 8 : index
%c64_15 = arith.constant 64 : index
%4 = pto.partition_view %1, offsets = [%c0_12, %c0_13], sizes = [%c8_14, %c64_15] : !pto.tensor_view<?x?xf32> -> !pto.partition_tensor_view<8x64xf32>
%c0_16 = arith.constant 0 : index
%c0_17 = arith.constant 0 : index
%c8_18 = arith.constant 8 : index
%c64_19 = arith.constant 64 : index
%5 = pto.partition_view %2, offsets = [%c0_16, %c0_17], sizes = [%c8_18, %c64_19] : !pto.tensor_view<?x?xf32> -> !pto.partition_tensor_view<8x64xf32>
%c0_i64 = arith.constant 0 : i64
%c2048_i64 = arith.constant 2048 : i64
%c4096_i64 = arith.constant 4096 : i64
%6 = pto.alloc_tile addr = %c0_i64 : !pto.tile_buf<vec, 8x64xf32>
%7 = pto.alloc_tile addr = %c2048_i64 : !pto.tile_buf<vec, 8x64xf32>
%8 = pto.alloc_tile addr = %c4096_i64 : !pto.tile_buf<vec, 8x64xf32>
pto.tload ins(%3 : !pto.partition_tensor_view<8x64xf32>) outs(%6 : !pto.tile_buf<vec, 8x64xf32>)
pto.tload ins(%4 : !pto.partition_tensor_view<8x64xf32>) outs(%7 : !pto.tile_buf<vec, 8x64xf32>)
%9 = pto.castptr %c0_i64 : i64 -> !pto.ptr<f32, ub>
%10 = pto.castptr %c2048_i64 : i64 -> !pto.ptr<f32, ub>
%11 = pto.castptr %c4096_i64 : i64 -> !pto.ptr<f32, ub>
%c64_20 = arith.constant 64 : index
%12 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result, %updated_source = pto.vlds_post %9[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_21, %updated_source_22 = pto.vlds_post %10[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%13 = pto.vadd %result, %result_21, %12 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%14 = pto.vsts_post %13, %11[%c64_20], %12 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%15 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_23, %updated_source_24 = pto.vlds_post %updated_source[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_25, %updated_source_26 = pto.vlds_post %updated_source_22[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%16 = pto.vadd %result_23, %result_25, %15 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%17 = pto.vsts_post %16, %14[%c64_20], %15 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%18 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_27, %updated_source_28 = pto.vlds_post %updated_source_24[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_29, %updated_source_30 = pto.vlds_post %updated_source_26[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%19 = pto.vadd %result_27, %result_29, %18 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%20 = pto.vsts_post %19, %17[%c64_20], %18 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%21 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_31, %updated_source_32 = pto.vlds_post %updated_source_28[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_33, %updated_source_34 = pto.vlds_post %updated_source_30[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%22 = pto.vadd %result_31, %result_33, %21 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%23 = pto.vsts_post %22, %20[%c64_20], %21 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%24 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_35, %updated_source_36 = pto.vlds_post %updated_source_32[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_37, %updated_source_38 = pto.vlds_post %updated_source_34[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%25 = pto.vadd %result_35, %result_37, %24 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%26 = pto.vsts_post %25, %23[%c64_20], %24 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%27 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_39, %updated_source_40 = pto.vlds_post %updated_source_36[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_41, %updated_source_42 = pto.vlds_post %updated_source_38[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%28 = pto.vadd %result_39, %result_41, %27 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%29 = pto.vsts_post %28, %26[%c64_20], %27 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%30 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_43, %updated_source_44 = pto.vlds_post %updated_source_40[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_45, %updated_source_46 = pto.vlds_post %updated_source_42[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%31 = pto.vadd %result_43, %result_45, %30 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%32 = pto.vsts_post %31, %29[%c64_20], %30 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
%33 = pto.pset_b32 "PAT_ALL" : !pto.mask
%result_47, %updated_source_48 = pto.vlds_post %updated_source_44[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%result_49, %updated_source_50 = pto.vlds_post %updated_source_46[%c64_20] : !pto.ptr<f32, ub> -> !pto.vreg<64xf32>, !pto.ptr<f32, ub>
%34 = pto.vadd %result_47, %result_49, %33 : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32>
%35 = pto.vsts_post %34, %32[%c64_20], %33 : !pto.vreg<64xf32>, !pto.ptr<f32, ub>, !pto.mask -> !pto.ptr<f32, ub>
pto.tstore ins(%8 : !pto.tile_buf<vec, 8x64xf32>) outs(%5 : !pto.partition_tensor_view<8x64xf32>)
}
return
}
}
Loading
Loading