Scalability Issues

[duansy3]

Scalability Issues
Although predictive search takes almost no time, but i found compute_hint costs some times that can't be ignored,which is necessary for reoderedArray.
During inferrence run time, the size of M varies, and generating a JSON file includding hint list for each size online is unrealistic.
Is it feasible to modify the code to be like this: to select a offline generated solution that is closest to the run time M ?
I'm not sure whether this kind of approximation will significantly affect the overlap effect.

[hkeee21]

Thanks for raising this issue.

1. I'm working on replacing the time-consuming compute_hint step by direct tile order calculation under block swizzling. Some code is updated in the ae branch.

2. I agree on the pre-search method for irregular M dimensions. Assuming a BM (tile size for the M dimension) of 128, the tile execution order with irregular M sizes should remain consistent with that of padding to 128. However, since the tail tile involves less workload, sampling is still necessary to accurately capture the behavior of irregular M sizes. But I don't think the tail effect is influential when M is large. Of course, all of those require experimental verification.