Investigate and, where appropriate, remove `#[expect(clippy::cast_precision_loss)]` instances

[arschles]

In some of the places we use this #[expect...], it may be important to not do casts in some cases. Find these cases and remove the unchecked casts.