Outline flattens rounded glyph tops (s, a, o) at 2px thickness

[vchelaru]

Description

With a 2-pixel outline enabled, rounded letters lose their curved tops — the outline forms a flat horizontal cap at the peak instead of following the arc. Vertical stems (h, d) look correct; the problem is specific to curved tops on glyphs like s, a, o, and n.

This appears to be an outline-shape bug, not normal Arial design.

Steps to reproduce

1. Rasterizer: KernSmith.Rasterizers.FreeType
2. Font: Arial
3. Size: 24
4. Outline thickness: 2
5. Generate text containing rounded glyphs — e.g. "Shadow and" or "ason"
6. Zoom in on s, a, o, n peaks

Expected behavior

The outline should follow the curved top of rounded glyphs, matching the underlying glyph silhouette.

Actual behavior

The outline at the top of rounded glyphs is flattened into a straight horizontal segment several pixels wide. Annotated zoom comparisons show the cap on s and a clearly; o and n show the same pattern.

Screenshots

Screenshots will be added in a follow-up comment (zoomed, with annotations on affected peaks).

Environment

• OS: Windows 10
• Rasterizer: KernSmith.Rasterizers.FreeType
• Reproduced in: KernSmith UI

Notes for investigation

Likely related to OutlineEffect EDT binarization — alpha is thresholded at >= 32 before the distance transform, which can turn a graduated anti-aliased peak into a flat binarized row; the 2px outline then traces that flattened silhouette:

// OutlineEffect.cs
binaryAlpha[i] = expandedAlpha[i] >= 32 ? (byte)255 : (byte)0;

OutlinePostProcessor uses a different path (EDT on raw alpha, no binarization) and may be worth comparing.

LayeredRenderingTests checks outline sizing but does not assert outline shape integrity on curved glyphs.

Possible regression test

Generate ason at size 24 with outline=2 via FreeType; sample pixels just outside the glyph peak on s/o and assert the outline boundary is not a perfectly flat horizontal run (or compare against a golden image of outline-only output).

[vchelaru]

[kaltinril]

Investigated alongside #126. PR #128 fixes #126 but does not address this issue — different root cause.

This issue is the exterior top of rounded glyphs being flattened, which (as the issue notes) traces to the EDT binarization in OutlineEffect:

binaryAlpha[i] = expandedAlpha[i] >= 32 ? (byte)255 : (byte)0;

PR #128 only changed counter (interior) handling and removed FloodFillExterior; the binarization and the exterior-edge path are unchanged, so the flat-top behavior is unchanged (verified by regenerating ason at 24px/outline 2 on the fixed build — rounded tops still flatten). Keeping this open for a focused follow-up that revisits the binarization threshold (or uses the raw-alpha EDT path like OutlinePostProcessor).

[kaltinril]

Candidate fix on branch fix/outline-rounded-tops (commit 63f7a71) — not merged, parked for a decision on approach.

It moves the OutlineEffect binarization to the 50% coverage contour (alpha >= 128, the glyph's true geometric edge) instead of >= 32. That rounds the flat cap on curved tops without bloating near-flat tops like lowercase a (an earlier > 0 attempt over-extended the fringe and added ~2px to the a cap).

Validated: unit tests (red→green on the threshold) + full bmfont-compare regression (only the outlined configs change, zero .fnt metadata changes).

Held because the binary-threshold approach is still a pixel-quantization compromise; the fully correct fix is a sub-pixel / coverage-weighted distance transform. This branch is preserved as the starting point for whichever direction we take.