DSF point-pool quantisation collapses near-coincident mesh vertices into degenerate triangles (holes in terrain, suspected cause of X-Plane ret_wind_acf NaN CTDs)

[Gummow]

Please see:

https://forums.x-plane.org/forums/topic/348168-macibm_alert-value-is-nan-or-inf-in-fm-triggered-by-ortho4xp-scenery/

https://forums.x-plane.org/forums/topic/339881-haab-ctd-100-reproducible-in-any-aircraft-1231-enginefmfm_central_mathfm/

I've been dealing with these NaN errors for 6+ months and finally just stuck Fable on it. Been using the Shred86 fork since it was released. I have not yet recompiled this myself to check to see if it fixes it, but I'm confident Fable knows the cause because I gave it the mesh data for Addis Ababa and nothing more and asked it to pinpoint for me, using OpenStreetMap, exactly where on the airport it would expect this NaN crash to occur if it did occur, and it nailed the exact spot.

DSF point-pool quantisation collapses near-coincident mesh vertices into degenerate triangles (holes in terrain, suspected cause of X-Plane ret_wind_acf NaN CTDs)

Summary

Ortho4XP writes mesh vertices into DSF point pools at a fixed initial quadtree level of 3, which gives a horizontal coordinate resolution of ~21 cm at the equator. Triangle4XP routinely produces valid triangles whose height is smaller than that. When such a triangle is written to DSF, its three vertices quantise onto collinear (or reordered) integer coordinates, and the triangle becomes zero-area or inverted.

X-Plane detects and discards these on load:

zOrtho4XP_-02+030 (1176534 tris, 11 skipped for -1.6 m^2)
zOrtho4XP_+08+038 (...  tris, 10 skipped ...)

Every Ortho4XP tile I have examined exhibits this; stock Global Scenery tiles do not. Each discarded triangle leaves a small hole in the terrain mesh.

Separately, there is a long-standing and hard-to-reproduce X-Plane 12 CTD that only occurs over Ortho4XP scenery:

E/ACF: Flight model error for aircraft: #0
.../fm_central_math/fm_wind_acf.cpp, line 729, function ret_wind_acf
The alert source is: vx_wrl
The alert note is: value is nan or inf!

I am not able to prove the two are connected, but the positional correlation is strong enough that I think it is worth reporting together — see Evidence.

Affected version: v1.40.13 (shred86 fork), Windows 11, packaged build. Also reported against: oscarpilote v1.40-beta, per this thread — the CTD reproduces on both forks, with and without overlays, and in a plugin-free X-Plane instance. X-Plane: 12.4.3-r2 (also reported on 12.4.2-r2).

---

Root cause

src/O4_DSF_Utils.py:

python

quad_init_level = 3                                    # line 21
quad_capacity_high = 50000
quad_capacity_low = 35000

python

pool_quadtree = QuadTree(quad_init_level, quad_capacity)      # line 505
for i in range(nbr_nodes):
    pool_quadtree.insert(
            float2qquad(node_coords[5 * i + 0] - tile.lon),
            float2qquad(node_coords[5 * i + 1] - tile.lat),
            quad_init_level)

python

node_icoords[...] = int(pool_quadtree.nodes[idx_node][0][level : level + 16], 2)   # 16-bit slice
...
scal_x = scal_y = 2 ** (-level)                               # line 558

Pool coordinates are uint16 with scal = 2**(-level), so the horizontal step is:

step = 2**(-level) / 65536  degrees

level | step (deg) | step at equator -- | -- | -- 3 | 1.907e-06 | 21.2 cm 4 | 9.537e-07 | 10.6 cm 5 | 4.768e-07 | 5.3 cm 6 | 2.384e-07 | 2.7 cm

Evidence

HRYR Kigali — tile -02+030

Two zero-area triangles at −1.9661669, 30.1244157, 69 m in from the runway 10 threshold, 16.5 m off the centreline (i.e. exactly on the runway rectangle edge, since width = 30 + 3476//1000 = 33 m for this untagged linear runway).

In the .mesh file, before DSF conversion, they are valid:

CentroidDist  AreaM2    ShortEdgeM  LongEdgeM    A       B       C
        0.7   0.048336      0.2239       1.05   207  148178  148212
        0.1   0.048336      0.2239       0.85  148212  126535     207

Heights are 2 * area / longest_edge = 9.2 cm and 11.4 cm — below the 21.2 cm pool step, hence collapse.

Vertex coordinates:

    Id  Lon                 Lat
     3  30.124464199999998  -1.965873700000000
   207  30.124413199999999  -1.966165700000000    <- vector-layer constraint point
126535  30.124419097000001  -1.966166730000000
148178  30.124403838210938  -1.966166730000000    <- Triangle4XP refinement points
148212  30.124411467605469  -1.966166730000000

Note that 126535, 148034, 148121, 148178 and 148212 all share latitude −1.96616673 exactly. That is an exact Web-Mercator tile boundary (integer y at ZL16 through ZL19), i.e. an orthophoto grid line. Vertex 207 — a runway-polygon point rounded to 7 decimal places — sits 1.03e-06° (≈11 cm) south of it.

So the immediate trigger here is the interaction of two things:

1. vector geometry rounded to 1e-7° (numpy.round(..., 7)), and
2. an unrounded orthophoto grid line,

producing a pair of constraints ~11 cm apart that the DSF writer cannot represent.

HAAB Addis Ababa — tile +08+038

Ten defective triangles in the tile; eight are scattered in open country, and exactly two are on the airport, 0.49 m apart:

AreaM2       Lat        Lon
     0  8.9767561  38.7817369
     0  8.9767605  38.7817369

Nearest constraints:

  4.8 m   way -5    aeroway=apron, name="Cargo Apron"
 18.2 m   way -122  aeroway=taxiway, ref=A1
 51.8 m   way -103  aeroway=parking_position, ref=C5

These are not on an orthophoto grid line — here it is the apron outline against the taxiway outline. The failure mode is therefore general to any pair of near-coincident constraints, not specific to the grid.

Correlation with the CTD

The user reports an intermittent CTD at both airports, always at the same physical spot at each, while taxiing. At Addis, the location was predicted from the DSF analysis before the user disclosed it, and matched.

Working hypothesis: a discarded triangle leaves a hole in the terrain mesh; a terrain-height query over that hole returns a non-finite value; that propagates into X-Plane's wind computation (ret_wind_acf / vx_wrl). Intermittency would then depend on exact taxi path and on whether wind conditions cause the terrain-influenced term to be evaluated. Unverified — it needs someone who can instrument X-Plane's side.

---

Reproduction

1. Build tile -2 / +30 (HRYR) with defaults: road_level=1, mesh_zl=19, curvature_tol=2.0, water_tech=XP11+bathy, no custom_dem.
2. Load the tile in X-Plane and check Log.txt:

   zOrtho4XP_-02+030 (1176534 tris, 11 skipped for -1.6 m^2)

3. Or check offline — convert the DSF and compute signed areas from the quantised coordinates:

   DSFTool --dsf2text -02+030.dsf out.txt

Parse PATCH_VERTEX triples inside BEGIN_PRIMITIVE 0 blocks; any triangle whose signed area is zero or of opposite sign to the majority is one X-Plane will discard. A PowerShell implementation is at the end of this report.

Reproduced identically across rebuilds (11 and 13 discards on two runs of the same tile), so it is deterministic given the input data, not a transient.

---

Suggested fixes

1. Raise quad_init_level (one line, immediate mitigation)

python

quad_init_level = 5     # ~5.3 cm step instead of ~21.2 cm

Costs more pools per tile (8×8 → 32×32 minimum) and a somewhat larger DSF. Would have prevented both cases documented here (9 cm and 11 cm heights). Does not fix the general problem, only moves the threshold down.

2. Subdivide pools on proximity, not just capacity (proper fix)

QuadTree.insert currently splits only on quad_capacity. Splitting also when the minimum pairwise separation within a bucket falls below the bucket's own step size would guarantee representability regardless of level, at the cost of deeper trees only where actually needed.

3. Drop degenerate triangles at write time (defensive)

After computing node_icoords, test each triangle's signed area in integer pool coordinates and skip any that are zero or inverted. X-Plane discards them anyway; omitting them at the source avoids shipping known-bad geometry and would let Ortho4XP report the count during the build.

4. Optional: snap vector constraints and orthogrid to a common resolution

The Kigali case arises partly because vector geometry is rounded to 1e-7° while orthogrid lines are not. Snapping both to the same grid before triangulation would prevent this specific class of near-coincidence, and would also reduce the count of near-degenerate triangles Triangle4XP has to work around (this tile reports Smallest area 1.7881e-16, Largest aspect ratio 1.9531e+06, Largest angle 178.79°).

Fixes 1 and 3 are independent and could ship together.

---

Appendix — verification script

Reports the triangles X-Plane will discard, with coordinates. Read-only; requires DSFTool --dsf2text output.

powershell

$sr = [System.IO.StreamReader]::new('out.txt')
$mLat = 110574.0
$mLon = 111320.0 * [math]::Cos(<TILE_LAT_DEG> * [math]::PI / 180.0)
$bad = New-Object System.Collections.ArrayList
$buf = New-Object 'double[]' 6
$k = 0; $inPrim = $false; $pos = 0; $neg = 0; $zero = 0
while ($null -ne ($line = $sr.ReadLine())) {
    if ($line.StartsWith('PATCH_VERTEX')) {
        if (-not $inPrim) { continue }
        $f = $line.Split(' ', [StringSplitOptions]::RemoveEmptyEntries)
        $buf[$k*2] = [double]$f[1]; $buf[$k*2+1] = [double]$f[2]; $k++
        if ($k -eq 3) {
            $a = 0.5 * (($buf[2]-$buf[0])*($buf[5]-$buf[1]) - ($buf[4]-$buf[0])*($buf[3]-$buf[1])) * $mLon * $mLat
            if ($a -gt 0) { $pos++ } elseif ($a -lt 0) { $neg++ } else { $zero++ }
            if ($a -ge 0) {
                [void]$bad.Add([pscustomobject]@{
                    AreaM2 = [math]::Round($a,4)
                    Lat = [math]::Round((($buf[1]+$buf[3]+$buf[5])/3),7)
                    Lon = [math]::Round((($buf[0]+$buf[2]+$buf[4])/3),7)
                })
            }
            $k = 0
        }
    }
    elseif ($line.StartsWith('BEGIN_PRIMITIVE')) {
        $inPrim = ([int]($line.Split(' ',[StringSplitOptions]::RemoveEmptyEntries)[1]) -eq 0); $k = 0
    }
    elseif ($line.StartsWith('END_PRIMITIVE')) { $inPrim = $false; $k = 0 }
}
$sr.Close()
"positive / zero / negative : $pos / $zero / $neg"
$bad | Format-Table -AutoSize

Ortho4XP winds its triangles clockwise in this coordinate convention, so the large count is negative and the defects are the positive and zero ones. Flip the comparison if a future change reverses the winding.

---

Things checked and excluded

• Elevation data. Not implicated. The HRYR tile reports Min altitude: 1167.0, Max altitude: 2375.0, Mean: 1494.5686, correct for Rwanda. Voids in the viewfinderpanoramas source were filled successfully. No NaN reaches the mesh. (Worth noting separately that O4_DEM_Utils.read_elevation_from_file tests only alt_dem == nodata and never numpy.isnan, so a source DEM containing NaN would pass through silently — not the cause here, but a latent hazard.)
• OSM data errors. None. At Kigali the runway and the turnaround taxiway correctly share nodes, which is standard OSM practice for routable aeroway features. Deleting the taxiway from the cached layer did not remove the defect (discards went 11 → 13).
• Roads. The big_roads layer has no geometry within 300 m of the Kigali defect.
• Plugins. The CTD reproduces in a plugin-free X-Plane instance per the linked forum report.

DSF point-pool quantisation collapses near-coincident mesh vertices into degenerate triangles (holes in terrain, suspected cause of X-Plane ret_wind_acf NaN CTDs) Summary

Ortho4XP writes mesh vertices into DSF point pools at a fixed initial quadtree level of 3, which gives a horizontal coordinate resolution of ~21 cm at the equator. Triangle4XP routinely produces valid triangles whose height is smaller than that. When such a triangle is written to DSF, its three vertices quantise onto collinear (or reordered) integer coordinates, and the triangle becomes zero-area or inverted.

X-Plane detects and discards these on load:

zOrtho4XP_-02+030 (1176534 tris, 11 skipped for -1.6 m^2)
zOrtho4XP_+08+038 (... tris, 10 skipped ...)

Every Ortho4XP tile I have examined exhibits this; stock Global Scenery tiles do not. Each discarded triangle leaves a small hole in the terrain mesh.

Separately, there is a long-standing and hard-to-reproduce X-Plane 12 CTD that only occurs over Ortho4XP scenery:

E/ACF: Flight model error for aircraft: #0
.../fm_central_math/fm_wind_acf.cpp, line 729, function ret_wind_acf
The alert source is: vx_wrl
The alert note is: value is nan or inf!

I am not able to prove the two are connected, but the positional correlation is strong enough that I think it is worth reporting together — see Evidence.

Affected version: v1.40.13 (shred86 fork), Windows 11, packaged build. Also reported against: oscarpilote v1.40-beta, per this thread — the CTD reproduces on both forks, with and without overlays, and in a plugin-free X-Plane instance. X-Plane: 12.4.3-r2 (also reported on 12.4.2-r2).
Root cause

src/O4_DSF_Utils.py:
python

quad_init_level = 3 # line 21
quad_capacity_high = 50000
quad_capacity_low = 35000

python

pool_quadtree = QuadTree(quad_init_level, quad_capacity) # line 505
for i in range(nbr_nodes):
pool_quadtree.insert(
float2qquad(node_coords[5 * i + 0] - tile.lon),
float2qquad(node_coords[5 * i + 1] - tile.lat),
quad_init_level)

python

node_icoords[...] = int(pool_quadtree.nodes[idx_node][0][level : level + 16], 2) # 16-bit slice
...
scal_x = scal_y = 2 ** (-level) # line 558

Pool coordinates are uint16 with scal = 2**(-level), so the horizontal step is:

step = 2**(-level) / 65536 degrees

level step (deg) step at equator
3 1.907e-06 21.2 cm
4 9.537e-07 10.6 cm
5 4.768e-07 5.3 cm
6 2.384e-07 2.7 cm

The quadtree does subdivide beyond the initial level, but only when a bucket exceeds quad_capacity (35–50k nodes). Subdivision is therefore driven purely by node count, never by node proximity. A pool containing a handful of nodes two centimetres apart is never split, so those nodes are emitted at 21 cm resolution and collapse.

This is confirmed directly in the DSF text output — all longitudes in a level-3 pool are exact multiples of 2⁻¹⁹ degrees:

PATCH_VERTEX 30.736797131 -1.527294575 ...
PATCH_VERTEX 30.736879149 -1.526918822 ...
PATCH_VERTEX 30.737001221 -1.526918822 ...
PATCH_VERTEX 30.737123293 -1.526918822 ...

Why airports, and why some airports

Constraint-forced vertices cluster wherever two hard constraints meet at a shallow angle. Airport pavement is the worst case because O4_Airport_Utils inserts runway, taxiway and apron outlines as hard constraints, and because runways aligned near east–west run nearly parallel to the orthophoto grid lines inserted at mesh_zl, so the two cross at a very acute angle and generate long, thin triangles.

Both airports where I reproduced this have shallow-angle runways:
Airport Runway True bearing Offset from E–W
HRYR Kigali 10/28 ~100° 10°
HAAB Addis Ababa Bole 07L/25R, 07R/25L 73.6° 16°
Evidence
HRYR Kigali — tile -02+030

Two zero-area triangles at −1.9661669, 30.1244157, 69 m in from the runway 10 threshold, 16.5 m off the centreline (i.e. exactly on the runway rectangle edge, since width = 30 + 3476//1000 = 33 m for this untagged linear runway).

In the .mesh file, before DSF conversion, they are valid:

CentroidDist AreaM2 ShortEdgeM LongEdgeM A B C
0.7 0.048336 0.2239 1.05 207 148178 148212
0.1 0.048336 0.2239 0.85 148212 126535 207

Heights are 2 * area / longest_edge = 9.2 cm and 11.4 cm — below the 21.2 cm pool step, hence collapse.

Vertex coordinates:

Id  Lon                 Lat
 3  30.124464199999998  -1.965873700000000

207 30.124413199999999 -1.966165700000000 <- vector-layer constraint point
126535 30.124419097000001 -1.966166730000000
148178 30.124403838210938 -1.966166730000000 <- Triangle4XP refinement points
148212 30.124411467605469 -1.966166730000000

Note that 126535, 148034, 148121, 148178 and 148212 all share latitude −1.96616673 exactly. That is an exact Web-Mercator tile boundary (integer y at ZL16 through ZL19), i.e. an orthophoto grid line. Vertex 207 — a runway-polygon point rounded to 7 decimal places — sits 1.03e-06° (≈11 cm) south of it.

So the immediate trigger here is the interaction of two things:

vector geometry rounded to 1e-7° (numpy.round(..., 7)), and
an unrounded orthophoto grid line,

producing a pair of constraints ~11 cm apart that the DSF writer cannot represent.
HAAB Addis Ababa — tile +08+038

Ten defective triangles in the tile; eight are scattered in open country, and exactly two are on the airport, 0.49 m apart:

AreaM2 Lat Lon
0 8.9767561 38.7817369
0 8.9767605 38.7817369

Nearest constraints:

4.8 m way -5 aeroway=apron, name="Cargo Apron"
18.2 m way -122 aeroway=taxiway, ref=A1
51.8 m way -103 aeroway=parking_position, ref=C5

These are not on an orthophoto grid line — here it is the apron outline against the taxiway outline. The failure mode is therefore general to any pair of near-coincident constraints, not specific to the grid.
Correlation with the CTD

The user reports an intermittent CTD at both airports, always at the same physical spot at each, while taxiing. At Addis, the location was predicted from the DSF analysis before the user disclosed it, and matched.

Working hypothesis: a discarded triangle leaves a hole in the terrain mesh; a terrain-height query over that hole returns a non-finite value; that propagates into X-Plane's wind computation (ret_wind_acf / vx_wrl). Intermittency would then depend on exact taxi path and on whether wind conditions cause the terrain-influenced term to be evaluated. Unverified — it needs someone who can instrument X-Plane's side.
Reproduction

Build tile -2 / +30 (HRYR) with defaults: road_level=1, mesh_zl=19, curvature_tol=2.0, water_tech=XP11+bathy, no custom_dem.
Load the tile in X-Plane and check Log.txt:

zOrtho4XP_-02+030 (1176534 tris, 11 skipped for -1.6 m^2)

Or check offline — convert the DSF and compute signed areas from the quantised coordinates:

DSFTool --dsf2text -02+030.dsf out.txt

Parse PATCH_VERTEX triples inside BEGIN_PRIMITIVE 0 blocks; any triangle whose signed area is zero or of opposite sign to the majority is one X-Plane will discard. A PowerShell implementation is at the end of this report.

Reproduced identically across rebuilds (11 and 13 discards on two runs of the same tile), so it is deterministic given the input data, not a transient.
Suggested fixes

1. Raise quad_init_level (one line, immediate mitigation)
   python

quad_init_level = 5 # ~5.3 cm step instead of ~21.2 cm

Costs more pools per tile (8×8 → 32×32 minimum) and a somewhat larger DSF. Would have prevented both cases documented here (9 cm and 11 cm heights). Does not fix the general problem, only moves the threshold down.
2. Subdivide pools on proximity, not just capacity (proper fix)

QuadTree.insert currently splits only on quad_capacity. Splitting also when the minimum pairwise separation within a bucket falls below the bucket's own step size would guarantee representability regardless of level, at the cost of deeper trees only where actually needed.
3. Drop degenerate triangles at write time (defensive)

After computing node_icoords, test each triangle's signed area in integer pool coordinates and skip any that are zero or inverted. X-Plane discards them anyway; omitting them at the source avoids shipping known-bad geometry and would let Ortho4XP report the count during the build.
4. Optional: snap vector constraints and orthogrid to a common resolution

The Kigali case arises partly because vector geometry is rounded to 1e-7° while orthogrid lines are not. Snapping both to the same grid before triangulation would prevent this specific class of near-coincidence, and would also reduce the count of near-degenerate triangles Triangle4XP has to work around (this tile reports Smallest area 1.7881e-16, Largest aspect ratio 1.9531e+06, Largest angle 178.79°).

Fixes 1 and 3 are independent and could ship together.
Appendix — verification script

Reports the triangles X-Plane will discard, with coordinates. Read-only; requires DSFTool --dsf2text output.
powershell

$sr = [System.IO.StreamReader]::new('out.txt')
$mLat = 110574.0
$mLon = 111320.0 * [math]::Cos(<TILE_LAT_DEG> * [math]::PI / 180.0)
$bad = New-Object System.Collections.ArrayList
$buf = New-Object 'double[]' 6
$k = 0; $inPrim = $false; $pos = 0; $neg = 0; $zero = 0
while ($null -ne ($line = $sr.ReadLine())) {
if ($line.StartsWith('PATCH_VERTEX')) {
if (-not $inPrim) { continue }
$f = $line.Split(' ', [StringSplitOptions]::RemoveEmptyEntries)
$buf[$k2] = [double]$f[1]; $buf[$k2+1] = [double]$f[2]; $k++
if ($k -eq 3) {
$a = 0.5 * (($buf[2]-$buf[0])($buf[5]-$buf[1]) - ($buf[4]-$buf[0])($buf[3]-$buf[1])) * $mLon * $mLat
if ($a -gt 0) { $pos++ } elseif ($a -lt 0) { $neg++ } else { $zero++ }
if ($a -ge 0) {
[void]$bad.Add([pscustomobject]@{
AreaM2 = [math]::Round($a,4)
Lat = [math]::Round((($buf[1]+$buf[3]+$buf[5])/3),7)
Lon = [math]::Round((($buf[0]+$buf[2]+$buf[4])/3),7)
})
}
$k = 0
}
}
elseif ($line.StartsWith('BEGIN_PRIMITIVE')) {
$inPrim = ([int]($line.Split(' ',[StringSplitOptions]::RemoveEmptyEntries)[1]) -eq 0); $k = 0
}
elseif ($line.StartsWith('END_PRIMITIVE')) { $inPrim = $false; $k = 0 }
}
$sr.Close()
"positive / zero / negative : $pos / $zero / $neg"
$bad | Format-Table -AutoSize

Ortho4XP winds its triangles clockwise in this coordinate convention, so the large count is negative and the defects are the positive and zero ones. Flip the comparison if a future change reverses the winding.
Things checked and excluded

Elevation data. Not implicated. The HRYR tile reports Min altitude: 1167.0, Max altitude: 2375.0, Mean: 1494.5686, correct for Rwanda. Voids in the viewfinderpanoramas source were filled successfully. No NaN reaches the mesh. (Worth noting separately that O4_DEM_Utils.read_elevation_from_file tests only alt_dem == nodata and never numpy.isnan, so a source DEM containing NaN would pass through silently — not the cause here, but a latent hazard.)
OSM data errors. None. At Kigali the runway and the turnaround taxiway correctly share nodes, which is standard OSM practice for routable aeroway features. Deleting the taxiway from the cached layer did not remove the defect (discards went 11 → 13).
Roads. The big_roads layer has no geometry within 300 m of the Kigali defect.
Plugins. The CTD reproduces in a plugin-free X-Plane instance per the linked forum report.