fix(statistics): align geographic grouping and show missing-parent entries

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## [Unreleased]
 
 ### Fixed
+- Location statistics now share exact ASCII-trimmed, parent-scoped grouping and combine “East Macedonia and Thrace” with “Eastern Macedonia and Thrace” only under “Greece”. Parent scoping can increase counts; the region correction can decrease them. Both Timeline views show missing-parent sections and safely encode labels; tied visits select one deterministic settlement coordinate (#573). See [Timeline statistics](docs/06-Timeline.md#statistics-grouping) for sources and the remaining string-only ambiguity: identical names within identical parents cannot be distinguished, while other alternate labels may still split one entity. Stored values and released-Mobile API shapes remain unchanged; no migration or provider calls are added.
 - Geoapify Location enrichment now stores street then house number, keeps settlement and state at their documented levels, and retains the independent provider line. Location maps, tables, timelines, groups and edit summaries prioritize structured addresses and show nearby feature metadata beneath them (#572).
 - CSV history exports use an explicit-offset enrichment timestamp so valid retained provenance survives backend round trips (#572).
 - Backend history imports preserve internal newlines and tabs in retained provider address lines; GPX/KML round trips normalize line endings to LF (#572).

docs/06-Timeline.md

@@ -70,6 +70,52 @@ Location Search & Filters
 
 ![Timeline Statistics](images/private-timeline-statistics.JPG)
 
+### Statistics grouping
+
+Statistics use recorded Country, Region and Place labels at read time. Only outer
+ASCII space (U+0020) and U+0009–U+000D are trimmed; null and empty results are
+missing. Case, accents, Unicode composition, internal whitespace and punctuation
+remain significant. Countries group by country; regions by country and region;
+settlements by country, region and place. Missing parents are separate from named
+parents and are never inferred. Summary counts equal the corresponding detailed
+group counts; Total Locations includes every record in the selected scope.
+
+The single geographic correction maps **East Macedonia and Thrace** to
+**Eastern Macedonia and Thrace** only under the exact trimmed country **Greece**,
+regardless of provider, manual entry or import origin. Other countries and
+spellings remain unchanged. Sources checked 2026-09-05:
+
+- [European Commission demographic-observatory project](https://reforms-investments.ec.europa.eu/technical-support-instrument-0/labour-market-and-social-protection/supporting-greece-establish-demographic-observatory-through-evidence-based-tools_en)
+- [Region of Eastern Macedonia and Thrace official website](https://www.pamth.gov.gr/en/)
+- [European Commission JRC regional report](https://publications.jrc.ec.europa.eu/repository/handle/JRC100503)
+
+These sources support the English label variation, not the identity of individual
+stored records. Parent scoping may increase visited counts; this region correction
+may decrease them. The existing API contracts, including released Mobile counts,
+are unchanged. Original labels, retained provider address lines, FullAddress and
+feature metadata are not rewritten, and this correction adds no migration or
+provider requests.
+
+Both User Timeline statistics views show children without recorded parents under
+presentation-only **Country not recorded** and **Region not recorded** sections.
+These sections do not add geographic entities or visited counts. Existing map links
+still navigate to averaged country/region coordinates or one settlement visit.
+
+All-time visits use Timestamp; date windows use LocalTimestamp with inclusive
+bounds. Visits and dates aggregate across corrected membership. Coordinate-average
+inputs are ordered by Location ID. A settlement uses its latest relevant timestamp,
+then highest Location ID to break ties. Countries sort by home status, visit count,
+then ordinal name; regions and settlements sort by their ordinal parent/name tuples.
+The home-country threshold remains the maximum of 40% of all records and three times
+the mean recorded-country visit count.
+
+Statistics labels are not new search identifiers. Search, Bulk Edit Notes,
+cascading choices, preview and update membership retain their existing semantics.
+The string-only limitation remains: settlements with identical names and recorded
+parents cannot be distinguished, and other alternate labels may still split one
+entity. Historical Place/Region administrative ambiguity is not resolved.
+
+
 Bulk Edit Notes
 - From Locations > Bulk Edit Notes, you can search by filters and update notes for many records at once.
 

tests/Wayfarer.Tests/Controllers/ApiLocationControllerTests.cs

@@ -391,12 +391,14 @@ public async Task CheckIn_ReturnsForbid_WhenUserInactive()
         Assert.IsType<ForbidResult>(result);
     }
 
+    /// <summary>Statistics accept the authenticated principal without requiring a token header.</summary>
     [Fact]
-    public async Task GetStats_ReturnsUnauthorized_WhenNoToken()
+    public async Task GetStats_ReturnsStats_ForAuthenticatedPrincipalWithoutToken()
     {
         var db = CreateDbContext();
         var user = SeedUserWithToken(db, "tok");
-        var controller = BuildApiController(db, user, includeAuthHeader: false);
+        var controller = BuildApiController(db, user, includeAuthHeader: false,
+            statsService: new StubStatsService(new UserLocationStatsDto()));
 
         var result = await controller.GetStats();
 

tests/Wayfarer.Tests/Services/LocationStatsServicePostgresTests.cs

@@ -1,4 +1,5 @@
 using NetTopologySuite.Geometries;
+using Microsoft.EntityFrameworkCore;
 using System.Text.Json;
 using Wayfarer.Models;
 using Wayfarer.Models.Dtos;
@@ -35,6 +36,11 @@ public async Task PopulatedStatistics_ReturnExpectedAllTimeAndDateRangeDetails()
         jsonOptions.Converters.Add(new PointJsonConverter());
         var json = JsonSerializer.Serialize(allTime, jsonOptions);
         using var document = JsonDocument.Parse(json);
+        Assert.Equal(new[] { "cities", "countries", "fromDate", "regions", "toDate", "totalLocations" },
+            document.RootElement.EnumerateObject().Select(p => p.Name).Order(StringComparer.Ordinal));
+        Assert.Equal(new[] { "coordinates", "countryName", "firstVisit", "lastVisit", "name", "regionName", "visitCount" },
+            document.RootElement.GetProperty("cities")[0].EnumerateObject().Select(p => p.Name).Order(StringComparer.Ordinal));
+        Assert.Equal(2, document.RootElement.GetProperty("totalLocations").GetInt32());
         Assert.Equal(25.87,
             document.RootElement.GetProperty("countries")[0].GetProperty("coordinates").GetProperty("longitude").GetDouble(),
             precision: 6);
@@ -62,6 +68,127 @@ public async Task PartialAddressHierarchy_RemainsReadable()
         AssertPartialHierarchy(dateRange);
     }
 
+    /// <summary>One dataset proves corrected membership, scope, representatives and read-only behavior.</summary>
+    [PostgresFact]
+    public async Task CorrectedGroups_AgreeAcrossScopes_AndPreserveStoredRows()
+    {
+        var user = await fixture.CreateUserAsync();
+        var other = await fixture.CreateUserAsync();
+        var start = new DateTime(2026, 9, 3, 8, 0, 0, DateTimeKind.Utc);
+        await using var context = fixture.CreateContext();
+        var labels = new (string? Country, string? Region, string? Place)[]
+        {
+            (" Greece ", " East Macedonia and Thrace\t", " Port "),
+            ("Greece", "Eastern Macedonia and Thrace", "Port"),
+            ("Other", "East Macedonia and Thrace", "Port"),
+            ("Other", "Eastern Macedonia and Thrace", "Port"),
+            (null, "East Macedonia and Thrace", "Port"),
+            ("", null, "Port"),
+            (" \t", "\r\n", "Port"),
+            ("Greece", null, "Port"),
+            ("Greece", "Another region", "Port"),
+            (null, null, null)
+        };
+        var rows = labels.Select((label, index) =>
+        {
+            var row = Location(user.Id, start, 20 + index, 40);
+            (row.Country, row.Region, row.Place) = label;
+            row.LocalTimestamp = index == 8 ? start.AddDays(1) : start.AddMinutes(index % 2);
+            row.FullAddress = $"Original {index}";
+            row.ProviderAddressLine1 = $"Provider {index}";
+            return row;
+        }).ToArray();
+        context.Locations.AddRange(rows);
+        context.Locations.Add(Location(other.Id, start, 99, 40));
+        await context.SaveChangesAsync();
+        var before = await context.Locations.AsNoTracking().Where(l => l.UserId == user.Id)
+            .OrderBy(l => l.Id).ToListAsync();
+        var service = new LocationStatsService(context);
+
+        var all = await service.GetDetailedStatsForUserAsync(user.Id);
+        var summary = await service.GetStatsForUserAsync(user.Id);
+        var window = await service.GetDetailedStatsForDateRangeAsync(user.Id, start, start.AddMinutes(1));
+        var windowSummary = await service.GetStatsForDateRangeAsync(user.Id, start, start.AddMinutes(1));
+        AssertAgreement(summary, all, 10, 2, 5, 7);
+        AssertAgreement(windowSummary, window, 9, 2, 4, 6);
+        foreach (var detail in new[] { all, window })
+        {
+            var region = Assert.Single(detail.Regions, r => r.CountryName == "Greece" &&
+                r.Name == "Eastern Macedonia and Thrace");
+            Assert.Equal(2, region.VisitCount);
+            Assert.Equal(20.5, region.Coordinates!.X);
+            var city = Assert.Single(detail.Cities, c => c.CountryName == "Greece" &&
+                c.RegionName == region.Name && c.Name == "Port");
+            Assert.Equal(2, city.VisitCount);
+            Assert.Equal(21, city.Coordinates!.X);
+            Assert.Equal(2, Assert.Single(detail.Cities, c => c.CountryName == "" && c.RegionName == "").VisitCount);
+        }
+        Assert.Equal(26, Assert.Single(all.Cities, c => c.CountryName == "" && c.RegionName == "").Coordinates!.X);
+        Assert.Equal(25, Assert.Single(window.Cities, c => c.CountryName == "" && c.RegionName == "").Coordinates!.X);
+        Assert.Equal(new[] { "Greece", "Other" }, all.Countries.Select(c => c.Name));
+        Assert.Equal(new[] { "", "Greece", "Greece", "Other", "Other" }, all.Regions.Select(r => r.CountryName));
+        Assert.Equal(start, all.ToDate);
+        Assert.Equal(start.AddMinutes(1), window.ToDate);
+        Assert.Equal(1, (await service.GetStatsForUserAsync(other.Id)).TotalLocations);
+        var after = await context.Locations.AsNoTracking().Where(l => l.UserId == user.Id)
+            .OrderBy(l => l.Id).ToListAsync();
+        // Compare every mapped scalar, including retained provider/feature fields, from fresh database reads.
+        var properties = context.Model.FindEntityType(typeof(Location))!.GetProperties();
+        foreach (var property in properties.Where(p => p.PropertyInfo != null))
+            Assert.Equal(before.Select(row => property.PropertyInfo!.GetValue(row)),
+                after.Select(row => property.PropertyInfo!.GetValue(row)));
+    }
+
+    /// <summary>Exercises the exact trim boundary and preserves non-ASCII, case and composition distinctions.</summary>
+    [PostgresTheory]
+    [InlineData(null, "")]
+    [InlineData("", "")]
+    [InlineData(" \t\n\v\f\r", "")]
+    [InlineData(" \t\n\v\f\rA \t B-'é\r\f\v\n\t ", "A \t B-'é")]
+    [InlineData("\u0085A\u0085", "\u0085A\u0085")]
+    [InlineData("\u00a0A\u00a0", "\u00a0A\u00a0")]
+    [InlineData("\u2003A\u2003", "\u2003A\u2003")]
+    [InlineData("\u001fA\u001f", "\u001fA\u001f")]
+    [InlineData("é", "é")]
+    [InlineData("e\u0301", "e\u0301")]
+    [InlineData("a", "a")]
+    public async Task Normalization_PreservesEveryCharacterOutsideOuterAsciiWhitespace(string? value, string expected)
+    {
+        var user = await fixture.CreateUserAsync();
+        await using var context = fixture.CreateContext();
+        var timestamp = new DateTime(2026, 9, 3, 8, 0, 0, DateTimeKind.Utc);
+        var row = Location(user.Id, timestamp, 20, 40);
+        (row.Country, row.Region, row.Place) = (value, value, value);
+        context.Locations.Add(row);
+        // The comparison row catches accidental Unicode/case folding and broader whitespace trimming.
+        var comparison = Location(user.Id, timestamp, 21, 40);
+        (comparison.Country, comparison.Region, comparison.Place) = ("A", "A", "A");
+        context.Locations.Add(comparison);
+        await context.SaveChangesAsync();
+        var service = new LocationStatsService(context);
+        var detail = await service.GetDetailedStatsForUserAsync(user.Id);
+        var count = expected == "" ? 1 : 2;
+        AssertAgreement(await service.GetStatsForUserAsync(user.Id), detail, 2, count, count, count);
+        Assert.Equal(expected == "" ? new[] { "A" } : new[] { "A", expected }.Order(StringComparer.Ordinal),
+            detail.Countries.Select(c => c.Name));
+        if (expected != "")
+        {
+            Assert.Contains(detail.Regions, r => r.Name == expected && r.CountryName == expected);
+            Assert.Contains(detail.Cities, c => c.Name == expected && c.CountryName == expected && c.RegionName == expected);
+        }
+    }
+
+    /// <summary>Summary counts describe exactly the detailed arrays, without synthetic parents.</summary>
+    private static void AssertAgreement(UserLocationStatsDto summary, UserLocationStatsDetailedDto detail,
+        int locations, int countries, int regions, int cities)
+    {
+        Assert.Equal((locations, countries, regions, cities),
+            (summary.TotalLocations, summary.CountriesVisited, summary.RegionsVisited, summary.CitiesVisited));
+        Assert.Equal((locations, countries, regions, cities),
+            (detail.TotalLocations, detail.Countries.Count, detail.Regions.Count, detail.Cities.Count));
+        Assert.Equal((summary.FromDate, summary.ToDate), (detail.FromDate, detail.ToDate));
+    }
+
     private static Location Location(string userId, DateTime timestamp, double longitude, double latitude) => new()
     {
         UserId = userId,