Complete HBNJ eight-day compatibility pass

README.md

@@ -47,9 +47,11 @@ actually worked, then giving them new data without replacing their identity.
 - The guide covers all 54 Japanese broadcast areas and 376 terrestrial
   services.
 - Area-coded requests receive compact regional payloads; Gunma currently has
-  11 stations and 397 programs.
+  11 stations and 3,475 programs across the channel's full eight-day window.
 - Current titles, times, genres, and Japanese program descriptions are packed
   into their native TV no Tomo records.
+- The original genre-search table is populated with 12 Japanese categories,
+  and the date carousel receives eight consecutive broadcast days.
 - The original activation, query, popularity, and synchronization CGI calls
   receive the response contracts expected by the channel.
 - A daily job collects, validates, packs, independently checks, and atomically
@@ -89,15 +91,16 @@ latest build validated:
 
 - 54 broadcast areas
 - 376 stations
-- 14,871 programs
-- 12,897 program descriptions
+- 123,803 programs across eight broadcast days
+- 104,665 program descriptions
+- 12 native genre-search categories
 - every header-to-EPG station key
 - every EPG-to-string record index
 - every regional native payload
 
-The next work is focused on in-app polish, multi-day guide rollover,
-popularity synchronization, easier local server setup, Wii Mail, and adapters
-for more Japanese WC24 channels.
+The next work is focused on visual testing of the completed genre and date
+flows, popularity synchronization, easier local server setup, Wii Mail, and
+adapters for more Japanese WC24 channels.
 
 ## Repository layout
 
@@ -118,10 +121,10 @@ build is:
 py -3 tools\update_hbnj_daily.py
 ```
 
-That command collects all regions into private staging, validates the complete
-guide, creates native HDPK payloads, independently parses them, and publishes
-only after every check passes. A failed build leaves the previous live guide
-untouched.
+That command collects eight days for all regions into private staging,
+validates the complete guide and native size ceilings, creates HDPK payloads,
+independently parses them, and publishes only after every check passes. A
+failed build leaves the previous live guide untouched.
 
 The shared command-line tools can audit WC24 state, inspect the local account,
 validate channel manifests, provision tasks, and run the replacement server:

channels/hbnj/README.md

@@ -4,10 +4,13 @@ This adapter targets the untouched Japanese v512 TV no Tomo WAD with title ID
 `0001000148424e4a` (`HBNJ`).
 
 On first-run, HBNJ natively creates a 4 MiB `wc24dl.vff`, `wc24pubk.mod`, and a
-`header.bin` download task in slot 10. The adapter adopts that native slot and
-redirects it to the replacement service. EPG and string payload fixtures remain
-available to the adapter, but they are not provisioned as speculative tasks;
-their native registration sequence must be observed first.
+`header.bin` download task in slot 10. After setup, the channel replaces that
+bootstrap entry with native `epg.bin` and `str.bin` tasks in slots 10 and 11.
+The two manifests model those observed phases separately, and the adapter
+adopts the channel-created records instead of inventing speculative tasks.
+The active adapter adds a JWC24-managed daily `header.bin` refresh in free slot
+12. This keeps station, area, and genre metadata current after first-run setup
+without replacing either native guide task.
 
 `prune_duplicate_tasks` removes only same-title, same-filename duplicates when
 the manifest adopts a task at a different canonical slot. This migrates the
@@ -29,18 +32,23 @@ served.
 schema audit, native packing, and independent binary audit in a private staging
 directory. It atomically publishes to `generated/current` only after every
 stage succeeds, so a failed upstream collection leaves the previous guide live.
-It also builds and validates one EPG per native area ID under
+It collects the channel's full eight-day date window, builds and validates one
+EPG/string pair per native area ID under
 `generated/current/areas/<id>`. Requests such as `/1016/epg.bin` resolve to the
-matching regional payload, with the national package retained as a fallback.
+matching regional payload. Unknown numeric area IDs fail closed instead of
+receiving the oversized national package. Each regional pair is checked against
+the Nintendo LZ10 24-bit limit and the channel's 4 MiB VFF capacity before
+publication.
 The collector also maps Bangumi's CSS genre categories to TV no Tomo's
-one-based ARIB genre IDs. It preserves program descriptions in the private
-guide JSON and packs them into the first text pointer of each native `str.bin`
-record. Every EPG detail record carries a validated one-based index into that
-table; the optional second text pointer remains empty.
-
-The task layout is based on confirmed reverse engineering from the retired
-workspace. It is not considered production-compatible until a clean WAD launch
-requests these slots and imports all three payloads without a scene bypass.
+one-based ARIB genre IDs. The native header contains the 12 Japanese labels
+used by the original genre-search screen. Program descriptions are preserved
+in the private guide JSON and packed into the first text pointer of each native
+`str.bin` record. Every EPG detail record carries a validated one-based index
+into that table; the optional second text pointer remains empty.
+
+The task layout and binary structures are based on confirmed reverse
+engineering and clean-WAD observation. A clean v512 WAD has imported all three
+payload types through the original WC24 path without a scene bypass.
 
 Channel-specific CGI endpoints (`activate.cgi`, `query.cgi`, `popularity.cgi`,
 and `/bin*`) are served by the adapter with the native `X-RESULT` success

channels/hbnj/channel.json

@@ -30,14 +30,19 @@
       "mode": "adopt",
       "compression": "nintendo-lz10",
       "envelope": "wc24-aes-ofb"
-    }
-  ],
-  "assets": [
+    },
     {
+      "slot": 12,
       "filename": "header.bin",
+      "route": "/header.bin",
       "payload": "generated/current/header.hdpk",
+      "refresh_minutes": 1440,
+      "retry_minutes": 5,
+      "unsigned": false,
+      "mode": "create",
       "compression": "nintendo-lz10",
       "envelope": "wc24-aes-ofb"
     }
-  ]
+  ],
+  "assets": []
 }

jwc24/server.py

@@ -138,8 +138,13 @@ def do_GET(self) -> None:  # noqa: N802
                         / route_parts[0]
                         / item.payload.name
                     )
-                    if area_payload.is_file():
-                        payload_path = area_payload
+                    if not area_payload.is_file():
+                        # Never fall back to the national package for an
+                        # unknown numeric area. Its station count exceeds the
+                        # channel's 24-station native model capacity.
+                        self.send_error(HTTPStatus.NOT_FOUND)
+                        return
+                    payload_path = area_payload
                 body = payload_path.read_bytes()
                 if item.compression == "nintendo-lz10":
                     body = _nintendo_lz10_literal(body)

tools/collect_hbnj_all.py

@@ -4,17 +4,78 @@
 import re
 import sys
 import time
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from pathlib import Path
 
 from collect_hbnj_region import atomic_json, fetch, parse_region
 from jwc24.hbnj_regions import PREFECTURES, broadcast_area_count
 
 
+def merge_duplicate_program(
+    previous: dict[str, object],
+    current: dict[str, object],
+) -> dict[str, object]:
+    comparable_previous = {
+        key: value for key, value in previous.items() if key != "source_program_id"
+    }
+    comparable_current = {
+        key: value for key, value in current.items() if key != "source_program_id"
+    }
+    if comparable_previous != comparable_current:
+        raise ValueError(f"program {current['id']} changed across broadcast pages")
+    # Bangumi sometimes exposes the real program ID on one side of the 05:00
+    # broadcast-day boundary and the placeholder -1 on the other.
+    if previous.get("source_program_id") == "-1" and current.get("source_program_id") != "-1":
+        return current
+    return previous
+
+
+def collect_with_retry(
+    *,
+    group_id: int,
+    broadcast_date: str,
+    area_id: int,
+    area_name: str,
+    prefecture_raw: int,
+    retries: int,
+    retry_delay: float,
+) -> tuple[str, dict[str, object]]:
+    for attempt in range(1, retries + 1):
+        try:
+            source_url, source = fetch(group_id, broadcast_date)
+            return source_url, parse_region(
+                source,
+                group_id=group_id,
+                area_id=area_id,
+                area_name=area_name,
+                prefecture_raw=prefecture_raw,
+                source_url=source_url,
+            )
+        except Exception as error:
+            if attempt == retries:
+                raise
+            wait = retry_delay * (2 ** (attempt - 1))
+            print(
+                f"  attempt {attempt}/{retries} failed: "
+                f"{type(error).__name__}: {error}; retrying in {wait:g}s",
+                file=sys.stderr,
+                flush=True,
+            )
+            if wait:
+                time.sleep(wait)
+    raise AssertionError("retry loop ended without a result")
+
+
 def main() -> int:
     parser = argparse.ArgumentParser(description="Collect all 54 HBNJ broadcast areas strictly.")
     parser.add_argument("--date", required=True, help="Broadcast date in YYYYMMDD form")
     parser.add_argument("--out", type=Path, required=True)
+    parser.add_argument(
+        "--days",
+        type=int,
+        default=8,
+        help="Consecutive broadcast days to collect (TV no Tomo displays eight)",
+    )
     parser.add_argument("--delay", type=float, default=0.5, help="Delay between region requests")
     parser.add_argument(
         "--retries",
@@ -33,6 +94,8 @@ def main() -> int:
         raise SystemExit("--date must use YYYYMMDD")
     if args.delay < 0:
         raise SystemExit("--delay cannot be negative")
+    if not 1 <= args.days <= 8:
+        raise SystemExit("--days must be between 1 and 8")
     if args.retries < 1:
         raise SystemExit("--retries must be at least 1")
     if args.retry_delay < 0:
@@ -42,55 +105,97 @@ def main() -> int:
     channels: list[dict[str, object]] = []
     programs: list[dict[str, object]] = []
     sources: list[dict[str, object]] = []
+    first_date = datetime.strptime(args.date, "%Y%m%d")
+    broadcast_dates = [
+        (first_date + timedelta(days=offset)).strftime("%Y%m%d")
+        for offset in range(args.days)
+    ]
     area_id = 1001
     for prefecture_raw, prefecture_name, regions in PREFECTURES:
         for group_id, region_name in regions:
             area_name = region_name if len(regions) > 1 else prefecture_name
-            print(
-                f"[{area_id - 1000:02d}/{broadcast_area_count()}] "
-                f"group={group_id} area={area_name}",
-                flush=True,
-            )
-            for attempt in range(1, args.retries + 1):
-                try:
-                    source_url, source = fetch(group_id, args.date)
-                    region = parse_region(
-                        source,
-                        group_id=group_id,
-                        area_id=area_id,
-                        area_name=area_name,
-                        prefecture_raw=prefecture_raw,
-                        source_url=source_url,
+            area: dict[str, object] | None = None
+            canonical_channels: list[dict[str, object]] | None = None
+            canonical_id_by_service: dict[str, int] = {}
+            area_programs: dict[int, dict[str, object]] = {}
+            source_urls: list[str] = []
+            daily_program_counts: list[int] = []
+            for day_number, broadcast_date in enumerate(broadcast_dates, start=1):
+                print(
+                    f"[{area_id - 1000:02d}/{broadcast_area_count()} "
+                    f"day {day_number}/{args.days}] group={group_id} "
+                    f"date={broadcast_date} area={area_name}",
+                    flush=True,
+                )
+                source_url, region = collect_with_retry(
+                    group_id=group_id,
+                    broadcast_date=broadcast_date,
+                    area_id=area_id,
+                    area_name=area_name,
+                    prefecture_raw=prefecture_raw,
+                    retries=args.retries,
+                    retry_delay=args.retry_delay,
+                )
+                region_channels = list(region["channels"])
+                service_by_region_channel = {
+                    int(channel["id"]): str(channel["service_id"])
+                    for channel in region_channels
+                }
+                if canonical_channels is None:
+                    area = dict(region["area"])
+                    canonical_channels = region_channels
+                    canonical_id_by_service = {
+                        str(channel["service_id"]): int(channel["id"])
+                        for channel in canonical_channels
+                    }
+                elif set(service_by_region_channel.values()) != set(canonical_id_by_service):
+                    raise ValueError(
+                        f"area {area_id} service lineup changed on {broadcast_date}"
                     )
-                    break
-                except Exception as error:
-                    if attempt == args.retries:
-                        raise
-                    wait = args.retry_delay * (2 ** (attempt - 1))
-                    print(
-                        f"  attempt {attempt}/{args.retries} failed: "
-                        f"{type(error).__name__}: {error}; retrying in {wait:g}s",
-                        file=sys.stderr,
-                        flush=True,
+
+                for program in region["programs"]:
+                    normalized = dict(program)
+                    service_id = service_by_region_channel[int(program["channel_id"])]
+                    normalized["channel_id"] = canonical_id_by_service[service_id]
+                    program_id = int(normalized["id"])
+                    previous = area_programs.get(program_id)
+                    area_programs[program_id] = (
+                        normalized
+                        if previous is None
+                        else merge_duplicate_program(previous, normalized)
                     )
-                    if wait:
-                        time.sleep(wait)
-            area = dict(region["area"])
+                source_urls.append(source_url)
+                daily_program_counts.append(len(region["programs"]))
+                if args.delay and not (
+                    area_id == 1000 + broadcast_area_count()
+                    and day_number == args.days
+                ):
+                    time.sleep(args.delay)
+
+            assert area is not None and canonical_channels is not None
+            merged_programs = sorted(
+                area_programs.values(),
+                key=lambda program: (
+                    int(program["channel_id"]),
+                    str(program["start"]),
+                    int(program["id"]),
+                ),
+            )
             areas.append(area)
-            channels.extend(region["channels"])
-            programs.extend(region["programs"])
+            channels.extend(canonical_channels)
+            programs.extend(merged_programs)
             sources.append(
                 {
                     "area_id": area_id,
                     "group_id": group_id,
-                    "source_url": source_url,
-                    "channels": len(region["channels"]),
-                    "programs": len(region["programs"]),
+                    "source_urls": source_urls,
+                    "broadcast_dates": broadcast_dates,
+                    "daily_program_counts": daily_program_counts,
+                    "channels": len(canonical_channels),
+                    "programs": len(merged_programs),
                 }
             )
             area_id += 1
-            if args.delay:
-                time.sleep(args.delay)
 
     channel_ids = [int(channel["id"]) for channel in channels]
     program_ids = [int(program["id"]) for program in programs]
@@ -106,6 +211,8 @@ def main() -> int:
         "format": "jwc24_hbnj_guide_v1",
         "source": "bangumi.org",
         "broadcast_date": args.date,
+        "broadcast_end_date": broadcast_dates[-1],
+        "days": args.days,
         "collected_at": datetime.now(timezone.utc).isoformat(),
         "areas": areas,
         "channels": channels,