PokemonAR

Setup

Requirements

• Xcode 16+ with iOS 26.2 SDK
• iPhone or iPad with LiDAR sensor (recommended) and A12+ chip
• CocoaPods (gem install cocoapods or brew install cocoapods)
• Python 3.10+ (for the debug server)

iOS App

# 1. Install CocoaPods dependencies (MediaPipe hand tracking)
cd PokemonAR
pod install

# 2. Open the workspace (not the .xcodeproj)
open PokemonAR.xcworkspace

# 3. GLTFKit2 is a Swift Package — Xcode resolves it automatically on first open

# 4. Place the MediaPipe hand landmark model in the bundle
#    Download hand_landmarker.task from MediaPipe and place it at:
#    PokemonAR/Models/hand_landmarker.task

# 5. Build and run on a physical device (AR requires a real device)

Debug Server

# 1. Install Python dependencies
pip install -r requirements.txt

# 2. Run the server
python Server/server.py

# The server starts on port 8000 by default.
# Connect from the app by entering your machine's local IP in the server connection UI.

What This Project Is

PokemonAR is an iOS AR app built with SwiftUI, ARKit, and SceneKit. It lets you spawn, walk with, catch, and battle Pokemon in augmented reality. The app supports multiple debug and gameplay modes: AR debug, hand debug, face debug, body debug, walk debug, catch debug, battle debug, PVE, sensor debug, and server debug. The runtime is organized as a coordinator plus focused extensions so each gameplay and rendering concern can be edited independently.

Main Runtime Logic

Core App State Machine

• Launch flow has three phases: preloading -> readyToStart -> running.
• Global app mode is driven by AppMode (arDebug, handDebug, faceDebug, bodyDebug, walkDebug, catchDebug, battleDebug, pve, pvp, sensorDebug, serverDebug).
• PVE has its own phase machine: walking -> battle -> catching -> walking.
• ContentView owns top-level state and routes events between overlays and AR runtime.

UI Architecture

• ARViewContainer stays mounted as the base AR renderer.
• SwiftUI overlays are swapped by mode/phase on top of that persistent AR layer.
• This keeps AR session continuity while UI surfaces change.
• Mode menu uses a 2-column grid layout to keep the growing mode list compact.
• Safe-area handling uses GeometryProxy.safeAreaInsets with a key-window fallback for reliability.

UI-to-AR Command Pattern (Important)

• UI sends one-shot AR commands by incrementing request IDs in ARDebugState (spawnRequestID, walkSpawnRequestID, catchSpawnRequestID, battle effect request IDs).
• ARCoordinator.syncDebugState() consumes those edge-triggered IDs and executes AR actions once.
• This avoids repeated side effects from SwiftUI re-renders and keeps AR actions deterministic.

AR Placement and Tracking Reliability

• Session uses ARWorldTrackingConfiguration with horizontal plane detection.
• LiDAR scene reconstruction is enabled on supported devices for mesh-based occlusion and surface detection.
• Person occlusion (personSegmentationWithDepth) is enabled on supported devices so people properly occlude virtual content.
• Most spawn paths require stable horizontal planes:
  • minimum plane extent: 0.30m
  • max center jitter: 0.02m
  • required stable duration: 0.35s
• Catch mode uses fallback tiers when stable planes are unavailable (stable plane -> relaxed plane -> estimated plane -> camera-forward fallback).
• Pokemon models are lifted above the ground plane to account for both model origin offset (bounding box correction) and LiDAR mesh surface height discrepancy (SceneKit hit test against mesh nodes).

ML/Vision Tracking Modes

• Hand Debug: MediaPipe HandLandmarker (primary, 21 joints per hand) with Vision VNDetectHumanHandPoseRequest fallback. Supports pokeball attachment and throw gestures. ML model is pre-warmed on a background thread when entering the mode.
• Face Debug: Vision VNDetectFaceLandmarksRequest (~76 landmarks including contour, eyes, eyebrows, nose, lips, pupils). CoreML model is pre-warmed with a dummy request on mode entry.
• Body Debug: ARKit ARBodyTrackingConfiguration (91-joint 3D skeleton in world space). Switches the AR session config on entry and restores world tracking on exit. Joints are projected to 2D via arView.projectPoint().

LiDAR Occlusion and Mesh

• LiDAR mesh anchors are rendered as invisible occlusion geometry (colorBufferWriteMask = [], writesToDepthBuffer = true) so real-world surfaces hide virtual content.
• A visible wireframe toggle shows what LiDAR is mapping.
• Both occlusion and mesh visibility are controlled independently via debug state toggles.

PVE Gameplay Loop

• In PVE walking, wild entities auto-spawn roughly every 5 seconds when active wild count is below 5.
• Wild entities roam with per-entity randomized behavior and publish radar contacts to UI.
• Encounter triggers when player-camera approaches a wild target within encounter distance.
• PVE battle enemy actions run on a simple timer (every ~3 seconds while battle is active).
• PVE win transitions to catch phase; catch result transitions back to walking.

PVP Gameplay

• PVP mode provides real-time player-vs-player battles between two devices via the server.
• PVP lobby UI handles server connection, device pairing (your device ID + opponent device ID), pokemon selection, and voice input for pokemon selection.
• Pokemon selections and battle actions are relayed through the server using pokemon_packet / opponent_pokemon_packet and action_packet / opponent_action_packet.
• Both players must select a pokemon and connect before starting; either player can trigger the battle start signal (pvp_start).
• PVP phase machine: lobby -> battle. After battle ends, players return to lobby.
• Voice input in PVP lobby works the same as PVE: record audio, upload to server, server resolves pokemon selection and sends it back via pokemon_packet.

Battle and Catch Rules

• Battle progression formulas:
  • maxHP = 10 * level
  • attackDamage = level * attackSlot(1...4)
  • heal = 5 * level (clamped by max HP)
  • xpDrop = defeatedLevel
  • xpToNextLevel = currentLevel
• Block window duration is 2.0s.
• Low HP warning threshold is < 15% HP.
• Type-effectiveness multipliers are applied in BattleRules (current species inference is asset/name based).
• Attack animations that include root motion (forward displacement baked into GLTF bones) are fully reset after playback by snapshotting and restoring both root and child node transforms.
• Catch chance is driven by ring timing/size and linearly mapped to probability (0.0 ... 0.8), then resolved with RNG after landing+wobble flow.

Persistence and Lifetime

• Pokedex entries persist through UserDefaults (PVEPokedexStore).
• PVE inventory is currently runtime memory (@State) and resets on app relaunch.
• Asset scene sources are preloaded/cached by AssetPreloader for faster first-use rendering.

Available Pokemon

• Bulbasaur (Grass/Poison) — Pokedex #1
• Charizard (Fire/Flying) — Pokedex #6
• Pikachu (Electric) — Pokedex #25
• Mewtwo (Psychic) — Pokedex #150
• Lugia (Psychic/Flying) — Pokedex #249
• Greninja (Water/Dark) — Pokedex #658

Pokemon are configured in PokemonConfig.json (stats, scale, animations, SFX) and PokemonSpeciesVisuals.swift (types, Pokedex IDs, sprite URLs). The server mirrors choices in POKEMON_CHOICES in server.py. New Pokemon can be added by placing a .glb asset in PokemonAR/Pokemon/, adding an entry to each of these three locations, and optionally adding a cry audio file to SoundTracks/.

Dependencies

• ARKit + SceneKit — AR session, 3D scene rendering
• GLTFKit2 — GLTF/GLB model loading and caching
• MediaPipeTasksVision (CocoaPod) — Hand landmark detection, pose landmark detection
• Vision — Face landmarks, hand pose fallback, body pose fallback
• AVFoundation — Audio playback (BGM, SFX, low HP warning)

Code File Reference

App Entry

• PokemonARApp.swift: App entry point that launches the root content view.

AR Runtime

• ARCoordinator.swift: Core AR coordinator class with shared runtime state, constants, and gesture entry points. Feature-specific behavior is intentionally moved into extension files so this root file stays focused on ownership/state.
• ARViewContainer.swift: SwiftUI bridge that hosts ARSCNView and wires coordinator lifecycle. Configures LiDAR mesh, scene reconstruction, person occlusion, and body detection support.
• ARCoordinator+CoreRuntime.swift: Main runtime loop, delegate callbacks, spawn/throw basics, and debug state sync. This is where request-ID driven commands from UI state are consumed and routed into scene actions.
• ARCoordinator+Math.swift: Shared vector/quaternion/math helpers used across AR features.
• ARCoordinator+TransformsAndAssets.swift: Asset loading, base scaling, transform application, camera-facing transforms, ML model warm-up, and mode change orchestration.
• ARCoordinator+PlaneTracking.swift: Plane detection, stability tracking, LiDAR occlusion/mesh material management, and debug plane visualization helpers.
• ARCoordinator+HandDebug.swift: Hand landmark detection, overlay projection, and hand-held pokeball attachment logic. Prefers MediaPipe when the task/model are present and falls back to Vision otherwise.
• ARCoordinator+FaceDebug.swift: Face landmark detection via Vision framework with overlay projection for face contour, eyes, eyebrows, nose, lips, and pupils.
• ARCoordinator+BodyDebug.swift: ARKit body tracking integration. Switches to ARBodyTrackingConfiguration on entry, projects 3D skeleton joints to 2D screen space, and restores world tracking on exit.
• ARCoordinator+SceneGeometry.swift: Shared SceneKit geometry/material builders for rings, shadows, helper nodes, ground-lift correction, and depth-read control.
• ARCoordinator+SceneAndVisuals.swift: Scene cleanup, debug axes, radar publishing, and core scene visual upkeep.
• ARCoordinator+WalkVisualEffects.swift: Walk startle effects, overlap resolution, walk circles, badge scale, and shadow geometry.
• ARCoordinator+WalkRoaming.swift: Walk entity roaming behavior scheduling and movement execution.
• ARCoordinator+WalkBadgeVisuals.swift: Walk badge node creation, badge texture drawing, palettes, and walk idle visual polish.
• ARCoordinator+SpawnWalkCatch.swift: Spawn entry points for AR debug, walk, and catch targets with placement fallbacks.
• ARCoordinator+CatchMode.swift: Catch ring update loop, throw lifecycle, and catch success/fail resolution flow.
• ARCoordinator+CatchWobbleAndText.swift: Pokeball wobble animation and catch result floating text rendering.
• ARCoordinator+BattleSpawn.swift: Battle entity spawning, arrangement, and battle-side node preparation.
• ARCoordinator+AnimationAndEffects.swift: Animation playback helpers, particle/effect utilities, and shared visual effects.
• ARCoordinator+SceneDebugPresentation.swift: Debug/battle text presentation, damage labels, and scene-side debug logging.

Core Logic

• AssetPreloader.swift: Shared async GLTF preload/cache service with startup progress reporting.
• DebugState.swift: Observable app/debug state model, data structs for hand/face/body tracking overlays, and request-ID triggers connecting UI and AR.
• PokemonConfig.swift: Loads/parses Pokemon config JSON and provides config lookups by asset.

Audio Logic

• AudioPlayers.swift: Centralized audio playback service for BGM, SFX, and mode-specific loops.

Battle Logic

• BattleRules.swift: Battle mechanics (damage resolution, type-effectiveness multipliers/tiers, block windows, HP bounds, result determination, low HP checks).
• BattleProgression.swift: XP/level progression formulas, HP scaling, attack/heal values, and level-up application. Pure logic with no AR/UI dependencies.

Algorithm Logic

• PokeballThrowPathAlgorithm.swift: Pure trajectory/release-position math for pokeball throw paths.
• WalkPathingAlgorithm.swift: Pure walk steering/path generation utilities for roaming entities.

PVE Logic Models

• PVEModels.swift: Shared PVE domain models (inventory pokemon, pokedex entries, and phase/state types).

Sensor Logic

• SensorDataPipeline.swift: Parses bundled sensor datasets into frame streams for preview/playback.
• SensorPlaybackController.swift: Legacy bundled-file playback controller for local preview/testing.
• SensorPreviewMode.swift: Sensor preview mode enum and mode-specific display intent.

Server Logic

• ServerDebugRemoteController.swift: Polls the local debug server for live IMU + joystick packets used by Server Debug mode. Also handles voice recording, upload, and PVP opponent pairing.

UI Root Composition

• ContentView.swift: Root screen composition and top-level mode routing.
• ContentView+ComputedState.swift: Derived/computed state used by root UI rendering.
• ContentView+FlowLogic.swift: Mode transitions, selection handling, encounter/catch event routing, and flow orchestration.
• ContentView+PVP.swift: PVP mode logic — lobby flow, pokemon selection relay, opponent action handling, battle start/return.
• ContentView+BattleAndPVECombat.swift: Battle/PVE combat loop, result transitions, audio-state updates, haptic feedback triggers.

Launch UI

• LaunchLoadingView.swift: Startup loading screen and preload progress presentation.
• LaunchStartView.swift: Start menu for entering debug or PVE flows.
• LaunchStyleComponents.swift: Shared visual primitives/styles used by launch screens.

AR/Mode Overlays

• AssetDebugOverlay.swift: AR debug HUD composition.
• AssetDebugComponents.swift: Reusable control components for AR debug overlay sections.
• HandDebugOverlay.swift: Hand landmark skeleton visualization overlay.
• FaceDebugOverlay.swift: Face landmark visualization overlay with contour/feature region lines.
• BodyDebugOverlay.swift: ARKit body skeleton visualization overlay.
• WalkDebugOverlay.swift: Walk debug HUD for spawn selection and controls.
• CatchDebugOverlay.swift: Catch debug HUD for target selection/spawn instructions.
• BattleDebugOverlay.swift: Battle HUD container and battle state presentation orchestration.
• BattleDebugHeaderViews.swift: Battle mode badge, intro/result views.
• BattleDebugSetupViews.swift: Battle setup panel with pokemon pickers.
• BattleDebugStatusViews.swift: Battle status cards, HP bars, and pokemon type badges.
• BattleDebugActionComponents.swift: Battle action controls (attack/block/heal/flee).
• PVEOverlayHUD.swift: Lightweight PVE phase/plane status HUD.
• PVEOverlayModels.swift: PVE UI-side selection/tab/profile model types.
• PVEStarterPickerOverlay.swift: Encounter-time pokemon selection overlay.
• PokemonInventoryOverlay.swift: PVE inventory + pokedex panel UI.
• ConnectionDebugOverlay.swift: Sensor debug overlay with local dataset and playback controls.
• ServerDebugOverlay.swift: Server debug overlay with live IMU telemetry and joystick state.
• PVPLobbyOverlay.swift: PVP lobby screen — server connection, device pairing, pokemon selection, voice input, and battle start.
• NormalModeOverlay.swift: Legacy minimal floating menu overlay utility.

Sensor UI

• SensorPreviewScene.swift: SceneKit-based sensor orientation/pose visualization scene.

Global Menu / Settings UI

• SettingMenu.swift: Global mode switch menu (2-column grid) and theme token definitions.

Shared UI Building Blocks

• OverlayDropdown.swift: Generic dropdown UI used by multiple overlays.
• OverlayChrome.swift: Shared floating chrome (mode/menu shell) for overlays.
• SpawnSelectionOverlay.swift: Shared spawn-selection shell used by walk/catch flows.
• AssetSpawnControlBar.swift: Shared asset dropdown + spawn action bar.
• DebugStatusAndEffects.swift: Shared debug status widgets and overlay effects.
• OptionCardColumnLayout.swift: Custom layout for option-card style panels.
• PokemonTextStyles.swift: Centralized typography tokens and text style modifiers.
• PokemonPanelSurfaces.swift: Shared panel shell shapes, surface treatments, and panel modifiers.
• PokemonButtonStyles.swift: Shared Pokemon-themed button styles and press feedback.
• PokemonDisplayComponents.swift: Shared reusable display components (mode badges, pokeball symbol, section headers).
• PokemonActionGlyphs.swift: Custom action glyph vector drawings for gameplay actions.
• PokemonSpeciesVisuals.swift: Shared pokemon visual/type catalog and icon/sprite lookup utilities.

Server UI

• ServerJoystickTelemetryView.swift: Joystick telemetry visualization.

Local Server Helper

• Server/server.py: Local FastAPI debug server. Replays IMU + joystick JSONL data for Server Debug mode. Handles pokemon selection, voice upload, action relay, and PVP opponent pairing across devices.
• Server/sensor_stream.py: Helper module that loads IMU + joystick JSONL streams.

Notes

• All large files were split by responsibility; no code file exceeds 500 lines.
• The project uses filesystem-synchronized Xcode groups, so Swift files added under the app folder are automatically included in the target.
• MediaPipe hand tracking is configured via the workspace Podfile and expects PokemonAR/Models/hand_landmarker.task in the app bundle.
• Body debug mode temporarily switches the AR session to ARBodyTrackingConfiguration and restores world tracking when leaving the mode.
• ML models (MediaPipe, Vision CoreML) are pre-warmed on background threads when entering detection modes to avoid first-frame lag.