Add EchoFace module

[evothreat]

Introduces EchoFace, a real-time webcam-driven facial animation module for multiplayer sessions. It uses Google MediaPipe's Face Landmarker task to capture the local user's facial expressions (blendshapes, head rotation, and eye look) and synchronizes the resulting avatar animation across the network.

For details on setup, architecture, and configuration, see the EchoFace Wiki Page.

[Copilot]

Pull request overview

Adds real-time MediaPipe facial tracking, UDP/network synchronization, avatar animation, and runtime controls.

Changes:

• Adds Unity facial animation and network bridge components.
• Adds the Python tracking client and setup documentation.
• Adds Python-related ignore rules and Unity metadata.

Reviewed changes

Copilot reviewed 10 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
Assets/SEE/Tools/EchoFace/FaceData.cs	Defines facial frame data.
Assets/SEE/Tools/EchoFace/FaceData.cs.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/EchoFace.cs	Applies facial animation to avatars.
Assets/SEE/Tools/EchoFace/EchoFace.cs.meta	Unity metadata and execution order.
Assets/SEE/Tools/EchoFace/EchoFaceNetworkBridge.cs	Synchronizes UDP data through Netcode.
Assets/SEE/Tools/EchoFace/EchoFaceNetworkBridge.cs.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs	Adds runtime toggling and popup UI.
Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/client.py	Orchestrates the tracking client.
Assets/SEE/Tools/EchoFace/Client/client.py.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/face_analyzer.py	Integrates MediaPipe detection.
Assets/SEE/Tools/EchoFace/Client/face_analyzer.py.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/face_data_sender.py	Serializes and sends tracking data.
Assets/SEE/Tools/EchoFace/Client/face_data_sender.py.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/helpers.py	Provides landmark drawing helpers.
Assets/SEE/Tools/EchoFace/Client/helpers.py.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/video_io.py	Handles capture, playback, and display.
Assets/SEE/Tools/EchoFace/Client/video_io.py.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/README.md	Documents Python client setup.
Assets/SEE/Tools/EchoFace/Client/README.md.meta	Unity metadata.
Assets/SEE/Tools/EchoFace/Client/face_landmarker.task.meta	Model asset metadata.
Assets/SEE/Tools/EchoFace/Client.meta	Client folder metadata.
Assets/SEE/Tools/EchoFace.meta	EchoFace folder metadata.
.gitignore	Adds Python environment exclusions.

Files not reviewed (13)

• Assets/SEE/Tools/EchoFace.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/README.md.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/client.py.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/face_analyzer.py.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/face_data_sender.py.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/face_landmarker.task.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/helpers.py.meta: Generated file
• Assets/SEE/Tools/EchoFace/Client/video_io.py.meta: Generated file
• Assets/SEE/Tools/EchoFace/EchoFace.cs.meta: Generated file
• Assets/SEE/Tools/EchoFace/EchoFaceNetworkBridge.cs.meta: Generated file
• Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs.meta: Generated file
• Assets/SEE/Tools/EchoFace/FaceData.cs.meta: Generated file

Suppressed comments (1)

Assets/SEE/Tools/EchoFace/Client/README.md:57

• The repeated PowerShell activation command is also invalid; invoke the generated Activate.ps1 script explicitly.

venv\Scripts\Activate

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[koschke]

In addition to Copilot, I also asked Claude Code for a review. Here is its response (which I hope are helpful):

I reviewed the branch against its merge-base with master (e95b5527fc) — 4 C# files, 5 Python files, a 3.7 MB model binary, and a .gitignore change. I read every line but did not compile or run anything; findings marked verify are ones I could not confirm without running Unity.

Must fix — correctness

1. Eye bones are never resolved when headTransform is set in the Inspector — [EchoFace.cs:475-491](Assets/SEE/Tools/EchoFace/EchoFace.cs:475)

FindEyeBones is called only inside the else of if (headTransform == null). If the prefab assigns headTransform (the documented, intended setup), the eye bones are never searched and leftEyeRestRotation/rightEyeRestRotation keep their default value (0,0,0,0) — not Quaternion.identity. If the eye transforms are also assigned in the Inspector, LateUpdate enters ApplyEyeRotation and assigns leftEyeRestRotation * currentLeftEyeRotation, i.e. a zero quaternion, to localRotation. Unity then logs "Quaternion To Matrix conversion failed" every frame and the eyes are broken. Move FindEyeBones(headTransform) out of the else, and initialize both rest rotations to Quaternion.identity.

2. CacheBlendshapeIndices dereferences a renderer it just warned was missing — [EchoFace.cs:868-871](Assets/SEE/Tools/EchoFace/EchoFace.cs:868)

Start logs "SkinnedMeshRenderer not found. Please assign it manually." and then unconditionally calls CacheBlendshapeIndices, whose first statement is skinnedMeshRenderer.sharedMesh → NullReferenceException. Guard on skinnedMeshRenderer == null and return.

3. EstimateHeadRotation guards on count, then indexes by key — [EchoFace.cs:703-714](Assets/SEE/Tools/EchoFace/EchoFace.cs:703)

landmarks.Count < 3 does not imply the three specific keys exist. Any payload carrying three other landmark keys throws KeyNotFoundException inside LateUpdate. Since the dictionary is built from network data, use TryGetValue on each of the three constants. The same class of problem exists in the viseme lambdas ([EchoFace.cs:252-344](Assets/SEE/Tools/EchoFace/EchoFace.cs:252)), which index arkit["jawOpen"] etc. directly: today the Python sender always emits all 52 values so it works, but a short or truncated "bs" array — trivially injectable, see #6 — throws. GetValueOrDefault is used consistently elsewhere in the same file; use it here too.

4. private new void OnDestroy() suppresses Netcode's own cleanup — [EchoFaceNetworkBridge.cs:269-282](Assets/SEE/Tools/EchoFace/EchoFaceNetworkBridge.cs:269)

The XML comment states NetworkBehaviour does not declare OnDestroy as virtual. It does — Library/PackageCache/com.unity.netcode.gameobjects@d43d2849/Runtime/Core/NetworkBehaviour.cs:1673 is public virtual void OnDestroy(), and its own doc says "If you override this, you must always invoke the base class version". Hiding it with new means m_NetworkObject.OnNetworkBehaviourDestroyed(this) and the NetworkVariable native-container disposal never run, leaving stale entries in ChildNetworkBehaviours and leaking native memory on despawn. Change to public override void OnDestroy() { Shutdown(); base.OnDestroy(); } and drop the remark.

5. EchoFaceToggleController grabs an arbitrary avatar's face — [EchoFaceToggleController.cs:166-175](Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs:166)

FindObjectOfType<EchoFace>() returns some EchoFace in the scene. In a multiplayer session that is as likely to be a remote player's avatar as the local one, so pressing the key toggles someone else's face. And because every avatar instance carries its own controller, each one polls the key and flips the same discovered component, so N avatars means N toggles per keypress — a net no-op for even N. Resolve via GetComponent<EchoFace>() on the own avatar and gate the whole component on local ownership, the way EchoFaceNetworkBridge.Start already does.

6. The UDP listener accepts packets from the whole network and relays them to every client — [EchoFaceNetworkBridge.cs:291](Assets/SEE/Tools/EchoFace/EchoFaceNetworkBridge.cs:291)

new UdpClient(port) binds 0.0.0.0:12345. Any host on the LAN can send face frames that this client forwards via SubmitFaceDataServerRpc to all peers, i.e. puppeteer another user's avatar (and, combined with #3, crash their LateUpdate). The Python client is local by design, so bind loopback: new UdpClient(new IPEndPoint(IPAddress.Loopback, port)). Consider validating bs.Count == BlendshapeOrder.Names.Length before use.

Architecture and integration

7. This duplicates SEE's existing blendshape replication and will fight it. [BlendshapeExpressionsSynchronizer](Assets/SEE/Game/Avatars/BlendshapeExpressionsSynchronizer.cs) already samples all weights off CC_Base_Body periodically and pushes them to every client through [BlendshapeExpressionsNetAction](Assets/SEE/Net/Actions/BlendshapeExpressionsNetAction.cs), which writes them on remote avatars. EchoFaceNetworkBridge adds a second, parallel replication path (raw JSON over ServerRpc/ClientRpc) so that remote clients recompute the same weights locally. If both components are on an avatar, two writers compete for the same SkinnedMeshRenderer — which is, I think, exactly what the commented-out line and its note at [EchoFace.cs:526](Assets/SEE/Tools/EchoFace/EchoFace.cs:526) ("Resetting the data will enable other components to manipulate the face causing jitter!") is working around. That symptom deserves a real fix rather than a disabled reset: either drive the existing net action from EchoFace, or document that BlendshapeExpressionsSynchronizer must be removed from the avatar when EchoFace is active.

Relatedly, the new path also bypasses SEE's AbstractNetAction convention that the rest of SEE.Net.Actions uses, and ships one JSON string per frame (~470 bytes measured from the sender's format) where a float[] in an INetworkSerializable would do — worth at least a sentence of justification in the thesis text.

8. Nothing is wired up. No prefab or scene in the diff references EchoFace, EchoFaceNetworkBridge, or EchoFaceToggleController — grepping Assets/ finds hits only in these four files and the Python/README. From the committed state the feature cannot be exercised at all; the avatar prefab changes appear to exist only in the author's working copy. That has to land with the branch.

9. Head pose is re-derived from three landmarks when MediaPipe can hand it over directly. output_facial_transformation_matrixes=False at [face_analyzer.py:94](Assets/SEE/Tools/EchoFace/Client/face_analyzer.py:94) explicitly turns off MediaPipe's own head-pose output, and [EchoFace.cs:717-733](Assets/SEE/Tools/EchoFace/EchoFace.cs:717) rebuilds a basis from chin plus two eyelid points instead. Two problems with the substitute: the landmark x/y are normalized independently by image width and height, so on a 4:3 frame one x-unit ≠ one y-unit and the basis is aspect-skewed; and ToUnityVector3 negates all three axes ([EchoFace.cs:539-542](Assets/SEE/Tools/EchoFace/EchoFace.cs:539)), a determinant −1 transform, so the resulting basis is left-handed and Quaternion.LookRotation yields a mirrored rotation. Together those plausibly explain why a hand-tuned tiltCorrection fudge is needed at all. Verify against the running avatar, but enabling the transformation matrix would remove the whole estimation step.

10. Duplicated helpers that SEE already provides. FindDeepChild is implemented twice, identically, in [EchoFace.cs:842](Assets/SEE/Tools/EchoFace/EchoFace.cs:842) and [EchoFaceToggleController.cs:335](Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs:335), and duplicates GameObjectExtensions.FindDescendant ([GameObjectExtensions.cs:1496](Assets/SEE/GameObjects/GameObjectExtensions.cs:1496)). The bone-name literals "CC_Base_Body", "CC_Base_Head" are hard-coded in both files although [AvatarSceleton](Assets/SEE/Game/Avatars/AvatarSceleton.cs) defines BaseBody and Head; the two eye bones belong there as new constants rather than as inspector strings.

11. The toggle key bypasses SEE's keybinding system and collides with an existing binding. Input.GetKeyDown(KeyCode.T) at [EchoFaceToggleController.cs:205](Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs:205) sidesteps the SEEInput / KeyBindings.IsDown(KeyAction…) pattern every other SEE component follows, so it ignores SEEInput.KeyboardShortcutsEnabled and will fire while the user is typing in a code window or chat. KeyCode.T is already registered for KeyAction.ToggleEvolutionCanvases ([KeyBindings.cs:220](Assets/SEE/Controls/KeyActions/KeyBindings.cs:220)). Register a new KeyAction and query it through SEEInput.

12. Toggling off leaves EchoFace's smoothing state stale, and mutates a shared Canvas. ResetAllBlendShapes zeroes the mesh weights but not EchoFace.currentBlendshapeValues, so re-enabling lerps from pre-disable values and the face snaps. Separately, EnsurePopupUI reuses whatever screen-space-overlay canvas it finds and force-raises its sortingOrder to ≥ 5000 ([EchoFaceToggleController.cs:398](Assets/SEE/Tools/EchoFace/EchoFaceToggleController.cs:398)) — a side effect on SEE's own UI — while every avatar instance adds another popup panel to that same canvas, all at the same anchored position.

Python client

13. The model path only resolves from one working directory — [face_analyzer.py:21](Assets/SEE/Tools/EchoFace/Client/face_analyzer.py:21)

MODEL_PATH = 'face_landmarker.task' is relative to the process CWD, which is why the README has to instruct cd Assets/SEE/Tools/EchoFace/Client. Use Path(__file__).with_name("face_landmarker.task") and the client runs from anywhere.

14. Video-file playback can violate MediaPipe's monotonic-timestamp contract — [client.py:153](Assets/SEE/Tools/EchoFace/Client/client.py:153)

frame_timestamp_ms = time.time_ns() // 1_000_000 combined with the acknowledged TODO at [client.py:86-89](Assets/SEE/Tools/EchoFace/Client/client.py:86) (the playback clock exists only when --fps is passed, source FPS is never used) means --video-path without --fps reads frames as fast as the file allows. Two frames landing in the same millisecond make detect_async raise "input timestamp must be monotonically increasing", which propagates out of the loop uncaught. Either finish that TODO by defaulting the clock to video_source.get_fps(), or derive the timestamp from a monotonic frame counter.

15. Smaller Python items. PlaybackClock.wait_for_next never rebases _start_time after a slow stretch, so once processing falls behind the target it runs unthrottled forever ([video_io.py:292-299](Assets/SEE/Tools/EchoFace/Client/video_io.py:292)). Pausing sleeps 300 ms per iteration while the OpenCV toolbar is only pumped from display_frame, so a click on "Pause" takes up to 300 ms to register ([client.py:124](Assets/SEE/Tools/EchoFace/Client/client.py:124)). resize_letterbox divides by w/h without a zero check. draw_centered_text ([helpers.py:61](Assets/SEE/Tools/EchoFace/Client/helpers.py:61)) is dead code and is also the only function in the client without a docstring. The blendshape/landmark ordering contract between BLENDSHAPE_ORDER/LM_ORDER and BlendshapeOrder.Names is correct — I checked all 52 names and the (152, 226, 446) triple index-by-index — but it is duplicated by hand in two languages with only a comment to keep them aligned; a generated file or a startup length/checksum handshake would be cheap insurance.

16. README gaps. It covers only venv creation and pip install mediapipe opencv-python — no pinned versions or requirements.txt, no mention of any CLI flag (--ip, --port, --headless, --video-path, --fps), no statement that --port must match the bridge's port field, and nothing at all about the Unity side (which components go on which prefab, the toggle key). Given #8, the Unity setup section is the one a reader most needs.

Repo hygiene

17. .gitignore silently loses /Assets/_Recovery. The diff replaces that line with the Python block; _Recovery appears nowhere in the resulting file. Restore it.

18. Three .meta files are truncated. EchoFaceNetworkBridge.cs.meta, EchoFaceToggleController.cs.meta, and FaceData.cs.meta contain only fileFormatVersion and guid — no MonoImporter: block, unlike EchoFace.cs.meta in the same commit. Unity will rewrite them on first import, producing a spurious diff for everyone who opens the project. Let Unity regenerate them and commit the result.

19. A 3.7 MB binary model is committed into Assets/. face_landmarker.task will be pulled into every clone forever and, sitting under Assets/, imported and shipped in builds. The project's convention is third-party content under Assets/Plugins; a download step in the README or Git LFS would both be kinder to the repo. Its MediaPipe licence/attribution is also not recorded anywhere.

Also note the project already ships Assets/Plugins/MediaPipeUnity/Runtime/Mediapipe.Runtime.asmdef — MediaPipe running in-process in Unity. Choosing an out-of-process Python client over it is a defensible thesis decision, but it should be an explicit, argued one.

Nits

var at [EchoFace.cs:561](Assets/SEE/Tools/EchoFace/EchoFace.cs:561), [630](Assets/SEE/Tools/EchoFace/EchoFace.cs:630) and [649](Assets/SEE/Tools/EchoFace/EchoFace.cs:649) — the SEE guidelines require explicit types. ApplyBlendshapes allocates a fresh Dictionary every frame per avatar; reuse a member and Clear() it. The Debug.LogWarning in EstimateHeadRotation fires every frame when landmarks are absent — flood the console once, not 60×/s. eyeSquintPower documents "optimal value is around 3" but defaults to 12 ([EchoFace.cs:100-107](Assets/SEE/Tools/EchoFace/EchoFace.cs:100)). "Mouth_Up" is added to the index cache at [EchoFace.cs:883](Assets/SEE/Tools/EchoFace/EchoFace.cs:883) but never written, so it only produces a startup warning. The //-------- Public Fields banner at [EchoFace.cs:50](Assets/SEE/Tools/EchoFace/EchoFace.cs:50) sits above fields that are all private. [Header] attributes are placed above the XML doc comments rather than adjacent to their field. Landmarks keys the three landmarks by stringified indices ("152") through a Dictionary<string, …>, when the payload already fixes them positionally — three named fields would remove the dictionary, the constants, and finding #3 at once. Commented-out alternative formulas at [EchoFace.cs:305](Assets/SEE/Tools/EchoFace/EchoFace.cs:305), [328](Assets/SEE/Tools/EchoFace/EchoFace.cs:328), [589-591](Assets/SEE/Tools/EchoFace/EchoFace.cs:589) and [video_io.py:156](Assets/SEE/Tools/EchoFace/Client/video_io.py:156) should go before merge. There are no tests, and the viseme-synthesis weights and blendshape mapping are pure functions — the easiest part of this to cover in SEETests.

The documentation quality throughout is genuinely above the repo average; almost every declaration carries a meaningful XML doc or docstring, and the compact-payload contract is explained well on both sides. The problems are concentrated in lifecycle wiring, the two-writers-to-one-mesh conflict with SEE's existing avatar replication, and reuse of what SEE already has.

[github-actions]

There are a few bad patterns I found which you should check.

[evothreat]

TODOs:

• Add documentation / user manual