[REQUEST] Add full hue_native_control option for native control of Philips Hue lights

[Hedda]

For reference also see this feature request for ZHA Device Handlers which asking about custom device support for this cluster:

• [Device Support Request] Hue bulbs missing Candle Effect zha-device-handlers#2517

Request add full support Philips Hue native control (a.k.a. "Hue Dynamic Scenes" or "Dynamic Lighting Scenes") with decode/encode of binary format used by Philips Hue Zigbee devices which can be used to command custom "Bifrost effects" (atomic Bifrost frames on cluster 0xFC03 to control different firmware-supported special animated scene effect like example "Candle", "Fireplace", and "Colorloop" animation) supported by a few Philips Hue lighting products and support Zigbee frames in a manufacturer specific cluster implementation format on cluster 0xFC03 to zha, zigpy, zha-quirks/zha-device-handlers, and the ZHA integration UI:

• https://github.com/chrivers/bifrost/blob/master/doc/hue-zigbee-format.md

• https://github.com/jtbandes/hue-zigbee-encoding

Philips Hue lights supports this via Zigbee frames in a manufacturer specific cluster implementation format, on cluster 0xFC03.

This is already supported by Zigbee2MQTT and is needed if want to achieve feature-parity with Z2M and zigbee-herdsman, and more importantlly really needed if want to encurage users of the official Philips Hue Bridge and app to move/migrate to ZHA:

• hue: add tzLocal.effect that support candle, fireplace, colorloop Koenkk/zigbee-herdsman-converters#5192

  • [Feature request]: Support Candle and Fireplace effects on Hue lights Koenkk/zigbee2mqtt#15699

    • https://community.home-assistant.io/t/a-simple-single-automation-candle-effect-for-hue-white-ambience-and-other-bulbs/235023/

Zigbee2MQTT supports these Hue Effects / animations (effect names):

• candle (Soft, flickering yellow/orange glow)
• fireplace (Dynamic flickering orange/red colors)
• colorloop (Cycles endlessly through the color spectrum)
• blink
• breathe
• okay
• sunset
• sunrise
• sparkle
• opal
• glisten
• underwater
• cosmos
• sunbeam
• enchant
• none, finish_effect, stop_hue_effect or stop_effect (Returns the light to its standard state)

These are supported by most Philips Hue series light products with color temperature, but see example these supported devices:

• https://www.zigbee2mqtt.io/devices/046677592530.html
• https://www.zigbee2mqtt.io/devices/8719514329843.html
• https://www.zigbee2mqtt.io/devices/8719514491229.html
• https://www.zigbee2mqtt.io/devices/046677592622.html
• https://www.zigbee2mqtt.io/devices/046677584658.html
• https://www.zigbee2mqtt.io/devices/046677577957.html
• https://www.zigbee2mqtt.io/devices/915005988101.html
• https://www.zigbee2mqtt.io/devices/929003711201.html
• https://www.zigbee2mqtt.io/devices/929002294102.html
• https://www.zigbee2mqtt.io/devices/9290030211.html

These dynamic scenes animations are also supported by deCONZ (Zigbee gateway from Dresden Elektronik) too:

• Hue gradient lightstrip dresden-elektronik/deconz-rest-plugin#5891 (comment)

  • Hue white and color ambiance ZHA lights dresden-elektronik/deconz-rest-plugin#6192

This type of specialized message is necessary to support many of the advanced features in Hue lights, such as:

• Multiple colors ("gradient") in LED strips.
• Light Effects ("Candle", "Fireplace", "Fireplace", "Colorloop", etc.).
• Combining effects with color settings.

The use case for this is not only to add feature-parity with the support for this manufacturer specific cluster previously only avalable via a proprietary Philips Hue Bridge, but also to allow users to fully replace their Philips Hue Bridge to instead get a better user experience when only using Zigbee2MQTT for their Philips Hue lighting Zigbee devices.

Note! This new feature requested for zha is directly related to fact that a developer named Mihonarium has this week now added hue_native_control option for native control of Philips lights to zigbee-herdsman-converters in an implementation where he has rewritted zigbee-herdsman's existing Philips manufacturer specific cluster encoder/decoder handling and adds full feature-parity support for Philips Hue native control with encode/decode of custom effects, with improved effect UX and configurable color wheel/effect interaction in Zigbee2MQTT UI with now now report state, support color+speed in a single command, and the color wheel/effect interaction is configurable.

• https://www.reddit.com/r/homeassistant/comments/1t17e1g/z2m_now_fully_supports_philips_hue/
  • philips: native control; fix effects and state reporting (attempt 2) Koenkk/zigbee-herdsman-converters#12008
  • philips: add hue native control Koenkk/zigbee-herdsman-converters#11655
  • philips: native control; fix effects and state reporting (attempt 2) Koenkk/zigbee-herdsman-converters#12008
  • Always add 'sunrise' to effects list Koenkk/zigbee-herdsman-converters#12087

Summary

Adds a hue_native_control option for native control of Philips lights.

Rewrites the Philips manuSpecificPhilips2 encoder/decoder drafted by LaurentvdBos based on the Bifrost spec, fixing several data corruption bugs and adding missing functionality. Also improves effect UX: effects now report state, support color+speed in a single command, and the color wheel/effect interaction is configurable.

Opt-in

By default, standard ZCL converters handle on/off, brightness, color, and color temperature. The behavior is unchanged from before this PR. The manuSpecificPhilips2 cluster can send all of these in a single atomic command, which is required for full effect support: changing color while an effect is running, stopping effects by changing color, and setting an effect with a specific color in one command. However, native control also changes some user-facing behavior (e.g., no simulated fade-to-off transition), so it is opt-in.

Users can enable it per-device in Z2M Settings → Settings (Specific), or globally via device_options: { hue_native_control: true } in configuration.yaml.

New features

• All Bifrost effects — Added sunset, sparkle, opal, glisten, underwater, cosmos, sunbeam, enchant (13 total including no_effect, candle, fireplace, prism/colorloop, sunrise)

• effect + color in one command — {"effect": "candle", "color": "#FF4400"} activates the effect at that color atomically. Also supports effect_speed in the same payload.

• effect_color (new expose) — Sets the base color of the active effect without stopping it. Accepts hex or XY.

• effect_color_mode (new per-device option) — Controls what happens when the regular color wheel is used while an effect is active:

  • "stop" (default): color change stops the effect (matches Hue app)
  • "update": color change re-sends the effect with the new color

• Deferred brightness on effect re-send — When re-sending an effect (via effect_color or "update" mode) with an explicit brightness, the brightness is sent as a separate command after the effect, since effects reset brightness on activation.

• effect and effect_speed access changed to STATE_SET — Current values are now visible in the Z2M frontend and HA.

• effect_speed — Settable via both the new path (philipsLightTz) and the effect converter.

• gradient_scale / gradient_offset — Settable. Fixed-point 5.3 format exposed as float.

• gradient_style — Fully settable (linear, scattered, mirrored). Was decoded but not writable.

• gradient_xy — Lossless XY coordinates published alongside RGB hex gradient.

• philips2_raw — Raw hex blob published for advanced clients (e.g. Bifrost) to decode independently.

• HSV→XY conversion — HSV colors are now converted correctly instead of being silently dropped.

• transition → fadeSpeed — Mapped to the Bifrost spec's per-message fade speed field.

• Delayed state read after effect activation — Reads device state 1s after activating an effect to sync actual brightness.

(Feel free to remove or ask me to remove the effect_color, effect_color_mode, and related stuff if it goes against the overall philosophy.)

Encoder/decoder rewrite

Replaced the ad-hoc hex string parsing with structured Encode/DecodeManuSpecificPhilips2 functions using DataView for proper binary field handling. The wire order follows the Bifrost spec (gradient colors before effect speed before gradient params — reversed relative to flag bit order). Each field type is defined as a HueTypeDetails struct with encode/decode/flag/maxLength, iterated in wire order.

Limitations

After activating an effect, we sync the device state via a 1-second setTimeout, as some Hue effects change light color/brightness. If the device goes offline in the window, the error is silently caught. There could be more robust approaches.

Other

I made a very cool hass dashboard card (inspired by the Hue app, but better than it)! See #12008 (comment)

PS: I also think it is relevant to Open Home Foundation's roadmap discussion about shaping ZHA to have feature-parity with Z2M:

• Further shape Zigbee support by implementing missing features in ZHA while retaining choice OpenHomeFoundation/roadmap#93

As well as being related to this guide for migrating from a Philips Hue Bridge to the ZHA (Zigbee Home Automation) integration in Home Assistant:

• https://community.home-assistant.io/t/migrating-from-a-philips-hue-bridge-to-the-zha-zigbee-home-automation-integration/620862

PPS: Another device that could also use native control of effect commands via quirks is Lidl Christmas led string HG0646:

• [Device Support Request] for Lidl Christmas led string HG0646 zha-device-handlers#1933

Lidl Christmas led string HG0646 uses cluster 0x00FC to invoke invoked native effects commands for their device:

• https://gist.github.com/enyone/e1e0182f41ab9924d33fd24d77765aa6#file-lidltuyaeffects-py-L91

  • https://gist.github.com/enyone/e1e0182f41ab9924d33fd24d77765aa6#file-lidltuyaeffects-py-L98

[Hedda]

For reference, there's a suggested draft implementation with prerequisite change to zha to allow stuff like this via quirks in the future:

• Migrate quirks out of zigpy and provide direct entity access #762

Not quite a solution but that is at least a suggested prerequisite change to zha which could allow Philips Hue native control via quirks in the future. But note that is just an example though and not actually an implementation, because as I understand that would then have to be done later as follow-up quirks to the zha-device-handlers repo after zigpy/zha#762 has been merged, (so only applicable if and when that pull request has been merged).

• Migrate quirks out of zigpy and provide direct entity access #762

As a test implementation, I've added support for the manufacturer-specific Hue lighting frame format in a fork of quirks. It look something like this (100% slop that manages to run):

"""Philips Hue native control via cluster 0xFC03 — v3 device-level quirk."""

from __future__ import annotations

import asyncio
import logging
from typing import Any, Final

from zigpy.quirks import CustomCluster
import zigpy.types as t
from zigpy.zcl import ClusterType
from zigpy.zcl.clusters.general import LevelControl, OnOff
from zigpy.zcl.clusters.lighting import Color
from zigpy.zcl.foundation import BaseCommandDefs, ZCLCommandDef

from zha.application import Platform
from zha.application.platforms.light import Light
from zha.zigbee.device import Device, DeviceMatch, Replace, register_device

class HueNativeLight(Light):
    """Light driven by atomic Bifrost frames on cluster 0xFC03."""

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)
        self._hue_cluster_handler = self.cluster_handlers[HueLightCluster.ep_attribute]

    @property
    def effect_list(self) -> list[str] | None:
        return list(EFFECT_NAME_TO_HUE)

    def on_add(self) -> None:
        super().on_add()
        self._hue_cluster_handler.cluster.add_listener(self)

    def cluster_command(self, tsn: int, command_id: int, args: list[Any]) -> None:
        """Handle a server-issued cluster command on 0xFC03."""
        if command_id != HUE_NATIVE_COMMAND_ID:
            return
        payload: t.SerializableBytes = args[0]
        try:
            frame = decode(bytes(payload.serialize()))
        except ValueError:
            _LOGGER.exception("Failed to decode Hue native frame: %s", payload)
            return
        self._apply_frame(frame)

    def _apply_frame(self, frame: HueFrame) -> None:
        if frame.on_off is not None:
            self._state = frame.on_off
        if frame.brightness is not None:
            self._brightness = frame.brightness
        if frame.color_xy is not None:
            self._xy_color = frame.color_xy
        if frame.color_mirek is not None:
            self._color_temp = frame.color_mirek
        if frame.effect_type is not None:
            self._effect = HUE_TO_EFFECT_NAME.get(frame.effect_type, "none")
        self.maybe_emit_state_changed_event()

    async def _async_turn_on_impl(
        self,
        *,
        transition: float | None,
        brightness: int | None,
        effect: str | None,
        flash: Any,
        color_temp: int | None,
        xy_color: tuple[float, float] | None,
        duration: float,
        **_unused: Any,
    ) -> None:
        del flash, _unused  # Flash uses the Identify cluster, not Bifrost.
        frame = HueFrame(
            on_off=True,
            brightness=_clamp_brightness(brightness),
            color_mirek=color_temp,
            color_xy=xy_color,
            fade_speed=_seconds_to_fade_speed(transition or duration),
            effect_type=EFFECT_NAME_TO_HUE.get(effect) if effect else None,
        )
        await self._send_frame(frame)
        if effect and EFFECT_NAME_TO_HUE.get(effect) not in (None, HueEffect.NO_EFFECT):
            asyncio.get_running_loop().call_later(
                _POST_EFFECT_REFRESH_DELAY,
                lambda: asyncio.create_task(self._refresh_state()),
            )

    async def async_turn_off(self, *, transition: float | None = None) -> None:
        await self._send_frame(
            HueFrame(
                on_off=False,
                fade_speed=_seconds_to_fade_speed(transition),
            )
        )

    async def _send_frame(self, frame: HueFrame) -> None:
        payload = encode(frame)
        _LOGGER.debug("Hue native frame: %s", payload.hex())
        await self._hue_cluster_handler.cluster.hue_native(t.SerializableBytes(payload))

    async def _refresh_state(self) -> None:
        # OnOff/Level/Color report standard attributes; trigger a re-poll.
        await self._on_off_cluster_handler.cluster.read_attributes(
            ["on_off"], allow_cache=False
        )

@register_device
class HueLcx004(Device):
    """Hue Play gradient lightstrip (LCX004) — Bifrost-native control."""

    _device_match = DeviceMatch(
        manufacturers=frozenset({PHILIPS, SIGNIFY}),
        models=frozenset({"LCX004"}),
    )
    _operations = (
        Replace(
            endpoint_id=11,
            cluster_id=HUE_FC03_CLUSTER_ID,
            cluster_type=ClusterType.Server,
            replacement=HueLightCluster,
        ),
    )

    def discover_entities(self):
        for entity in super().discover_entities():
            # Skip existing Light entities
            if entity.PLATFORM == Platform.LIGHT:
                continue
            yield entity

        # Register new HueNativeLight entities
        for ep_id, endpoint in self.endpoints.items():
            if ep_id == 0:
                continue

            handlers = endpoint.all_cluster_handlers
            on_off = handlers.get(f"{ep_id}:0x{OnOff.cluster_id:04x}")
            fc03 = handlers.get(f"{ep_id}:0x{HUE_FC03_CLUSTER_ID:04x}")
            if on_off is None or fc03 is None:
                continue

            # Will be cleaned up once device handlers are removed
            cluster_handlers = [on_off, fc03]
            for opt_id in (LevelControl.cluster_id, Color.cluster_id):
                ch = handlers.get(f"{ep_id}:0x{opt_id:04x}")
                if ch is not None:
                    cluster_handlers.append(ch)

            yield HueNativeLight(
                cluster_handlers=cluster_handlers,
                endpoint=endpoint,
                device=self,
            )

I think we can greatly minimize the Python code required to implement custom entities with a better API. This has a few benefits: