Transfer all Sinopé devices to quirks V2

[claudegel]

Proposed change

Transfer all Sinopé devices to quirks V2 and add support for some new devices.
Add many new attributes in manufacturer cluster.

Additional information

Some manufacturer cluster attributes are already supported in ZHA but attributes values are not the same. ex. attribute 0x0002 for keypad_lockout in manufacturer cluster have three values. Unlocked = 0x00, Locked = 0x01, Partial_lock = 0x02. In ZHA there is 5 values Unlock, lock1, lock2, lock3 and lock4 which have no meaning for Sinopé devices.
Same for on_led_color which have only the color used in Neviweb. We should be able to add mode color.
They should be included in this quirks as they are only available for Sinopé devices.
This PR replace PR #3821.

Checklist

• The changes are tested and work correctly
• pre-commit checks pass / the code has been formatted using Black
• Tests have been added to verify that the new code works

[claudegel]

Look like there is a bug with some quirks:
AssertionError: Manufacturer-model pair 'Sinope Technologies' 'RM3250ZB' is shared by multiple quirks: ['/home/runner/work/zha-device-handlers/zha-device-handlers/zhaquirks/sinope/switch.py:444', '/home/runner/work/zha-device-handlers/zha-device-handlers/zhaquirks/sinope/switch.py:399']
The to devices have the same SKU RM3250ZB but there cluster are different.
Can I just rename one from QuirkBuilder(SINOPE, "RM3250ZB") to QuirkBuilder(SINOPE, "RM3250ZB-V2") to fix this. I have many devices like this and it was ok in the previous quirks.

[TheJulianJES]

Quirks v2 no longer match by the whole ep/cluster signature. They only match by manufacturer and model.

If you really need to distinguish two identically named devices depending on their cluster signature, I believe you should be able to do it something like this:

sinope_device_1_signature = {
    MODELS_INFO: [(SINOPE, "RM3250ZB")],
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha_p.PROFILE_ID,
            DEVICE_TYPE: zha_p.DeviceType.ON_OFF_OUTPUT,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                DeviceTemperature.cluster_id,
                Identify.cluster_id,
                Groups.cluster_id,
                Scenes.cluster_id,
                OnOff.cluster_id,
                Metering.cluster_id,
                ElectricalMeasurement.cluster_id,
                Diagnostic.cluster_id,
                SINOPE_MANUFACTURER_CLUSTER_ID,
            ],
            OUTPUT_CLUSTERS: [
                Identify.cluster_id,
                Groups.cluster_id,
                Ota.cluster_id,
            ],
        }
    },
}

# and when calling the v2 QuirkBuilder methods, add:
.filter(signature_matches(sinope_device_1_signature))
# ...

That should additionally match by signature, though you ideally don't want to do this.
The tests will likely still fail, as we haven't needed this behavior so far. If we need to go this route, I can adjust the tests to ignore quirks with a specified filter though.

If this is about preventing an entity to be created for one firmware version, there's no good way to do that at the moment.
Currently, v2 quirks entities are force-created, even if the attribute value cannot be read during pairing, for cases when the attribute isn't exposed on the device.
Ideally, we'd add an additional parameter that can be enabled to only create specified entities if the attribute is actually present in zigpy cache, so exposed by the device. This would work for older devices returning `None" and newer ones actually returning the attribute.

We do have v2 quirks that also expose the entity on older firmware versions though, even if it's not supported.
There's no big harm in that, but we'll try to fix that in the future.
Related issue: zigpy/zha#373

[TheJulianJES]

Can I just rename one from QuirkBuilder(SINOPE, "RM3250ZB") to QuirkBuilder(SINOPE, "RM3250ZB-V2") to fix this. I have many devices like this and it was ok in the previous quirks.

So, to answer that: No, if the model name of the device is "RM3250ZB", then the "RM3250ZB-V2" quirk wouldn't match it. Only the first one with "RM3250ZB" would match it.

[claudegel]

If I keep the two as QuirkBuilder(SINOPE, "RM3250ZB") it load correctly and I can see both device in ZHA. But it is shared-ci / Run tests Python 3.12 that fail with AssertionError

[TheJulianJES]

No, it's the same manufacturer and model. Only one of the quirks will apply. Keep both as RM3250ZB and remove one. Everything will seem work fine. Now, add that quirk back and remove the other RM3250ZB. Everything will still work fine.
It's because only one of the quirks will ever work.
I'd have to see if it's the one added first (i.e. further up in the file) or the one further below, but that doesn't really matter. Only one of them will apply if they have the same manufacturer and model.

v2 quirks only match by manufacturer and model and you're adding two quirks with the same manufacturer and model.
ZHA will only apply one of them, since they're meant for the exact same device.
The test just informs you that you're doing something that you likely don't want to do.

v1 quirks had the signature. v2 quirks only have manufacturer and model to match against by default, no signature.
The # SimpleDescriptor comments are just comments, they don't do anything.

[TheJulianJES]

The great thing about v2 quirks is that we don't need multiple quirk variants if firmware updates or other variants changed the device signature ever so slightly.
Like, to v2 quirks, it doesn't matter if there's an additional Time cluster on one of the endpoints after a firmware update.

For v1 quirks, you'd have to add another quirk, with a new signature and have two quirks in the end if there are different variants of devices (e.g. with different firmware).
With v2 quirks, you only have one quirk, as it only matches the device by the manufacturer and model name. The rest of the signature (endpoints and clusters) are completely ignored.

[claudegel]

Both device expose there signature for cluster but one device have more cluster than the old one:

    # <SimpleDescriptor(endpoint=1, profile=260,
    # device_type=2, device_version=0,
    # input_clusters=[0, 3, 4, 5, 6, 1794, 2820, 2821, 65281]
    # output_clusters=[3, 4, 25]>
    QuirkBuilder(SINOPE, "RM3250ZB")

    # <SimpleDescriptor(endpoint=1, profile=260,
    # device_type=2, device_version=0,
    # input_clusters=[0, 2, 3, 4, 5, 6, 1794, 2820, 2821, 65281]
    # output_clusters=[3, 4, 25]>
    QuirkBuilder(SINOPE, "RM3250ZB")

But they share the same QuirkBuilder(SINOPE, "RM3250ZB") and same device_version=0.

Can I put only the device quirks that have all cluster like the second one above. Does ZHA will load the first one above if one cluster is missing, 0x0002

[TheJulianJES]

You shouldn't be doing this in v2 quirks. Only add clusters if they're missing from the device signature by default.
This won't do anything, since they're most likely all already exposed by the device itself.

Like, all of the .adds(...) should be completely removed here.

[claudegel]

I'm doing this only in the light.py because I had to add line
.replaces_endpoint(1, device_type=zha_p.DeviceType.ON_OFF_LIGHT)
and this was removing all cluster that I didn't .replace

    .replaces(CustomDeviceTemperatureCluster)
    .replaces(LightManufacturerCluster)

Adding those lines fix the problem.
In Sinopé light DevicesType are reported wrong so I need to do that .replace to change the DeviceType. This work ok in V2.

[TheJulianJES]

To re-iterate on the differences vs v1 quirks and v2 quirks:
v1 quirks had a signature and replacement. If the signature matches a device, it'll present the replacement signature to ZHA. So, if you want to replace a cluster, you'd just replace it in the replacement signature, keeping everything the same.

For v2 quirks, it works nothing like that. They match by manufacturer and model only. Now, if I want to replace a cluster only, I just do this:

(
    QuirkBuilder("IKEA", "TRADFRI control outlet")
    .replaces(MyCustomCluster, endpoint_id=1) # the endpoint_id=1 is optional for ep 1, as 1 is the default
    .add_to_registry()
)

All other clusters on the device are left untouched. For example, all devices have to expose a Basic cluster on at least one endpoint, but we don't care about that here, as we don't need to modify it. So we don't do anything with that.
Only the MyCustomCluster is replaced with one that has further attribute definitions, for example.

The example IKEA control outlet above also has an OnOff cluster exposed by default. We don't touch it, because that basic functionality already works fine.

---

You can kind of consider v2 quirks as the difference of v1 quirks when comparing the signature and replacement.
All clusters that you don't care about are exactly the same in signature and replacement for v1 quirks.
So, for v2 quirks, you don't do anything with those clusters.

But if you want to replace/remove/add a cluster in a v1 quirk, you change that in the replacement.
Considering the difference between the signature and replacement, you'll then want to use v2 quirk equivalents like .replaces(cluster),.remove(cluster_id), or .adds(cluster).

Is that a better explanation?

[claudegel]

So I will keep only one copy of QuirkBuilder(SINOPE, "RM3250ZB") and make sure that it contain all .number, .enum, .sensor that are present in the other QuirkBuilder(SINOPE, "RM3250ZB").
I'll do it for all duplicate I have mostly in thermostat.py, switch.py and sensor.py.
What happens for the device that is missing one cluster ? The related .number, .sensor etc will be ignored or just grey out in the device card ?

[claudegel]

There is an import error in test_sinope.py. Is it you who fix this ?

Also there is a AssertionError: Translation key 'valve_countdown' is shared by quirks with different fallback names:
Do I need to pay Tuya :)

[claudegel]

Also where can I add missing french translation.

[claudegel]

Look like there is only the Valve countdown collision with Tuya that we need to fix. maybe I can change it to closing countdown. I hope nobody have used this one :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.00%. Comparing base (5556090) to head (3925782).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #4042      +/-   ##
==========================================
- Coverage   91.19%   91.00%   -0.20%     
==========================================
  Files         335      335              
  Lines       10886    10956      +70     
==========================================
+ Hits         9928     9971      +43     
- Misses        958      985      +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[claudegel]

@TheJulianJES There is one last test that do not work but this one I don't know how to fix. Can you help ?

[TheJulianJES]

The Codecov project check doesn't really matter.
I think it just fails, as the overall number of lines was drastically reduced with the v2 quirks, thus also causing the whole project coverage percentage to be lower, failing the check.

All lines of the v2 quirks are covered, so that's good.

[claudegel]

Thank you. check it and let me know if we can merge to V2

[claudegel]

I've found my last bug for dimmer, DM2500ZB. Now all entities are up and running
Thank you for your help

[claudegel]

@TheJulianJES , I've few questions:

• For number is it possible to force display as a number instead of a slider. I have possible value from 0 to 60.
• For number do we need to put a unit= ???, I've a number which is not in seconds, celsius or percent. Just no unit of measure. Can I omit unit= or it will use a default unit.
• for IAS 0x0002 ZoneStatus I have different values for different status. This attribute is not in the quirks and It is set as binary_sensor, on / off. Is it possible to make a select even if that attribute is uint16 in order to list the error message instead of a value. or anything else that work.

[claudegel]

Ok I've found how to make an enum out of t.uint16_t
I just put t.enum_factory(t.uint16_t, "manufacturer_specific") and it work.

class ZoneStatus(t.enum_factory(t.uint16_t, "manufacturer_specific")):
    """IAS zone status."""

    Ok = 0x0030
    Connector_1 = 0x0031
    Connector_2 = 0x0032
    Low_battery = 0x0038
    Connector_low_bat = 0x003A

Now can you tell me how to limit a value. For example I receive the value for battery_percentage_remaining which goes to 200% with new battery. How can I set the value red from the attribute not to go above 100%

[TheJulianJES]

For number is it possible to force display as a number instead of a slider. I have possible value from 0 to 60.

Not possible via quirks v2 entities, yet. There's a request for it here: zigpy/zha#364

For number do we need to put a unit= ???, I've a number which is not in seconds, celsius or percent. Just no unit of measure. Can I omit unit= or it will use a default unit.

You can omit the unit and no unit will be shown in HA.
Even if you want to use a unit that's not already built-in here, you can just define a custom string, e.g. unit="°C" (though that's built-in).

• for IAS 0x0002 ZoneStatus I have different values for different status. This attribute is not in the quirks and It is set as binary_sensor, on / off. Is it possible to make a select even if that attribute is uint16 in order to list the error message instead of a value. or anything else that work.

An IasZone entity should already be created by ZHA, though that will always trigger for bit 1 and bit 2.
If you need to disable the default entity to create your own entities, you can do something like this:

.prevent_default_entity_creation(endpoint_id=1, cluster_id=IasZone.cluster_id)  # disable ZHA entity
.binary_sensor(
    IasZone.AttributeDefs.zone_status.name,  # though you may want to use your own attribute definition here
    IasZone.cluster_id,
    unique_id_suffix="ias_1",  # to not conflict with unique ID of other entity, as only the attr name is used by default
    translation_key="leak_status",
    fallback_name="Leak status",
    attribute_converter=lambda x: x & 0b00000001,  # use only bit 1
)
.binary_sensor(
    IasZone.AttributeDefs.zone_status.name,  # though you may want to use your own attribute definition here
    IasZone.cluster_id,
    unique_id_suffix="ias_2",  # to not conflict with unique ID of other entity, as only the attr name is used by default
    translation_key="tamper_status",
    fallback_name="Tamper status",
    `,  # use only bit 2
)
.add_to_registry()

If you have multiple quirks v2 entities for the same attribute, as might be possible for zone_status, you'll need to add a unique_id_suffix to each entity like shown above to not conflict in HA, as it's normally just the attribute name, so it would be zone_status for both entities (but we specified a different unique_id_suffix above, so it's fine).

The same might also apply if you have the same quirks v2 entity for multiple endpoint ids, but I can't fully remember right now.

Above, I've also shown how you can use attribute_converter to only use some bits of the attribute value.

Ok I've found how to make an enum out of t.uint16_t I just put t.enum_factory(t.uint16_t, "manufacturer_specific") and it work.

class ZoneStatus(t.enum_factory(t.uint16_t, "manufacturer_specific")):
    """IAS zone status."""

    Ok = 0x0030
    Connector_1 = 0x0031
    Connector_2 = 0x0032
    Low_battery = 0x0038
    Connector_low_bat = 0x003A

Eh, I don't think you should/want to do that generally.
There's also a zcl_type you can set on attribute definitions, though there might still be issues when reporting needs to be set up for that. But you can have the zigpy (enum) type and the zcl_type being the integer type.

Like, the above way shows how to make separate binary sensors for that (might need to keep the type as an int?).
If you want it to be one read-only sensor entity, you can keep using .enum and just add entity_platform=EntityPlatform.SENSOR to use the sensor platform, instead of the default select platform.

Now can you tell me how to limit a value. For example I receive the value for battery_percentage_remaining which goes to 200% with new battery. How can I set the value red from the attribute not to go above 100%

ZHA already creates a default entity for battery_percentage_remaining, so you shouldn't need to create that on your own.

ZCL specifies that it's percentage = value / 2. So, 100 % is reported as value 200.
I think you should be able to add multiplier=0.5 on your sensor or number entity to divide the number by 2.

Otherwise, attribute_converter=lambda x: x / 2 should also be possible. You can also do more sophisticated calculations using attribute_converter (e.g. call a "proper method").

[TheJulianJES]

I've not had a proper look at everything changed in this PR, but if you have multiple quirks that use duplicated entities, you might be able to add a base quirk without any model info, and then clone that base quirk multiple times with added model info and more custom entities.

See this example, but try to ignore all the Tuya stuff:

zha-device-handlers/zhaquirks/tuya/tuya_valve.py

Lines 226 to 351 in 5556090

	gx02_base_quirk = (
	TuyaQuirkBuilder()
	.tuya_battery(dp_id=108, battery_type=BatterySize.AA, battery_qty=4)
	.tuya_metering(dp_id=111)
	.tuya_onoff(dp_id=2)
	.tuya_number(
	dp_id=103,
	attribute_name="irrigation_cycles",
	type=t.uint8_t,
	min_value=0,
	max_value=100,
	step=1,
	translation_key="irrigation_cycles",
	fallback_name="Irrigation cycles",
	)
	.tuya_dp_attribute(
	dp_id=1,
	attribute_name="irrigation_mode",
	type=t.Bool,
	)
	.enum(
	attribute_name="irrigation_mode",
	cluster_id=TUYA_CLUSTER_ID,
	enum_class=GiexIrrigationMode,
	translation_key="irrigation_mode",
	fallback_name="Irrigation mode",
	)
	.tuya_enum(
	dp_id=107,
	attribute_name="weather_delay",
	enum_class=GiexIrrigationWeatherDelay,
	translation_key="weather_delay",
	fallback_name="Weather delay",
	initially_disabled=True,
	)
	.tuya_sensor(
	dp_id=114,
	attribute_name="irrigation_duration",
	type=t.uint32_t,
	converter=lambda x: giex_string_to_td(x),
	state_class=SensorStateClass.MEASUREMENT,
	device_class=SensorDeviceClass.DURATION,
	unit=UnitOfTime.SECONDS,
	translation_key="irrigation_duration",
	fallback_name="Irrigation duration",
	)
	.tuya_sensor(
	dp_id=101,
	attribute_name="irrigation_start_time",
	type=t.CharacterString,
	converter=lambda x: giex_string_to_ts(x),
	device_class=SensorDeviceClass.TIMESTAMP,
	translation_key="irrigation_start_time",
	fallback_name="Irrigation start time",
	)
	.tuya_sensor(
	dp_id=102,
	attribute_name="irrigation_end_time",
	type=t.CharacterString,
	converter=lambda x: giex_string_to_ts(x),
	device_class=SensorDeviceClass.TIMESTAMP,
	translation_key="irrigation_end_time",
	fallback_name="Irrigation end time",
	)
	.skip_configuration()
	)
	(
	gx02_base_quirk.clone()
	.applies_to("_TZE200_sh1btabb", "TS0601")
	.tuya_number(
	dp_id=104,
	attribute_name="irrigation_target",
	type=t.uint32_t,
	min_value=0,
	max_value=GIEX_24HRS_AS_MIN,
	step=1,
	translation_key="irrigation_target",
	fallback_name="Irrigation target",
	)
	.tuya_number(
	dp_id=105,
	attribute_name="irrigation_interval",
	min_value=0,
	type=t.uint32_t,
	max_value=GIEX_24HRS_AS_MIN,
	step=1,
	unit=UnitOfTime.MINUTES,
	translation_key="irrigation_interval",
	fallback_name="Irrigation interval",
	)
	.add_to_registry()
	)
	(
	gx02_base_quirk.clone()
	.applies_to("_TZE200_a7sghmms", "TS0601")
	.applies_to("_TZE204_a7sghmms", "TS0601")
	.applies_to("_TZE200_7ytb3h8u", "TS0601") # Giex GX02 Var 1
	.applies_to("_TZE204_7ytb3h8u", "TS0601") # Giex GX02 Var 1
	.applies_to("_TZE284_7ytb3h8u", "TS0601") # Giex GX02 Var 3
	.tuya_number(
	dp_id=104,
	attribute_name="irrigation_target",
	type=t.uint32_t,
	min_value=0,
	max_value=GIEX_12HRS_AS_SEC,
	step=1,
	translation_key="irrigation_target",
	fallback_name="Irrigation target",
	)
	.tuya_number(
	dp_id=105,
	attribute_name="irrigation_interval",
	type=t.uint32_t,
	min_value=0,
	max_value=GIEX_12HRS_AS_SEC,
	step=1,
	unit=UnitOfTime.SECONDS,
	translation_key="irrigation_interval",
	fallback_name="Irrigation interval",
	)
	.add_to_registry()
	)

gx02_base_quirk is the base quirk. Its entities will be cloned to the other two quirks below (the first for _TZE200_sh1btabb only and the second for _TZE200_a7sghmms and others).
That way, the entities added to the base gx02_base_quirk quirk don't have to be copied to the two other quirks.

Doing it this way isn't always helpful, but it might be for this PR (if you have very similar, but slightly differing devices).

[claudegel]

It there a way to express data like battery voltage as integer not float. 100 instead of 100.0

[puddly]

You can use suggested_display_precision=0 when creating a sensor. It defaults to 1.

[claudegel]

The sensor for battery_percentage_remaining is not created by my quirk but come from ZHA.
So I didn't create the sensor and I can't add suggested_display_precision=0. Another option?
But it's good to know for my quirks. Thanks