KeyError with custom_field_data on plugin

[geckigek]

KeyError: 'custom_field_data' When Adding VideoIPathNMOSMultiDeviceConfiguration

Environment

• Django Version: 4.2.10
• Python Version: 3.10.12
• NetBox Version: 3.7.3
• Plugin Name and Version: VIP Plugin (version not specified)

Description

I'm developing a NetBox plugin and encountered a KeyError while trying to add an instance of VideoIPathNMOSMultiDeviceConfiguration via the plugin's form. The issue seems to arise during the serialization process of custom fields when saving the form data.

Steps to Reproduce

1. Navigate to the add form for VideoIPathNMOSMultiDeviceConfiguration at http://127.0.0.1:8000/plugins/vip/videoipathnmosmultideviceconfiguration/add.
2. Fill out the form fields and submit.
3. The server responds with a 500 error, and the traceback points to a KeyError related to custom_field_data.

Expected Behavior

I expected the form to save the new VideoIPathNMOSMultiDeviceConfiguration instance without errors, and then redirect to the detail view of the newly created instance.

Actual Behavior

The server throws a KeyError exception, specifically mentioning that 'custom_field_data' is missing, during the attempt to save the new instance.

Traceback (Most Relevant Call Last)

File "/opt/netbox-3.7.3/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
...
File "/opt/netbox-3.7.3/netbox/utilities/utils.py", line 175, in serialize_object
data['custom_fields'] = data.pop('custom_field_data')
KeyError: 'custom_field_data'

Additional Information

I suspect the issue might be related to how custom fields are serialized or perhaps a mismatch in expectations between the plugin and NetBox's core functionalities regarding custom fields. I've looked into the NetBox documentation and source code but haven't found a clear indication of what might be causing this issue. The error I am getting only shows up on an inherited model. Does this maybe have to do something with my error?

Any insights, suggestions, or guidance on how to resolve this error would be greatly appreciated. Thank you in advance for your time and help.

Here are some code snippets from the source code:

models.py:

class VideoIPathDevicetype(NetBoxModel):
    name = models.CharField(max_length=100)
    devicetype = models.ForeignKey(
        to="dcim.DeviceType",
        on_delete=models.CASCADE,
        blank=False,
    )
    extradata = models.JSONField(blank=True, null=True)

    class Meta:
        ordering = ("name",)

    def __str__(self):
        return self.name

    def get_absolute_url(self):
        return reverse("plugins:vip_plugin:videoipathdevicetype", args=[self.pk])


class VideoIPathDeviceConfiguration(NetBoxModel):
    devicetype = models.ForeignKey(
        VideoIPathDevicetype, on_delete=models.CASCADE, related_name="configurations"
    )
    driver_name = models.CharField(max_length=100)
    organization = models.CharField(max_length=100, default="SWR")
    version = models.CharField(max_length=20, default="0.0")

    class Meta:
        ordering = ("driver_name",)

    def __str__(self):
        return self.driver_name

    def get_absolute_url(self):
        return reverse(
            "plugins:vip_plugin:videoipathdeviceconfiguration", args=[self.pk]
        )


class VideoIPathNMOSMultiDeviceConfiguration(VideoIPathDeviceConfiguration):
    always_enable_rtp = models.BooleanField(default=False)
    disable_rx_sdp = models.BooleanField(default=False)
    disable_rx_sdp_with_null = models.BooleanField(default=True)
    enable_bulk_config = models.BooleanField(default=False)
    enable_experimental_alarms = models.BooleanField(default=False)
    experimental_alarm_port = models.IntegerField(default=0)
    use_indices_in_ids = models.BooleanField(default=True)
    port = models.IntegerField(default=3210)

    class Meta:
        ordering = ("driver_name",)

    def __str__(self):
        return self.driver_name

    def get_absolute_url(self):
        return reverse(
            "plugins:vip_plugin:videoipathnmosmultideviceconfiguration", args=[self.pk]
        )

forms.py:

class VideoIPathDevicetypeForm(NetBoxModelForm):
    class Meta:
        model = VideoIPathDevicetype
        fields = ("name", "devicetype", "tags")


class VideoIPathDeviceConfigurationForm(NetBoxModelForm):
    class Meta:
        model = VideoIPathDeviceConfiguration
        fields = ("devicetype", "driver_name", "organization", "version")


class VideoIPathNMOSMultiDeviceConfigurationForm(VideoIPathDevicetypeForm):
    class Meta:
        model = VideoIPathNMOSMultiDeviceConfiguration
        fields = (
            "devicetype",
            "organization",
            "version",
            "driver_name",
            "always_enable_rtp",
            "disable_rx_sdp",
            "disable_rx_sdp_with_null",
            "enable_bulk_config",
            "enable_experimental_alarms",
            "experimental_alarm_port",
            "use_indices_in_ids",
            "port",
            "tags",
        )

views.py:

class VideoIPathNMOSMultiDeviceConfigurationEditView(generic.ObjectEditView):
    queryset = models.VideoIPathNMOSMultiDeviceConfiguration.objects.all()
    form = forms.VideoIPathNMOSMultiDeviceConfigurationForm

api/serializers.py:

class VideoIPathDevicetypeSerializer(NetBoxModelSerializer):
    url = serializers.HyperlinkedIdentityField(
        view_name="plugins-api:vip_plugin-api:videoipathdevicetype-detail"
    )
    devicetype = NestedDeviceTypeSerializer()

    class Meta:
        model = VideoIPathDevicetype
        fields = (
            "id",
            "url",
            "display",
            "name",
            "devicetype",
            "extradata",
            "tags",
            "custom_fields",
            "created",
            "last_updated",
        )


class VideoIPathDeviceConfigurationSerializer(NetBoxModelSerializer):
    url = serializers.HyperlinkedIdentityField(
        view_name="plugins-api:vip_plugin-api:videoipathdeviceconfiguration-detail"
    )
    devicetype = NestedVideoIPathDevicetypeSerializer()

    class Meta:
        model = VideoIPathDeviceConfiguration
        fields = (
            "id",
            "url",
            "display",
            "devicetype",
            "driver_name",
            "organization",
            "version",
            "custom_fields",
            "created",
            "last_updated",
        )


class VideoIPathNMOSMultiDeviceConfigurationSerializer(VideoIPathDeviceConfigurationSerializer):
    url = serializers.HyperlinkedIdentityField(
        view_name="plugins-api:vip_plugin-api:videoipathnmos

[candlerb]

class VideoIPathDevicetypeSerializer(NetBoxModelSerializer):
    url = serializers.HyperlinkedIdentityField(
        view_name="plugins-api:vip_plugin-api:videoipathdevicetype-detail"
    )
    devicetype = NestedDeviceTypeSerializer()

    class Meta:
        model = VideoIPathDevicetype
        fields = (
            "id",
            "url",
            "display",
            "name",
            "devicetype",
            "extradata",
            "tags",
            "custom_fields",    <<<<<
            "created",
            "last_updated",
        )

You explicitly asked the serializer to include custom_fields (ditto for VideoIPathDeviceConfigurationSerializer). But your model does not inherit from CustomFieldsMixin

[geckigek]

Hi,

thanks for your advise. I eliminated all "custom_fields" from the serializers.py, but the error is still occuring. And it is only occuring on the inherited VideoIPathNMOSDeviceConfiguration class. All the other serializers work with the "custom_fields" included in the fields on the serializer classes.

The inheritance on my models.py looks like this:

class VideoIPathDeviceConfiguration(NetBoxModel) -> inherited from NetBoxModel

class VideoIPathNMOSMultiDeviceConfiguration(VideoIPathDeviceConfiguration) -> inherited from VideoIPathDeviceConfiguration

Does the NetBoxModel not inherit from the CustomFieldsMixin Class or why do you mention that the model does not inherit from CustomFieldsMixin?

Best regards

Stefan

[candlerb]

Does the NetBoxModel not inherit from the CustomFieldsMixin Class

Sorry you're right, it does.

>>> import netbox.models
>>> netbox.models.NetBoxModel.__mro__
(<class 'netbox.models.NetBoxModel'>,
 <class 'netbox.models.NetBoxFeatureSet'>,
 <class 'netbox.models.features.BookmarksMixin'>,
 <class 'netbox.models.features.ChangeLoggingMixin'>,
 <class 'netbox.models.features.CloningMixin'>,
 <class 'netbox.models.features.CustomFieldsMixin'>,
 <class 'netbox.models.features.CustomLinksMixin'>,
 <class 'netbox.models.features.CustomValidationMixin'>,
 <class 'netbox.models.features.ExportTemplatesMixin'>,
 <class 'netbox.models.features.JournalingMixin'>,
 <class 'netbox.models.features.TagsMixin'>,
 <class 'netbox.models.features.EventRulesMixin'>,
 <class 'django.db.models.base.Model'>,
 <class 'django.db.models.utils.AltersData'>,
 <class 'object'>)

[candlerb]

Then I suspect it's because you've subclassed VideoIPathNMOSMultiDeviceConfiguration from VideoIPathDeviceConfiguration.

This isn't something you'd normally do in Django. For a start, which table will each type of object reside in? Either you will have multi-table inheritance, which is generally a bad thing, or you're trying to do some sort of single-table polymorphism which probably requires a plugin.

You can see that Netbox itself uses mixins, or abstract base classes: if you're looking to share code or attributes between two models, that would be the best way to go about it IMO.

[divad1196]

"Generally a bad thing" is based on the short opinion of a random guy. Of course, there are downsides and performance is probably the worst one, but this is not necessarily an issue, typically in my case.
Without going in details I personally think that this guy is wrong.

I searched for "polymorphic django model" and stubble upon this plugin before finding out about multi table inheritance. The issue with this plugin is that I still need to redefine Netbox's Primary model.

[geckigek]

Thanks for your help. A Mixin class did the job!

[Sundsja]

Hi, I am working as a plugin developer too and ran into the same exact issue. If I understood correctly, Netbox doesn't block using multi-table inheritance but it is not the recommended way to go?

More spesifically: I would have to create a Mixin class to handle the 'custom_field_data' serializer issue on the model that is inheriting from the parent class that inherits from NetboxModel?

[satyam-pw]

is your issue resolved?

[Sundsja]

No it's not. The only issue is that I would like to know if this type of solution is 'blocked' by Netbox.