IEventClientV1 fails to convert time value

[brobichaud]

We recently upgraded our k8s clusters to v1.18.4 and some code I've used for many months has suddenly started throwing errors trying to deserialize datetime values from IEventClientV1.List(). I'm running v2.3.13 of KubeClient and v12.0.3 of Newtonsoft.Json. So I'm thinking something in v1.18 of k8s is serving up times in a format that is a problem. Exception and stack below.

Any thoughts on a solution?

Error converting value 9/2/2020 8:04:22 PM to type 'KubeClient.Models.MicroTimeV1'. Path 'items[4].eventTime', line 1, position 6250.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(TextReader reader, Type objectType)
   at HTTPlease.Formatters.Json.JsonFormatter.ReadAsync(InputFormatterContext context, Stream stream)
   at HTTPlease.Formatters.ContentExtensions.<ReadAsAsync>d__1`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__14`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__12`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__11`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at KubeClient.ResourceClients.KubeResourceClient.<GetResourceList>d__21`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at KubeClient.ResourceClients.EventClientV1.<List>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Digimarc.Tools.DeployMgr.K8sControl.<GetClusterEvents>d__10.MoveNext() in C:\dev\madras\Tools\DeployMgr\Integrations\K8sControl.cs:line 139

Inner Exception:

Could not cast or convert from System.DateTime to KubeClient.Models.MicroTimeV1.
   at Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable(Object value, Type initialType, Type targetType)
   at Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast(Object initialValue, CultureInfo culture, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)

[tintoy]

Hmm - MicroTimeV1 probably needs some custom fields and logic (e.g. cast operators to / from DateTime) and a custom JsonConverter (to handle serialisation / deserialisation). From memory the OmniSharp LSP project had to do something similar for JSON-RPC. I think the model generator may already be doing something similar for at least one other type but I’m on my phone at the moment so can’t really check yet.

I can have a crack at that this weekend I think - shouldn’t be too difficult 🙂

[tintoy]

Hope to get this done on Sunday (tomorrow).

[tintoy]

Hey - I don’t have a working K8s install at the moment; are you able to try this out on the feature/micro-time-serialization branch?

[brobichaud]

Hmm, is there any chance you could give me a nuget to try?

[tintoy]

Let me dig up the MyGet feed URL :)

[tintoy]

https://www.myget.org/F/dotnet-kube-client/api/v3/index.json

[tintoy]

Package version:

https://travis-ci.org/github/tintoy/dotnet-kube-client/builds/724597945#L546

[brobichaud]

I'm still seeing the problem after building against these versions:

KubeClient v2.4.0-micros-time-seria0045
KubeClient.Extensions.KubeConfig v2.4.0-micros-time-seria0045

The error I'm seeing:

KubeClient.Models.Converters.MicroTimeV1Converter cannot deserialise a value of type 'System.Nullable`1[[KubeClient.Models.MicroTimeV1, KubeClient, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null]]'.
   at KubeClient.Models.Converters.MicroTimeV1Converter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(TextReader reader, Type objectType)
   at HTTPlease.Formatters.Json.JsonFormatter.ReadAsync(InputFormatterContext context, Stream stream)
   at HTTPlease.Formatters.ContentExtensions.<ReadAsAsync>d__1`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__14`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__12`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at HTTPlease.FormatterResponseExtensions.<ReadContentAsAsync>d__11`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at KubeClient.ResourceClients.KubeResourceClient.<GetResourceList>d__21`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at KubeClient.ResourceClients.EventClientV1.<List>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Digimarc.Tools.DeployMgr.K8sControl.<GetClusterEvents>d__10.MoveNext() in C:\dev\madras\Tools\DeployMgr\Integrations\K8sControl.cs:line 139

[tintoy]

Ah. I suspect it’s because the converter doesn’t recognise the nullable version of the struct. Ok, I’ll try updating that first thing tomorrow morning and add a test for deserialisation. Are you able to post sample event JSON I can use?

[brobichaud]

Indeed I can, here you go.

{
    "apiVersion": "v1",
    "items": [
        {
            "action": "Binding",
            "apiVersion": "v1",
            "eventTime": "2020-09-11T14:59:24.491964Z",
            "firstTimestamp": null,
            "involvedObject": {
                "apiVersion": "v1",
                "kind": "Pod",
                "name": "live-tillamook-admin-7d8fb47687-jrhmf",
                "namespace": "default",
                "resourceVersion": "4596853",
                "uid": "3570c016-21c9-4aed-ae4b-03aaa65e7fff"
            },
            "kind": "Event",
            "lastTimestamp": null,
            "message": "Successfully assigned default/live-tillamook-admin-7d8fb47687-jrhmf to akswin1000002",
            "metadata": {
                "creationTimestamp": "2020-09-11T14:59:24Z",
                "managedFields": [
                    {
                        "apiVersion": "events.k8s.io/v1beta1",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:action": {},
                            "f:deprecatedSource": {
                                "f:component": {}
                            },
                            "f:eventTime": {},
                            "f:note": {},
                            "f:reason": {},
                            "f:regarding": {
                                "f:apiVersion": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:namespace": {},
                                "f:resourceVersion": {},
                                "f:uid": {}
                            },
                            "f:reportingController": {},
                            "f:reportingInstance": {},
                            "f:type": {}
                        },
                        "manager": "kube-scheduler",
                        "operation": "Update",
                        "time": "2020-09-11T14:59:24Z"
                    }
                ],
                "name": "live-tillamook-admin-7d8fb47687-jrhmf.1633c2b27c4bec21",
                "namespace": "default",
                "resourceVersion": "4596860",
                "selfLink": "/api/v1/namespaces/default/events/live-tillamook-admin-7d8fb47687-jrhmf.1633c2b27c4bec21",
                "uid": "15209988-a1eb-484d-8001-1c7f7712b3c1"
            },
            "reason": "Scheduled",
            "reportingComponent": "default-scheduler",
            "reportingInstance": "default-scheduler-kube-scheduler-65bfb7b57f-4sfxp",
            "source": {
                "component": "default-scheduler"
            },
            "type": "Normal"
        },
        {
            "apiVersion": "v1",
            "count": 1,
            "eventTime": null,
            "firstTimestamp": "2020-09-11T14:59:25Z",
            "involvedObject": {
                "apiVersion": "v1",
                "fieldPath": "spec.containers{tillamook-admin}",
                "kind": "Pod",
                "name": "live-tillamook-admin-7d8fb47687-jrhmf",
                "namespace": "default",
                "resourceVersion": "4596857",
                "uid": "3570c016-21c9-4aed-ae4b-03aaa65e7fff"
            },
            "kind": "Event",
            "lastTimestamp": "2020-09-11T14:59:25Z",
            "message": "Container image \"myrepo.docker.com/tillamook/admin/master:master.23\" already present on machine",
            "metadata": {
                "creationTimestamp": "2020-09-11T14:59:25Z",
                "managedFields": [
                    {
                        "apiVersion": "v1",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:count": {},
                            "f:firstTimestamp": {},
                            "f:involvedObject": {
                                "f:apiVersion": {},
                                "f:fieldPath": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:namespace": {},
                                "f:resourceVersion": {},
                                "f:uid": {}
                            },
                            "f:lastTimestamp": {},
                            "f:message": {},
                            "f:reason": {},
                            "f:source": {
                                "f:component": {},
                                "f:host": {}
                            },
                            "f:type": {}
                        },
                        "manager": "kubelet.exe",
                        "operation": "Update",
                        "time": "2020-09-11T14:59:25Z"
                    }
                ],
                "name": "live-tillamook-admin-7d8fb47687-jrhmf.1633c2b2d5306f44",
                "namespace": "default",
                "resourceVersion": "4596864",
                "selfLink": "/api/v1/namespaces/default/events/live-tillamook-admin-7d8fb47687-jrhmf.1633c2b2d5306f44",
                "uid": "4c2ee673-54b5-452e-9fcb-85a771d07316"
            },
            "reason": "Pulled",
            "reportingComponent": "",
            "reportingInstance": "",
            "source": {
                "component": "kubelet",
                "host": "akswin1000002"
            },
            "type": "Normal"
        },
        {
            "apiVersion": "v1",
            "count": 1,
            "eventTime": null,
            "firstTimestamp": "2020-09-11T14:59:26Z",
            "involvedObject": {
                "apiVersion": "v1",
                "fieldPath": "spec.containers{tillamook-admin}",
                "kind": "Pod",
                "name": "live-tillamook-admin-7d8fb47687-jrhmf",
                "namespace": "default",
                "resourceVersion": "4596857",
                "uid": "3570c016-21c9-4aed-ae4b-03aaa65e7fff"
            },
            "kind": "Event",
            "lastTimestamp": "2020-09-11T14:59:26Z",
            "message": "Created container tillamook-admin",
            "metadata": {
                "creationTimestamp": "2020-09-11T14:59:26Z",
                "managedFields": [
                    {
                        "apiVersion": "v1",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:count": {},
                            "f:firstTimestamp": {},
                            "f:involvedObject": {
                                "f:apiVersion": {},
                                "f:fieldPath": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:namespace": {},
                                "f:resourceVersion": {},
                                "f:uid": {}
                            },
                            "f:lastTimestamp": {},
                            "f:message": {},
                            "f:reason": {},
                            "f:source": {
                                "f:component": {},
                                "f:host": {}
                            },
                            "f:type": {}
                        },
                        "manager": "kubelet.exe",
                        "operation": "Update",
                        "time": "2020-09-11T14:59:26Z"
                    }
                ],
                "name": "live-tillamook-admin-7d8fb47687-jrhmf.1633c2b2d7097b44",
                "namespace": "default",
                "resourceVersion": "4596865",
                "selfLink": "/api/v1/namespaces/default/events/live-tillamook-admin-7d8fb47687-jrhmf.1633c2b2d7097b44",
                "uid": "bab42137-f33e-4cb4-9e09-1da24cf5c4b7"
            },
            "reason": "Created",
            "reportingComponent": "",
            "reportingInstance": "",
            "source": {
                "component": "kubelet",
                "host": "akswin1000002"
            },
            "type": "Normal"
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

[tintoy]

Thanks! Ok, have added a working test that verifies it can deserialise both nullable and non-nullable versions of MicroTimeV1.

https://travis-ci.org/github/tintoy/dotnet-kube-client/builds/726466033

(had to bypass a bit of yak-shaving along the way 😉)

[brobichaud]

Excellent, that's seems to have fixed things. I am now able to view events as expected. Thank you so much!

Can you let me know when this is available in the mainline release of the nugets?

[tintoy]

Great! Will do :)