`ActiveModel::from_json` fails to deserialize TimeDateTimeWithTimezone fields to NotSet when field is missing from JSON payload

[PacificBird]

Description

This is a continuation of #3160, which was closed due to thinking that things were closed by SeaQL/sea-query#1093. This sea-query fixed the issues happening for optional values, but didn't remedy the deserialization problems for non-optional ones.

When deserializing an ActiveValue using from_json, if a TimeDateTimeWithTimezone field doesn't have a corresponding value in the JSON payload, somewhere along the way it defaults to a stringified version of the unix epoch, which causes the following error down the line:

Json Error: invalid type: string "'1970-01-01 00:00:00.000000 +00:00'", expected an `OffsetDateTime`

Steps to Reproduce

This is a minimal example of the exact problem I'm having:

        use sea_orm::entity::prelude::*;
        use serde::{Deserialize, Serialize};
        #[sea_orm::model]
        #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, DeriveEntityModel)]
        #[sea_orm(table_name = "example")]
        #[serde(rename_all = "UPPERCASE")]
        pub struct Model {
            #[sea_orm(primary_key, auto_increment = false)]
            #[serde(rename = "STATION_ID")]
            pub unit_id: i32,
            pub created_at: TimeDateTimeWithTimeZone,
        }

        impl ActiveModelBehavior for ActiveModel {}

        let payload = r#"{ "data": [
            {"STATION_ID": 100000}
        ]}"#;

        let result = serde_json::from_str::<serde_json::Value>(&payload)
            .unwrap()
            .get("data")
            .and_then(serde_json::Value::as_array)
            .unwrap()
            .into_iter()
            .filter_map(|val| {
                let x = <ActiveModel>::from_json(val.clone());
                dbg!(&x); // Json Error: invalid type: string "'1970-01-01 00:00:00.000000 +00:00'", expected an `OffsetDateTime`
                x.ok()
            })
            .collect::<Vec<_>>();

        assert!(!result.is_empty());

Expected Behavior

Any missing fields from the JSON payload should just deserialize as NotSet in the ActiveModel, which they did in 2.0.0-rc32.

[PacificBird]

Additionally, for "PlainDateTime"s, the error is instead that the string is "'-9999-01-01 00:00:00.000000", if that helps.

[Huliiiiii]

Most of the issues can be fixed by generate from_json impl in the derive macro instead of the default trait method. I haven't had time lately, anyone want to try it?

[PacificBird]

do we know what ended up breaking it? It used to work. I would think it would be easier to patch over that than overhaul the macro

[Huliiiiii]

In fact, implementing this with macros would be simpler. The current implementation is quite complicated, and fixing one bug may introduce another bug.

A quick fix should be to replace CommonSqlQueryBuilder.value_to_string with serde_json::to_value in sea_value_to_json_value.

[PacificBird]

After trying to play around with implementing from_json in the macros for a while, I am coming up upon a pretty serious level of complexity and redundant work that is already done with Models. It would require duplicating a lot of serde attribute parsing and I don't even know where to begin with the problem of making sure that both Model and ActiveModel respect custom deserializers. If there's a more elegant way to do JSON deserialization with ActiveModels it is much beyond my extreme inexperience with designing macros. I suspect doing it in a more elegant way would require a lot of thoughtful design and a big refactor. I will try to make a pull in sea-query to do the fix there. I am in pretty urgent need of this working!

[PacificBird]

Running into further issues regarding the fact that serialization and deserialization is often not round-trippable, and this entire system assumes that it is. Even if I fix the issue in sea-query, that doesn't change the fact that whatever JSON value serde_json::to_value produces in sea_value_to_json_value could be in an entirely wrong format for the Model if deserialize_with is used on that field. My only other idea was to make a private struct that is just a mirror of the Model where every field is wrapped in an extra Option, but that will also break custom deserializers. I am really confused as to how this functioned properly in earlier versions. I have to be missing something because this feels like an intractable problem...

[Huliiiiii]

Previous implementation had the serde(rename) issue, and fixing that cause the current issue.

[PacificBird]

This PR still doesn't fix the problem of fields with asymmetric ser-de loops, but as far as I can think of that is just not possible to solve. In any case that was always how the system worked and it's enough of an edge case that it's not a big priority to support. This pull fixes the date time issues.