[SPARK-60108][SQL] Scope docs to from_json, fix PERMISSIVE wording, test container values

Author: srielau

docs/sql-migration-guide.md

@@ -28,7 +28,7 @@ license: |
 
 ## Upgrading from Spark SQL 4.3 to 4.4
 
-- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is true, JSON object names used as `MAP<CHAR(n), _>` or `MAP<VARCHAR(n), _>` keys in `from_json` and the JSON datasource are length-checked without padding or trimming. A `CHAR(n)` key must already be exactly `n` characters, and a `VARCHAR(n)` key must already be at most `n` characters. A mismatched key turns the row into a bad record: in `PERMISSIVE` mode it is dropped (the output is `null`), while in `FAILFAST` mode parsing fails with `UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY` (SQLSTATE `0A000`) as the cause of `MALFORMED_RECORD_IN_PARSING`. Duplicate names are kept, exactly as for STRING keys; `spark.sql.mapKeyDedupPolicy` is not applied.
+- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is true, `from_json` length-checks JSON object names used as `MAP<CHAR(n), _>` or `MAP<VARCHAR(n), _>` keys without padding or trimming them. A `CHAR(n)` key must already be exactly `n` characters, and a `VARCHAR(n)` key must already be at most `n` characters; duplicate names are kept, exactly as for STRING keys, and `spark.sql.mapKeyDedupPolicy` is not applied. A key that fails the check makes the enclosing map a bad record: in `PERMISSIVE` mode the map is set to `null` while sibling fields are preserved (the whole `from_json` result is `null` only when the map is the top-level type), and in `FAILFAST` mode parsing fails with `UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY` (SQLSTATE `0A000`) as the cause of `MALFORMED_RECORD_IN_PARSING`. The same parser check runs when reading JSON files with a user-specified CHAR/VARCHAR reader schema. Reads whose CHAR/VARCHAR keys are handled read-side instead (for example catalog tables) are unchanged: there keys are padded, `spark.sql.mapKeyDedupPolicy` is applied, and an over-long key raises `EXCEED_LIMIT_LENGTH`.
 - Since Spark 4.4, when `spark.sql.preserveCharVarcharTypeInfo` is true and `spark.sql.charVarchar.standardSemantics.enabled` is false, ORC reads that apply a CHAR/VARCHAR schema over STRING storage return the stored values without ORC truncation, matching Parquet. Previously the ORC reader requested `char(n)`/`varchar(n)` and truncated STRING-stored values to `n`. Read-side length checks (`EXCEED_LIMIT_LENGTH`) apply only when `spark.sql.charVarchar.standardSemantics.enabled` is true.
 - Since Spark 4.4, the options maps passed to `from_csv`, `to_csv`, `schema_of_csv`, `from_json`, `to_json`, `schema_of_json`, `from_xml`, `to_xml`, and `schema_of_xml` must be foldable after replacing `RuntimeReplaceable` expressions. Previously, Spark evaluated non-foldable options during analysis, which allowed some constant expressions but could fail with an internal error or incorrectly evaluate row-dependent expressions. To allow deterministic and row-independent non-foldable options, set `spark.sql.legacy.allowNonFoldableOptions` to `true`. Row-dependent, unevaluable, and nondeterministic options are always rejected.
 - Since Spark 4.4, when an already-analyzed Data Source V2 query is refreshed after a compatible schema change, connectors can return more data columns from the current table schema in `Scan.readSchema()` than requested by `SupportsPushDownRequiredColumns.pruneColumns`. Previously, this partial pruning could fail planning because the scan reported columns absent from the analyzed relation output.

sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala

@@ -3235,6 +3235,32 @@ class BasicCharVarcharTestSuite extends SharedSparkSession {
         }
       }
 
+      // A rejected key whose value is an object or array exercises skipChildren(): the entire
+      // nested value is consumed so the map is nulled and the sibling `tail` field survives, and
+      // in FAILFAST the key is still surfaced. This holds in both partial-results modes.
+      Seq(true, false).foreach { partial =>
+        withSQLConf(SQLConf.JSON_ENABLE_PARTIAL_RESULTS.key -> partial.toString) {
+          checkAnswer(
+            sql("""SELECT from_json('{"m":{"a":{"x":1}},"tail":2}',
+              |  'm MAP<CHAR(3), STRUCT<x: INT>>, tail INT')""".stripMargin),
+            Row(Row(null, 2)))
+          checkAnswer(
+            sql("""SELECT from_json('{"m":{"a":[1,2,3]},"tail":2}',
+              |  'm MAP<CHAR(3), ARRAY<INT>>, tail INT')""".stripMargin),
+            Row(Row(null, 2)))
+          assertUnsupportedJsonMapKey(
+            """SELECT from_json('{"a":{"x":1}}',
+              |  'MAP<CHAR(3), STRUCT<x: INT>>', map('mode', 'FAILFAST'))""".stripMargin,
+            key = "a",
+            dataTypeSql = "CHAR(3)")
+          assertUnsupportedJsonMapKey(
+            """SELECT from_json('{"a":[1,2,3]}',
+              |  'MAP<CHAR(3), ARRAY<INT>>', map('mode', 'FAILFAST'))""".stripMargin,
+            key = "a",
+            dataTypeSql = "CHAR(3)")
+        }
+      }
+
       // multiLine + streaming top-level array: a bad key in one element must not add spurious
       // rows. Two elements in, two rows out, with the bad element reported in _corrupt_record.
       withTempPath { file =>