feat: Add support for Spicepod version v2 (spiceai#9583)

* feat: Add support for Spicepod version v2

- Updated spicepod.schema.json to include "v2" in the version enum.
- Changed default version in SpicepodDefinition to v2.
- Updated init command to create spicepod.yaml with version v2.
- Added comprehensive tests for v1 and v2 versions, ensuring backward compatibility.
- Created new test fixtures for v2 spicepods and updated existing v1 fixtures.
- Implemented migration logic for deprecated fields from v1 to v2.
- Enhanced caching configuration for v2, including new fields and defaults.
- Updated documentation and comments to reflect changes in versioning and features.

* feat: Remove support for v1beta1 and update examples to v2

* fix: ensure newline at end of iceberg-duckdb[file].yaml

* Apply suggestion from @lukekim

* feat: Add support for Spicepod version v2

- Updated spicepod.schema.json to include "v2" in the version enum.
- Changed default version in SpicepodDefinition to v2.
- Updated init command to create spicepod.yaml with version v2.
- Added comprehensive tests for v1 and v2 versions, ensuring backward compatibility.
- Created new test fixtures for v2 spicepods and updated existing v1 fixtures.
- Implemented migration logic for deprecated fields from v1 to v2.
- Enhanced caching configuration for v2, including new fields and defaults.
- Updated documentation and comments to reflect changes in versioning and features.

* feat: Remove support for v1beta1 and update examples to v2

* fix: ensure newline at end of iceberg-duckdb[file].yaml

* Apply suggestion from @lukekim

* fix: Update JSON schema for v2 features and fix remaining v1beta1 reference

- Add read_write_create to AccessMode enum in schema
- Add partition_management to Scheduler and PartitionManagement  in schema
- Update test/adbc/python/spicepod.yaml from v1beta1 to v2

* style: Refactor code for consistency and readability in version tests

Author: lukekim

.schema/spicepod.schema.json

@@ -468,8 +468,8 @@
     "SpicepodVersion": {
       "type": "string",
       "enum": [
-        "v1beta1",
-        "v1"
+        "v1",
+        "v2"
       ]
     },
     "SpicepodKind": {
@@ -1216,13 +1216,50 @@
               "type": "null"
             }
           ]
+        },
+        "partition_management": {
+          "description": "Partition management configuration.",
+          "anyOf": [
+            {
+              "$ref": "#/$defs/PartitionManagement"
+            },
+            {
+              "type": "null"
+            }
+          ]
         }
       },
       "additionalProperties": false,
       "required": [
         "state_location"
       ]
     },
+    "PartitionManagement": {
+      "type": "object",
+      "properties": {
+        "interval": {
+          "type": "string",
+          "default": "30s"
+        },
+        "max_assignments_per_cycle": {
+          "type": "integer",
+          "format": "uint",
+          "minimum": 0,
+          "default": 100
+        },
+        "max_partitions_per_executor": {
+          "type": "integer",
+          "format": "uint",
+          "minimum": 0,
+          "default": 1000
+        },
+        "discovery_timeout": {
+          "type": "string",
+          "default": "60s"
+        }
+      },
+      "additionalProperties": false
+    },
     "Params": {
       "type": "object",
       "additionalProperties": {
@@ -1488,7 +1525,8 @@
       "type": "string",
       "enum": [
         "read",
-        "read_write"
+        "read_write",
+        "read_write_create"
       ]
     },
     "ComponentReference": {

bin/spice/src/commands/init.rs

@@ -112,7 +112,7 @@ fn determine_names(name_arg: &str) -> (String, String) {
 /// Create the spicepod.yaml content.
 fn create_spicepod_yaml(name: &str) -> String {
     format!(
-        r"version: v1
+        r"version: v2
 kind: Spicepod
 name: {name}
 "