Skip to content

Commit 3959a60

Browse files
committed
fix
1 parent fe3d4ab commit 3959a60

File tree

3 files changed

+119
-242
lines changed

3 files changed

+119
-242
lines changed

docs/content/pypaimon/cli.md

Lines changed: 48 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ Output:
127127

128128
### Table Get
129129

130-
Get and display detailed schema information about a Paimon table.
130+
Get and display table schema information in JSON format. The output format is the same as the schema JSON format used in table create, making it easy to export and reuse table schemas.
131131

132132
```shell
133133
paimon table get DATABASE.TABLE
@@ -140,107 +140,67 @@ paimon table get mydb.users
140140
```
141141

142142
Output:
143+
```json
144+
{
145+
"fields": [
146+
{"id": 0, "name": "user_id", "type": "BIGINT"},
147+
{"id": 1, "name": "username", "type": "STRING"},
148+
{"id": 2, "name": "email", "type": "STRING"},
149+
{"id": 3, "name": "age", "type": "INT"},
150+
{"id": 4, "name": "city", "type": "STRING"},
151+
{"id": 5, "name": "created_at", "type": "TIMESTAMP"},
152+
{"id": 6, "name": "is_active", "type": "BOOLEAN"}
153+
],
154+
"partitionKeys": ["city"],
155+
"primaryKeys": ["user_id"],
156+
"options": {
157+
"bucket": "4",
158+
"changelog-producer": "input"
159+
},
160+
"comment": "User information table"
161+
}
143162
```
144-
================================================================================
145-
Table: mydb.users
146-
================================================================================
147-
148-
Schema ID: 0
149-
150-
Comment: User information table
151163

152-
================================================================================
153-
Fields:
154-
================================================================================
155-
ID Name Type Nullable Description
156-
----- ------------------ -------------------- --------- --------------------------
157-
0 user_id BIGINT YES
158-
1 username STRING YES
159-
2 email STRING YES
160-
3 age INT YES
161-
4 city STRING YES
162-
5 created_at TIMESTAMP(6) YES
163-
6 is_active BOOLEAN YES
164-
165-
================================================================================
166-
Partition Keys: city
167-
168-
================================================================================
169-
Primary Keys: user_id
170-
171-
================================================================================
172-
Table Options:
173-
================================================================================
174-
bucket = 4
175-
changelog-producer = input
176-
```
164+
**Note:** The output JSON can be saved to a file and used directly with the `table create` command to recreate the table structure.
177165

178166
### Table Create
179167

180-
Create a new Paimon table with a schema defined in a JSON or YAML file.
168+
Create a new Paimon table with a schema defined in a JSON file. The schema JSON format is the same as the output from `table get`, ensuring consistency and easy schema reuse.
181169

182170
```shell
183-
paimon table create DATABASE.TABLE --schema-file SCHEMA_FILE
171+
paimon table create DATABASE.TABLE --schema SCHEMA_FILE
184172
```
185173

186174
**Options:**
187175

188-
- `--schema-file, -s`: Path to schema definition file (JSON or YAML) - **Required**
176+
- `--schema, -s`: Path to schema JSON file - **Required**
189177
- `--ignore-if-exists, -i`: Do not raise error if table already exists
190178

191-
The schema file should be a JSON or YAML file with the following structure:
179+
The schema JSON file follows the same format as output by `table get`:
192180

193-
**JSON Example (`schema.json`):**
181+
**Field Properties:**
194182

195-
```json
196-
{
197-
"fields": [
198-
{"name": "user_id", "type": "BIGINT"},
199-
{"name": "username", "type": "STRING"},
200-
{"name": "email", "type": "STRING"},
201-
{"name": "age", "type": "INT"},
202-
{"name": "city", "type": "STRING"},
203-
{"name": "created_at", "type": "TIMESTAMP"},
204-
{"name": "is_active", "type": "BOOLEAN"}
205-
],
206-
"partition_keys": ["city"],
207-
"primary_keys": ["user_id"],
208-
"options": {
209-
"bucket": "4",
210-
"changelog-producer": "input"
211-
},
212-
"comment": "User information table"
213-
}
214-
```
183+
- `id`: Field ID (integer, typically starts from 0) - **Required**
184+
- `name`: Field name - **Required**
185+
- `type`: Field data type (e.g., `INT`, `BIGINT`, `STRING`, `TIMESTAMP`, `DECIMAL(10,2)`) - **Required**
186+
- `description`: Optional field description
215187

216-
**YAML Example (`schema.yaml`):**
188+
**Schema Properties:**
217189

218-
```yaml
219-
fields:
220-
- name: user_id
221-
type: BIGINT
222-
- name: username
223-
type: STRING
224-
- name: email
225-
type: STRING
226-
- name: age
227-
type: INT
228-
- name: city
229-
type: STRING
230-
- name: created_at
231-
type: TIMESTAMP
232-
- name: is_active
233-
type: BOOLEAN
234-
235-
partition_keys:
236-
- city
237-
238-
primary_keys:
239-
- user_id
240-
241-
options:
242-
bucket: "4"
243-
changelog-producer: input
244-
245-
comment: User information table
246-
```
190+
- `fields`: List of field definitions - **Required**
191+
- `partitionKeys`: List of partition key column names
192+
- `primaryKeys`: List of primary key column names
193+
- `options`: Table options as key-value pairs
194+
- `comment`: Table comment
195+
196+
**Example Workflow:**
197+
198+
1. Export schema from an existing table:
199+
```shell
200+
paimon table get mydb.users > users_schema.json
201+
```
202+
203+
2. Create a new table with the same schema:
204+
```shell
205+
paimon table create mydb.users_copy --schema users_schema.json
206+
```

paimon-python/pypaimon/cli/cli_table.py

Lines changed: 18 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
"""
2323

2424
import sys
25+
from pypaimon.common.json_util import JSON
2526

2627

2728
def cmd_table_read(args):
@@ -85,7 +86,7 @@ def cmd_table_get(args):
8586
"""
8687
Execute the 'table get' command.
8788
88-
Gets and displays table schema information.
89+
Gets and displays table schema information in JSON format.
8990
9091
Args:
9192
args: Parsed command line arguments.
@@ -116,52 +117,9 @@ def cmd_table_get(args):
116117
print(f"Error: Failed to get table '{table_identifier}': {e}", file=sys.stderr)
117118
sys.exit(1)
118119

119-
# Get table schema
120-
schema = table.table_schema
121-
122-
# Display table information
123-
print("=" * 80)
124-
print(f"Table: {database_name}.{table_name}")
125-
print("=" * 80)
126-
127-
# Display schema ID
128-
print(f"\nSchema ID: {schema.id}")
129-
130-
# Display comment if exists
131-
if schema.comment:
132-
print(f"\nComment: {schema.comment}")
133-
134-
# Display fields
135-
print(f"\n{'='*80}")
136-
print("Fields:")
137-
print(f"{'='*80}")
138-
print(f"{'ID':<5} {'Name':<18} {'Type':<20} {'Nullable':<9} {'Description'}")
139-
print(f"{'-'*5} {'-'*18} {'-'*20} {'-'*9} {'-'*26}")
140-
141-
for field in schema.fields:
142-
nullable = "YES" if field.type.nullable else "NO"
143-
description = field.description or ""
144-
print(f"{field.id:<5} {field.name:<18} {str(field.type):<20} {nullable:<9} {description}")
145-
146-
# Display partition keys
147-
if schema.partition_keys:
148-
print(f"\n{'='*80}")
149-
print(f"Partition Keys: {', '.join(schema.partition_keys)}")
150-
151-
# Display primary keys
152-
if schema.primary_keys:
153-
print(f"\n{'='*80}")
154-
print(f"Primary Keys: {', '.join(schema.primary_keys)}")
155-
156-
# Display options
157-
if schema.options:
158-
print(f"\n{'='*80}")
159-
print("Table Options:")
160-
print(f"{'='*80}")
161-
for key, value in sorted(schema.options.items()):
162-
print(f" {key:<40} = {value}")
163-
164-
print(f"\n{'='*80}\n")
120+
# Get table schema and convert to Schema, then output as JSON
121+
schema = table.table_schema.to_schema()
122+
print(JSON.to_json(schema, indent=2))
165123

166124

167125
def cmd_table_create(args):
@@ -174,7 +132,6 @@ def cmd_table_create(args):
174132
args: Parsed command line arguments.
175133
"""
176134
import json
177-
import yaml
178135
from pypaimon.cli.cli import load_catalog_config, create_catalog
179136
from pypaimon import Schema
180137

@@ -195,72 +152,29 @@ def cmd_table_create(args):
195152

196153
database_name, table_name = parts
197154

198-
# Load schema from file
199-
schema_file = args.schema_file
155+
# Load schema from JSON file
156+
schema_file = args.schema
200157
if not schema_file:
201-
print("Error: Schema file is required. Use --schema-file option.", file=sys.stderr)
158+
print("Error: Schema is required. Use --schema option.", file=sys.stderr)
202159
sys.exit(1)
203160

204161
try:
205162
with open(schema_file, 'r', encoding='utf-8') as f:
206-
if schema_file.endswith('.json'):
207-
schema_def = json.load(f)
208-
elif schema_file.endswith('.yaml') or schema_file.endswith('.yml'):
209-
schema_def = yaml.safe_load(f)
210-
else:
211-
print("Error: Unsupported schema file format. Use .json, .yaml, or .yml", file=sys.stderr)
212-
sys.exit(1)
163+
schema_json = f.read()
164+
paimon_schema = JSON.from_json(schema_json, Schema)
165+
213166
except FileNotFoundError:
214167
print(f"Error: Schema file not found: {schema_file}", file=sys.stderr)
215168
sys.exit(1)
169+
except json.JSONDecodeError as e:
170+
print(f"Error: Invalid JSON format in schema file: {e}", file=sys.stderr)
171+
sys.exit(1)
216172
except Exception as e:
217-
print(f"Error: Failed to read schema file: {e}", file=sys.stderr)
173+
print(f"Error: Failed to parse schema: {e}", file=sys.stderr)
218174
sys.exit(1)
219175

220-
# Parse schema definition
176+
# Create table
221177
try:
222-
# Validate required fields
223-
if 'fields' not in schema_def:
224-
print("Error: Schema must contain 'fields' section", file=sys.stderr)
225-
sys.exit(1)
226-
227-
# Build PyArrow schema from definition
228-
import pyarrow as pa
229-
pa_fields = []
230-
for field in schema_def['fields']:
231-
field_name = field.get('name')
232-
field_type = field.get('type')
233-
234-
if not field_name or not field_type:
235-
print("Error: Each field must have 'name' and 'type'", file=sys.stderr)
236-
sys.exit(1)
237-
238-
# Convert type string to PyArrow type
239-
from pypaimon.schema.data_types import DataTypeParser, PyarrowFieldParser
240-
241-
# Parse type string to Paimon DataType, then convert to PyArrow type
242-
paimon_type = DataTypeParser.parse_data_type(field_type)
243-
pa_type = PyarrowFieldParser.from_paimon_type(paimon_type)
244-
pa_fields.append(pa.field(field_name, pa_type))
245-
246-
pa_schema = pa.schema(pa_fields)
247-
248-
# Extract optional parameters
249-
partition_keys = schema_def.get('partition_keys', [])
250-
primary_keys = schema_def.get('primary_keys', [])
251-
options = schema_def.get('options', {})
252-
comment = schema_def.get('comment')
253-
254-
# Create Paimon schema
255-
paimon_schema = Schema.from_pyarrow_schema(
256-
pa_schema,
257-
partition_keys=partition_keys if partition_keys else None,
258-
primary_keys=primary_keys if primary_keys else None,
259-
options=options if options else None,
260-
comment=comment
261-
)
262-
263-
# Create table
264178
ignore_if_exists = args.ignore_if_exists
265179
catalog.create_table(f"{database_name}.{table_name}", paimon_schema, ignore_if_exists)
266180

@@ -309,9 +223,9 @@ def add_table_subcommands(table_parser):
309223
help='Table identifier in format: database.table'
310224
)
311225
create_parser.add_argument(
312-
'--schema-file', '-s',
226+
'--schema', '-s',
313227
required=True,
314-
help='Path to schema definition file (JSON or YAML)'
228+
help='Path to schema JSON file'
315229
)
316230
create_parser.add_argument(
317231
'--ignore-if-exists', '-i',

0 commit comments

Comments
 (0)