Skip to content

catalog.md

Latest commit

 

History

History
334 lines (239 loc) · 7.23 KB

catalog.md

File metadata and controls

334 lines (239 loc) · 7.23 KB

Catalog Operations

This guide covers catalog operations including listing, retrieving, and navigating the Dremio catalog.

Commands

List Catalog

List all items in the catalog.

alt-dremio-cli catalog list [OPTIONS]

Options:

  • --include TEXT - Include additional fields (e.g., permissions, datasetCount)

Examples:

# List all catalog items
alt-dremio-cli catalog list

# List with permissions
alt-dremio-cli catalog list --include permissions

# List with dataset count
alt-dremio-cli catalog list --include datasetCount

# JSON output
alt-dremio-cli --output json catalog list

# Use specific profile
alt-dremio-cli --profile software catalog list

Get Catalog Item by ID

Retrieve a specific catalog item by its ID.

alt-dremio-cli catalog get <ITEM_ID> [OPTIONS]

Arguments:

  • ITEM_ID - The catalog item ID (UUID)

Options:

  • --include TEXT - Include additional fields

Examples:

# Get catalog item
alt-dremio-cli catalog get 4cc92138-34e8-4c84-ad03-abfb23b6d5f3

# Get with SQL definition
alt-dremio-cli catalog get 4cc92138-34e8-4c84-ad03-abfb23b6d5f3 --include sql

# Get with permissions
alt-dremio-cli catalog get 4cc92138-34e8-4c84-ad03-abfb23b6d5f3 --include permissions

# YAML output
alt-dremio-cli --output yaml catalog get 4cc92138-34e8-4c84-ad03-abfb23b6d5f3

Get Catalog Item by Path

Retrieve a catalog item by its path.

alt-dremio-cli catalog get-by-path <PATH> [OPTIONS]

Arguments:

  • PATH - The catalog path (dot-separated or slash-separated)

Options:

  • --include TEXT - Include additional fields

Examples:

# Get by dot-separated path
alt-dremio-cli catalog get-by-path "MySpace.MyTable"

# Get by slash-separated path
alt-dremio-cli catalog get-by-path "MySpace/MyFolder/MyView"

# Cloud: source.namespace.object
alt-dremio-cli catalog get-by-path "evangelism-2026.testing.my_table"

# Software: space.object or catalog.namespace.object
alt-dremio-cli catalog get-by-path "Analytics.sales_data"
alt-dremio-cli catalog get-by-path "dremio-catalog.alexmerced.testing"

# With additional fields
alt-dremio-cli catalog get-by-path "MySpace.MyView" --include sql

Scenarios

Exploring the Catalog

# 1. List all top-level items
alt-dremio-cli catalog list

# 2. Find a specific space or source
alt-dremio-cli catalog list | grep "MySpace"

# 3. Get details about a space
alt-dremio-cli catalog get-by-path "MySpace"

# 4. Explore nested items
alt-dremio-cli catalog get-by-path "MySpace/Reports"

Finding Datasets

# List all items with dataset counts
alt-dremio-cli catalog list --include datasetCount

# Get specific dataset
alt-dremio-cli catalog get-by-path "Sales.customers"

# Check dataset permissions
alt-dremio-cli catalog get-by-path "Sales.customers" --include permissions

Working with Views

# Get view definition
alt-dremio-cli catalog get-by-path "Analytics.monthly_summary" --include sql

# Get view metadata
alt-dremio-cli --output json catalog get-by-path "Analytics.monthly_summary"

Cross-Environment Comparison

# Compare catalog between environments
alt-dremio-cli --profile dev catalog list > dev_catalog.json
alt-dremio-cli --profile prod catalog list > prod_catalog.json
diff dev_catalog.json prod_catalog.json

Output Formats

Table (Default)

alt-dremio-cli catalog list

Output:

┌────────────────────┬──────┬─────────────┬──────────┐
│ ID                 │ Path │ Type        │ Created  │
├────────────────────┼──────┼─────────────┼──────────┤
│ abc-123-def-456    │ ...  │ SPACE       │ 2024-... │
│ xyz-789-ghi-012    │ ...  │ SOURCE      │ 2024-... │
└────────────────────┴──────┴─────────────┴──────────┘

JSON

alt-dremio-cli --output json catalog list

Output:

{
  "data": [
    {
      "id": "abc-123-def-456",
      "path": ["MySpace"],
      "type": "CONTAINER",
      "containerType": "SPACE",
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ]
}

YAML

alt-dremio-cli --output yaml catalog list

Output:

data:
  - id: abc-123-def-456
    path:
      - MySpace
    type: CONTAINER
    containerType: SPACE
    createdAt: '2024-01-01T00:00:00Z'

Path Formats

Cloud

source.namespace.object

Examples:

  • evangelism-2026.testing.my_table
  • my-s3-source.data.customers

Software

space.object
catalog.namespace.object

Examples:

  • Analytics.sales_data
  • dremio-catalog.alexmerced.testing
  • @user@company.com.my_view

Common Use Cases

1. Inventory Management

# Export full catalog inventory
alt-dremio-cli --output json catalog list > catalog_inventory.json

# Count items by type
alt-dremio-cli --output json catalog list | jq '[.data[] | .containerType] | group_by(.) | map({type: .[0], count: length})'

2. Finding Specific Items

# Find all spaces
alt-dremio-cli --output json catalog list | jq '.data[] | select(.containerType == "SPACE")'

# Find all sources
alt-dremio-cli --output json catalog list | jq '.data[] | select(.containerType == "SOURCE")'

# Find all views
alt-dremio-cli --output json catalog list | jq '.data[] | select(.type == "VIRTUAL_DATASET")'

3. Validation

# Verify item exists
alt-dremio-cli catalog get-by-path "MySpace.MyTable" && echo "Exists" || echo "Not found"

# Check if path is accessible
alt-dremio-cli catalog get-by-path "Sales.customers" --include permissions

4. Migration Planning

# List all items in source environment
alt-dremio-cli --profile source catalog list --include datasetCount > source_catalog.json

# List all items in target environment
alt-dremio-cli --profile target catalog list --include datasetCount > target_catalog.json

# Compare and plan migration
diff source_catalog.json target_catalog.json

Tips

  1. Use JSON output for scripting:

    alt-dremio-cli --output json catalog list | jq '.data[] | .path'

  2. Filter results with grep:

    alt-dremio-cli catalog list | grep "Analytics"

  3. Save catalog snapshots:

    alt-dremio-cli --output json catalog list > catalog_$(date +%Y%m%d).json

  4. Check permissions before operations:

    alt-dremio-cli catalog get-by-path "MySpace.MyTable" --include permissions

Error Handling

Item Not Found

$ alt-dremio-cli catalog get-by-path "NonExistent.Table"
Error: Resource not found

Solution: Verify the path exists:

alt-dremio-cli catalog list | grep "NonExistent"

Permission Denied

$ alt-dremio-cli catalog get abc-123
Error: Access forbidden

Solution: Check your profile has appropriate permissions.

Invalid Path Format

$ alt-dremio-cli catalog get-by-path "Invalid Path With Spaces"
Error: Invalid path format

Solution: Use proper path separators:

alt-dremio-cli catalog get-by-path "Space.Folder.Object"