chore: add helpers to monitor API parity

[mitchnielsen]

Summary

• (WIP) Adds a Go script that will identify APIs not implemented in our provider
• (coming) script to list all APIs
• (coming) script to list all Terraform resources and datasources
• adds a Python script that fetches the Prefect Cloud openapi.json + Terraform provider's catalog of resources/datasources from registry.terraform.io and creates a table of implementation statuses

Related to https://linear.app/prefect/issue/PLA-802/automate-the-process-of-checking-api-parity

Testing

Go script

$ go run scripts/compare.go
2025/02/26 17:52:30 API 'artifacts' has no Terraform resource equivalent
2025/02/26 17:52:30 API 'collections' has no Terraform resource equivalent
2025/02/26 17:52:30 API 'events' has no Terraform resource equivalent
2025/02/26 17:52:30 API 'slas' has no Terraform resource equivalent

Python script

$ uv run scripts/compare-and-output-markdown.py

Notes

Here are some scripts I had from my first implementation that outgrew bash a bit:

Click to expand scripts

#!/usr/bin/env bash

resources=$(find internal/provider/resources | grep -v 'test' | awk -F '\/' '{print $4}' | sed 's;\.go;;g')
datasources=$(find internal/provider/datasources | grep -v 'test' | awk -F '\/' '{print $4}' | sed 's;\.go;;g')
unique_resources='automation account_member'

all_resources=( ${resources} ${datasources} ${unique_resources} )

# Print out all the resources.
#
# Sort them and remove the ones that are just the plural equivalent,
# such as "work_pool" and "work_pools".
#
# Also remove provider-specific resources.
printf '%s\n' "${all_resources[@]}" | sort | uniq \
  | grep -v 'base_model' \
  | grep -v 'automation_' \
  | grep -v 'account_members' \
  | grep -v 'teams' \
  | grep -v 'work_pools' \
  | grep -v 'work_queues'

#!/usr/bin/env bash

# Accounts endpoints
account_endpoints=$(curl -s https://api.prefect.cloud/api/openapi.json | jq '.paths | keys' -r | grep '\/api\/accounts\/{account_id}' | grep -v 'workspaces' | sed 's;\/api\/accounts\/{account_id}\/;;g' \
  | awk -F '\/' '{print $1}' \
  | sed 's;\";;g' \
  | sed 's;,;;g' \
  | sed 's;\ ;;g' \
  | sed 's;s$;;g' \
  | sort \
  | uniq)

# Workspace-level endpoints
# ignored: ui/
# ignored: v2/concurrency_limits
workspace_endpoints=$(curl -s https://api.prefect.cloud/api/openapi.json | jq '.paths | keys' -r | grep '\/api\/accounts\/{account_id}' | grep 'workspaces' | sed 's;\/api\/accounts\/{account_id}\/workspaces\/{workspace_id}\/;;g' \
  | grep -v 'v2' \
  | grep -v 'ui' \
  | grep -v 'capabilities' \
  | grep -v 'saved_searches' \
  | grep -v 'bot_access' \
  | grep -v 'access' \
  | grep -v 'team_access' \
  | awk -F '\/' '{print $1}' \
  | sed 's;\";;g' \
  | sed 's;,;;g' \
  | sed 's;\ ;;g' \
  | sed 's;s$;;g' \
  | sort \
  | uniq)

ignored_endpoints='task_run_concurrency_limit global_concurrency_limit block_capabilities saved_searches team_access bot_access access'

all_endpoints=( ${account_endpoints} ${workspace_endpoints} ${ignored_endpoints} )

printf '%s\n' "${all_endpoints[@]}" | sort | uniq

#!/usr/bin/env bash

cleanup() {
  rm -f api-resources.txt provider-resources.txt
}

cleanup

./scripts/api-resources      > api-resources.txt
./scripts/provider-resources > provider-resources.txt

printf 'API resources                                                   Provider resources \n'
printf '==============================================================|============================\n'
diff \
  --color=always \
  --side-by-side \
  --suppress-common-lines \
  --ignore-space-change \
  --ignore-blank-lines \
  --ignore-all-space \
  api-resources.txt provider-resources.txt

cleanup

Requirements

General

• The contributing guide has been read
• Title follows the conventional commits format
• Body includes Closes <issue>, if available
• Relevant labels have been added
• Draft status is used until ready for review

Code-level changes

• Unit tests are added/updated
• Acceptance tests are added/updated (including import tests, when needed)

New or updated resource/datasource

• Documentation is added (generated by make docs from source code)
  - When applicable, provide a link back to the relevant page in the Prefect documentation site.
• For resources, the following are added:
  - Resource example under examples/resources/prefect_<name>/resource.tf
  - Import example under examples/resources/prefect_<name>/import.sh
• For datasources, the following is added:
  - Datasource example under examples/data-sources,resources>/prefect_<name>/data-source.tf

[parkedwards]

@mitchnielsen im going to keep your go/bash scripts, as based on our convo there might be a next iteration of this parity checking that does something more sophisticated, and I didn't want to drop the context of what you had been noodling on.