diff --git a/docs/data/analytics/hubble/analyst-guide/queries-for-horizon-like-data.mdx b/docs/data/analytics/hubble/analyst-guide/queries-for-horizon-like-data.mdx index 980c981c30..e01602388a 100644 --- a/docs/data/analytics/hubble/analyst-guide/queries-for-horizon-like-data.mdx +++ b/docs/data/analytics/hubble/analyst-guide/queries-for-horizon-like-data.mdx @@ -813,17 +813,60 @@ group by 1,2,3,4,5 Fee stats are used to predict what fee to set for a transaction before submitting it to the network. -### Retrieve Fee Stats +:::note + +The `fee_stats_agg` model has been deprecated and replaced by two new models: `ledger_fee_stats_agg` (ledger-grain) and `daily_fee_stats_agg` (daily-grain). The new models provide separate Classic and Soroban fee breakdowns, inclusion fee metrics, and surge pricing statistics. + +::: + +### Retrieve Daily Fee Stats ```sql select day_agg - , fee_charged_max - , fee_charged_min - -- please see table schema for more available columns and data -from `crypto-stellar.crypto_stellar_dbt.fee_stats_agg` + , total_fee_charged + , max_fee_charged + , txn_count + , failed_txn_count + , classic_txn_count + , classic_sum_fee_charged + , classic_max_fee_charged + , classic_pct_ledgers_in_surge + , soroban_txn_count + , soroban_sum_fee_charged + , soroban_max_fee_charged + , soroban_sum_inclusion_fee_charged + , soroban_sum_resource_fee + , soroban_pct_ledgers_in_surge + -- please see table schema for more available columns and data +from `crypto-stellar.crypto_stellar_dbt.daily_fee_stats_agg` +where true + and day_agg between and +``` + +### Retrieve Ledger-Level Fee Stats + +```sql +select + day_agg + , ledger_sequence + , total_fee_charged + , max_fee_charged + , txn_count + , classic_txn_count + , classic_sum_fee_charged + , classic_is_surge_ledger + , soroban_txn_count + , soroban_sum_fee_charged + , soroban_sum_inclusion_fee_charged + , soroban_sum_resource_fee + , soroban_is_surge_ledger + -- please see table schema for more available columns and data +from `crypto-stellar.crypto_stellar_dbt.ledger_fee_stats_agg` where true and day_agg between and + -- optional filter by ledger + --and ledger_sequence = ``` ## getEvents diff --git a/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/daily-fee-stats-agg.mdx b/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/daily-fee-stats-agg.mdx new file mode 100644 index 0000000000..5fe1abeb59 --- /dev/null +++ b/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/daily-fee-stats-agg.mdx @@ -0,0 +1,83 @@ +--- +title: Daily Fee Stats Agg +sidebar_position: 82 +description: "" +--- + +
+ +## Table Metadata + +| Property | Configuration | +| --- | --- | +| Natural Key(s) | day_agg | +| Partition Field(s) | N/A | +| Clustered Field(s) | day_agg | +| Documentation | [dbt docs](http://www.stellar-dbt-docs.com/#!/model/model.stellar_dbt_public.daily_fee_stats_agg) | + +## Column Details + +| Name | Description | Data Type | Domain Values | Required? | Notes | +| --- | --- | --- | --- | --- | --- | +| day_agg | Day when the fee stats were aggregated. | DATE | | Yes | | +| total_fee_charged | Sum of fee_charged across all transactions (Classic + Soroban) for the day. | INTEGER | | Yes | | +| max_fee_charged | Maximum fee_charged across all transactions (Classic + Soroban) for the day. | INTEGER | | Yes | | +| txn_count | Total number of transactions (Classic + Soroban) for the day. | INTEGER | | Yes | | +| failed_txn_count | Number of failed transactions (Classic + Soroban) for the day. Failed transactions are still included in all fee aggregates because inclusion_fee_charged and non_refundable_resource_fee_charged are charged regardless of transaction success. | INTEGER | | No | | +| total_effective_txn_operation_count | Total number of effective operations across all transactions for the day. Uses effective_txn_operation_count, which adds 1 for fee-bump transactions to account for the extra inner transaction. | INTEGER | | Yes | | +| total_raw_txn_operation_count | Total number of raw operations (txn_operation_count) across all transactions for the day. Unlike effective operation count, this does NOT include fee-bump transactions. | INTEGER | | Yes | | +| min_ledger_sequence | First (minimum) ledger sequence number for the day. | INTEGER | | Yes | | +| max_ledger_sequence | Last (maximum) ledger sequence number for the day. | INTEGER | | Yes | | +| total_ledgers | Total number of ledgers processed for the day, regardless of whether they contain Classic or Soroban transactions. Used as the denominator for total_pct_ledgers_in_surge. | INTEGER | | Yes | | +| classic_txn_count | Number of Classic transactions for the day. Classic transactions have resource_fee = 0. | INTEGER | | No | | +| classic_failed_txn_count | Number of failed Classic transactions for the day. | INTEGER | | No | | +| classic_total_effective_operation_count | Total number of effective operations across Classic transactions for the day. Adds 1 for fee-bump transactions. | INTEGER | | No | | +| classic_total_raw_operation_count | Total number of raw operations across Classic transactions for the day. Does NOT include fee-bump transactions. | INTEGER | | No | | +| classic_sum_fee_charged | Sum of fee_charged across Classic transactions for the day. For Classic txns, fee_charged is the inclusion fee (no resource_fee component). | INTEGER | | No | | +| classic_max_fee_charged | Maximum fee_charged across Classic transactions for the day. | INTEGER | | No | | +| classic_sum_max_fee | Sum of COALESCE(new_max_fee, max_fee) across Classic transactions for the day. Represents total willingness-to-pay. | INTEGER | | No | | +| classic_max_max_fee | Maximum of COALESCE(new_max_fee, max_fee) across Classic transactions for the day. | INTEGER | | No | | +| classic_max_inclusion_fee_per_op | Maximum inclusion fee per operation (fee_charged / effective_txn_operation_count) across Classic transactions for the day. | FLOAT | | No | | +| classic_min_inclusion_fee_per_op | Minimum inclusion fee per operation across Classic transactions for the day. | FLOAT | | No | | +| classic_total_ledgers | Total number of ledgers containing at least one Classic transaction for the day. | INTEGER | | No | | +| classic_surge_ledger_count | Number of ledgers where at least one Classic transaction experienced surge pricing for the day. | INTEGER | | No | | +| classic_total_surge_txn_count | Total number of Classic transactions that experienced surge pricing across all ledgers for the day. | INTEGER | | No | | +| classic_total_surge_operation_count | Total operations in Classic transactions that experienced surge pricing across all ledgers for the day. | INTEGER | | No | | +| classic_pct_ledgers_in_surge | Percentage of Classic-containing ledgers that experienced surge pricing for the day. Calculated as 100 \* classic_surge_ledger_count / classic_total_ledgers. | FLOAT | | No | | +| soroban_txn_count | Number of Soroban transactions for the day. Soroban transactions have resource_fee > 0. | INTEGER | | No | | +| soroban_failed_txn_count | Number of failed Soroban transactions for the day. Even failed Soroban transactions are charged inclusion_fee_charged and non_refundable_resource_fee_charged, so they are included in all fee aggregates. | INTEGER | | No | | +| soroban_total_effective_operation_count | Total number of effective operations across Soroban transactions for the day. Adds 1 for fee-bump transactions. | INTEGER | | No | | +| soroban_total_raw_operation_count | Total number of raw operations across Soroban transactions for the day. Does NOT include fee-bump transactions. | INTEGER | | No | | +| soroban_sum_fee_charged | Sum of fee_charged across Soroban transactions for the day. fee_charged = inclusion_fee_charged + non_refundable_resource_fee_charged + refundable_resource_fee_charged. | INTEGER | | No | | +| soroban_max_fee_charged | Maximum fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_sum_inclusion_fee_charged | Sum of inclusion_fee_charged across Soroban transactions for the day. The base inclusion fee is 100 stroops per operation. If inclusion_fee_charged exceeds this base, the ledger experienced inclusion fee surge pricing. | INTEGER | | No | | +| soroban_max_inclusion_fee_charged | Maximum inclusion_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_sum_inclusion_fee_bid | Sum of inclusion_fee_bid across Soroban transactions for the day. inclusion_fee_bid = COALESCE(new_max_fee, max_fee) - resource_fee. | INTEGER | | No | | +| soroban_max_inclusion_fee_bid | Maximum inclusion_fee_bid across Soroban transactions for the day. | INTEGER | | No | | +| soroban_max_inclusion_fee_per_op | Maximum inclusion fee per operation (inclusion_fee_charged / effective_txn_operation_count) across Soroban transactions for the day. | FLOAT | | No | | +| soroban_min_inclusion_fee_per_op | Minimum inclusion fee per operation across Soroban transactions for the day. | FLOAT | | No | | +| soroban_sum_resource_fee | Sum of resource_fee (the pre-execution budget for Soroban resource consumption) across Soroban transactions for the day. | INTEGER | | No | | +| soroban_max_resource_fee | Maximum resource_fee across Soroban transactions for the day. | INTEGER | | No | | +| soroban_min_resource_fee | Minimum resource_fee across Soroban transactions for the day. | INTEGER | | No | | +| soroban_sum_non_refundable_resource_fee_charged | Sum of non_refundable_resource_fee_charged across Soroban transactions for the day. Covers CPU instructions, read bytes, write bytes, and bandwidth. Charged based on declared resources regardless of tx success/failure. | INTEGER | | No | | +| soroban_max_non_refundable_resource_fee_charged | Maximum non_refundable_resource_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_min_non_refundable_resource_fee_charged | Minimum non_refundable_resource_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_sum_refundable_resource_fee_charged | Sum of refundable_resource_fee_charged across Soroban transactions for the day. Covers rent, events, and return value. Based on actual usage; 0 for failed transactions. | INTEGER | | No | | +| soroban_max_refundable_resource_fee_charged | Maximum refundable_resource_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_min_refundable_resource_fee_charged | Minimum refundable_resource_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_sum_resource_fee_refund | Sum of resource_fee_refund across Soroban transactions for the day. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_max_resource_fee_refund | Maximum resource_fee_refund across Soroban transactions for the day. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_min_resource_fee_refund | Minimum resource_fee_refund across Soroban transactions for the day. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_sum_rent_fee_charged | Sum of rent_fee_charged across Soroban transactions for the day. This is the portion of refundable_resource_fee_charged that went to ledger entry TTL extensions. | INTEGER | | No | | +| soroban_max_rent_fee_charged | Maximum rent_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_min_rent_fee_charged | Minimum rent_fee_charged across Soroban transactions for the day. | INTEGER | | No | | +| soroban_total_ledgers | Total number of ledgers containing at least one Soroban transaction for the day. | INTEGER | | No | | +| soroban_surge_ledger_count | Number of ledgers where at least one Soroban transaction experienced surge pricing for the day. | INTEGER | | No | | +| soroban_total_surge_txn_count | Total number of Soroban transactions that experienced surge pricing across all ledgers for the day. | INTEGER | | No | | +| soroban_total_surge_operation_count | Total operations in Soroban transactions that experienced surge pricing across all ledgers for the day. | INTEGER | | No | | +| soroban_pct_ledgers_in_surge | Percentage of Soroban-containing ledgers that experienced surge pricing for the day. Calculated as 100 \* soroban_surge_ledger_count / soroban_total_ledgers. | FLOAT | | No | | +| total_pct_ledgers_in_surge | Percentage of all ledgers for the day where at least one transaction (Classic or Soroban) experienced surge pricing. | FLOAT | | No | | +| fee_pool | Sum of fee_pool across all ledgers for the day. Represents all fees collected across the network for the day. | INTEGER | | No | | +| airflow_start_ts | Timestamp when the Airflow DAG run started. Used for pipeline metadata and debugging. | STRING | | No | | + +
diff --git a/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/ledger-fee-stats-agg.mdx b/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/ledger-fee-stats-agg.mdx new file mode 100644 index 0000000000..d226cc8b58 --- /dev/null +++ b/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/ledger-fee-stats-agg.mdx @@ -0,0 +1,77 @@ +--- +title: Ledger Fee Stats Agg +sidebar_position: 81 +description: "" +--- + +
+ +## Table Metadata + +| Property | Configuration | +| --- | --- | +| Natural Key(s) | ledger_sequence | +| Partition Field(s) | N/A | +| Clustered Field(s) | day_agg, ledger_sequence | +| Documentation | [dbt docs](http://www.stellar-dbt-docs.com/#!/model/model.stellar_dbt_public.ledger_fee_stats_agg) | + +## Column Details + +| Name | Description | Data Type | Domain Values | Required? | Notes | +| --- | --- | --- | --- | --- | --- | +| day_agg | Day when the ledger fee stats were aggregated. | DATE | | Yes | | +| ledger_sequence | The unique ledger sequence number. This is the grain of the table -- one row per ledger. | INTEGER | | Yes | | +| total_fee_charged | Sum of fee_charged across all transactions (Classic + Soroban) in the ledger. This is the total fees actually paid. | INTEGER | | Yes | | +| max_fee_charged | Maximum fee_charged across all transactions (Classic + Soroban) in the ledger. | INTEGER | | Yes | | +| txn_count | Total number of transactions (Classic + Soroban) in the ledger. | INTEGER | | Yes | | +| failed_txn_count | Number of failed transactions (Classic + Soroban) in the ledger. Failed transactions are still included in all fee aggregates because inclusion_fee_charged and non_refundable_resource_fee_charged are charged regardless of transaction success. | INTEGER | | No | | +| total_effective_txn_operation_count | Total number of effective operations across all transactions in the ledger. Uses effective_txn_operation_count, which adds 1 for fee-bump transactions to account for the extra inner transaction. | INTEGER | | Yes | | +| total_raw_txn_operation_count | Total number of raw operations (txn_operation_count) across all transactions in the ledger. Unlike effective operation count, this does NOT include fee-bump transactions. | INTEGER | | Yes | | +| classic_txn_count | Number of Classic transactions in the ledger. Classic transactions have resource_fee = 0. | INTEGER | | No | | +| classic_failed_txn_count | Number of failed Classic transactions in the ledger. | INTEGER | | No | | +| classic_total_effective_operation_count | Total number of effective operations across Classic transactions in the ledger. Adds 1 for fee-bump transactions. | INTEGER | | No | | +| classic_total_raw_operation_count | Total number of raw operations across Classic transactions in the ledger. Does NOT include fee-bump transactions. | INTEGER | | No | | +| classic_sum_fee_charged | Sum of fee_charged across Classic transactions. For Classic txns, fee_charged is the inclusion fee (no resource_fee component). | INTEGER | | No | | +| classic_max_fee_charged | Maximum fee_charged across Classic transactions in the ledger. | INTEGER | | No | | +| classic_sum_max_fee | Sum of COALESCE(new_max_fee, max_fee) across Classic transactions. Represents total willingness-to-pay. | INTEGER | | No | | +| classic_max_max_fee | Maximum of COALESCE(new_max_fee, max_fee) across Classic transactions in the ledger. | INTEGER | | No | | +| classic_max_inclusion_fee_per_op | Maximum inclusion fee per operation (fee_charged / effective_txn_operation_count) across Classic transactions. Fee-bump txns add 1 to the operation count. | FLOAT | | No | | +| classic_min_inclusion_fee_per_op | Minimum inclusion fee per operation across Classic transactions in the ledger. | FLOAT | | No | | +| classic_surge_txn_count | Number of Classic transactions in the ledger where fee_charged exceeded the base fee (100 stroops per operation), indicating surge pricing. | INTEGER | | No | | +| classic_surge_operation_count | Total operations in Classic transactions that experienced surge pricing in the ledger. | INTEGER | | No | | +| classic_is_surge_ledger | True if any Classic transaction in the ledger experienced surge pricing. | BOOLEAN | | No | | +| soroban_txn_count | Number of Soroban transactions in the ledger. Soroban transactions have resource_fee > 0. | INTEGER | | No | | +| soroban_failed_txn_count | Number of failed Soroban transactions in the ledger. Even failed Soroban transactions are charged inclusion_fee_charged and non_refundable_resource_fee_charged, so they are included in all fee aggregates. | INTEGER | | No | | +| soroban_total_effective_operation_count | Total number of effective operations across Soroban transactions in the ledger. Adds 1 for fee-bump transactions. | INTEGER | | No | | +| soroban_total_raw_operation_count | Total number of raw operations across Soroban transactions in the ledger. Does NOT include fee-bump transactions. | INTEGER | | No | | +| soroban_sum_fee_charged | Sum of fee_charged across Soroban transactions. fee_charged = inclusion_fee_charged + non_refundable_resource_fee_charged + refundable_resource_fee_charged. | INTEGER | | No | | +| soroban_max_fee_charged | Maximum fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_sum_inclusion_fee_charged | Sum of inclusion_fee_charged across Soroban transactions. The base inclusion fee is 100 stroops per operation. If inclusion_fee_charged exceeds this base, the ledger experienced inclusion fee surge pricing. | INTEGER | | No | | +| soroban_max_inclusion_fee_charged | Maximum inclusion_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_sum_inclusion_fee_bid | Sum of inclusion_fee_bid across Soroban transactions. inclusion_fee_bid = COALESCE(new_max_fee, max_fee) - resource_fee. | INTEGER | | No | | +| soroban_max_inclusion_fee_bid | Maximum inclusion_fee_bid across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_max_inclusion_fee_per_op | Maximum inclusion fee per operation (inclusion_fee_charged / effective_txn_operation_count) across Soroban transactions in the ledger. | FLOAT | | No | | +| soroban_min_inclusion_fee_per_op | Minimum inclusion fee per operation across Soroban transactions in the ledger. | FLOAT | | No | | +| soroban_sum_resource_fee | Sum of resource_fee (the pre-execution budget for Soroban resource consumption) across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_max_resource_fee | Maximum resource_fee across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_min_resource_fee | Minimum resource_fee across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_sum_non_refundable_resource_fee_charged | Sum of non_refundable_resource_fee_charged across Soroban transactions. Covers CPU instructions, read bytes, write bytes, and bandwidth. Charged based on declared resources regardless of tx success/failure. | INTEGER | | No | | +| soroban_max_non_refundable_resource_fee_charged | Maximum non_refundable_resource_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_min_non_refundable_resource_fee_charged | Minimum non_refundable_resource_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_sum_refundable_resource_fee_charged | Sum of refundable_resource_fee_charged across Soroban transactions. Covers rent, events, and return value. Based on actual usage; 0 for failed transactions. | INTEGER | | No | | +| soroban_max_refundable_resource_fee_charged | Maximum refundable_resource_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_min_refundable_resource_fee_charged | Minimum refundable_resource_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_sum_resource_fee_refund | Sum of resource_fee_refund across Soroban transactions. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_max_resource_fee_refund | Maximum resource_fee_refund across Soroban transactions. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_min_resource_fee_refund | Minimum resource_fee_refund across Soroban transactions. NOTE: Currently broken -- always 0. | INTEGER | | No | | +| soroban_sum_rent_fee_charged | Sum of rent_fee_charged across Soroban transactions. This is the portion of refundable_resource_fee_charged that went to ledger entry TTL extensions. | INTEGER | | No | | +| soroban_max_rent_fee_charged | Maximum rent_fee_charged across Soroban transactions in the ledger. | INTEGER | | No | | +| soroban_min_rent_fee_charged | Minimum rent_fee_charged across Soroban transactions in the ledger. Will be 0 for ledgers where all Soroban transactions failed. | INTEGER | | No | | +| soroban_surge_txn_count | Number of Soroban transactions in the ledger where inclusion_fee_charged exceeded the base fee (100 stroops per operation), indicating surge pricing. | INTEGER | | No | | +| soroban_surge_operation_count | Total operations in Soroban transactions that experienced surge pricing in the ledger. | INTEGER | | No | | +| soroban_is_surge_ledger | True if any Soroban transaction in the ledger experienced surge pricing. | BOOLEAN | | No | | +| closed_at | Timestamp in UTC when the ledger closed and was committed to the network. Sourced from history_ledgers. | TIMESTAMP | | No | | +| fee_pool | The total fee pool for the ledger, sourced from history_ledgers. | INTEGER | | No | | +| airflow_start_ts | Timestamp when the Airflow DAG run started. Used for pipeline metadata and debugging. | STRING | | No | | + +
diff --git a/routes.txt b/routes.txt index 712002b1a5..cad5756414 100644 --- a/routes.txt +++ b/routes.txt @@ -216,7 +216,9 @@ /docs/data/analytics/hubble/data-catalog/data-dictionary/bronze/ttl /docs/data/analytics/hubble/data-catalog/data-dictionary/gold /docs/data/analytics/hubble/data-catalog/data-dictionary/gold/asset-balances-daily-agg +/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/daily-fee-stats-agg /docs/data/analytics/hubble/data-catalog/data-dictionary/gold/fee-stats-agg +/docs/data/analytics/hubble/data-catalog/data-dictionary/gold/ledger-fee-stats-agg /docs/data/analytics/hubble/data-catalog/data-dictionary/gold/trade-agg /docs/data/analytics/hubble/data-catalog/data-dictionary/gold/tvl-agg /docs/data/analytics/hubble/data-catalog/data-dictionary/silver