Skip to content

Live Table UDTF's #449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Live Table UDTF's #449

wants to merge 19 commits into from

Conversation

ShahNewazKhan
Copy link
Contributor

@ShahNewazKhan ShahNewazKhan commented Apr 9, 2025

This PR ports over Live Table UDTF's from livequery-models due to the collisions of bronze layer models across projects ( ex bronze__blocks, bronze__transactions etc existing in evm, aptos & near).

The Live Table UDTF's are designed to retrieve data live from Near nodes to fill the gap between current chain head and the Near data batch ingested via Streamline.

The following Live Table UDTF's have been created:

  • livetable.tf_fact_blocks
  • livetable.tf_fact_transaction
  • livetable.tf_fact_receipts
  • livetable.tf_ez_actions

We use the livequery_base.get_rendered_model() macro to reuse the transformation applied in the gold/silver models and override the bronze layer models. In order to do so we need to materialize the gold/silver layer as ephemeral models, we achieve this via the model's config block with a conditional materialization:

{{ config(
    ...
    materialized = var('LIVE_TABLE_MATERIALIZATION', 'view')
   ...
) }}

For the bronze layer override we use the ENABLE_LIVE_TABLE conditional to render the Live Table RPC logic or the default batch logic.

{{ config (
    materialized = var('LIVE_TABLE_MATERIALIZATION', 'view'),
    tags = ['streamline_helper']
) }}

{% if var('ENABLE_LIVE_TABLE', false) %}
    
    {%- set blockchain = this.schema -%}
    {%- set network = this.identifier -%}
    {%- set schema = blockchain ~ "_" ~ network -%}

    WITH spine AS (
        {{ near_live_table_target_blocks(start_block='_block_height', block_count='row_count') | indent(4) -}}
    ),
    raw_blocks AS (
        {{ near_live_table_get_raw_block_data('spine') | indent(4) -}}
    )

    SELECT
        OBJECT_INSERT(
            rb.rpc_data_result,          
            'BLOCK_ID',                 
            rb.block_height,             
            TRUE                         
        ) AS value,
        rb.rpc_data_result AS data,
        round(rb.block_height, -3) AS partition_key,
        CURRENT_TIMESTAMP() AS _inserted_timestamp
    FROM
        raw_blocks rb
{% else %}
    -- BATCH LOGIC: Default
    {{ streamline_external_table_FR_query_v2(
        model = "blocks_v2",
        partition_function = "CAST(SPLIT_PART(SPLIT_PART(file_name, '/', 3), '_', 1) AS INTEGER )") 
    }}
{% endif %}

In addition to the above, this PR add's the following:

  • Make directive to deploy live table UDTF's
  • blocks,tx,chunks specific udf_api's to enable granular throughput tuning

@ShahNewazKhan ShahNewazKhan marked this pull request as draft April 9, 2025 23:15
@ShahNewazKhan ShahNewazKhan changed the title Ephemeral deploy LT UDTF Success \0/ | WIP: Bronze override Live Table UDTF's Apr 11, 2025
@ShahNewazKhan ShahNewazKhan marked this pull request as ready for review April 11, 2025 01:09
@forgxyz
Copy link
Collaborator

forgxyz commented Apr 30, 2025

Let's create a 2nd Near endpoint on Quicknode for the livetable workflow to easily segment call volume metrics

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants