-
Notifications
You must be signed in to change notification settings - Fork 13
Common Issues
Gaps, nulls, late or missing data are common when working with Blockchain data. This page outlines some common occurrences and how to fix them.
Gaps can be tested for by running dbt test -s tag:gap_test. By default, this will run over the past 7 days for efficiency. A test over the full data can be run by passing DBT_FULL_TEST with the value True.
If there are block gaps, there may be chunk gaps that initially do not alert due to the nature of the data. Blocks contain an array of chunk hashes within the block. The content of the chunk is then delivered by the shard. The gap test checks for chucks that we should have based on the array in blocks against the chunk hashes we do have flattened in the chunks table. Thus, if the block is missing, the test will not check for chunks that we'd expect to find in that (missing) block. Once we fill the block gap, the chunk gap test should accurately assess if we have all chunks.