v2.16.2

• Fix substreams support for requests with 'application/grpc-web*' content-type (old connectweb library)

v2.16.1

• Fix substreams/firehose endpoints detection of supported compression: do not fail on 'algo;q=x.y' syntax
• Fix relayer failing to get back to live if reader blocks are unlinkable after a long period, and merger has removed one-blocks: it will now shutdown in that case, so it can be restarted.
• Fix substreams tier2 jobs behind load balancer: will now retry forever on 'Unavailable: no healthy upstream' errors
• Add substreams-tier2-authenticator flag to specify the authenticator to use for tier2 requests. Can be 'trust://' (default, same as previous behavior) or 'secret://'
• Add substreams-tier1-subrequests-secret-key flag to specify the secret key to use for tier1 subrequests authentication when using 'secret://' authenticator on tier2
• Add reader-node-grpc-secret-key flag to specify the secret key to use for reader node gRPC authentication
• Add ?secret=... parsing to relayer-sources
• Add Prometheus metrics for reader test mode: track blocks compared, success/failure counts, and success/failure percentages for easy monitoring at interval stats.

v2.16.0

Substreams Performance (RPC V4, New blocks from last partial)

• RPC V4 protocol with BlockScopedDatas batching: Multiple BlockScopedData messages are now batched into a single BlockScopedDatas response, reducing gRPC round-trips and message framing overhead during backfill. Clients automatically fall back V4 → V3 → V2 when connecting to older servers, so no flag changes are required.
• S2 compression is now the default: Replaces gzip as the default compression algorithm, providing ~3-5x faster compression/decompression with comparable ratios. The client automatically negotiates compression with the server.
• VTProtobuf fast serialization: Both client and server now use vtprotobuf for protobuf marshaling/unmarshaling, providing ~2-3x faster serialization with reduced memory allocations.
• Server-side message buffering: Configurable via --substreams-tier1-output-buffer-size flag (default: 100 blocks) or MESSAGE_BUFFER_MAX_DATA_SIZE environment variable (default: 10MB).
• Improved Connect/gRPC protocol selection: Server now efficiently routes requests to the appropriate handler based on content-type, improving performance by ~15% for pure gRPC clients.
• New blocks from last partial: "Last partial blocks" are now accepted interchangeably with new blocks, allowing faster full blocks for requests that do not ask for partial blocks.

Added

• Add firecore tools substreams logs connections <user_id> command to query Cloud Logging and show Substreams connections for an organization. Correlates incoming requests with stats by trace ID and presents a summary table showing active, closed, and error connections with details like network, source IP, module, duration, and blocks processed.

Removed

• Remove alpha partial blocks support in firehose service (only exposed via substreams)

v2.15.9

• Substreams: Improved 'partial blocks': support new pbbstream's "LastPartial" field, fix 'undo' scenarios for stores
• Reduce RAM usage with partial blocks (relayer, substreams, firehose)
• Prevent panic if transactionTrace.receipt is nil in LogFilter (even if it is not a normal scenario)

v2.15.8

• Fix eth poller issue with marshalling block number (regression in v2.15.5)

v2.15.7

• Bump Golang to build to 1.25

Substreams

• Fix issue where a retry on dstore while writing a fullKV would corrupt the file, making it unreadable. Fix prevents this and also now deletes affected files when they are detected
• Fix bug where so request could get stuck forever (until the clients drops or server restarts).
• Fix issue where transient HTTP/2 stream errors (e.g., INTERNAL_ERROR) from dstore were being treated as fatal errors instead of being retried. These transient network errors are now detected and retried with exponential backoff.

v2.15.6

Added

• Added bucketed prometheus metrics head_block_relative_time_sum to help investigate latency and pipeline performance:
  • "app=firehose_output" and "app=substreams_output" that shows latency between outputing live blocks and their blocktime.
  • "app=relayer" for latency at relayer's input
• Substreams: Fixed underflow in 'FailedPrecondition desc = request needs to process a total of x blocks' error when running from 'substreams run' with a start-block in the future.
• Bump base64 library to use a much faster one in reader
• dstore: bumped google storage lib to v1.59.1 to fix a bug in their multi-range downloader, in case it affects us

v2.15.5

Substreams fixes

• Fix issue where "live backfiller" would not create segments after reconnecting with a cursor starting from a previous quicksave, causing delays in future reconnection
• Prevent "panic" when log messages are too large: instead, they will be truncated with a 'some logs were truncated' message.
• Raise max individual log message size from 128k to 512k
• Raise max log message size for a full block from 128k to 5MiB
• Reduce log level from Warn to Debug when we fail to get or set the store size (for backends that don't support it)

Substreams Partial blocks (experimental)

• Removed PartialsData message and brought back this data inside the good old BlockScopedData
• added the following fields to BlockScopedData:
  • bool is_partial to indicate if this block is a partial block. The following two fields are only present when is_partial==true
  • optional bool is_last_partial to indicate if this is the last partial of a given block (with correct block hash)
  • optional uint32 partial_index to indicate the index of this partial block within the full block
• renamed partial_blocks_only flag to partial_blocks on substreams Blocks request
• removed include_partial_blocks flag from substreams Blocks request

AWS Store

• Migrated from AWS SDK for Go v1 to v2 (github.com/aws/aws-sdk-go → github.com/aws/aws-sdk-go-v2)

Azure store

• Added support for "workload identity credentials" in Azure. Order of preference is:
  • If AZURE_STORAGE_KEY is set, use shared key credential (previous behavior)
  • Otherwise, use DefaultAzureCredential which supports:
    • Managed Identity (for Azure resources)
    • Service Principal (via AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID)
    • Azure CLI credentials
    • Visual Studio Code credentials

RPC Poller

• Added requests_hash (EIP-7685) field handling to the RPC poller.

v2.14.4

Partial blocks

Added experimental support for partial blocks (e.g. Flashblocks on Base)

See https://docs.substreams.dev/reference-material/chains-and-endpoints/flashblocks for details about how they work in Substreams.

• SUBSTREAMS_BIGGEST_PARTIAL_BLOCK_INDEX environment variable to specify the index to use when bundling the "last partial block" from the full block. (default: 10, for Base)
• Added flag --include-partial-blocks on tools firehose-client

Eth calls

• Introduced ETH_CALL_USE_BLOCK_NUMBER_DURATION environment variable. When set, all RPC calls older than that duration will be done targeting the block by number instead of hash.
  The ETH_CALL_FALLBACK_TO_LATEST_DURATION mechanism can still be applied over this, as it has precedence.

Bugfixes

• Substreams: fixed issue where "live backfiller" would not create segments after reconnecting with a cursor starting from a previous quicksave, causing delays in future reconnection

v2.14.3

Substreams

• Fixed substreams regression from v2.14.2 where some jobs would not get scheduled correctly, resulting in failure with the mssage get size of store "...": opening file: not found.
• Introduced ETH_CALL_FALLBACK_TO_LATEST_DURATION environment variable. When set, all RPC calls older than that duration will be done targeting "latest" block instead of current.
  Calls that return error code -32602 (invalid params) will also be retried using the latest block reference instead of the original block hash.
  This is to support chains where "archive nodes" do not exist, allowing serving recent data nonetheless.