Skip to content

CHANGELOG.md

Latest commit

 

History

History
337 lines (232 loc) · 31.2 KB

CHANGELOG.md

File metadata and controls

337 lines (232 loc) · 31.2 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.7.2] - 2026-06-30

This version does include some minor fixes, some caused by the overall change of the database schema. There were multiple refactors which should improve the maintainability and readability of the codebase.

Added

  • Feat(indexer): add option to accept log level as a flag in the cli 7c295e2
  • Feat: add initial support for the ssl to postgres 395d536

Changes

  • Chore: small text corrections dae2116
  • Refac(indexer/orchestrator): historic run awaits the context now 72fad55
  • Deps: update chi and gnark-crypto 6088cec
  • Refac(indexer/schema):adjust aggregate tables similarly to db tables 5c48476
  • Chore(data_processor): remove dead code f55e7ce
  • Refac(schema&cli): move setup logic closer to schema 35324dd
  • Refac(pkgs/schema): add data type assertion and reflaction testing 21b01eb
  • Refac(indexer/decoder): add registry based system d998fd1
  • Refac: use message interface for data grouping ecafb9e
  • Refac(pkgs/schema): add generics for inserting data eca95f9
  • Refac(pkgs/schema): move the copy the row logic closer to the schemas cd43e3b
  • Refac(pkgs): rename sql_data_types to schema 8874feb
  • Docs: update grammer and add quick start section 46f9d0b
  • Change(api): add function that will provide a mean block prod value 5531175

Fixed

  • Fix(indexer/dp): return errors if processing message fails 2426e89
  • Fix(indexer): a bug where the database pool closes before the ingestion 076f3f5
  • Fix(indexer/cli): add missing tables for adding privilages to the user 74be438
  • Fix(ci): revert the go version for govulncheck 13ba5fc
  • Fix(api): fix empty volume response 3011c77

Tests and Code Check

  • Audit(coderabbit): add additional error handle for query block prod 814b575
  • Audit(coderabbit): add buildDSN for database connection dc810b4
  • Audit(sonarqube): adjust code to use variable instead of typed in params 747da47
  • Audit(sonarqube): drop complexity for some api handler function f29123c
  • Audit(sonarqube): reduce complexity of copyraw test 62070b5
  • Audit(sonarqube): drop complexity of aggregation table test b7b03c0
  • Test(pkgs/schema): add validation for aggregate tables bf55820

[0.7.1] - 2026-06-24

This version should add support to process new Gnoland testnet 13. Added support for Auth message types, and the bank multi send should be supported.

Added

  • Feat(api): add new message types to the tx message route e38c9f3
  • Feat(cli): add to setup db to init auth tables and multi send 1b56f5f
  • Feat(decoder): add convert to auth methods 72e2334
  • Feat(sql_data_types): add schemas for the new auth msg types 0ecf0c7
  • Feat(decoder): add auth msg types 32fb8df
  • Feat(timescaledb): add auth insert methods 33f4d34

Changes

  • Refac(timescaledb): move the GetAllBlockSigners to query_block.go a4f9bf3
  • Refac(data_processor): add new auth types and partially move decoding 81ff897
  • Refactor(decoder): refactor the decoder to use smaller fn per msg type c71f39c
  • Deps: update gno to chain/test13 f08460d

Fixed

  • Fix: add missing add address f681153
  • Fix(sql_data_types): fix chain_name to use enum type in database ca36797
  • Fix: add missing data to the create session ac2130c
  • Fix: dockerfile indexer.go path updated and force the toolcahin to auto a04c48c
  • Fix(ci): fix the release.yml to use correct path to indexer.go 61b96b3

[0.7.0] - 2026-06-14

This release brings a versioned REST API, several new routes, real-time statistics, partial support for the bank multi send message type, and a large number of query fixes and performance refactors. The database layer received significant work, including a schema migration table for future releases and a refactor of the transaction lookups to use tx_hash_id. The Go toolchain and dependencies were updated and the TimescaleDB code was separated out of the database package.

This version will make a freeze on any new feature unless it is related to performance, stability, or security until the v1 release. All of the development will be focused on bug fixes and improvements of the existing functionality.

Added

  • REST API routes are now versioned. All routes are served under a v1 group using Huma route groups. 70b7330, 5dafaf3
  • Real time statistics served from an in-memory handler. 1b6d563
  • New route to get all validator signings in the last 24h. df369ad
  • Several new REST API routes added and existing ones fixed. 34360b7
  • Partial support for the bank multi send message type in the decoder, SQL data types and the DataProcessor. Full support will be added once Gno Amino encoder has it implemented. d8c9e2f
  • Schema migration table to ease database migrations in future releases. 588500b
  • Option to use a base64 hash directly in the transactions/{hash} query. 641e5f7
  • Sort by date/time on queries that can support it.
  • Error log and success/execution status to transaction and address transaction queries.
  • User-agent support for the RPC client.
  • SECURITY.md file outlining the security policy and reporting guidelines. 74cdd48

Changes

  • The TimescaleDB code was separated out of the database package into its own package. afdc20d
  • Refactored database queries to use tx_hash_id for transaction lookups, and refactored the hypertables that contained the tx_hash column. a4f4f3d, f13d980
  • Optimized the indexer orchestrator and adjusted the codebase to the recent changes. 8d53baa
  • Implemented rate limiting for specific API routes. 0e39720
  • Refactored hypertable creation to use a structured HypertableParams struct for better configuration management, including OrderBy and SegmentBy options. 79c4fd8
  • Moved the CLI commands to the cli directory and indexer.go to the cmd file. 59999fa
  • Force the pgx driver to use UTC time and refactored date handling across the API and database queries. 0a1cfcb
  • Daily query views now include data up to the end of the day, and every daily query returns 0 when there is no data instead of returning nothing.
  • Changed NextCursor and PrevCursor fields to be non-omittable in the JSON output.
  • Refactored error handling in the address and block handlers, and removed the sort order parameter from address transaction handling.
  • Adjusted the RPC HTTP client to use a better HTTP configuration.
  • Refactored the Valkey client implementation and used pure Go implementation.
  • Updated the Go toolchain to go1.26.4 and updated other dependencies.

Fixed

  • The indexer querying the new database for the last block height returned an error.
  • The validator 24h signing query incorrectly counted all blocks instead of only the blocks with validator signings. 6c8a920
  • Fixed duplicate entries being made for the address_tx table. 04b1a01
  • The rate limiter added a TTL of a couple of thousand years; fixed alongside the rate limiting work. 0e39720
  • Fixed missing query columns in GetBlock on the TimescaleDB query.
  • Fixed unique detection for the SQL database initialization and added unique keys to some tables.
  • extractCoins now inserts a 0 value and the coins are inserted at the index.
  • Various fixes to the view and transaction queries, including the daily block production query.
  • Reverted performRequest to first read the body and then parse the JSON.

[0.6.0] - 2026-03-14

This release has some new features added and minor improvements.

Added

  • Continuous aggregates for the data. Now the database will automatically aggregate the data by the time bucket and the data is aggregated by the chain name. This feature adds metrics for the blocks, transactions, validator signings, daily active accounts and fee volume.
  • Added API keys authentication to the API. It is optional and can be disabled in the configuration.
  • Add ratelimit by IP address and by API key.
  • Docker service Valkey for rate limiting.
  • Add multiple new routes to the API to get the data from the continuous aggregates.

Changes

  • Renamed routes of some API routes to be more descriptive.
  • Changed the blocks table, now it doesn't store tx hashes.
  • Some slight performance improvements were made in data processing.
  • Adjusted the SQL commands to use newer TimescaleDB API commands and functions.

[0.5.0] - 2026-03-05

This release has some new features added and minor improvements.

The zstd compression has been added. This it is not production ready and still in development, however initial testing has been done and it seems to work, about 30% less storage is required.

The API has been improved with some new routes and minor improvements. The cursor based pagination has been added to the API on certain routes. The API now also has POST utilities to convert between Base64 and Base64URL.

The docker image for the API has been added. This allows for easy deployment of the API via docker. The docker compose file has been adjusted for full deployment via docker(for production and development).

Added

  • Indexer can now compress the events using zstandard compression with the use of -e or --compress-events flag. Still in development.
  • CLI tool to train the zstandard dictionary from the database.
  • POST utilities to convert between Base64 and Base64URL
  • Docker image for the API
  • Cursor based pagination to the API on certain routes.

Changes

  • Renamed routes of some API routes to be more descriptive.
  • Docker compose adjusted for full deployment via docker(for production and development)
  • Some minor performance improvements during data processing.
  • Updated the go version to 1.25.7 and all of the dependencies to the latest version.

[0.4.0] - 2025-11-26

Mostly it has some bug fixes.

Fixed

  • The indexer would go into the blocks data and store the signers from the last commit, which is actually all of the block signers from the previous block. So it would insert it like it was meant for that block height. From now on the indexer will fetch data from the /commit method and insert it properly. de40740
  • Gnoland can indeed execute multiple message types in the same transaction. The indexer wasn't able to hold this data properly and would cause an error because in the postgres primary key was attached making each message type unique. Now each message type has a message_counter which is a smallint(int16) which is used as a index for that transaction. d647901

Changes

  • The REST API now returns a map of int16 and message data for the /transaction/{tx_hash}/message route. 74a35b30
  • The indexer should stop using sync.Map and use sync.Mutex with regular map to store the addresses before handing the operation to the AddressSolver function. dde82a4
  • The Orchestrator needs to make a fetch to the commit RPC method. This operation is done side by side with fetching the blocks method. A side effect of this is that indexer at that moment will fetch 2 times more request to the RPC, and if the Gnoland node has a limit on the amount of RPC clients that can use it, it might cause the indexer to slow down and throw errors on this requests. In future releases this should be improved. d647901
  • Updated the go version to 1.25.4

[0.3.0] - 2025-11-10

In this release there are some fixes and improvements. The live process should work properly now and the REST API has some new routes. CLI commands are now combined with the ones from the setup cli. Some processes have been improved to use less memory.

Added

  • The REST API has some new routes. The API can now return last block height, last number(x) of blocks, last x transactions. 789c24b

Changed

  • The CLI commands are now combined all together so the cmd/setup.go is removed and the users can now only download the main cli and initiate everything they needs for the indexer to work. 805513b
  • Changed the data and query operators to use mutex and store any data they process/collect directly into the type they need to return. There shouldn't be any major perfromance difference but it should allocate less memory. 89e5b6d

Fixed

  • When Indexer started to run in live mode without any previous data it should start to process the data from first block height. But instead it tried to query block height 0. 3517e5a

[0.2.1] - 2025-10-06

Not really much of a change just added dockerignore file, small changes to the release.yml so it pushes the api also.

[0.2.0] - 2025-10-04

Added the REST API with some basic routes that will come in handy. Small bug fixes and changes.

Added

  • Rest API with 5 basic routes that will come in handy. 90bef0e,f2a39d6,d875be2
  • Some basic documentation for the API and modified the existing docs. b2c20d2
  • Database queries for the API ( althoguh they can be used for any other app or service if needed ) d875be2, f2a39d6

Fixed

  • Some table fixes, the msg_types would insert the empty string because when the make slice function was called by the accident it added the empty string instead of just allocating the size of the slice. d875be2
  • The validator signing has the column name addresses for the signed validators. While this is not a bug it wasn't intended. 90bef0e

[0.1.1] - 2025-10-02

Mostly some fixes. The live should work now there was a bug with the RPC client when making a request to the last block height recorded. The retry worker could sometimes send the data to the closed channel. Now the query operator calls the wg.Done functions directly. It should work now but there might be some other bugs.

Fixed

  • There was a bug with the RPC client when making a request to the last block height recorded. f44b3f3
  • The live process was not working because the RPC client was not sending the height parameter when making a request to the last block height recorded. f44b3f3

Changed

  • Moved the database to the pkgs directory f44b3f3

[0.1.0] - 2025-10-01

This is officially first working version of the indexer. The historic version was successfuly tested on the real data. It took about 2m30s on 2vCPU for 10K blocks and about 1m30s on 4vCPU for 10K blocks. The live version was not tested on the real data yet mostly because there is active pullic Gnoland testnet. The live will be tested properly on the testnet 9 when it is released. So the index will probably work but expect some bugs. Some features are still missing and this is still a work in progress.

Added

  • Added a GitHub Actions workflow to build the indexer and release it as a binary and docker image 5f20967
  • Added docker file and docker compose 18129be, 5f20967
  • Docs are now available 18129be, 5f20967

Changes

  • The cmd setup can now add tables to already existing database 5f20967

Fixed

  • There were some bugs related to poinetrs if the value was nil for block responsers related to validator signing c7f229a

[0.1.0-beta.2] - 2025-09-29

The indexer had some bug fixes and some small improvments. The integration test was technically successful but there seems there is some kind of bug with the indexer. The indexer is not fully tested yet only the historic process has been tested. But not any runs were made on the real data. You can try to run this version on the real data but be advised it is not fully tested and might not work as expected.

Added

  • Makefile has been added. If you feel advanterous you can try to build the indexer with greentea garbage collection. 9fdad03
  • Apperently the previous versions didn't had the method to insert the data for the table address_tx. Now every transaction that was executed can be tied to each address that was involved in the transaction. 9fdad03, 6dd764
  • The CLI for the database setup now has a new command to create a new user for the database and appoint privileges to the user. It can be a reader(for APIs and some other programs that need SELECT privileges) or a writer(example indexer for historical data). c39c1f7
  • Added a retry mechanism for the query operator. 900ee4f

Changed

  • Updated the go version to 1.25.1 b3e02b0

[0.1.0-beta.1] - 2025-09-25

The indexer had some bug fixes and some small improvments. The integration test was technically successful but there seems there is some kind of bug with the indexer. The indexer is not ready for production use.

Changed

  • When the indexer decodes the data using Amino decoder it unloads the data into a map[string]any, then from there it would make 2 conversions, one for the general data struct and the second for the sql data types. The idea was to have seperated logic for the general data struct and sql types. However at this point the indexer already needs to call the copy from method where the data is again being unloaded into some sort of tuple. So the first conversion was removed. 50ca1f2
  • Updated all of the dependencies to the latest version 5370a5c

Fixed

  • Fixed the sql related bugs, added some missing types, switched to pgtype.Numeric for the amount type 8aea191, 2b7ed52,ddfdcc1
  • In some instances the pgx data need to be in the pgtype.Array for instance Txs for the block need to be stored into the pgtype.Array. The indexer now uses a generics function to convert the data into the pgtype.Array 2b7ed52
  • The chunk end height was incremented by 1 when the indexer started the historic process. This caused the chunks to overlap and the indexer to throw an error about the duplication. The indexer now correctly sets the chunk end height to the max height ddfdcc1
  • Fixed a bug where the data processor would ask the address from the regular address cache instead from the validator address cache ddfdcc1

[0.1.0-alpha.2] - 2025-09-21

This is a second alpha release although the indexer is not yet ready.

Added

  • Updated all of the dependencies to the latest version 5fb6b8d
  • Added the generator functions for the integration test d61cfa6, 34a46fa
  • Added the synthetic integration test 76e42f6
  • Moved some of the code logic to it's own package 9cc12e9, 76e42f6, 9ca2214

[0.1.0-alpha.1] - 2025-09-15

This is the first alpha release of the Spectra Gnoland indexer. Technically most of the indexer components are done but it is not tested fully so this version is not recommended for production use.

Added

  • CLI for the Indexer
  • Config and env loaders
  • RPC client with rate limiting
  • PGX pool Postgres Client
  • Address cache for regular and validator addresses
  • Signal hook for graceful shutdown and emergency shutdown
  • Amino decoder for the data from the Gnoland Chain
  • Major operator/worker pattern for the indexer have been implemented
  • Basic database setup

Known Issues

  • The indexer is not tested and it is not recommended for production use.
  • The setup program only sets the database and ties it to the admin user. This could be bad for security.
  • The proto encoding for the events is not tested yet and might not even end in the final release.
  • Zstandard compression has been added but it has only been used in some minor test nothing more. For this to work properly a synthetic dataset would need to be created and used to train the dictionary. Alternatively it can be trained on the real data but given that the chain is still in the development stage there is no gurantee it will have enough data to train a good dictionary.