Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3011c77
fix(api): fix empty volume response
nman98 Jun 24, 2026
5531175
change(api): add function that will provide a mean block prod value
nman98 Jun 24, 2026
13ba5fc
fix(ci): revert the go version for govulncheck
nman98 Jun 27, 2026
74be438
fix(indexer/cli): add missing tables for adding privilages to the user
nman98 Jun 27, 2026
395d536
feat: add initial support for the ssl to postgres
nman98 Jun 27, 2026
076f3f5
fix(indexer): a bug where the database pool closes before the ingestion
nman98 Jun 28, 2026
7c295e2
feat(indexer): add option to accept log level as a flag in the cli
nman98 Jun 28, 2026
46f9d0b
docs: update grammer and add quick start section
nman98 Jun 28, 2026
8874feb
refac(pkgs): rename sql_data_types to schema
nman98 Jun 29, 2026
cd43e3b
refac(pkgs/schema): move the copy the row logic closer to the schemas
nman98 Jun 29, 2026
eca95f9
refac(pkgs/schema): add generics for inserting data
nman98 Jun 29, 2026
ecafb9e
refac: use message interface for data grouping
nman98 Jun 29, 2026
d998fd1
refac(indexer/decoder): add registry based system
nman98 Jun 29, 2026
21b01eb
refac(pkgs/schema): add data type assertion and reflaction testing
nman98 Jun 29, 2026
35324dd
refac(schema&cli): move setup logic closer to schema
nman98 Jun 29, 2026
f55e7ce
chore(data_processor): remove dead code
nman98 Jun 29, 2026
5c48476
refac(indexer/schema):adjust aggregate tables similarly to db tables
nman98 Jun 30, 2026
bf55820
test(pkgs/schema): add validation for aggregate tables
nman98 Jun 30, 2026
6088cec
deps: update chi and gnark-crypto
nman98 Jun 30, 2026
4a944db
chore: release v0.7.2
nman98 Jun 30, 2026
b7b03c0
audit(sonarqube): drop complexity of aggregation table test
nman98 Jun 30, 2026
62070b5
audit(sonarqube): reduce complexity of copyraw test
nman98 Jun 30, 2026
f29123c
audit(sonarqube): drop complexity for some api handler function
nman98 Jun 30, 2026
747da47
audit(sonarqube): adjust code to use variable instead of typed in params
nman98 Jun 30, 2026
2426e89
fix(indexer/dp): return errors if processing message fails
nman98 Jun 30, 2026
72fad55
refac(indexer/orchestrator): historic run awaits the context now
nman98 Jun 30, 2026
dc810b4
audit(coderabbit): add buildDSN for database connection
nman98 Jun 30, 2026
814b575
audit(coderabbit): add additional error handle for query block prod
nman98 Jun 30, 2026
dae2116
chore: small text corrections
nman98 Jun 30, 2026
04c1680
chode: update CHANGELOG.md
nman98 Jun 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/govulncheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4
with:
go-version: "1.25.11"
go-version: "1.26.4"

- name: Setup Go Vulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,4 @@ server.key

# Docker
.dockerignore
docker-compose-local.yml
docker-compose-local.yml
46 changes: 46 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,52 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.7.2] - 2026-06-30

This version does include some minor fixes, some caused by the overall change of the database schema.
There were multiple refactors which should improve the maintainability and readability of the codebase.

### Added

- Feat(indexer): add option to accept log level as a flag in the cli [7c295e2](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/7c295e276a6d10f7f0ca6af5a35041721089debe)
- Feat: add initial support for the ssl to postgres [395d536](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/395d5361d5c52bde384fa31ce0afd05574970690)

### Changes

- Chore: small text corrections [dae2116](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/dae211697cc5f588644a02829d4c4c71060f3073)
- Refac(indexer/orchestrator): historic run awaits the context now [72fad55](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/72fad55adc9c16173f5745fa5c5417e49e2e717e)
- Deps: update chi and gnark-crypto [6088cec](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/6088ceccde2f489828c47fcf6fbb4f3f4474b93f)
- Refac(indexer/schema):adjust aggregate tables similarly to db tables [5c48476](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/5c484764f2138c4c125fb4e3c823f84dcb12c7eb)
- Chore(data_processor): remove dead code [f55e7ce](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/f55e7ce535e2dbb54e4f76291f883cb85d20926f)
- Refac(schema&cli): move setup logic closer to schema [35324dd](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/35324dddf59968cc0e766b35ee0cc6997fce37a9)
- Refac(pkgs/schema): add data type assertion and reflaction testing [21b01eb](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/21b01eb9a72756721fd010cc464ed79c37f90f42)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix typo: "reflaction" → "reflection".

User-facing changelog contains a spelling error in the commit description.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` at line 24, Fix the spelling mistake in the changelog entry
from “reflaction” to “reflection” in the affected commit description. Update the
user-facing text in CHANGELOG.md so the summary line reads correctly and remains
consistent with the existing entry format.

- Refac(indexer/decoder): add registry based system [d998fd1](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/d998fd12990cc1472e1dd88b8598d6f4bff768ff)
- Refac: use message interface for data grouping [ecafb9e](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/ecafb9e61927e09064d83cc8cd622e05326f051f)
- Refac(pkgs/schema): add generics for inserting data [eca95f9](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/eca95f9d419977a79121c35b7523d4b080b9473e)
- Refac(pkgs/schema): move the copy the row logic closer to the schemas [cd43e3b](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/cd43e3bbe2ca566b2a5a40b4116c35758af16918)
- Refac(pkgs): rename sql_data_types to schema [8874feb](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/8874feb9ee8face4188a19deafbf8bada2feef36)
- Docs: update grammer and add quick start section [46f9d0b](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/46f9d0b79b42dd64da847d008e64ce00e9165144)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix typo: "grammer" → "grammar".

User-facing changelog contains a spelling error in the commit description.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` at line 30, Update the changelog entry text in CHANGELOG.md to
fix the spelling error in the commit description by changing “grammer” to
“grammar”; adjust the specific changelog item that mentions the quick start
section so the user-facing release note is spelled correctly.

- Change(api): add function that will provide a mean block prod value [5531175](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/553117588e78686056b23633a7719862a042c8dc)

### Fixed

- Fix(indexer/dp): return errors if processing message fails [2426e89](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/2426e89195563f95c71240ad25c9fe68b09abd84)
- Fix(indexer): a bug where the database pool closes before the ingestion [076f3f5](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/076f3f5f2417085cfd5266de54e1da5924f72372)
- Fix(indexer/cli): add missing tables for adding privilages to the user [74be438](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/74be438702ec7197b3d7be7c0dabe6e45ac60b76)
- Fix(ci): revert the go version for govulncheck [13ba5fc](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/13ba5fc2e0403c767c3a41e1eeab864e2d111099)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix typo: "privilages" → "privileges".

User-facing changelog contains a spelling error in the commit description.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` at line 37, The changelog entry in CHANGELOG.md has a spelling
error in the commit description. Update the text in that entry to change
“privilages” to “privileges” while keeping the rest of the commit message and
link intact.

- Fix(api): fix empty volume response [3011c77](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/3011c7747e814545701f5134442a1ff7d3522f70)

### Tests and Code Check

- Audit(coderabbit): add additional error handle for query block prod [814b575](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/814b575a33ee5e9db689e311ccef7b2a140e2546)
- Audit(coderabbit): add buildDSN for database connection [dc810b4](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/dc810b41d6061cd74cada3771c4c787979d5ba0b)
- Audit(sonarqube): adjust code to use variable instead of typed in params [747da47](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/747da477ff6481ebd020d7eafe0af9cefa60eebd)
- Audit(sonarqube): drop complexity for some api handler function [f29123c](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/f29123c1ceddc03462a6ae59817bc08233725458)
- Audit(sonarqube): reduce complexity of copyraw test [62070b5](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/62070b5c06fd98882e2888d93a10ce1ed29921f7)
- Audit(sonarqube): drop complexity of aggregation table test [b7b03c0](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/b7b03c05854ab34a5d2a8afeceafa8b7e47e5f63)
- Test(pkgs/schema): add validation for aggregate tables [bf55820](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/bf55820063fa9ae7b38d05359855859bb7879495)

## [0.7.1] - 2026-06-24

This version should add support to process new Gnoland testnet 13. Added support for Auth message
Expand Down
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,17 @@ train-zstd:
@read -p "Enter the amount of events to collect (default: 10000): " amount; \
amount=$${amount:-10000}; \
go run compression/cmd/main.go --config training-config.yml --amount $$amount --chain-name gnoland --dict-path ./pkgs/dict_loader/events.zstd.bin

########################################################
# Add changes to the CHANGELOG.md
########################################################

.PHONY: changelog
changelog:
@echo "Adding changes to the CHANGELOG.md"
@read -p "Enter new git tag that the changes are associated with: " tag; \
new_tag=$${tag}; \
last_tag=$$(git describe --tags --abbrev=0 --match "v*" | tail -n 1); \
last_tag+="..HEAD"; \
git cliff $$last_tag --tag $$new_tag --prepend CHANGELOG.md
@echo "Changes added to CHANGELOG.md"
73 changes: 73 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ not the best choice. And finally the nodes are not the best when it comes to sto
## Table of content

- [Table of content](#table-of-content)
- [Quick Start](#quick-start)
- [The solution](#the-solution)
- [Why TimescaleDB? And can it work on other SQL databases?](#why-timescaledb-and-can-it-work-on-other-sql-databases)
- [How does the indexer work?](#how-does-the-indexer-work)
Expand All @@ -34,6 +35,78 @@ not the best choice. And finally the nodes are not the best when it comes to sto
- [🐞 Cons](#-cons)
- [In depth documentation](#in-depth-documentation)

## Quick Start

The fastest way to get the indexer running is with Docker Compose for the database and a local binary for setup and ingestion.

**Prerequisites:** Go 1.26+ and Docker with Docker Compose.

### 1. Clone and build

```bash
git clone https://github.com/Cogwheel-Validator/spectra-gnoland-indexer.git
cd spectra-gnoland-indexer
make build-indexer
```

### 2. Configure

```bash
cp config.yml.example config.yml
```

Open `config.yml` and set the `rpc` field to your Gnoland node RPC endpoint (e.g. `https://gnoland-testnet-rpc.cogwheel.zone`). Make sure `chain_name` matches the name you want to use for the database.

### 3. Start the database

```bash
docker compose up -d timescaledb
```

This starts a TimescaleDB instance on port 5432 with the default password `12345678`. Change the password in `docker-compose.yml` for any non-local deployment.

### 4. Initialize the database

```bash
./build/indexer setup create-db \
--db-host localhost --db-port 5432 \
--db-user postgres --db-name postgres \
--ssl-mode disable --new-db-name gnoland --chain-name gnoland
```

When prompted, enter the postgres password. This creates the schema and all necessary tables.

Optionally create a dedicated writer user:

```bash
./build/indexer setup create-user writer \
--db-host localhost --db-port 5432 \
--db-user postgres --db-name postgres \
--ssl-mode disable --privilege writer
```

### 5. Run the indexer

**Historic mode** — index a specific block range (useful for initial sync):

```bash
./build/indexer run historic --config config.yml --from-height 1 --to-height 50000
```

**Live mode** — follow the chain tip in real time after historic sync is done:

```bash
./build/indexer run live --config config.yml
```

If you want to start from the current chain tip without syncing history, add `--skip-db-check`:

```bash
./build/indexer run live --config config.yml --skip-db-check
```

For production deployment, systemd service examples, Docker usage, and configuration details see [docs/setup.md](docs/setup.md).

## The solution

The data in the blockchain is mostly tied to blocks, however when any kind of analytics is needed we need some
Expand Down
3 changes: 3 additions & 0 deletions api/config/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ type ApiEnv struct {
ApiDbPassword string `env:"API_DB_PASSWORD" envDefault:"12345678"`
ApiDbName string `env:"API_DB_NAME" envDefault:"gnoland"`
ApiDbSslmode string `env:"API_DB_SSLMODE" envDefault:"disable"`
ApiDbSslRootCert string `env:"API_DB_SSLROOTCERT" envDefault:""`
ApiDbSslCert string `env:"API_DB_SSLCERT" envDefault:""`
ApiDbSslKey string `env:"API_DB_SSLKEY" envDefault:""`
ApiDbPoolMaxConns int `env:"API_DB_POOL_MAX_CONNS" envDefault:"50"`
ApiDbPoolMinConns int `env:"API_DB_POOL_MIN_CONNS" envDefault:"10"`
ApiDbPoolMaxConnLifetime time.Duration `env:"API_DB_POOL_MAX_CONN_LIFETIME" envDefault:"10m"`
Expand Down
111 changes: 59 additions & 52 deletions api/handlers/address.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,11 @@ func (h *AddressHandler) GetAddressTxs(
ctx context.Context,
input *humatypes.AddressGetInput,
) (*humatypes.AddressGetOutput, error) {
var fromTs, toTs *time.Time
if !input.FromTimestamp.IsZero() {
fromTs = &input.FromTimestamp
}
if !input.ToTimestamp.IsZero() {
toTs = &input.ToTimestamp
}
if (fromTs == nil) != (toTs == nil) {
return nil, badRequest("from_timestamp and to_timestamp must both be set or both be unset")
fromTs, toTs, err := parseTimestampPair(input.FromTimestamp, input.ToTimestamp)
if err != nil {
return nil, err
}
timestampMode := fromTs != nil && toTs != nil
timestampMode := fromTs != nil

var limit *uint64
if input.Limit != 0 {
Expand All @@ -91,57 +85,70 @@ func (h *AddressHandler) GetAddressTxs(
}

addressTxs, hasMore, err := h.db.GetAddressTxs(
ctx,
input.Address,
h.chainName,
fromTs,
toTs,
limit,
cursor,
direction,
ctx, input.Address, h.chainName, fromTs, toTs, limit, cursor, direction,
)
if err != nil {
return nil, mapDbError("GetAddressTxs", "address not found", err)
}

body := humatypes.AddressTxsBody{
AddressTxs: *addressTxs,
body, err := buildAddressTxsBody(*addressTxs, hasMore, direction, cursor)
if err != nil {
return nil, err
}
return &humatypes.AddressGetOutput{Body: body}, nil
}

if len(*addressTxs) > 0 {
rows := *addressTxs
newest := rows[0]
oldest := rows[len(rows)-1]
newestCur, err := makeTxCursor(newest.BlockHeight, newest.Hash)
if err != nil {
return nil, internalError("GetAddressTxs.makeTxCursor", err)
}
oldestCur, err := makeTxCursor(oldest.BlockHeight, oldest.Hash)
if err != nil {
return nil, internalError("GetAddressTxs.makeTxCursor", err)
}
func parseTimestampPair(from, to time.Time) (*time.Time, *time.Time, error) {
var fromPtr, toPtr *time.Time
if !from.IsZero() {
fromPtr = &from
}
if !to.IsZero() {
toPtr = &to
}
if (fromPtr == nil) != (toPtr == nil) {
return nil, nil, badRequest("from_timestamp and to_timestamp must both be set or both be unset")
}
return fromPtr, toPtr, nil
}

switch direction {
case database.Next:
body.HasNext = hasMore
if hasMore {
body.NextCursor = &oldestCur
}
if cursor != nil {
body.HasPrev = true
body.PrevCursor = &newestCur
}
case database.Prev:
body.HasPrev = hasMore
if hasMore {
body.PrevCursor = &newestCur
}
body.HasNext = true
func buildAddressTxsBody(
rows []database.AddressTx,
hasMore bool,
direction database.Direction,
cursor *string,
) (humatypes.AddressTxsBody, error) {
body := humatypes.AddressTxsBody{AddressTxs: rows}
if len(rows) == 0 {
return body, nil
}

newestCur, err := makeTxCursor(rows[0].BlockHeight, rows[0].Hash)
if err != nil {
return body, internalError("GetAddressTxs.makeTxCursor", err)
}
oldestCur, err := makeTxCursor(rows[len(rows)-1].BlockHeight, rows[len(rows)-1].Hash)
if err != nil {
return body, internalError("GetAddressTxs.makeTxCursor", err)
}

switch direction {
case database.Next:
body.HasNext = hasMore
if hasMore {
body.NextCursor = &oldestCur
}
if cursor != nil {
body.HasPrev = true
body.PrevCursor = &newestCur
}
case database.Prev:
body.HasPrev = hasMore
if hasMore {
body.PrevCursor = &newestCur
}
body.HasNext = true
body.NextCursor = &oldestCur
}

return &humatypes.AddressGetOutput{
Body: body,
}, nil
return body, nil
}
Loading