Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/golangci.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
name: Golangci-lint
permissions:
contents: read
packages: read

on:
push:
branches: [dev, release/**]
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/gotest.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
name: Go Test
permissions:
contents: read
packages: read

on:
push:
branches: [dev, release/**]
Expand Down
10 changes: 6 additions & 4 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
name: Integration Tests

permissions:
contents: read
packages: read
on:
pull_request:
branches: [ main, dev ]
Expand Down Expand Up @@ -55,7 +57,7 @@ jobs:

- name: Initialize database schema
run: |
go run indexer/main.go setup create-db --db-user postgres --db-name postgres
go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Database schema mismatch in small test setup.

The setup command creates the schema in the postgres database (--db-name postgres), but the test config connects to the gnoland database (line 75). The schema won't exist in the target database, causing test failures.

The small test's container (line 33) initializes with POSTGRES_DB: postgres, but the test config should target postgres database, not gnoland. Alternatively, pass --db-name gnoland to the setup command.

Proposed fix (option 1: align setup with test config):

-          go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
+          go run indexer/indexer.go setup create-db --db-user postgres --db-name gnoland

Or (option 2: align test config with setup):

-          dbname: gnoland
+          dbname: postgres

Which approach is correct depends on your data model. Please verify.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
go run indexer/indexer.go setup create-db --db-user postgres --db-name gnoland
🤖 Prompt for AI Agents
In .github/workflows/integration-tests.yml around line 60, the setup command
creates the schema in the postgres database (--db-name postgres) while the test
config targets gnoland (line 75), causing a schema mismatch; fix by either (A)
changing the setup command to use --db-name gnoland so the schema is created in
the database the tests connect to, or (B) change the test configuration (and any
DB env like POSTGRES_DB) to use postgres so it matches the setup step—pick the
correct option for your data model and update the workflow lines accordingly so
setup and test DB names are identical.

env:
DB_HOST: localhost
DB_PORT: 5432
Expand Down Expand Up @@ -129,7 +131,7 @@ jobs:

- name: Initialize database schema
run: |
go run indexer/main.go setup create-db
go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Database schema mismatch in medium test setup.

Same issue as the small test: the setup command creates the schema in the postgres database (--db-name postgres), but the test config connects to gnoland (line 149). The schema won't exist in the target database.

Unlike the small test, the container (line 107) initializes with POSTGRES_DB: gnoland, suggesting the intended target is gnoland. Pass --db-name gnoland to the setup command:

-          go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
+          go run indexer/indexer.go setup create-db --db-user postgres --db-name gnoland
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
go run indexer/indexer.go setup create-db --db-user postgres --db-name gnoland
🤖 Prompt for AI Agents
.github/workflows/integration-tests.yml around line 134: the setup step runs the
indexer with --db-name postgres while the Postgres container is initialized with
POSTGRES_DB: gnoland and the test config connects to gnoland, causing a schema
mismatch. Update the setup command to pass --db-name gnoland (so the schema is
created in the same database the tests connect to) and verify any other DB-name
references in this workflow match gnoland.

env:
DB_HOST: localhost
DB_PORT: 5432
Expand Down Expand Up @@ -203,7 +205,7 @@ jobs:

- name: Initialize database schema
run: |
go run indexer/main.go setup create-db
go run indexer/indexer.go setup create-db

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

Script executed:

#!/bin/bash
# Examine the workflow file around the relevant lines
cat -n .github/workflows/integration-tests.yml | sed -n '55,65p; 130,140p; 205,215p'

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

    55	        env:
    56	          PGPASSWORD: test_password_12345678
    57	
    58	      - name: Initialize database schema
    59	        run: |
    60	          go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
    61	        env:
    62	          DB_HOST: localhost
    63	          DB_PORT: 5432
    64	          DB_USER: gnoland
    65	          DB_PASSWORD: test_password_12345678
   130	          PGPASSWORD: test_password_12345678
   131	
   132	      - name: Initialize database schema
   133	        run: |
   134	          go run indexer/indexer.go setup create-db --db-user postgres --db-name postgres
   135	        env:
   136	          DB_HOST: localhost
   137	          DB_PORT: 5432
   138	          DB_USER: postgres
   139	          DB_PASSWORD: test_password_12345678
   140	          DB_NAME: gnoland
   205	
   206	      - name: Initialize database schema
   207	        run: |
   208	          go run indexer/indexer.go setup create-db
   209	        env:
   210	          DB_HOST: localhost
   211	          DB_PORT: 5432
   212	          DB_USER: postgres
   213	          DB_PASSWORD: test_password_12345678
   214	          DB_NAME: gnoland
   215	

</shell_result>


Script executed:

#!/bin/bash
# Find indexer.go and examine its setup and create-db command handling
find . -name "indexer.go" -type f | head -5

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

./indexer/indexer.go

</shell_result>


Script executed:

#!/bin/bash
# Check file size then examine setup create-db command handling
wc -l ./indexer/indexer.go

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

13 ./indexer/indexer.go

</shell_result>


Script executed:

#!/bin/bash
# Read the full indexer.go file and search for setup command implementation
cat -n ./indexer/indexer.go && find ./indexer -name "*.go" -type f | grep -E "(setup|cmd)" | head -10

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

     1	package main
     2	
     3	import (
     4		"log"
     5	
     6		"github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd"
     7	)
     8	
     9	func main() {
    10		if err := cmd.RootCmd.Execute(); err != nil {
    11			log.Fatalf("failed to execute command: %v", err)
    12		}
    13	}
./indexer/cmd/func.go
./indexer/cmd/historic.go
./indexer/cmd/live.go
./indexer/cmd/root.go
./indexer/cmd/run.go
./indexer/cmd/setup.go

</shell_result>


Script executed:

#!/bin/bash
# Check setup.go size and examine it
wc -l ./indexer/cmd/setup.go
cat -n ./indexer/cmd/setup.go

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

272 ./indexer/cmd/setup.go
     1	package cmd
     2	
     3	import (
     4		"context"
     5		"log"
     6		"time"
     7	
     8		dbinit "github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/db_init"
     9		"github.com/Cogwheel-Validator/spectra-gnoland-indexer/pkgs/database"
    10		"github.com/Cogwheel-Validator/spectra-gnoland-indexer/pkgs/sql_data_types"
    11		"github.com/spf13/cobra"
    12	)
    13	
    14	var allowedSslModes = []string{"disable", "require", "verify-ca", "verify-full", "allow", "prefer"}
    15	
    16	// dbParams holds common database connection parameters
    17	type dbParams struct {
    18		host     string
    19		port     int
    20		user     string
    21		name     string
    22		password string
    23		sslMode  string
    24	}
    25	
    26	var setupCmd = &cobra.Command{
    27		Use:   "setup",
    28		Short: "Database setup tools",
    29		Long:  `A collection of tools to set up and manage the database for the gnoland indexer.`,
    30	}
    31	
    32	func init() {
    33		// Add subcommands
    34		setupCmd.AddCommand(createDbCmd)
    35		setupCmd.AddCommand(createUserCmd)
    36		setupCmd.AddCommand(createConfigCmd)
    37	
    38		// Common flags for both database setup commands
    39		for _, cmd := range []*cobra.Command{createDbCmd, createUserCmd} {
    40			cmd.Flags().StringP("db-host", "b", "", "The database host, default is localhost")
    41			cmd.Flags().IntP("db-port", "p", 0, "The database port, default is 5432")
    42			cmd.Flags().StringP("db-user", "u", "", "The database user, default is postgres")
    43			cmd.Flags().StringP("db-name", "d", "", "The database name, default is postgres")
    44			cmd.Flags().StringP("ssl-mode", "s", "", "The SSL mode for the database connection, default is disable")
    45		}
    46	
    47		// create-user specific flags
    48		createUserCmd.Flags().StringP("privilege", "r", "", "The privilege level for the user (reader or writer)")
    49		createUserCmd.Flags().String("user", "", "The user name for the user to create")
    50	
    51		// create-db specific flags
    52		createDbCmd.Flags().String("new-db-name", "", "The database name to create, default is gnoland")
    53		createDbCmd.Flags().String("chain-name", "", "The chain name for the database type enum, default is gnoland")
    54	
    55		// create-config specific flags
    56		createConfigCmd.Flags().StringP("config", "c", "config.yaml", "The config file name, default is config.yaml")
    57		createConfigCmd.Flags().BoolP("overwrite", "o", false, "Overwrite the existing config file, default is false")
    58	}
    59	
    60	var createDbCmd = &cobra.Command{
    61		Use:   "create-db",
    62		Short: "Create a new database named gnoland",
    63		Long: `Create a new database named gnoland for the indexer. It goes\n
    64		throught a lot of steps to create the database and insert the tables and data.`,
    65		RunE: func(cmd *cobra.Command, args []string) error {
    66			log.Printf("Initiating the cmd to set up the database for the indexer...")
    67	
    68			// Parse and validate common database flags
    69			params, err := parseCommonFlags(cmd, "postgres")
    70			if err != nil {
    71				log.Fatalf("failed to parse flags: %v", err)
    72			}
    73	
    74			// get the new database name from the flags
    75			newDbName, _ := cmd.Flags().GetString("new-db-name")
    76			if newDbName == "" {
    77				newDbName = "gnoland"
    78			}
    79	
    80			// get the chain name from the flags
    81			chainName, _ := cmd.Flags().GetString("chain-name")
    82			if chainName == "" {
    83				chainName = "gnoland"
    84			}
    85	
    86			// Prompt for password
    87			params.password, err = promptPassword()
    88			if err != nil {
    89				log.Fatalf("failed to read password: %v", err)
    90			}
    91	
    92			// Create database config
    93			dbConfig := params.createDatabaseConfig()
    94	
    95			// create a new database connection
    96			db := database.NewTimescaleDbSetup(dbConfig)
    97	
    98			// create a new database named "gnoland"
    99			// but check if the current database is "gnoland"
   100			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
   101			defer cancel()
   102			currentDb, err := db.CheckCurrentDatabaseName(ctx)
   103			if err != nil {
   104				log.Fatalf("failed to check current database name: %v", err)
   105			}
   106			log.Printf("Logged in into the database %s", currentDb)
   107	
   108			// if the current database is not "gnoland", create a new database named "gnoland"
   109			// and insert all of the tables and data from the "gnoland" database
   110			if currentDb != newDbName {
   111				// create a new database named "gnoland"
   112				log.Printf("Creating a new database named %s", newDbName)
   113				err = database.CreateDatabase(db, newDbName)
   114				if err != nil {
   115					log.Fatalf("failed to create database: %v", err)
   116				}
   117				// switch to the new database
   118				log.Printf("Switching to the new database %s", newDbName)
   119				// only for now later add dbName value, it is only for the testing now
   120				err = database.SwitchDatabase(db, dbConfig, newDbName)
   121				if err != nil {
   122					log.Fatalf("failed to switch database: %v", err)
   123				}
   124				// insert all of the tables and data from the new database
   125				// First create special types (custom postgres types that tables depend on)
   126				// and type enums
   127				specialTypes := []sql_data_types.DBSpecialType{
   128					sql_data_types.Amount{},
   129					sql_data_types.Attribute{}, // this needs to be inserted prior to event type
   130					sql_data_types.Event{},
   131				}
   132				typeEnums := []string{
   133					chainName,
   134				}
   135	
   136				// Initialize database initializer
   137				dbInit := dbinit.NewDBInitializer(db.GetPool())
   138	
   139				// Create special types first (they need to exist before tables that use them)
   140				log.Printf("Inserting all of the special types into the %s database", chainName)
   141				for _, specialType := range specialTypes {
   142					err = dbInit.CreateSpecialTypeFromStruct(specialType, specialType.TypeName())
   143					if err != nil {
   144						log.Fatalf("failed to create special type %s: %v", specialType.TypeName(), err)
   145					}
   146				}
   147	
   148				// Create type enums
   149				log.Printf("Inserting all of the type enums into the %s database", chainName)
   150				err = dbInit.CreateChainTypeEnum(typeEnums)
   151				if err != nil {
   152					log.Fatalf("failed to create type enum %s: %v", typeEnums, err)
   153				}
   154	
   155				// Create regular tables (non-time-series tables)
   156				log.Printf("Inserting all of the regular tables into the %s database", chainName)
   157				regularTables := []sql_data_types.DBTable{
   158					sql_data_types.GnoAddress{},
   159					sql_data_types.GnoValidatorAddress{},
   160				}
   161	
   162				for _, dataType := range regularTables {
   163					err = dbInit.CreateTableFromStruct(dataType, dataType.TableName())
   164					if err != nil {
   165						log.Fatalf("failed to create table %s: %v", dataType.TableName(), err)
   166					}
   167				}
   168	
   169				// Create hypertables (time-series tables with timestamp columns)
   170				log.Printf("Inserting all of the hypertables into the %s database", chainName)
   171				hypertables := []struct {
   172					table           sql_data_types.DBTable
   173					partitionColumn string
   174					chunkInterval   string
   175				}{
   176					{sql_data_types.Blocks{}, "timestamp", "1 week"},
   177					{sql_data_types.ValidatorBlockSigning{}, "timestamp", "1 week"},
   178					{sql_data_types.AddressTx{}, "timestamp", "1 week"},
   179					{sql_data_types.TransactionGeneral{}, "timestamp", "1 week"},
   180					{sql_data_types.MsgSend{}, "timestamp", "1 week"},
   181					{sql_data_types.MsgCall{}, "timestamp", "1 week"},
   182					{sql_data_types.MsgAddPackage{}, "timestamp", "1 week"},
   183					{sql_data_types.MsgRun{}, "timestamp", "1 week"},
   184				}
   185	
   186				for _, ht := range hypertables {
   187					err = dbInit.CreateHypertableFromStruct(ht.table, ht.table.TableName(), ht.partitionColumn, ht.chunkInterval)
   188					if err != nil {
   189						log.Fatalf("failed to create hypertable %s: %v", ht.table.TableName(), err)
   190					}
   191				}
   192				log.Printf("Successfully created all of the hypertables into the %s database", chainName)
   193			} else {
   194				log.Printf("The current database is %s, and it already exists", currentDb)
   195				// TODO else if the current database is "gnoland" then we need to check if the tables exist
   196				// and if they don't exist then we need to create them
   197				// also any kind of future updates to the database should be done here
   198			}
   199			return nil
   200		},
   201	}
   202	
   203	var createUserCmd = &cobra.Command{
   204		Use:   "create-user",
   205		Short: "Create a new user for the database",
   206		Long:  `Create a new user for the database. It will ask for the password and create the user.`,
   207		RunE: func(cmd *cobra.Command, args []string) error {
   208			// Parse and validate common database flags
   209			params, err := parseCommonFlags(cmd, "gnoland")
   210			if err != nil {
   211				log.Fatalf("failed to parse flags: %v", err)
   212			}
   213	
   214			// Get privilege flag
   215			privilege, _ := cmd.Flags().GetString("privilege")
   216			if privilege == "" {
   217				log.Fatalf("privilege is required")
   218			} else if privilege != "reader" && privilege != "writer" {
   219				log.Fatalf("invalid privilege: %s", privilege)
   220			}
   221	
   222			// get the user name from the flags
   223			userName, _ := cmd.Flags().GetString("user")
   224			if userName == "" {
   225				log.Fatalf("user name is required")
   226			}
   227	
   228			// Prompt for password
   229			params.password, err = promptPassword()
   230			if err != nil {
   231				log.Fatalf("failed to read password: %v", err)
   232			}
   233	
   234			// Create database config and connection
   235			dbConfig := params.createDatabaseConfig()
   236			db := database.NewTimescaleDbSetup(dbConfig)
   237			dbInit := dbinit.NewDBInitializer(db.GetPool())
   238	
   239			// Create a new user
   240			err = dbInit.CreateUser(userName)
   241			if err != nil {
   242				log.Fatalf("failed to create user: %v", err)
   243			}
   244	
   245			// Appoint privileges to the user
   246			err = dbInit.AppointPrivileges(params.user, privilege, []string{})
   247			if err != nil {
   248				log.Fatalf("failed to appoint privileges to user: %v", err)
   249			}
   250	
   251			log.Printf("Successfully created user %s with privilege %s", params.user, privilege)
   252			return nil
   253		},
   254	}
   255	
   256	var createConfigCmd = &cobra.Command{
   257		Use:   "create-config",
   258		Short: "Generate a config with default values.",
   259		Long: `Generate a config with default values. It will make a config file with default values. 
   260		You can add --overwrite to overwrite the existing config file. And you can use --config to specifly the path`,
   261		RunE: func(cmd *cobra.Command, args []string) error {
   262			// get the config file name from the flags
   263			configFileName, _ := cmd.Flags().GetString("config")
   264			if configFileName == "" {
   265				configFileName = "config.yaml"
   266			}
   267			overwrite, _ := cmd.Flags().GetBool("overwrite")
   268			createConfig(overwrite, configFileName)
   269			log.Printf("Successfully created config file %s", configFileName)
   270			return nil
   271		},
   272	}

</shell_result>


Script executed:

#!/bin/bash
# Find parseCommonFlags function definition
grep -n "func parseCommonFlags" ./indexer/cmd/*.go

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

./indexer/cmd/func.go:20:func parseCommonFlags(cmd *cobra.Command, defaultDbName string) (*dbParams, error) {

</shell_result>


Script executed:

#!/bin/bash
# Read func.go to understand parseCommonFlags and env var handling
cat -n ./indexer/cmd/func.go

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

     1	package cmd
     2	
     3	import (
     4		"fmt"
     5		"log"
     6		"os"
     7		"path/filepath"
     8		"slices"
     9		"syscall"
    10		"time"
    11	
    12		"github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/config"
    13		"github.com/Cogwheel-Validator/spectra-gnoland-indexer/pkgs/database"
    14		"github.com/spf13/cobra"
    15		"go.yaml.in/yaml/v4"
    16		"golang.org/x/term"
    17	)
    18	
    19	// parseCommonFlags extracts and validates common database flags
    20	func parseCommonFlags(cmd *cobra.Command, defaultDbName string) (*dbParams, error) {
    21		params := &dbParams{}
    22	
    23		params.host, _ = cmd.Flags().GetString("db-host")
    24		params.port, _ = cmd.Flags().GetInt("db-port")
    25		params.user, _ = cmd.Flags().GetString("db-user")
    26		params.sslMode, _ = cmd.Flags().GetString("ssl-mode")
    27		params.name, _ = cmd.Flags().GetString("db-name")
    28	
    29		// Apply environment variable fallbacks (for CI/CD)
    30		if params.host == "" {
    31			if envHost := os.Getenv("DB_HOST"); envHost != "" {
    32				params.host = envHost
    33			}
    34		}
    35		if params.port == 0 {
    36			if envPort := os.Getenv("DB_PORT"); envPort != "" {
    37				if _, err := fmt.Sscanf(envPort, "%d", &params.port); err != nil {
    38					return nil, fmt.Errorf("failed to scan port: %v", err)
    39				}
    40			}
    41		}
    42		if params.user == "" {
    43			if envUser := os.Getenv("DB_USER"); envUser != "" {
    44				params.user = envUser
    45			}
    46		}
    47		if params.name == "" {
    48			if envDbName := os.Getenv("DB_NAME"); envDbName != "" {
    49				params.name = envDbName
    50			}
    51		}
    52	
    53		// Apply defaults if still empty
    54		if params.sslMode == "" {
    55			params.sslMode = "disable"
    56		}
    57		if params.host == "" {
    58			params.host = "localhost"
    59		}
    60		if params.port == 0 {
    61			params.port = 5432
    62		}
    63		if params.user == "" {
    64			params.user = "postgres"
    65		}
    66		if params.name == "" {
    67			params.name = defaultDbName
    68		}
    69	
    70		// Validate
    71		if !slices.Contains(allowedSslModes, params.sslMode) {
    72			return nil, fmt.Errorf("invalid ssl mode: %s", params.sslMode)
    73		}
    74		if params.port < 1 || params.port > 65535 {
    75			return nil, fmt.Errorf("invalid port: %d", params.port)
    76		}
    77	
    78		return params, nil
    79	}
    80	
    81	// promptPassword prompts user for password input or reads from environment
    82	func promptPassword() (string, error) {
    83		// First check if password is provided via environment variable (for CI/CD)
    84		if envPassword := os.Getenv("DB_PASSWORD"); envPassword != "" {
    85			return envPassword, nil
    86		}
    87	
    88		// Interactive mode: prompt user for password
    89		fmt.Print("Enter the database password: ")
    90		bytePassword, err := term.ReadPassword(int(syscall.Stdin))
    91		if err != nil {
    92			return "", fmt.Errorf("failed to read password: %v", err)
    93		}
    94		fmt.Println()
    95		return string(bytePassword), nil
    96	}
    97	
    98	// createDatabaseConfig creates a DatabasePoolConfig from dbParams
    99	func (p *dbParams) createDatabaseConfig() database.DatabasePoolConfig {
   100		return database.DatabasePoolConfig{
   101			Host:                      p.host,
   102			Port:                      p.port,
   103			User:                      p.user,
   104			Dbname:                    p.name,
   105			Password:                  p.password,
   106			Sslmode:                   p.sslMode,
   107			PoolMaxConns:              10,
   108			PoolMinConns:              1,
   109			PoolMaxConnLifetime:       10 * time.Minute,
   110			PoolMaxConnIdleTime:       5 * time.Minute,
   111			PoolHealthCheckPeriod:     1 * time.Minute,
   112			PoolMaxConnLifetimeJitter: 1 * time.Minute,
   113		}
   114	}
   115	
   116	func createConfig(overwrite bool, fileName string) {
   117		if fileName == "" {
   118			fileName = "config.yml"
   119		}
   120		absolutePath, err := filepath.Abs(fileName)
   121		if err != nil {
   122			log.Fatalf("failed to get absolute path: %v", err)
   123		}
   124		fileName = absolutePath
   125		// config file
   126		config := config.Config{
   127			RpcUrl:                    "http://localhost:26657",
   128			PoolMaxConns:              50,
   129			PoolMinConns:              10,
   130			PoolMaxConnLifetime:       5 * time.Minute,
   131			PoolMaxConnIdleTime:       5 * time.Minute,
   132			PoolHealthCheckPeriod:     1 * time.Minute,
   133			PoolMaxConnLifetimeJitter: 1 * time.Minute,
   134			LivePooling:               5 * time.Second,
   135			MaxBlockChunkSize:         50,
   136			MaxTransactionChunkSize:   100,
   137			ChainName:                 "gnoland",
   138			RetryAmount:               &[]int{6}[0],
   139			Pause:                     &[]int{3}[0],
   140			PauseTime:                 &[]time.Duration{15 * time.Second}[0],
   141			ExponentialBackoff:        &[]time.Duration{2 * time.Second}[0],
   142		}
   143		// marshal the config to yaml
   144		yamlFile, err := yaml.Marshal(config)
   145		if err != nil {
   146			log.Fatalf("failed to marshal config: %v", err)
   147		}
   148		// check if the config file exists
   149		if _, err = os.Stat(fileName); err == nil {
   150			if overwrite {
   151				if writeErr := os.WriteFile(fileName, yamlFile, 0644); writeErr != nil {
   152					log.Fatalf("failed to overwrite config file: %v", writeErr)
   153				}
   154			} else {
   155				log.Fatalf("config file already exists, use --overwrite to overwrite it")
   156			}
   157		} else if os.IsNotExist(err) {
   158			if writeErr := os.WriteFile(fileName, yamlFile, 0644); writeErr != nil {
   159				log.Fatalf("failed to create config file: %v", writeErr)
   160			}
   161		} else {
   162			log.Fatalf("failed to stat config file: %v", err)
   163		}
   164	}

</shell_result>


Script executed:

#!/bin/bash
# Verify the default db name logic for create-db
sed -n '65,78p' ./indexer/cmd/setup.go

Repository: Cogwheel-Validator/spectra-gnoland-indexer


<shell_result>
Repository: Cogwheel-Validator/spectra-gnoland-indexer
Exit code: 0

stdout:

	RunE: func(cmd *cobra.Command, args []string) error {
		log.Printf("Initiating the cmd to set up the database for the indexer...")

		// Parse and validate common database flags
		params, err := parseCommonFlags(cmd, "postgres")
		if err != nil {
			log.Fatalf("failed to parse flags: %v", err)
		}

		// get the new database name from the flags
		newDbName, _ := cmd.Flags().GetString("new-db-name")
		if newDbName == "" {
			newDbName = "gnoland"
		}

</shell_result>


Based on my verification, I have confirmed the inconsistency claim. Here's what I found:

Verification Summary:

The parseCommonFlags function implements a clear precedence: command-line flags → environment variables → defaults.

  • Small/Medium tests (lines 60, 134): Explicitly pass --db-user postgres --db-name postgres, which forces connections to the "postgres" database regardless of environment variables.

  • Large test (line 208): Omits these flags and relies on DB_NAME=gnoland (line 214), causing it to connect to the "gnoland" database instead.

This creates functionally different behavior:

  • Small/Medium: Connect to "postgres" DB, then create "gnoland"
  • Large: Connect to "gnoland" DB (from env var), then attempt creation (which is skipped if DB already exists due to line 110 check in setup.go)

The inconsistency is real and impacts initialization behavior across test sizes.


Inconsistency: Large test omits --db-user and --db-name flags, causing different connection database.

The small and medium tests explicitly pass --db-user postgres --db-name postgres (lines 60, 134), connecting to the "postgres" database for initialization. The large test omits these flags (line 208) and instead relies on the DB_NAME=gnoland environment variable (line 214), connecting to "gnoland" instead.

This creates inconsistent initialization behavior: small/medium tests connect to "postgres" to create/initialize "gnoland", while the large test connects directly to "gnoland". If the large test intends to initialize from "postgres" like the other tests, add the explicit flags. If "gnoland" is the intended starting database for large tests, document this intentional difference and verify the schema initialization handles both scenarios correctly.

🤖 Prompt for AI Agents
.github/workflows/integration-tests.yml around line 208: the large test run
omits the explicit db flags and relies on DB_NAME=gnoland which causes it to
connect to a different database than the small/medium tests; to fix, make the
large test consistent with the others by adding the same flags used in
small/medium (e.g. append --db-user postgres --db-name postgres to the go run
indexer/indexer.go setup create-db invocation) or alternatively set
DB_NAME=postgres for that step and ensure the env/flag precedence is documented;
choose the option that matches intended behavior and keep all three test sizes
consistent.

env:
DB_HOST: localhost
DB_PORT: 5432
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.0] - 2025-11-26

Mostly it has some bug fixes.

### Fixed

- The indexer would go into the blocks data and store the signers from the last commit, which is actually all of the block signers from the previous block. So it would insert it like it was meant for that block height. From now on the indexer will fetch data from the /commit method and insert it properly. [de40740](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/de407407e67ae588141b361307fae18215f53a18)
- Gnoland can indeed execute multiple message types in the same transaction. The indexer wasn't able to hold this data properly and would cause an error because in the postgres primary key was attached making each message type unique. Now each message type has a message_counter which is a smallint(int16) which is used as a index for that transaction. [d647901](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/d64790181c626e2ac0137dd9cce08d10ec2b6a7c)

### Changes

- The REST API now returns a map of int16 and message data for the `/transaction/{tx_hash}/message` route. [74a35b30](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/74a35b301070bb1dab5cbadec6fe64b16ad7eb3b)
- The indexer should stop using sync.Map and use sync.Mutex with regular map to store the addresses before handing the operation to the AddressSolver function. [dde82a4](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/dde82a4ecc9bce50963fbff3f4e13a3a20b47f9d)
- The Orchestrator needs to make a fetch to the commit RPC method. This operation is done side by side with fetching the blocks method. A side effect of this is that indexer at that moment will fetch 2 times more request to the RPC, and if the Gnoland node has a limit on the amount of RPC clients that can use it, it might cause the indexer to slow down and throw errors on this requests. In future releases this should be improved. [d647901](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/d64790181c626e2ac0137dd9cce08d10ec2b6a7c)
- Updated the go version to 1.25.4

## [0.3.0] - 2025-11-10

In this release there are some fixes and improvements. The live process should work properly now and the REST API has some new routes. CLI commands are now combined with the ones from the setup cli. Some processes have been improved to use less memory.
Expand Down
32 changes: 23 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
.PHONY: build install clean build-experimental install-experimental build-api
.PHONY: build install clean build-experimental install-experimental build-api test-race integration-test test

########################################################
# Build and install the indexer
########################################################


# Get git information
GIT_COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo "unknown")
GIT_TAG := $(shell git describe --tags --exact-match 2>/dev/null || echo "")
VERSION := $(if $(GIT_TAG),$(GIT_TAG),dev-$(GIT_COMMIT))
GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "")
VERSION := $(if $(GIT_TAG),$(GIT_TAG),$(GIT_BRANCH)-$(GIT_COMMIT))

build:
mkdir -p build
go build -ldflags="-X main.Commit=$(GIT_COMMIT) -X main.Version=$(VERSION)" -o build/indexer indexer/main.go
go build -ldflags="-X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Commit=$(GIT_COMMIT) -X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Version=$(VERSION)" -o build/indexer indexer/indexer.go

install:
cd indexer && go install ./... -ldflags="-X main.Commit=$(GIT_COMMIT) -X main.Version=$(VERSION)"
go install -ldflags="-X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Commit=$(GIT_COMMIT) -X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Version=$(VERSION)" indexer/indexer.go

build-api:
mkdir -p build
Expand All @@ -22,9 +28,17 @@ clean:
# experimental build with greentea garbage collection
# use at your own risk
build-experimental:
GOEXPERIMENT=greenteagc go build -ldflags="-X main.Commit=$(GIT_COMMIT) -X main.Version=$(VERSION)" -o build/indexer-tea indexer/main.go
GOEXPERIMENT=greenteagc go build -ldflags="-X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Commit=$(GIT_COMMIT) -X github.com/Cogwheel-Validator/spectra-gnoland-indexer/indexer/cmd.Version=$(VERSION)" -o build/indexer-tea indexer/indexer.go

# experimental install with greentea garbage collection
# use at your own risk
install-experimental:
cd indexer && GOEXPERIMENT=greenteagc go install ./... -ldflags="-X main.Commit=$(GIT_COMMIT) -X main.Version=$(VERSION)"
########################################################
# Test the indexer
########################################################

test:
go test -v ./...

test-race:
go test -v -race ./...

integration-test:
cd integration && go test -v -tags=integration -timeout=20m ./...
8 changes: 7 additions & 1 deletion api/handlers/address.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,13 @@ func (h *AddressHandler) GetAddressTxs(
ctx context.Context,
input *humatypes.AddressGetInput,
) (*humatypes.AddressGetOutput, error) {
address, err := h.db.GetAddressTxs(input.Address, h.chainName, input.FromTimestamp, input.ToTimestamp)
address, err := h.db.GetAddressTxs(
ctx,
input.Address,
h.chainName,
input.FromTimestamp,
input.ToTimestamp,
)
if err != nil {
return nil, huma.Error404NotFound("Address not found", err)
}
Expand Down
10 changes: 5 additions & 5 deletions api/handlers/blocks.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ func NewBlocksHandler(db DatabaseHandler, chainName string) *BlocksHandler {
// GetBlock retrieves a block by height
func (h *BlocksHandler) GetBlock(ctx context.Context, input *humatypes.BlockGetInput) (*humatypes.BlockGetOutput, error) {
// Fetch from database
block, err := h.db.GetBlock(input.Height, h.chainName)
block, err := h.db.GetBlock(ctx, input.Height, h.chainName)
if err != nil {
return nil, huma.Error404NotFound(fmt.Sprintf("Block at height %d not found", input.Height), err)
}
Expand Down Expand Up @@ -51,7 +51,7 @@ func (h *BlocksHandler) GetFromToBlocks(
if input.FromHeight > input.ToHeight {
return nil, huma.Error400BadRequest("From height must be less than to height", nil)
}
blocks, err := h.db.GetFromToBlocks(input.FromHeight, input.ToHeight, h.chainName)
blocks, err := h.db.GetFromToBlocks(ctx, input.FromHeight, input.ToHeight, h.chainName)
if err != nil {
return nil, huma.Error404NotFound(fmt.Sprintf("Blocks from height %d to height %d not found", input.FromHeight, input.ToHeight), err)
}
Expand All @@ -76,7 +76,7 @@ func (h *BlocksHandler) GetAllBlockSigners(
input *humatypes.AllBlockSignersGetInput,
) (*humatypes.AllBlockSignersGetOutput, error) {
// Fetch from database
blockSigners, err := h.db.GetAllBlockSigners(h.chainName, input.BlockHeight)
blockSigners, err := h.db.GetAllBlockSigners(ctx, h.chainName, input.BlockHeight)
if err != nil {
return nil, huma.Error404NotFound("Block signers not found", err)
}
Expand All @@ -92,7 +92,7 @@ func (h *BlocksHandler) GetAllBlockSigners(

// Get latest block height
func (h *BlocksHandler) GetLatestBlock(ctx context.Context, _ *humatypes.LatestBlockHeightGetInput) (*humatypes.LatestBlockHeightGetOutput, error) {
block, err := h.db.GetLatestBlock(h.chainName)
block, err := h.db.GetLatestBlock(ctx, h.chainName)
if err != nil {
return nil, huma.Error404NotFound("Latest block height not found", err)
}
Expand All @@ -112,7 +112,7 @@ func (h *BlocksHandler) GetLatestBlock(ctx context.Context, _ *humatypes.LatestB
// Get last x blocks
func (h *BlocksHandler) GetLastXBlocks(ctx context.Context, input *humatypes.LastXBlocksGetInput) (*humatypes.LastXBlocksGetOutput, error) {
// Fetch from database
blocks, err := h.db.GetLastXBlocks(h.chainName, input.Amount)
blocks, err := h.db.GetLastXBlocks(ctx, h.chainName, input.Amount)
if err != nil {
return nil, huma.Error404NotFound("Last x blocks not found", err)
}
Expand Down
45 changes: 23 additions & 22 deletions api/handlers/database_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package handlers_test

import (
"context"
"fmt"
"time"

Expand All @@ -18,13 +19,13 @@ type MockDatabase struct {
msgCall map[string]*database.MsgCall
msgAddPackage map[string]*database.MsgAddPackage
msgRun map[string]*database.MsgRun
msgTypes map[string]string
msgTypes map[string][]string

shouldError bool
errorMsg string
}

func (m *MockDatabase) GetBlock(height uint64, chainName string) (*database.BlockData, error) {
func (m *MockDatabase) GetBlock(ctx context.Context, height uint64, chainName string) (*database.BlockData, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -35,7 +36,7 @@ func (m *MockDatabase) GetBlock(height uint64, chainName string) (*database.Bloc
return block, nil
}

func (m *MockDatabase) GetFromToBlocks(fromHeight uint64, toHeight uint64, chainName string) ([]*database.BlockData, error) {
func (m *MockDatabase) GetFromToBlocks(ctx context.Context, fromHeight uint64, toHeight uint64, chainName string) ([]*database.BlockData, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -50,7 +51,7 @@ func (m *MockDatabase) GetFromToBlocks(fromHeight uint64, toHeight uint64, chain
return result, nil
}

func (m *MockDatabase) GetAllBlockSigners(chainName string, blockHeight uint64) (*database.BlockSigners, error) {
func (m *MockDatabase) GetAllBlockSigners(ctx context.Context, chainName string, blockHeight uint64) (*database.BlockSigners, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -61,7 +62,7 @@ func (m *MockDatabase) GetAllBlockSigners(chainName string, blockHeight uint64)
return blockSigners, nil
}

func (m *MockDatabase) GetTransaction(txHash string, chainName string) (*database.Transaction, error) {
func (m *MockDatabase) GetTransaction(ctx context.Context, txHash string, chainName string) (*database.Transaction, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -72,7 +73,7 @@ func (m *MockDatabase) GetTransaction(txHash string, chainName string) (*databas
return transaction, nil
}

func (m *MockDatabase) GetAddressTxs(address string, chainName string, fromTimestamp time.Time, toTimestamp time.Time) (*[]database.AddressTx, error) {
func (m *MockDatabase) GetAddressTxs(ctx context.Context, address string, chainName string, fromTimestamp time.Time, toTimestamp time.Time) (*[]database.AddressTx, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -83,7 +84,7 @@ func (m *MockDatabase) GetAddressTxs(address string, chainName string, fromTimes
return addressTxs, nil
}

func (m *MockDatabase) GetLatestBlock(chainName string) (*database.BlockData, error) {
func (m *MockDatabase) GetLatestBlock(ctx context.Context, chainName string) (*database.BlockData, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -93,7 +94,7 @@ func (m *MockDatabase) GetLatestBlock(chainName string) (*database.BlockData, er
return m.latestBlock, nil
}

func (m *MockDatabase) GetLastXBlocks(chainName string, x uint64) ([]*database.BlockData, error) {
func (m *MockDatabase) GetLastXBlocks(ctx context.Context, chainName string, x uint64) ([]*database.BlockData, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -104,7 +105,7 @@ func (m *MockDatabase) GetLastXBlocks(chainName string, x uint64) ([]*database.B
return blocks, nil
}

func (m *MockDatabase) GetLastXTransactions(chainName string, x uint64) ([]*database.Transaction, error) {
func (m *MockDatabase) GetLastXTransactions(ctx context.Context, chainName string, x uint64) ([]*database.Transaction, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
Expand All @@ -115,57 +116,57 @@ func (m *MockDatabase) GetLastXTransactions(chainName string, x uint64) ([]*data
return transactions, nil
}

func (m *MockDatabase) GetMsgType(txHash string, chainName string) (string, error) {
func (m *MockDatabase) GetMsgTypes(ctx context.Context, txHash string, chainName string) ([]string, error) {
if m.shouldError {
return "", fmt.Errorf("%s", m.errorMsg)
return nil, fmt.Errorf("%s", m.errorMsg)
}
msgType, ok := m.msgTypes[txHash]
msgTypes, ok := m.msgTypes[txHash]
if !ok {
return "", fmt.Errorf("message type not found")
return nil, fmt.Errorf("message type not found")
}
return msgType, nil
return msgTypes, nil
}

func (m *MockDatabase) GetBankSend(txHash string, chainName string) (*database.BankSend, error) {
func (m *MockDatabase) GetBankSend(ctx context.Context, txHash string, chainName string) ([]*database.BankSend, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
bankSend, ok := m.bankSend[txHash]
if !ok {
return nil, fmt.Errorf("bank send not found")
}
return bankSend, nil
return []*database.BankSend{bankSend}, nil
}

func (m *MockDatabase) GetMsgCall(txHash string, chainName string) (*database.MsgCall, error) {
func (m *MockDatabase) GetMsgCall(ctx context.Context, txHash string, chainName string) ([]*database.MsgCall, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
msgCall, ok := m.msgCall[txHash]
if !ok {
return nil, fmt.Errorf("message call not found")
}
return msgCall, nil
return []*database.MsgCall{msgCall}, nil
}

func (m *MockDatabase) GetMsgAddPackage(txHash string, chainName string) (*database.MsgAddPackage, error) {
func (m *MockDatabase) GetMsgAddPackage(ctx context.Context, txHash string, chainName string) ([]*database.MsgAddPackage, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
msgAddPackage, ok := m.msgAddPackage[txHash]
if !ok {
return nil, fmt.Errorf("message add package not found")
}
return msgAddPackage, nil
return []*database.MsgAddPackage{msgAddPackage}, nil
}

func (m *MockDatabase) GetMsgRun(txHash string, chainName string) (*database.MsgRun, error) {
func (m *MockDatabase) GetMsgRun(ctx context.Context, txHash string, chainName string) ([]*database.MsgRun, error) {
if m.shouldError {
return nil, fmt.Errorf("%s", m.errorMsg)
}
msgRun, ok := m.msgRun[txHash]
if !ok {
return nil, fmt.Errorf("message run not found")
}
return msgRun, nil
return []*database.MsgRun{msgRun}, nil
}
27 changes: 14 additions & 13 deletions api/handlers/interface.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package handlers

import (
"context"
"time"

"github.com/Cogwheel-Validator/spectra-gnoland-indexer/pkgs/database"
Expand All @@ -9,21 +10,21 @@ import (
// DatabaseHandler interface for the database
type DatabaseHandler interface {
// Addresses
GetAddressTxs(address string, chainName string, fromTimestamp time.Time, toTimestamp time.Time) (*[]database.AddressTx, error)
GetAddressTxs(ctx context.Context, address string, chainName string, fromTimestamp time.Time, toTimestamp time.Time) (*[]database.AddressTx, error)

// Blocks
GetBlock(height uint64, chainName string) (*database.BlockData, error)
GetFromToBlocks(fromHeight uint64, toHeight uint64, chainName string) ([]*database.BlockData, error)
GetAllBlockSigners(chainName string, blockHeight uint64) (*database.BlockSigners, error)
GetLatestBlock(chainName string) (*database.BlockData, error)
GetLastXBlocks(chainName string, x uint64) ([]*database.BlockData, error)
GetBlock(ctx context.Context, height uint64, chainName string) (*database.BlockData, error)
GetFromToBlocks(ctx context.Context, fromHeight uint64, toHeight uint64, chainName string) ([]*database.BlockData, error)
GetAllBlockSigners(ctx context.Context, chainName string, blockHeight uint64) (*database.BlockSigners, error)
GetLatestBlock(ctx context.Context, chainName string) (*database.BlockData, error)
GetLastXBlocks(ctx context.Context, chainName string, x uint64) ([]*database.BlockData, error)

// Transactions
GetTransaction(txHash string, chainName string) (*database.Transaction, error)
GetLastXTransactions(chainName string, x uint64) ([]*database.Transaction, error)
GetMsgType(txHash string, chainName string) (string, error)
GetBankSend(txHash string, chainName string) (*database.BankSend, error)
GetMsgCall(txHash string, chainName string) (*database.MsgCall, error)
GetMsgAddPackage(txHash string, chainName string) (*database.MsgAddPackage, error)
GetMsgRun(txHash string, chainName string) (*database.MsgRun, error)
GetTransaction(ctx context.Context, txHash string, chainName string) (*database.Transaction, error)
GetLastXTransactions(ctx context.Context, chainName string, x uint64) ([]*database.Transaction, error)
GetMsgTypes(ctx context.Context, txHash string, chainName string) ([]string, error)
GetBankSend(ctx context.Context, txHash string, chainName string) ([]*database.BankSend, error)
GetMsgCall(ctx context.Context, txHash string, chainName string) ([]*database.MsgCall, error)
GetMsgAddPackage(ctx context.Context, txHash string, chainName string) ([]*database.MsgAddPackage, error)
GetMsgRun(ctx context.Context, txHash string, chainName string) ([]*database.MsgRun, error)
}
Loading