v0.7.2#20
Merged
Merged
Conversation
This function did exist before however there was a possibility to create a bug if indexer worked for a limited time or there simply isn't enough data. There was a temporary solution to return a true mean value. Now the API should be able to return the needed data even with limited data inserted into the database
This only marks some initial progress to support the send data using SSL
and away from the db
The process now uses typed system so the program doesn't need to work with map[string]any. It should provide more maintainable code.
|
Warning Review limit reached
Next review available in: 56 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. 📝 WalkthroughWalkthroughThis PR renames ChangesSchema, decoder, database, and runtime wiring
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
indexer/db_init/tables.go (1)
579-588: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winQuote the role name and password before executing
CREATE USER.userNameand the prompted password are concatenated into SQL as-is, so a crafted value can break the statement or inject extra SQL. Use PostgreSQL-safe quoting/sanitization instead offmt.Sprintf.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@indexer/db_init/tables.go` around lines 579 - 588, The CREATE USER statement in tables.go is built by concatenating userName and the prompted password directly into SQL, so update the user creation flow to use PostgreSQL-safe quoting or parameterization instead of fmt.Sprintf. Locate the SQL assembly around the password prompt in the user creation path and ensure both the role name and password are safely escaped before db.pool.Exec is called.indexer/data_processor/operator.go (1)
153-167: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winAvoid inserting zero-value block rows after per-block failures.
processBlockreturns early on decode/parse errors, butProcessBlocksstill inserts every slot inblocksData. Track valid indexes like the transaction/validator paths before callingInsertRows.Proposed fix
blocksData := make([]s.Blocks, blockAmount) + valid := make([]bool, blockAmount) wg := sync.WaitGroup{} wg.Add(blockAmount) for idx, block := range blocks { - go d.processBlock(idx, block, &wg, blocksData) + go d.processBlock(idx, block, &wg, &valid[idx], blocksData) } wg.Wait() + + result := make([]s.Blocks, 0, blockAmount) + for idx, ok := range valid { + if ok { + result = append(result, blocksData[idx]) + } + } // add multiplier for the timeout depending on the block amount - timeout := 10*time.Second + (time.Duration(blockAmount) * time.Second / 5) + timeout := 10*time.Second + (time.Duration(len(result)) * time.Second / 5) ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() - err := d.dbPool.InsertRows(ctx, s.AsInsertable(blocksData)) + err := d.dbPool.InsertRows(ctx, s.AsInsertable(result))func (d *DataProcessor) processBlock( idx int, block *rpcClient.BlockResponse, wg *sync.WaitGroup, + valid *bool, blocksData []s.Blocks, ) { @@ blocksData[idx] = s.Blocks{ Hash: hash, Height: height, Timestamp: block.Result.Block.Header.Time, ChainID: block.Result.Block.Header.ChainID, ChainName: d.chainName, } + *valid = true }Also applies to: 187-210
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@indexer/data_processor/operator.go` around lines 153 - 167, ProcessBlocks is inserting every entry from blocksData even when processBlock exits early on decode/parse errors, leaving zero-value rows behind. Update ProcessBlocks to track only successfully processed block indexes, similar to the transaction/validator paths, and build the insert payload from that valid set before calling dbPool.InsertRows. Make the change in the code path that fills blocksData and the corresponding insertion logic so failed blocks are skipped entirely.
🧹 Nitpick comments (6)
indexer/orchestrator/operator.go (1)
270-270: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueTypo in doc comment: "smae" → "same".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@indexer/orchestrator/operator.go` at line 270, Fix the typo in the doc comment for collectTransactionsFromBlocks by changing “smae” to “same”; update the comment text only and leave the function logic unchanged.README.md (1)
63-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUpdate to modern Docker Compose command.
docker-compose(hyphen) is the deprecated v1 command. Modern Docker usesdocker compose(plugin). Update to prevent setup failures for new users.- docker-compose up -d timescaledb + docker compose up -d timescaledb🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` at line 63, The README setup command still uses the deprecated docker-compose v1 syntax. Update the documented command to the modern docker compose form so new users can start the timescaledb service without setup issues. Refer to the README setup step containing docker-compose up -d timescaledb and replace it with the current Docker Compose plugin command.cliff.toml (1)
19-19: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueTrailing
\nlikely produces an extra blank line in the regenerated header.In a triple-quoted TOML string this line already ends with a real newline; the appended
\nadds a second one, so the full-file (-o) header gains a blank line after the Semantic Versioning sentence before the closing of the block. Drop it unless the extra spacing is intended.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cliff.toml` at line 19, The generated header text in cliff.toml includes an unnecessary literal newline escape after the Semantic Versioning sentence, which can introduce an extra blank line in the output. Update the header string so the Semantic Versioning line in the TOML block ends naturally without the appended newline escape, keeping the template aligned with the surrounding header text.docs/requirements.md (1)
64-64: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueGo version drift between docs and CI.
Docs state
go 1.26.1(matchinggo.mod), but.github/workflows/govulncheck.ymlpins1.26.4. Not a defect since the doc reflects the module floor, but consider aligning the stated version with the toolchain you actually validate against to avoid confusion.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/requirements.md` at line 64, The Go version noted in the docs is out of sync with the toolchain version used in CI, which can confuse readers about what is actually validated. Update the version referenced in the requirements documentation to match the pinned Go version used by the govulncheck workflow, and keep the `go.mod`-based minimum separate if needed. Use the existing Go version reference in `requirements.md` and the CI workflow’s version pin as the source of truth when aligning the wording.indexer/decoder/registry.go (1)
162-164: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low value
strings.Join(m.Args, ",")loses argument boundaries when an arg itself contains a comma.Joining args with a bare
,makes the stored value ambiguous (e.g.["a,b","c"]is indistinguishable from["a","b","c"]). If exact reconstruction of call args matters downstream, consider a non-ambiguous encoding (JSON array, or a separator that's escaped). If this matches the prior column semantics, it can be left as-is.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@indexer/decoder/registry.go` around lines 162 - 164, The `strings.Join(m.Args, ",")` assignment in `registry.go` makes `Args` ambiguous when individual arguments can contain commas. Update the `Args` serialization in the registry path that builds the record from `m.PkgPath`, `m.Func`, and `m.Args` to use an unambiguous encoding such as a JSON array, or otherwise escape separators so the original arguments can be reconstructed exactly. If the downstream column is intentionally a legacy comma-separated field, keep it unchanged only if that semantics is required.indexer/cli/setup.go (1)
241-245: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the repeated aggregate interval.
SonarCloud is failing on the duplicated
"1 month"literal; a local constant keeps this list aligned.+ const aggregateChunkInterval = "1 month" views := []struct { agg dbinit.ContinuousAggregateDefinition segmentByCols []string chunkInterval string }{ - {s.TxCounter{}, []string{"chain_name"}, "1 month"}, + {s.TxCounter{}, []string{"chain_name"}, aggregateChunkInterval},Apply the same constant to the remaining entries.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@indexer/cli/setup.go` around lines 241 - 245, The repeated "1 month" aggregate interval in the setup list should be extracted into a local constant to keep the entries aligned and satisfy SonarCloud. Update the slice in setup.go where the metric definitions are assembled in the setup logic to use a shared constant instead of repeating the literal, and apply that constant consistently to all remaining aggregate entries in the same list.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Line 37: The changelog entry in CHANGELOG.md has a spelling error in the
commit description. Update the text in that entry to change “privilages” to
“privileges” while keeping the rest of the commit message and link intact.
- Line 30: Update the changelog entry text in CHANGELOG.md to fix the spelling
error in the commit description by changing “grammer” to “grammar”; adjust the
specific changelog item that mentions the quick start section so the user-facing
release note is spelled correctly.
- Line 24: Fix the spelling mistake in the changelog entry from “reflaction” to
“reflection” in the affected commit description. Update the user-facing text in
CHANGELOG.md so the summary line reads correctly and remains consistent with the
existing entry format.
In `@docs/requirements.md`:
- Around line 25-30: Fix the small wording typos in the storage requirements
paragraph by editing the prose in the relevant documentation section: change
“you might needs SSD” to “you might need an SSD,” and correct “arround” to
“around.” Keep the rest of the paragraph intact and make the wording read
naturally in the surrounding requirements text.
In `@docs/setup.md`:
- Around line 76-80: The privilege description in the setup docs is incomplete
and the `keymgr` role is introduced without explanation. Update the prose near
the privilege list to complete the writer privileges description in the setup
documentation, and add a short sentence explaining what `keymgr` grants and when
it should be used. Refer to the existing privilege wording around the
reader/writer/keymgr section so the documentation stays consistent.
- Line 24: Remove the duplicated word in the documentation sentence so the
phrase reads naturally; update the text in the setup guide where it mentions the
GitHub project packages, and ensure the wording around the linked packages URL
is corrected without changing the meaning.
In `@indexer/cli/setup.go`:
- Around line 83-85: The help text for the database creation command is using a
raw string literal with escaped newline text, so Cobra will display the
characters literally instead of a line break. Update the help text in the setup
command definition to use an actual newline inside the backtick string for the
Long description, keeping the command metadata in the same place so the help
output renders correctly.
- Line 70: The create-user command still exposes a stale --user flag even though
user_name is now passed as a positional argument, so remove the unused flag from
the setup in createUserCmd to keep the help text and behavior aligned. Update
the command wiring in setup so only the positional user_name path is supported,
and verify any related parsing or validation around createUserCmd no longer
references the old flag.
In `@indexer/data_processor/operator.go`:
- Around line 443-448: Message conversion failures are currently being dropped
because nil entries from ConvertToDbMessages are skipped when building
aggregatedDbMessages, allowing ProcessMessages to succeed with missing rows.
Update the message conversion flow in ProcessMessages and the affected batch
handling around msgResults so ConvertToDbMessages errors are collected in a
separate error channel/slice and returned immediately instead of continuing to
Phase 2. Make sure the aggregation step that calls aggregatedDbMessages.Merge
only runs when all conversions succeeded, and abort before any insert if any
conversion error was encountered.
In `@indexer/main_operator/operator.go`:
- Around line 107-108: `HistoricProcess` is not observing the shutdown context
even though `signalHandler.RegisterOperation()` is used before it starts. Update
`orch.HistoricProcess` (and any chunk-processing path it calls) to accept
`signalHandler.Context()` and check `ctx.Done()` inside the historic chunk loop
so a SIGTERM can stop processing early. Use the existing
`signalHandler.Context()` and `HistoricProcess` symbols to wire the context
through the historic range processing flow.
In `@pkgs/database/timescaledb/insert.go`:
- Around line 43-47: InsertRows should validate that every element in the rows
batch targets the same table and column layout before building the COPY call.
Add a homogeneity check in InsertRows around the first-row metadata usage so
rows[0].TableName() and rows[0].TableColumns() are only used after confirming
all Insertable items match; otherwise reject the batch early with a clear error.
Keep the fix localized to InsertRows and the CopyFromSlice/CopyFrom path so
mixed batches cannot be copied with the wrong table metadata.
In `@pkgs/database/timescaledb/pool.go`:
- Around line 72-91: The buildDSN logic in pool.go is concatenating libpq
keyword/value pairs directly, so fields like password, SslRootCert, SslCert, and
SslKey can break parsing when they contain spaces, quotes, or backslashes.
Update buildDSN to either construct a URL-style DSN or quote/escape each value
before appending it to the string, keeping the existing config fields and
pgxpool.ParseConfig flow intact.
In `@pkgs/database/timescaledb/query_block.go`:
- Around line 313-318: The error handling in the block-time query is too broad:
the row.Scan path in the query block collapses every database failure into the
same “not enough block data” message. Update the logic around row.Scan so only
pgx.ErrNoRows maps to the friendly missing-data error, and any other scan/query
error is wrapped and returned with its original details. Keep the heightDiff ==
0 check as the data-shortage case, and ensure the query block uses errors and
github.com/jackc/pgx/v5 to distinguish these paths cleanly.
---
Outside diff comments:
In `@indexer/data_processor/operator.go`:
- Around line 153-167: ProcessBlocks is inserting every entry from blocksData
even when processBlock exits early on decode/parse errors, leaving zero-value
rows behind. Update ProcessBlocks to track only successfully processed block
indexes, similar to the transaction/validator paths, and build the insert
payload from that valid set before calling dbPool.InsertRows. Make the change in
the code path that fills blocksData and the corresponding insertion logic so
failed blocks are skipped entirely.
In `@indexer/db_init/tables.go`:
- Around line 579-588: The CREATE USER statement in tables.go is built by
concatenating userName and the prompted password directly into SQL, so update
the user creation flow to use PostgreSQL-safe quoting or parameterization
instead of fmt.Sprintf. Locate the SQL assembly around the password prompt in
the user creation path and ensure both the role name and password are safely
escaped before db.pool.Exec is called.
---
Nitpick comments:
In `@cliff.toml`:
- Line 19: The generated header text in cliff.toml includes an unnecessary
literal newline escape after the Semantic Versioning sentence, which can
introduce an extra blank line in the output. Update the header string so the
Semantic Versioning line in the TOML block ends naturally without the appended
newline escape, keeping the template aligned with the surrounding header text.
In `@docs/requirements.md`:
- Line 64: The Go version noted in the docs is out of sync with the toolchain
version used in CI, which can confuse readers about what is actually validated.
Update the version referenced in the requirements documentation to match the
pinned Go version used by the govulncheck workflow, and keep the `go.mod`-based
minimum separate if needed. Use the existing Go version reference in
`requirements.md` and the CI workflow’s version pin as the source of truth when
aligning the wording.
In `@indexer/cli/setup.go`:
- Around line 241-245: The repeated "1 month" aggregate interval in the setup
list should be extracted into a local constant to keep the entries aligned and
satisfy SonarCloud. Update the slice in setup.go where the metric definitions
are assembled in the setup logic to use a shared constant instead of repeating
the literal, and apply that constant consistently to all remaining aggregate
entries in the same list.
In `@indexer/decoder/registry.go`:
- Around line 162-164: The `strings.Join(m.Args, ",")` assignment in
`registry.go` makes `Args` ambiguous when individual arguments can contain
commas. Update the `Args` serialization in the registry path that builds the
record from `m.PkgPath`, `m.Func`, and `m.Args` to use an unambiguous encoding
such as a JSON array, or otherwise escape separators so the original arguments
can be reconstructed exactly. If the downstream column is intentionally a legacy
comma-separated field, keep it unchanged only if that semantics is required.
In `@indexer/orchestrator/operator.go`:
- Line 270: Fix the typo in the doc comment for collectTransactionsFromBlocks by
changing “smae” to “same”; update the comment text only and leave the function
logic unchanged.
In `@README.md`:
- Line 63: The README setup command still uses the deprecated docker-compose v1
syntax. Update the documented command to the modern docker compose form so new
users can start the timescaledb service without setup issues. Refer to the
README setup step containing docker-compose up -d timescaledb and replace it
with the current Docker Compose plugin command.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c8208ce1-0fad-49f5-8198-dcaec371a4f0
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (45)
.github/workflows/govulncheck.ymlCHANGELOG.mdMakefileREADME.mdapi/config/types.goapi/serve.gocliff.tomlconfig-api-example.ymlconfig.yml.exampledocs/README.mddocs/requirements.mddocs/setup.mdgo.modindexer/cli/func.goindexer/cli/root.goindexer/cli/setup.goindexer/cmd/indexer.goindexer/config/types.goindexer/data_processor/event.goindexer/data_processor/operator.goindexer/data_processor/operator_test.goindexer/data_processor/types.goindexer/db_init/tables.goindexer/decoder/convert.goindexer/decoder/decoder.goindexer/decoder/process.goindexer/decoder/product.goindexer/decoder/registry.goindexer/decoder/registry_test.goindexer/decoder/types.goindexer/main_operator/operator.goindexer/orchestrator/operator.gopkgs/database/timescaledb/insert.gopkgs/database/timescaledb/pool.gopkgs/database/timescaledb/query_block.gopkgs/database/timescaledb/query_view.gopkgs/schema/aggregates_validation_test.gopkgs/schema/assertions.gopkgs/schema/copyrow.gopkgs/schema/copyrow_test.gopkgs/schema/hypertables.gopkgs/schema/materialize.gopkgs/schema/table.gopkgs/schema/tables_validation_test.gopkgs/schema/types.go
💤 Files with no reviewable changes (3)
- docs/README.md
- indexer/decoder/process.go
- indexer/decoder/convert.go
| - Refac(indexer/schema):adjust aggregate tables similarly to db tables [5c48476](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/5c484764f2138c4c125fb4e3c823f84dcb12c7eb) | ||
| - Chore(data_processor): remove dead code [f55e7ce](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/f55e7ce535e2dbb54e4f76291f883cb85d20926f) | ||
| - Refac(schema&cli): move setup logic closer to schema [35324dd](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/35324dddf59968cc0e766b35ee0cc6997fce37a9) | ||
| - Refac(pkgs/schema): add data type assertion and reflaction testing [21b01eb](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/21b01eb9a72756721fd010cc464ed79c37f90f42) |
There was a problem hiding this comment.
Fix typo: "reflaction" → "reflection".
User-facing changelog contains a spelling error in the commit description.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.md` at line 24, Fix the spelling mistake in the changelog entry
from “reflaction” to “reflection” in the affected commit description. Update the
user-facing text in CHANGELOG.md so the summary line reads correctly and remains
consistent with the existing entry format.
| - Refac(pkgs/schema): add generics for inserting data [eca95f9](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/eca95f9d419977a79121c35b7523d4b080b9473e) | ||
| - Refac(pkgs/schema): move the copy the row logic closer to the schemas [cd43e3b](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/cd43e3bbe2ca566b2a5a40b4116c35758af16918) | ||
| - Refac(pkgs): rename sql_data_types to schema [8874feb](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/8874feb9ee8face4188a19deafbf8bada2feef36) | ||
| - Docs: update grammer and add quick start section [46f9d0b](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/46f9d0b79b42dd64da847d008e64ce00e9165144) |
There was a problem hiding this comment.
Fix typo: "grammer" → "grammar".
User-facing changelog contains a spelling error in the commit description.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.md` at line 30, Update the changelog entry text in CHANGELOG.md to
fix the spelling error in the commit description by changing “grammer” to
“grammar”; adjust the specific changelog item that mentions the quick start
section so the user-facing release note is spelled correctly.
|
|
||
| - Fix(indexer): a bug where the database pool closes before the ingestion [076f3f5](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/076f3f5f2417085cfd5266de54e1da5924f72372) | ||
| - Fix(indexer/cli): add missing tables for adding privilages to the user [74be438](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/74be438702ec7197b3d7be7c0dabe6e45ac60b76) | ||
| - Fix(ci): revert the go version for govulncheck [13ba5fc](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/commit/13ba5fc2e0403c767c3a41e1eeab864e2d111099) |
There was a problem hiding this comment.
Fix typo: "privilages" → "privileges".
User-facing changelog contains a spelling error in the commit description.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.md` at line 37, The changelog entry in CHANGELOG.md has a spelling
error in the commit description. Update the text in that entry to change
“privilages” to “privileges” while keeping the rest of the commit message and
link intact.
Comment on lines
+25
to
+30
| It can work on HDD for development but for any bigger projects where write and read | ||
| speeds are important you might needs SSD depending on what you are doing. The SATA SSD is a good choice for the | ||
| most part but you can also use the NVMe SSD for better performance. | ||
|
|
||
| As for the size it depends on the amount of the data that you are indexing. | ||
| For example the integration test did about 400K blocks and 1 million | ||
| transactions. So it took around 2.2 GB of disk space. There are a lot of things that can affect the size of the | ||
| database but at least use this as some reference point to how much space you might need. | ||
| For the storage capacity it depends. Given that the indexer is still in development it may vary and it can't be said for certain. For Gnoland Testnet 13 at arround 500K blocks and 250K transactions it used a | ||
| bit over 1.2 GB. This should give you some sort of image of how much space you might need. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix small typos in this paragraph.
you might needs SSD → you might need an SSD, and arround → around.
✏️ Proposed wording fixes
-speeds are important you might needs SSD depending on what you are doing. The SATA SSD is a good choice for the
+speeds are important you might need an SSD depending on what you are doing. The SATA SSD is a good choice for the
@@
-For the storage capacity it depends. Given that the indexer is still in development it may vary and it can't be said for certain. For Gnoland Testnet 13 at arround 500K blocks and 250K transactions it used a
+For the storage capacity it depends. Given that the indexer is still in development it may vary and it can't be said for certain. For Gnoland Testnet 13 at around 500K blocks and 250K transactions it used a 📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| It can work on HDD for development but for any bigger projects where write and read | |
| speeds are important you might needs SSD depending on what you are doing. The SATA SSD is a good choice for the | |
| most part but you can also use the NVMe SSD for better performance. | |
| As for the size it depends on the amount of the data that you are indexing. | |
| For example the integration test did about 400K blocks and 1 million | |
| transactions. So it took around 2.2 GB of disk space. There are a lot of things that can affect the size of the | |
| database but at least use this as some reference point to how much space you might need. | |
| For the storage capacity it depends. Given that the indexer is still in development it may vary and it can't be said for certain. For Gnoland Testnet 13 at arround 500K blocks and 250K transactions it used a | |
| bit over 1.2 GB. This should give you some sort of image of how much space you might need. | |
| It can work on HDD for development but for any bigger projects where write and read | |
| speeds are important you might need an SSD depending on what you are doing. The SATA SSD is a good choice for the | |
| most part but you can also use the NVMe SSD for better performance. | |
| For the storage capacity it depends. Given that the indexer is still in development it may vary and it can't be said for certain. For Gnoland Testnet 13 at around 500K blocks and 250K transactions it used a | |
| bit over 1.2 GB. This should give you some sort of image of how much space you might need. |
🧰 Tools
🪛 LanguageTool
[grammar] ~29-~29: Ensure spelling is correct
Context: ... for certain. For Gnoland Testnet 13 at arround 500K blocks and 250K transactions it us...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/requirements.md` around lines 25 - 30, Fix the small wording typos in
the storage requirements paragraph by editing the prose in the relevant
documentation section: change “you might needs SSD” to “you might need an SSD,”
and correct “arround” to “around.” Keep the rest of the paragraph intact and
make the wording read naturally in the surrounding requirements text.
Source: Linters/SAST tools
| For now there aren't any prebuilt docker images. You can build your own by using the Dockerfile. It will also assume that you have the docker installed and that user has the docker group. | ||
| To build the image you can use the following command: | ||
| There are prebuilt docker images. Check the [`docker-compose.yml`](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/blob/main/docker-compose.yml) | ||
| to see how can you use them. All of the images are available at the the Github project [packages](https://github.com/orgs/Cogwheel-Validator/packages?repo_name=spectra-gnoland-indexer). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Duplicated word: "the the Github".
✏️ Fix
-to see how can you use them. All of the images are available at the the Github project [packages](https://github.com/orgs/Cogwheel-Validator/packages?repo_name=spectra-gnoland-indexer).
+to see how you can use them. All the images are available at the Github project [packages](https://github.com/orgs/Cogwheel-Validator/packages?repo_name=spectra-gnoland-indexer).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| to see how can you use them. All of the images are available at the the Github project [packages](https://github.com/orgs/Cogwheel-Validator/packages?repo_name=spectra-gnoland-indexer). | |
| to see how you can use them. All the images are available at the Github project [packages](https://github.com/orgs/Cogwheel-Validator/packages?repo_name=spectra-gnoland-indexer). |
🧰 Tools
🪛 LanguageTool
[style] ~24-~24: Consider removing “of” to be more concise
Context: ...pose.yml) to see how can you use them. All of the images are available at the the Github ...
(ALL_OF_THE)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/setup.md` at line 24, Remove the duplicated word in the documentation
sentence so the phrase reads naturally; update the text in the setup guide where
it mentions the GitHub project packages, and ensure the wording around the
linked packages URL is corrected without changing the meaning.
Source: Linters/SAST tools
Comment on lines
+43
to
+47
| pgxSlice := pgx.CopyFromSlice(len(rows), func(i int) ([]any, error) { | ||
| return rows[i].CopyRow(), nil | ||
| }) | ||
|
|
||
| columns := blocks[0].TableColumns() | ||
| tableName := blocks[0].TableName() | ||
|
|
||
| _, err := t.pool.CopyFrom(ctx, pgx.Identifier{tableName}, columns, pgxSlice) | ||
| return err | ||
| } | ||
|
|
||
| // InsertValidatorBlockSignings inserts a slice of validator block signings using COPY FROM. | ||
| func (t *TimescaleDb) InsertValidatorBlockSignings( | ||
| ctx context.Context, | ||
| validatorBlockSigning []sql_data_types.ValidatorBlockSigning, | ||
| ) error { | ||
| if len(validatorBlockSigning) == 0 { | ||
| return nil | ||
| } | ||
|
|
||
| pgxSlice := pgx.CopyFromSlice(len(validatorBlockSigning), func(i int) ([]any, error) { | ||
| return []any{ | ||
| validatorBlockSigning[i].BlockHeight, | ||
| validatorBlockSigning[i].Timestamp, | ||
| validatorBlockSigning[i].Proposer, | ||
| makePgxArray(validatorBlockSigning[i].SignedVals), | ||
| validatorBlockSigning[i].ChainName}, nil | ||
| }) | ||
|
|
||
| columns := validatorBlockSigning[0].TableColumns() | ||
| tableName := validatorBlockSigning[0].TableName() | ||
|
|
||
| _, err := t.pool.CopyFrom(ctx, pgx.Identifier{tableName}, columns, pgxSlice) | ||
| return err | ||
| } | ||
|
|
||
| // InsertTransactionsGeneral inserts a slice of transaction general data using COPY FROM. | ||
| func (t *TimescaleDb) InsertTransactionsGeneral( | ||
| ctx context.Context, | ||
| transactionsGeneral []sql_data_types.TransactionGeneral, | ||
| ) error { | ||
| if len(transactionsGeneral) == 0 { | ||
| return nil | ||
| } | ||
|
|
||
| pgxSlice := pgx.CopyFromSlice(len(transactionsGeneral), func(i int) ([]any, error) { | ||
| return []any{ | ||
| transactionsGeneral[i].TxId, | ||
| transactionsGeneral[i].ChainName, | ||
| transactionsGeneral[i].Timestamp, | ||
| transactionsGeneral[i].BlockHeight, | ||
| makePgxArray(transactionsGeneral[i].MsgTypes), | ||
| makePgxArray(transactionsGeneral[i].TxEvents), | ||
| transactionsGeneral[i].TxEventsCompressed, | ||
| transactionsGeneral[i].CompressionOn, | ||
| transactionsGeneral[i].GasUsed, | ||
| transactionsGeneral[i].GasWanted, | ||
| transactionsGeneral[i].FeeAmount, | ||
| transactionsGeneral[i].FeeDenom, | ||
| transactionsGeneral[i].Success, | ||
| transactionsGeneral[i].ErrorLog, | ||
| }, nil | ||
| }) | ||
|
|
||
| columns := transactionsGeneral[0].TableColumns() | ||
| tableName := transactionsGeneral[0].TableName() | ||
|
|
||
| _, err := t.pool.CopyFrom(ctx, pgx.Identifier{tableName}, columns, pgxSlice) | ||
| _, err := t.pool.CopyFrom(ctx, pgx.Identifier{rows[0].TableName()}, rows[0].TableColumns(), pgxSlice) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate homogeneous COPY batches before using the first row’s table metadata.
InsertRows accepts any []s.Insertable, but COPY uses only rows[0].TableName() and rows[0].TableColumns(). A mixed batch can fail late or map same-width rows into the wrong columns.
Proposed guard
+ tableName := rows[0].TableName()
+ tableColumns := rows[0].TableColumns()
+ for i := 1; i < len(rows); i++ {
+ if rows[i].TableName() != tableName || !equalStrings(rows[i].TableColumns(), tableColumns) {
+ return fmt.Errorf("mixed InsertRows batch: row %d targets %s", i, rows[i].TableName())
+ }
+ }
+
pgxSlice := pgx.CopyFromSlice(len(rows), func(i int) ([]any, error) {
return rows[i].CopyRow(), nil
})
- _, err := t.pool.CopyFrom(ctx, pgx.Identifier{rows[0].TableName()}, rows[0].TableColumns(), pgxSlice)
+ _, err := t.pool.CopyFrom(ctx, pgx.Identifier{tableName}, tableColumns, pgxSlice)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkgs/database/timescaledb/insert.go` around lines 43 - 47, InsertRows should
validate that every element in the rows batch targets the same table and column
layout before building the COPY call. Add a homogeneity check in InsertRows
around the first-row metadata usage so rows[0].TableName() and
rows[0].TableColumns() are only used after confirming all Insertable items
match; otherwise reject the batch early with a clear error. Keep the fix
localized to InsertRows and the CopyFromSlice/CopyFrom path so mixed batches
cannot be copied with the wrong table metadata.
The historic now also uses ctx.Done() like live run does.
|
There was a problem hiding this comment.
🧹 Nitpick comments (1)
api/handlers/transactions.go (1)
67-77: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueOptional: avoid rebuilding the dispatch map per request.
dispatchis reallocated on everyGetTransactionMessagecall. Since the values are receiver-bound methods, you could build it once per handler (e.g. lazily viasync.Onceor in the constructor and store onTransactionsHandler) to avoid the per-request allocation. Low impact given the small map.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@api/handlers/transactions.go` around lines 67 - 77, The dispatch map in GetTransactionMessage is rebuilt on every request, causing avoidable allocation overhead. Move the map creation out of the hot path by storing it on TransactionsHandler or initializing it once with sync.Once, and keep the receiver-bound handlers like getBankSendResponse and getMsgRunResponse wired through that cached map so the per-request call only performs lookup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@api/handlers/transactions.go`:
- Around line 67-77: The dispatch map in GetTransactionMessage is rebuilt on
every request, causing avoidable allocation overhead. Move the map creation out
of the hot path by storing it on TransactionsHandler or initializing it once
with sync.Once, and keep the receiver-bound handlers like getBankSendResponse
and getMsgRunResponse wired through that cached map so the per-request call only
performs lookup.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c6fc0388-9064-4c45-9fa7-1bbe776b9013
📒 Files selected for processing (17)
.gitignoreREADME.mdapi/handlers/address.goapi/handlers/transactions.godocs/api.mdindexer/cli/setup.goindexer/data_processor/operator.goindexer/main_operator/operator.goindexer/orchestrator/operator.goindexer/orchestrator/operator_test.gointegration/synthetic/init.gopkgs/database/timescaledb/pool.gopkgs/database/timescaledb/pool_test.gopkgs/database/timescaledb/query_block.gopkgs/schema/aggregates_validation_test.gopkgs/schema/copyrow_test.gopkgs/schema/table.go
✅ Files skipped from review due to trivial changes (3)
- docs/api.md
- .gitignore
- README.md
🚧 Files skipped from review as they are similar to previous changes (8)
- indexer/main_operator/operator.go
- pkgs/schema/aggregates_validation_test.go
- pkgs/schema/table.go
- pkgs/schema/copyrow_test.go
- pkgs/database/timescaledb/pool.go
- pkgs/database/timescaledb/query_block.go
- indexer/data_processor/operator.go
- indexer/cli/setup.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
changelogmake target and a new “Quick Start” guide.