Skip to content

fix raw writer when batch size is too small #285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tmcgroul wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix raw writer when batch size is too small #285

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ea5b017
fix raw writer when batch size is too small
tmcgroul May 20, 2024
1174c20
rework fix
tmcgroul May 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions common/changes/@subsquid/util-internal-archive-layout/dumper-fix_2024-05-20-10-22.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"packageName": "@subsquid/util-internal-archive-layout",
"comment": "fix raw writer when batch size is too small",
"type": "patch"
}
],
"packageName": "@subsquid/util-internal-archive-layout"
}
1 change: 1 addition & 0 deletions util/util-internal-archive-layout/src/layout.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,4 +444,5 @@ function* pack<T>(items: T[], size: number): Iterable<T[]> {
}

items.splice(0, offset)
yield items
Copy link
Collaborator

@eldargab eldargab May 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks the whole idea of committing the data in fixed sized chunks. This is crucial for reproducibility.

Copy link
Contributor Author

@tmcgroul tmcgroul May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but if we keep this code as it is then some chunks will not be complete.
recently i discovered that solana raw dataset had ~5 blocks gap. 2 dump isntances were running: one from a specific block and without --last-block and another with --last-block. so that one with --last-block instead of writing a final chunk from 5 blocks were finishing its process without writing anything at all.
and it happend only because the dumper restarted before this final chunk otherwise it still would be able to write this final chunk

Copy link
Collaborator

@eldargab eldargab May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bug is not here. It's just that, blocks left unwritten after the end of ingest loop, should be saved -

squid-sdk/util/util-internal-archive-layout/src/layout.ts

Line 261 in ea5b017

if (firstBlock) {

}