Code

Feat/preorg tsv event replay#1

Code

Open

csgui wants to merge 134 commits intodevelopcsgui/stacks-blockchain-api:developfrom

feat/preorg-tsv-event-replaycsgui/stacks-blockchain-api:feat/preorg-tsv-event-replayCopy head branch name to clipboard

Commits

Commits on Apr 18, 2022

• refactor: split postgres-store into several files

  rafaelcr

  committed
  8718112

  Copy full SHA for 8718112

• refactor: inherit all from PgStore

  rafaelcr

  committed
  fcbe844

  Copy full SHA for fcbe844

Commits on Apr 19, 2022

• refactor: start using PgStore everywhere

  rafaelcr

  committed
  491a632

  Copy full SHA for 491a632

• refactor: fix test imports

  rafaelcr

  committed
  c817352

  Copy full SHA for c817352

• refactor: restore notifier into PgStore

  rafaelcr

  committed
  c8891fa

  Copy full SHA for c8891fa

• refactor: rename to PgWriteStore

  rafaelcr

  committed
  e6f6119

  Copy full SHA for e6f6119

• Merge branch 'develop' into feat/pg-refactor

  rafaelcr

  authored
  30f80d9

  Copy full SHA for 30f80d9

Commits on Apr 20, 2022

• fix: tests and lint

  rafaelcr

  committed
  2bba410

  Copy full SHA for 2bba410

• fix: unused exports

  rafaelcr

  committed
  fffec65

  Copy full SHA for fffec65

Commits on Apr 22, 2022

• refactor: pg-store converted

  rafaelcr

  committed
  737bd38

  Copy full SHA for 737bd38

Commits on Apr 25, 2022

• refactor: pg-write-store

  rafaelcr

  committed
  56050ca

  Copy full SHA for 56050ca

• refactor: use legacy pg for event requests, notifier and migrations

  rafaelcr

  committed
  9588295

  Copy full SHA for 9588295

Commits on Apr 26, 2022

• refactor: update tests

  rafaelcr

  committed
  543bb86

  Copy full SHA for 543bb86

• refactor: tx insert mode

  rafaelcr

  committed
  e02e3ce

  Copy full SHA for e02e3ce

• refactor: first test passing

  rafaelcr

  committed
  d105d91

  Copy full SHA for d105d91

Commits on Apr 27, 2022

• refactor: transformed some inserts

  rafaelcr

  committed
  77af0b9

  Copy full SHA for 77af0b9

• refactor: transform all inserts

  rafaelcr

  committed
  c309b26

  Copy full SHA for c309b26

• refactor: fix token-tests

  rafaelcr

  committed
  4e6484a

  Copy full SHA for 4e6484a

• refactor: some more tests fixed

  rafaelcr

  committed
  e5ecbc7

  Copy full SHA for e5ecbc7

• fix: more tests

  rafaelcr

  committed
  8706f98

  Copy full SHA for 8706f98

• fix: nasty mempool query

  rafaelcr

  committed
  e36459a

  Copy full SHA for e36459a

• fix: microblock hash test

  rafaelcr

  committed
  6190ed5

  Copy full SHA for 6190ed5

• fix: events weird test

  rafaelcr

  committed
  0a72a8c

  Copy full SHA for 0a72a8c

Commits on Apr 28, 2022

• fix: api tests all passing

  rafaelcr

  committed
  1de3145

  Copy full SHA for 1de3145

• fix: mb tests

  rafaelcr

  committed
  df0e291

  Copy full SHA for df0e291

• fix: bns code

  rafaelcr

  committed
  d20d546

  Copy full SHA for d20d546

• fix: datastore connection tests

  rafaelcr

  committed
  780b253

  Copy full SHA for 780b253

• fix: rosetta, bns, token tests

  rafaelcr

  committed
  dec3711

  Copy full SHA for dec3711

Commits on Apr 29, 2022

• fix: faucet write db

  rafaelcr

  committed
  4f42d3d

  Copy full SHA for 4f42d3d

• fix: write store to api server for faucet

  rafaelcr

  committed
  5384367

  Copy full SHA for 5384367

• style: cleanup

  rafaelcr

  committed
  e70da5f

  Copy full SHA for e70da5f

• fix: faucet launch

  rafaelcr

  committed
  359a9b8

  Copy full SHA for 359a9b8

• fix: upgrade postgres

  rafaelcr

  committed
  02e44d4

  Copy full SHA for 02e44d4

• fix: test launch

  rafaelcr

  committed
  bffdd82

  Copy full SHA for bffdd82

• fix: remove type transform

  rafaelcr

  committed
  1687660

  Copy full SHA for 1687660

• refactor: go back to buffer fields

  rafaelcr

  committed
  eb4f0eb

  Copy full SHA for eb4f0eb

• fix: insert numeric as string so values can fit

  rafaelcr

  committed
  29e0380

  Copy full SHA for 29e0380

• refactor: move all types to common.ts

  rafaelcr

  committed
  99ad314

  Copy full SHA for 99ad314

• style: comments

  rafaelcr

  committed
  b083fe4

  Copy full SHA for b083fe4

Commits on Apr 30, 2022

• fix: rosetta cli tests

  rafaelcr

  committed
  98e4e68

  Copy full SHA for 98e4e68

• fix: undo launch custom

  rafaelcr

  committed
  8eb2a3a

  Copy full SHA for 8eb2a3a

• refactor: bytea type, api tests pass

  rafaelcr

  committed
  22e666a

  Copy full SHA for 22e666a

Commits on May 1, 2022

• fix: pg uri test

  rafaelcr

  committed
  8d6f6bc

  Copy full SHA for 8d6f6bc

• chore: define PgNumeric type

  rafaelcr

  committed
  63a030d

  Copy full SHA for 63a030d

• style: sql object references

  rafaelcr

  committed
  87bbabb

  Copy full SHA for 87bbabb

• style: clean up tx events query

  rafaelcr

  committed
  a74ea3c

  Copy full SHA for a74ea3c

• style: add explicit PgJsonb type

  rafaelcr

  committed
  4902975

  Copy full SHA for 4902975

Commits on May 4, 2022

• feat: dont parse into buffer from pg

  rafaelcr

  committed
  a0ea63d

  Copy full SHA for a0ea63d

• fix: datastore tests

  rafaelcr

  committed
  154c329

  Copy full SHA for 154c329

• fix: rosetta tests

  rafaelcr

  committed
  74195fa

  Copy full SHA for 74195fa

• fix: bns tests

  rafaelcr

  committed
  5f45bfe

  Copy full SHA for 5f45bfe

• fix: microblock tests

  rafaelcr

  committed
  d0b8e77

  Copy full SHA for d0b8e77

• chore: add nvmrc set to v16.15.0

  rafaelcr

  committed
  ea63864

  Copy full SHA for ea63864

• chore: update package-lock.json

  rafaelcr

  committed
  9b6f0f7

  Copy full SHA for 9b6f0f7

• chore: restore vscode config

  rafaelcr

  committed
  2e6dde2

  Copy full SHA for 2e6dde2

Commits on May 5, 2022

• chore: shorten poll connection name to avoid hitting pg limits

  rafaelcr

  committed
  697d5c3

  Copy full SHA for 697d5c3

Commits on May 10, 2022

• fix: remove returning statements when not required, use count instead

  rafaelcr

  committed
  2a0c1d5

  Copy full SHA for 2a0c1d5

Commits on May 12, 2022

• fix: nvmrc back to v16

  rafaelcr

  committed
  4e4978e

  Copy full SHA for 4e4978e

• fix: pool max env config

  rafaelcr

  committed
  f7e68ef

  Copy full SHA for f7e68ef

Commits on May 13, 2022

• feat: implement new "pre-org" event-replay mode

  zone117x

  committed
  0e5b90d

  Copy full SHA for 0e5b90d

• feat: batch sql inserts

  zone117x

  committed
  2e5c983

  Copy full SHA for 2e5c983

• test(perf): use pg COPY stream for streaming insert

  zone117x

  committed
  90eb686

  Copy full SHA for 90eb686

• test: for reference, commit several approaches for fast pg inserting

  zone117x

  committed
  eddbeed

  Copy full SHA for eddbeed

• temp: commit read cpu profiling output

  zone117x

  committed
  b61e232

  Copy full SHA for b61e232

• feat: pre-org mode inserts for /new_burn_block data

  zone117x

  committed
  8d9730a

  Copy full SHA for 8d9730a

• feat: pre-org insert into `blocks` table from /new_block

  zone117x

  committed
  07843d1

  Copy full SHA for 07843d1

• feat: pre-org insert into `microblocks` table from /new_block

  zone117x

  committed
  24a3b80

  Copy full SHA for 24a3b80

• feat: pre-org insert into `txs` table from /new_block

  zone117x

  committed
  60b4949

  Copy full SHA for 60b4949

• feat: pre-org insert into `stx_events` table from /new_block

  zone117x

  committed
  b1a75b4

  Copy full SHA for b1a75b4

• feat: pre-org insert into `principal_stx_txs` table from /new_block

  zone117x

  committed
  a4be310

  Copy full SHA for a4be310

• feat: pre-org insert into `contract_logs` table from /new_block

  zone117x

  committed
  666bdd0

  Copy full SHA for 666bdd0

• feat: pre-org insert into `stx_lock_events` table from /new_block

  zone117x

  committed
  b8c7e0f

  Copy full SHA for b8c7e0f

• feat: pre-org insert into `ft_events`, `nft_events`, and `smart_contracts` tables from /new_block

  zone117x

  committed
  6ebdea8

  Copy full SHA for 6ebdea8

• feat: pre-org insert into `zonefiles`, `names`, and `namespaces` tables from /new_block

  zone117x

  committed
  874e1da

  Copy full SHA for 874e1da

• feat: pre-org insert into `zonefiles` tables from /attachments/new event data

  zone117x

  committed
  a80b84e

  Copy full SHA for a80b84e

• chore: cleanup duplicate attachment parsing code

  zone117x

  committed
  157ac2f

  Copy full SHA for 157ac2f

• feat: pre-org insert into `subdomains` and `zonefiles` tables from /attachments/new event data

  zone117x

  committed
  115741e

  Copy full SHA for 115741e

• chore: logging improvements

  zone117x

  committed
  77f5654

  Copy full SHA for 77f5654

• chore: move `preorg` logic into separate file

  zone117x

  committed
  daa746b

  Copy full SHA for daa746b

• chore: add more logging to `insertRawEvents`

  zone117x

  committed
  9a71d93

  Copy full SHA for 9a71d93

• feat: cumulative time tracking and reporting for each sql function during `preorg` event import

  zone117x

  committed
  d03d7b4

  Copy full SHA for d03d7b4

• feat: batch inserts for 2x speedup to `txs` table updates

  zone117x

  committed
  e24a0c2

  Copy full SHA for e24a0c2

• feat: batch inserts for 3x speedup to `principal_stx_txs` table updates

  zone117x

  committed
  88b1883

  Copy full SHA for 88b1883

• feat: batch inserts for 25% speedup to `event_observer_requests` table updates

  zone117x

  committed
  f2cdf76

  Copy full SHA for f2cdf76

• feat: batch inserts for 25% speedup to `stx_events` table updates

  zone117x

  committed
  b39ac4a

  Copy full SHA for b39ac4a

• feat: batch inserts for 2x speedup to `ft_events` table updates

  zone117x

  committed
  cd01ff8

  Copy full SHA for cd01ff8

• feat: batch inserts for 3x speedup to n`ft_events` table updates

  zone117x

  committed
  48edc7b

  Copy full SHA for 48edc7b

• feat: batch inserts for 2x speedup to `contract_logs` table updates

  zone117x

  committed
  79371f3

  Copy full SHA for 79371f3

• chore: fix log strings

  zone117x

  committed
  a1fbf07

  Copy full SHA for a1fbf07

• feat: batched inserts for `blocks` and `microblocks` tables

  zone117x

  committed
  cdb1ac9

  Copy full SHA for cdb1ac9

• feat: batch inserts for 10x speedup to `names` and `zonefiles` table updates

  zone117x

  committed
  3a9f1e6

  Copy full SHA for 3a9f1e6

• chore: fix typos

  zone117x

  committed
  4ba0e4b

  Copy full SHA for 4ba0e4b

• chore: revert no longer used code

  zone117x

  committed
  3c081d4

  Copy full SHA for 3c081d4

• feat: run `db.FinishEventReplay()` in `preorg` mode

  zone117x

  committed
  ffd872f

  Copy full SHA for ffd872f

• feat: support for inserting unpruned & un-org'd for last N (default 256) events

  zone117x

  committed
  c051915

  Copy full SHA for c051915

• fix: populate bns subdomain block data during bulk insertion

  zone117x

  committed
  5760246

  Copy full SHA for 5760246

• feat: configurable sql index method for identifier columns (default to `btree`, opt-in to `hash`)

  zone117x

  committed
  f73ac5d

  Copy full SHA for f73ac5d

• chore: post-rebase lint fixes

  zone117x

  committed
  ae23acc

  Copy full SHA for ae23acc

• chore: fix more linter errors

  zone117x

  committed
  343ffaa

  Copy full SHA for 343ffaa

• chore(ci): test event-import perf in gh actions

  zone117x

  committed
  d2c15e1

  Copy full SHA for d2c15e1

• fix(tests): repair `Block execution cost` unit test

  zone117x

  committed
  d9ab137

  Copy full SHA for d9ab137

• chore: logging improvements

  zone117x

  committed
  9f1d271

  Copy full SHA for 9f1d271

• perf: run importing in parallel, about 20% faster

  zone117x

  committed
  d2e70f1

  Copy full SHA for d2e70f1

• perf: leave indexes alone on `event_observer_requests` table

  zone117x

  committed
  ec76c56

  Copy full SHA for ec76c56

• feat: write temporary tsv preprocessing file output to /tmp dir

  zone117x

  committed
  8530774

  Copy full SHA for 8530774

• feat: perform table re-indexing in parallel, around 40% faster

  zone117x

  committed
  aee7d1a

  Copy full SHA for aee7d1a

• chore: lint fixes

  zone117x

  committed
  d6c0bb0

  Copy full SHA for d6c0bb0

• chore: time tracking for parallel /new_block table reindexing

  zone117x

  committed
  4781372

  Copy full SHA for 4781372

Commits on Jun 6, 2022

• Merge branch 'develop' into feat/preorg-tsv-event-replay

  Show description for e03c656
  zone117x

  committed
  e03c656

  Copy full SHA for e03c656

• chore: update `postgres` lib to latest stable

  zone117x

  committed
  4d74742

  Copy full SHA for 4d74742

• chore: restore vscode debug config to archival event replay

  zone117x

  committed
  0711613

  Copy full SHA for 0711613

• chore: always use default (btree) index method for `sender` and `recipient` text columns in events tables

  zone117x

  committed
  07dcaf7

  Copy full SHA for 07dcaf7

• chore: remove cpuprofile files, should not have been committed

  zone117x

  committed
  2d64210

  Copy full SHA for 2d64210

Commits on Jun 7, 2022

• chore: reduce in-memory buffer sizes

  zone117x

  committed
  54a4151

  Copy full SHA for 54a4151

• chore: pin client and docs package.json versions

  zone117x

  committed
  747ccaf

  Copy full SHA for 747ccaf

• chore: use npm ci everywhere

  zone117x

  committed
  b72c162

  Copy full SHA for b72c162

• chore: update client and docs package-lock.json files

  zone117x

  committed
  b706764

  Copy full SHA for b706764

• chore: fix root package-lock.json

  zone117x

  committed
  e624fbd

  Copy full SHA for e624fbd

• chore: fix docs package-lock.json

  zone117x

  committed
  2bbb3f7

  Copy full SHA for 2bbb3f7

• chore: fix docs package-lock.json

  zone117x

  committed
  70d0028

  Copy full SHA for 70d0028

• chore: fix docs package-lock.json, attempt 3

  zone117x

  committed
  8f5243f

  Copy full SHA for 8f5243f

• chore: fix docs package-lock.json, attempt 4

  zone117x

  committed
  dbda2ae

  Copy full SHA for dbda2ae

Commits on Jun 9, 2022

• chore: log process exit code

  zone117x

  committed
  b7f8d84

  Copy full SHA for b7f8d84

Commits on Jun 16, 2022

• chore: preorg vscode debug config

  zone117x

  committed
  533b56a

  Copy full SHA for 533b56a

• Merge branch 'develop' into feat/preorg-tsv-event-replay

  Show description for 5bf8e67
  zone117x

  committed
  5bf8e67

  Copy full SHA for 5bf8e67

Commits on Aug 15, 2022

• chore: breakup large `preOrgTsvImport` fn

  zone117x

  committed
  724ac4c

  Copy full SHA for 724ac4c

• Merge branch 'develop' into feat/preorg-tsv-event-replay

  Show description for 2232a4d
  zone117x

  committed
  2232a4d

  Copy full SHA for 2232a4d

• chore: post-merge fixes

  zone117x

  committed
  f3589fd

  Copy full SHA for f3589fd

• chore: remove parallel event pg inserts, simplify table index toggling, remove needless sql transactions

  zone117x

  committed
  2a238db

  Copy full SHA for 2a238db

• chore: progress on more memory-friendly tsv streaming and transforming

  zone117x

  committed
  1d70ee3

  Copy full SHA for 1d70ee3

• chore: rewrite the /new_block preorg ingest loop to use less nodejs memory, postgres memory leak workaround

  zone117x

  committed
  0892e9e

  Copy full SHA for 0892e9e

Commits on Aug 16, 2022

• chore: fix pg memory usage issue caused by large batch inserts into the `txs` table

  zone117x

  committed
  80446ec

  Copy full SHA for 80446ec

• chore: lint fix

  zone117x

  committed
  813964c

  Copy full SHA for 813964c

Commits on Oct 28, 2022

• ci: patch

  rafaelcr

  authored
  405a0f3

  Copy full SHA for 405a0f3