Skip to content

Commit f6ec1b3

Browse files
authored
Speed up codd by parsing SQL concurrently (#209)
Parse concurrently to sending SQL to postgresql for a 19% reduction in application time in some cases
1 parent 8e76873 commit f6ec1b3

File tree

10 files changed

+803
-528
lines changed

10 files changed

+803
-528
lines changed

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,7 @@ We recommend following these instructions closely to catch as many possible issu
217217
`pg_dump` does not dump all of the schema state that codd checks. A few examples include role related state, the database's default transaction isolation level and deferredness, among other settings. So check that it isn't the case that you get different schemas when that happens. If you've checked with `psql` and everything looks to be the same please report a bug in codd.
218218

219219
2. ### Will codd run out of memory or system resources if my migration files are too large or too many?
220-
Most likely not. Codd reads migrations from disk in streaming fashion and keeps in memory only a single statement at a time. For `COPY` statements, codd uses a constant-size buffer to stream-read the contents and achieve bounded memory usage while staying fast. Also, codd does not open more than one migration file simultaneously to stay well below typical file handle limits imposed by the shell or operating system, and that is also assured through an automated test that runs in CI with `strace`. Codd does keep metadata about all pending migrations and the expected schema in memory, but those should be fairly small.
220+
Most likely not. Codd reads migrations from disk in streaming fashion and keeps in memory only a constant number of SQL statements at a time. For `COPY` statements, codd uses a constant-size buffer to stream-read the contents and achieve bounded memory usage while staying fast. Also, codd does not open more than one migration file simultaneously to stay well below typical file handle limits imposed by the shell or operating system, and that is also assured through an automated test that runs in CI with `strace`. Codd does keep metadata about all pending migrations and the expected schema in memory, but those should be fairly small.
221221

222222
3. ### Will codd handle SQL errors nicely?
223-
Codd tries to do the "best possible thing" even in rather unusual situations. It will retry sets of consecutive in-txn migrations atomically so as not to leave your database in an intermediary state. Even for no-txn migrations, codd will retry the failing statement instead of entire migrations, and _even_ if you write explicit `BEGIN..COMMIT` sections in no-txn migrations, codd will be smart enough to retry from the `BEGIN` if a statement inside that section fails. See the [retry examples](/docs/SQL-MIGRATIONS.md#examples) if you're interested. Basically, we hope you should be able to write your migrations however you want and rely comfortably on the fact that codd should do the reasonable thing when handling errors.
223+
Codd tries to do the "best possible thing" even in rather unusual situations. It will retry sets of consecutive in-txn migrations atomically so as not to leave your database in an intermediary state. Even for no-txn migrations, codd will retry only the failing statement instead of entire migrations, and _even_ if you write explicit `BEGIN..COMMIT` sections in no-txn migrations, codd will be smart enough to retry from the `BEGIN` if a statement inside that section fails. See the [retry examples](/docs/SQL-MIGRATIONS.md#examples) if you're interested. Basically, we hope you should be able to write your migrations however you want and rely comfortably on the fact that codd should do the reasonable thing when handling errors.

0 commit comments

Comments
 (0)