Support TursoDB database imports - #1062
Conversation
| // sidecars are empty. | ||
| func prepareTursoDBFile(file string) error { | ||
| output, err := exec.Command("tursodb", "-q", "-m", "list", file, | ||
| "PRAGMA journal_mode = mvcc; PRAGMA wal_checkpoint(TRUNCATE); PRAGMA quick_check;").CombinedOutput() |
There was a problem hiding this comment.
In #1012, I added a spinner during quick_check, because for large DBs, it can make appear like the CLI is hanging. With this change, it doesn't look like there's a spinner.
See the runQuickCheck() function.
| // checkpoint any logical-log entries into the main file, and validate the | ||
| // resulting database. Uploading only the main file is safe once all data-bearing | ||
| // sidecars are empty. | ||
| func prepareTursoDBFile(file string) error { |
There was a problem hiding this comment.
it's weird that the checkpointing/preparing/pre-upload logic is in a file called group_flag.go 🤔
| if cipher != "" { | ||
| return nil, errors.New("remote encryption is not supported when importing tursodb (MVCC) format files") | ||
| } | ||
| case fileFormatNotSQLite: |
There was a problem hiding this comment.
fileFormatUnknown is not handled
| if err := checkpointWALBeforeUpload(file); err != nil { | ||
| return nil, err | ||
| } | ||
| if err := sqliteFileIntegrityChecks(file, cipher); err != nil { |
There was a problem hiding this comment.
This function contains some nice code (a good error message if the user uploads a dump, spinner on quick_check, max size check, page size, etc.) that isn't replicated in the Turso code. Maybe reuse this function with turso instead of sqlite?
| } | ||
| } | ||
|
|
||
| fileInfo, err := os.Stat(file) |
There was a problem hiding this comment.
This check is now performed twice for WAL-mode databases (other one is in sqliteFileIntegrityChecks).
| func checkTursoDBAvailable() error { | ||
| _, err := exec.LookPath("tursodb") | ||
| if errors.Is(err, exec.ErrNotFound) { | ||
| return errors.New("could not find tursodb on your system. Please install it to import into a TursoDB database") |
There was a problem hiding this comment.
This could be a sub-par first experience for a new user looking to import their DB into the cloud, because they'll need to install turso, then get an error, and figure out how to download tursodb.
Could we give them instructions on installing tursodb?
I wonder if there's a way we could avoid this. It doesn't make sense for a user to have to install 2 CLI's.
There was a problem hiding this comment.
we could use turso-go. I did this because we also require the user to have sqlite3 installed. Don't know if this introduces any issues for building the cli.
Co-authored-by: Mikaël Francoeur <mikael.francoeur@hotmail.com>
ec3d239 to
1b8cb41
Compare
|
|
||
| type databaseFileChecker struct { | ||
| name string | ||
| binary string |
There was a problem hiding this comment.
nit: this type is now weird, because the binary field either has value sqlite3 or ""
There was a problem hiding this comment.
yeah i've seen it and will adjust it, Im first just trying to the run code locally first to confirm it works, then will move to an interface of functions probably
There was a problem hiding this comment.
or maybe just an int to identify
There was a problem hiding this comment.
ok changed it now
b4721e9 to
3e94c78
Compare
2 participants
Summary
db importto TursoDB and route file seeds through the upload flowWhy
TursoDB imports could be routed through an incompatible create path or rejected because the uploaded file format did not match the target database. The import flow now validates and converts supported inputs without silently dropping sidecar data.
Validation
go test ./internal/tursopassesgo test ./internal/cmdcurrently fails in pre-existingTestGetReservedBytes: expected 0 reserved bytes, got 12