-
Notifications
You must be signed in to change notification settings - Fork 161
fix: respect access mode and file open flags #1962
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,4 +6,5 @@ | |
/duckdb_webapi.* | ||
/dist | ||
/docs | ||
/test_temp | ||
*.d.ts.map |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
import * as duckdb from '../src/'; | ||
import { randomUUID } from 'crypto'; | ||
import { unlink } from 'fs/promises'; | ||
import { mkdirSync } from 'fs'; | ||
import path from 'path'; | ||
|
||
const tmpdir = () => { | ||
const tmp = path.resolve(__dirname, '../test_temp'); | ||
mkdirSync(tmp, { recursive: true }); | ||
return tmp; | ||
} | ||
|
||
export function testNodeFS(db: () => duckdb.AsyncDuckDB): void { | ||
const files: string[] = []; | ||
|
||
afterAll(async () => { | ||
await Promise.all(files.map(file => unlink(file).catch(() => {}))); | ||
await db().flushFiles(); | ||
await db().dropFiles(); | ||
}); | ||
|
||
describe('Node FS', () => { | ||
it('Should not create an empty DB file in read-only mode for non-existent path', async () => { | ||
const tmp = tmpdir(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Usage of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hello @carlopi, I've updated the code to use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks a lot! I will let CI have a go, if it fails I will re-run with debug logging enabled (unsure if then it becomes visible, if not I can share it). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, we are doing a bunch of CI heavy stuff elsewhere, it will take a while.. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's okay. I was also trying to run tests in my local environment with some of the intermediate artifacts produced by the CI to reduce the build time, but it didn't run smoothly as the CI did 😿 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The CI still failed. There might be something changed in the codebase since the last successful run (cleaned up after force pushes) months ago. I will take a deeper look soon |
||
const filename = `duckdb_test_${randomUUID().replace(/-/g, '')}`; | ||
files.push(path.join(tmp, filename)); | ||
|
||
await expectAsync( | ||
db().open({ | ||
path: path.join(tmp, filename), | ||
accessMode: duckdb.DuckDBAccessMode.READ_ONLY, | ||
}), | ||
).toBeRejectedWithError(/database does not exist/); | ||
}); | ||
|
||
it('Should create DB file in read-write mode for non-existent path', async () => { | ||
const tmp = tmpdir(); | ||
const filename = `duckdb_test_${randomUUID().replace(/-/g, '')}`; | ||
files.push(path.join(tmp, filename)); | ||
|
||
await expectAsync( | ||
db().open({ | ||
path: path.join(tmp, filename), | ||
accessMode: duckdb.DuckDBAccessMode.READ_WRITE, | ||
}), | ||
).toBeResolved(); | ||
}); | ||
|
||
it('Should create an empty DB file in read-only mode for non-existent path with direct I/O', async () => { | ||
const tmp = tmpdir(); | ||
const filename = `duckdb_test_${randomUUID().replace(/-/g, '')}`; | ||
files.push(path.join(tmp, filename)); | ||
|
||
await expectAsync( | ||
db().open({ | ||
path: path.join(tmp, filename), | ||
accessMode: duckdb.DuckDBAccessMode.READ_ONLY, | ||
useDirectIO: true, | ||
}), | ||
).toBeRejectedWithError(/database does not exist/); | ||
}); | ||
|
||
it('Should create DB file in read-write mode for non-existent path with direct I/O', async () => { | ||
const tmp = tmpdir(); | ||
const filename = `duckdb_test_${randomUUID().replace(/-/g, '')}`; | ||
files.push(path.join(tmp, filename)); | ||
|
||
await expectAsync( | ||
db().open({ | ||
path: path.join(tmp, filename), | ||
accessMode: duckdb.DuckDBAccessMode.READ_WRITE, | ||
useDirectIO: true, | ||
}), | ||
).toBeResolved(); | ||
}); | ||
}); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Five lines below this, where we call createSyncAccessHandle, we can pass a new optional mode to OPFS to further enforce the access mode: https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle#mode
If the access mode is passed to createSyncAccessHandle, it would allow read-only connections to be opened simultaneously (for example when using the same app in multiple tabs). A big win for duckdb-wasm + OPFS.
The option is only implemented in Chrome and Edge right now, but issues exist to implement in both Webkit and Firefox:
https://bugs.webkit.org/show_bug.cgi?id=283959
https://bugzilla.mozilla.org/show_bug.cgi?id=1949462
And tests have already been added to the Web Platform Tests dashboard:
https://wpt.fyi/results/fs/FileSystemFileHandle-sync-access-handle-lock-modes.https.tentative.worker.html?label=experimental&label=master&aligned
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@amiller-gh
I believe it's fine for this PR to focus only on a single window.
I think we should first handle single-window support before addressing multiple windows.
Because the mode argument of the createSyncAccessHandle method was introduced to specify the file access mode. However, this argument has not yet been enabled in lib.webworker.d.ts (the Web Worker type definition file). As a result, using this argument in TypeScript may cause a type error. To resolve this issue, you need to either wait for the type definition file to be updated or extend the type definitions within your project.
And, I plan to create a PR for handling multiple windows, so this PR needs to be merged for that.