Skip to content

Commit 8ea4619

Browse files
feat: fetch metadata sources concurrently (#111)
## Summary - run independent DOI metadata sources concurrently while preserving deterministic merge priority - run PMID and arXiv fallback candidates concurrently only after their primary source fails - add `async_from_identifier()` and async feeder methods without removing the synchronous API - use the existing rate-limited `requests` session in native worker threads - use `pyodide.http.pyfetch` with `asyncio.gather`, retries, timeouts, and browser-side rate limiting in Pyodide - keep the Web UI as a thin wrapper around the Python package - retain a synchronous Web fallback until the async package version is published to PyPI - add concurrency and transport regression tests ## Why The source chain previously waited for each independent network service before starting the next. DOI metadata is merged from several independent sources, so most of that latency can overlap. Pyodide cannot rely on native Python worker threads, so the browser path uses an async Fetch-based transport instead. Fallback-only services remain lazy to avoid unnecessary API traffic. Source priority and merge order are unchanged. ## Validation - `python -m compileall` on the changed Python sources: passed - `node --check docs/wenxian.js`: passed - branch is based directly on the current upstream `master` (`a1e2c8f`) - full repository CI is expected to validate lint, typing, tests, and coverage on this PR Agent: ChatGPT Model: GPT-5.6 Sol --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent a1e2c8f commit 8ea4619

14 files changed

Lines changed: 1877 additions & 352 deletions

docs/wenxian.js

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,13 @@ import { asyncRun } from "./pyworker.js";
33
async function from_identifier(identifier) {
44
const pythonIdentifier = JSON.stringify(identifier);
55
const { results, error } = await asyncRun(`
6-
from wenxian.from_identifier import from_identifier
7-
reference = from_identifier(${pythonIdentifier})
6+
try:
7+
from wenxian.from_identifier import async_from_identifier
8+
except ImportError:
9+
from wenxian.from_identifier import from_identifier
10+
reference = from_identifier(${pythonIdentifier})
11+
else:
12+
reference = await async_from_identifier(${pythonIdentifier})
813
reference.bibtex if reference is not None and not reference.is_empty() else None
914
`);
1015
return { results, error };
@@ -21,7 +26,6 @@ document.getElementById("submit").addEventListener("click", function (event) {
2126
if (results) {
2227
output_text.textContent = results;
2328
Prism.highlightElement(output_text);
24-
// show the output
2529
output.style.display = "block";
2630
message.textContent = "";
2731
} else if (!error) {
@@ -36,9 +40,7 @@ document.getElementById("submit").addEventListener("click", function (event) {
3640
});
3741

3842
function run_example(identifier) {
39-
// fill the input
4043
document.getElementById("identifier").value = identifier;
41-
// submit the form
4244
document.getElementById("submit").click();
4345
}
4446
window.run_example = run_example;

0 commit comments

Comments
 (0)