Repository navigation
Expand file tree
/
Copy pathdescription.yml
More file actions
84 lines (73 loc) · 3.96 KB
/
Copy pathdescription.yml
File metadata and controls
84 lines (73 loc) · 3.96 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# Descriptor for submission to https://github.com/duckdb/community-extensions
# (goes in extensions/marc21/description.yml in that repo).
extension:
name: marc21
description: Read, write, validate and edit MARC 21 bibliographic data (ISO 2709, MARCXML, MARC-in-JSON, breaker, Aleph; UTF-8 and MARC-8)
version: 2.0.1
language: C++
build: cmake
license: MIT
maintainers:
- mgbilby
repo:
github: mgbilby/duckdb-marc21
ref: d4a821d289fb0ee33c2bcf2fdbed8874a64169fa
# The registry's v1.5.6 lane (scripts/build.py keys it on DUCKDB_VERSION ==
# v1.5.6) builds this ref, which must stay on the 1.5.6 code: 9fcea062 is
# the marc21 1.5.6 release, the last commit built and served for v1.5.6.
variegata: 9fcea0628f6c8853a5f8e96d4e789f08ba801f7f
docs:
hello_world: |
-- 1. Read a binary MARC dump (gzip and globs work; MARC-8 and UTF-8
-- are detected per record and decoded to NFC Unicode)
SELECT control_number,
marc_subfield(fields, '245', 'a') AS title,
marc_subfields(fields, '650', 'a') AS subjects
FROM read_marc('catalogue/*.mrc.gz');
-- 2. One row per subfield, for frequency and completeness work
SELECT tag, code, count(*) FROM read_marc_subfields('dump.mrc')
GROUP BY ALL ORDER BY 3 DESC;
-- 3. MARCXML, MARC-in-JSON (incl. FOLIO SRS shape), breaker, Aleph
SELECT * FROM read_marcxml('export.xml');
SELECT * FROM read_marcjson('records.ndjson');
-- 4. Address by MARCspec; validate structurally or against Avram
SELECT marc_spec(leader, fields, '008/35-37') FROM read_marc('dump.mrc');
SELECT marc_validate(leader, fields) FROM read_marc('dump.mrc');
-- 5. Batch-edit and write back — full encoding crosswalk
COPY (SELECT marc_set_subfield(fields, '040', 'a', 'XX-XxUND') AS fields,
leader
FROM read_marc('in.mrc'))
TO 'out.mrc' (FORMAT marc, ENCODING 'marc8');
-- 6. Or write MARCXML / breaker / MARC-in-JSON
COPY (SELECT * FROM read_marc('in.mrc')) TO 'out.xml' (FORMAT marcxml);
extended_description: |
A cleanly licensed MARC 21 toolkit for DuckDB, aimed at cataloging and
metadata librarians: the parser, MARC-8 tables and writer are implemented
from ISO 2709 and the Library of Congress specifications only, not from
previous software, and verified by differential fuzzing and round-trip
testing.
### Read
`read_marc` (nested LIST(STRUCT) per record), `read_marc_subfields`
(one row per subfield), `read_marc_raw` (original bytes + error column),
`read_marcxml`, `read_marc_breaker`, `read_marcjson` (community shape,
NDJSON and FOLIO SRS envelopes), `read_alephseq`. Glob patterns, gzip,
and DuckDB FileSystem protocols (httpfs URLs — including SRU and OAI-PMH
responses, whose envelopes are handled). `read_z3950` retrieves records
live from Z39.50 servers (Library of Congress, OCLC, most ILSes) with a
built-in dependency-free client. On DuckDB v2.0 the readers are
multi-file readers (lists of paths, `filename` and `file_index` columns,
`union_by_name`, `hive_partitioning`, parallel across files). Scans
parallelise across and within files.
### Write
`COPY ... TO (FORMAT marc | marcxml | mrk | marcjson)`. ISO 2709 output
in UTF-8 or MARC-8 (full LC code tables: ANSEL, Cyrillic, Greek, Hebrew,
Arabic, EACC with escape designations; NCR fallback), round-trip tested.
### Query, validate, edit
MARCspec addressing (`marc_spec`), structural and Avram-schema validation
(`marc_validate`, `marc_validate_avram`), format detection, record
editing (`marc_set_subfield`, `marc_remove_fields`, `marc_replace_values`,
...), merge with field protection (`marc_merge`), record diff
(`marc_diff`), identifier normalisation (`marc_isbn13`, `marc_issn`,
`marc_lccn`, `marc_oclc`), NACO normalisation and match keys for
deduplication, and a FOLIO Source Record Storage bridge
(`marc_parse_json`) for in-place analytics on an ATTACHed FOLIO database.