Bad MARC data is being imported

[LeadSongDog]

#1620 should not have been closed.

We are still seeing errors in MARC encoding messing up edition import of umlauts in the titles in IA records.

See https://openlibrary.org/recentchanges/2025/12/18/add-book/158663607 for a very recent example, importing https://openlibrary.org/show-records/ia:b22014792_0001 incompletely, but with the defective encoding.

Observe also that the unimported OCoLC shown in the MARC links to https://search.worldcat.org/title/23745933 which has the correctly encoded title.

We don’t need Instruction f©ơr die ©œffentlich angestellten Aerzte und Wund©Þrzte in lieu of Instruction für die öffentlich angestellten Aerzte und Wundärzte

[tfmorris]

The correct reference is #1620. Comparing the binary MARC, XML MARC, OL data, and IA data shows the key problem.

IA is autodetecting the correct character set and using it to rewrite the MARC XML and its metadata, while OL is using the declared character set in the binary MARC and importing corrupted data. There's really no reason for OpenLibrary to get it wrong if Internet Archive can figure out how to get it correct.

https://openlibrary.org/show-records/ia:b22014792_0001
https://dn721908.ca.archive.org/0/items/b22014792_0001/b22014792_0001_marc.xml
https://archive.org/details/b22014792_0001

[LeadSongDog]

Corrected the ref to #1620, but still, it has been closed without solving the problem. It should be reopened.

[hornc]

Techincal investigation results:

archive.org is fetching some of these MARC records from specific sources that have been marked (via configuration) as always providing UTF-8 encoded MARC records, even if they specify MARC-8 in the leader.

When extracting this metadata, archive.org uses the override encoding to:

• write the archive.org item metadata
• generate a corrected MARC XML

It currently does not correct the leader of the stored MARC binary, which is what is being picked up by Open Library.

This will cause the same problem for any person or process that consumes the archive.org MARC records that claim the incorrect encoding.

Proposed fix:

At the source archive.org should correct the binary MARC it stores based on the connection configuration.

To re-iterate: archive.org is not using any kind of character-set detection smarts to determine these records are UTF-8 , it knows they are UTF-8 based on the z39.50 source.

[tfmorris]

At the source archive.org should correct the binary MARC it stores based on the connection configuration.

That sounds like an improvement. It might not catch all cases, but it's better than what's in place now.

As an aside, I'll note that the author of MarcEdit has said that they ignore the declared character set encoding, because it is often unreliable, and always use character set detection to determine the character encoding to use.

[pranitaurlam]

I’ve recently been investigating IA import issues related to Unicode handling
and MARC parsing (see #11637 and PR #11639).

Happy to help investigate this issue as well. Could you clarify which MARC
fields or example IA records are known to be importing incorrectly here?
If there’s a specific reproduction case, I can try to add a focused test or
help narrow down where the data corruption is occurring.