Epic: Fix importation of bogus authors and cleanup old data

[LeadSongDog]

There are tens of thousands of bogus author records with names * Publishing or * Books. Somewhat fewer with * Editions and other-language equivalents.

Many originate with the import of low quality records from BWB or AMZ such as https://www.betterworldbooks.com/product/detail/9783110367737
which was imported as
https://openlibrary.org/books/OL34526350M/Quantenmechanik
where the authors include
https://openlibrary.org/authors/OL9711355A/Perseus_Books_Perseus_Books_LLC.

Many (about 30%) of these author records have no associated work record. Those are low-hanging fruit that could simply be bulk removed.

More have only work records that are misattributed to the “author” with these publisher names and the “publisher” shown as "Independently Published", “CreateSpace” or the like. For these there is often another correct work record of similar title showing the correct authorship. Some heuristics might help with these.

A substantial group however are corporate authorships by publisher staff writers with no public attribution to an individual. This is particularly common in bibliographies, reference works, study notes, and textbooks.

Suggestions?

[hornc]

Junk publishers here: https://openlibrary.org/search/authors?q=Publisher&mode=everything

I'll try to get a bot task that can prune an entire edition + work + author tree based on the worst of these results.

[LeadSongDog]

Thank you. A few are real publishers with human names, sometimes shown where the author is unk or staff, but in most cases you’ll want a chainsaw, not pruning shears.

[hornc]

This is hard going -- I have removed about 2k empty publisher-as-authors which have already been cleared out and had no editions or works assigned to them, and removed many entirely non-book publishers and their work, which has between a dozen and hundreds of junk non-book records (sometimes repeated over and over again)...

This leave many 100s of junk publisher names with only one or two non-book items, inter-dispersed with more legitimate publisher-recorded-as-author, where there is probably some kind of clean up required other than simply deleting junk, but I've cleared out as much as I can easily do in an automate sweep.

I'll think on what can be done with what remains, it'll probably be more of a general identification of obvious non-book items to remove them completely.

[LeadSongDog]

@hornc Thank you for tackling this, but the job is rather bigger than that.

By way of quantification, there are currently

4448 “authors” with “Editions” in their name: https://openlibrary.org/search/authors?q=Editions&mode=everything

23104 with “Books”: https://openlibrary.org/search/authors?q=Books*

52698 with “Publish*”: https://openlibrary.org/search/authors?q=Publish*

of all these, about 60% show just one edition.

[FelixMuregi]

Hey everyone,

This is a great initiative—cleaning up these misattributed author records will make Open Library much more accurate and useful. I have a few ideas that could help speed up the process:

Automation Ideas for Cleanup:

1. Pattern-Based Filtering:

Use regex or keyword-based detection for names containing “Publish”, “Books”, “Editions”, etc., to flag likely publisher-as-author records.

Cross-check these with actual publisher databases to identify legitimate cases.

2. Edition Count Heuristic:

If an “author” has only one or two editions, it's likely junk.

Authors with many editions could be reviewed manually to avoid removing real corporate authors.

3. Duplicate Detection:

Compare book titles and metadata to find cases where the same work exists twice—one with a correct author and one with a misattributed publisher-as-author.

Where possible, merge these into the correct record.

4. Automated Merging & Deletion:

For cases where an author has zero work records, they can likely be bulk deleted.

If a work is attributed to both a real author and a “publisher” author, the publisher name could be removed automatically.

Would love to contribute if there's a way to help with the automation process! Is there a dataset or API access available for running tests on this?

Looking forward to hearing your thoughts!

[LeadSongDog]

Some nits to pick, but I’m happy to see this get some attention.

Re your #1, equivalent terms in other languages will also be useful.

Re your #2, the “author” should be matched to works, not editions, but there are many legitimately one-work authors.

Re your #3, merges need to be approached with great care to avoid conflation. Before assuming the other work record is correct it should be checked to see that there are independent reliable identifiers (LCCN, oclcn, etc) supporting that author-work pairing. Even then, disambiguating synonymous authors is nontrivial.

Re your #4, void author records may be transient for recently added works, so it may be necessary to “touch” the author record to trigger a current work count.

Thank you for looking into this.

[FelixMuregi]

Good call on these points—I see how multilingual support and identifier checks could make the process more reliable. Instead of outright merging or deleting, what if we flagged questionable records for a semi-automated review? Maybe a system that prioritizes them based on confidence scores (e.g., low edition count + no identifier match = high priority for review)?

I'm curious if Open Library has any internal tools for this or if it’d need to be built from scratch. Any thoughts?

[LeadSongDog]

Try a few manual author merges and you’ll see there is an intermediate Merge Authors page listing just the preselected author records with some of their works. On this page one selects the primary and the other records to be merged, so effectively getting a second chance to choose not merge some of them. A semi-automatic approach might be to generate links to such intermediate pages for human review.

Example, if I start from:
https://openlibrary.org/search/authors?q=Melville+Hen*

then I preselect

https://openlibrary.org/authors/merge?records=OL350124A,OL6978237A

Once I chooses the two records and click Request Merge, the merger is queued up for a second look by a librarian, and if approved it is usually done in a day or so. Simple cases are quick.