Import bookdash.org as a Trusted Book Provider (with editions for all languages)

[mekarpeles]

Feature Request

Here is the sitemap of books for bookdash
https://bookdash.org/books-sitemap.xml

This python code gives us a list of the ~850 urls:

import requests
import xml.etree.ElementTree as ET

# Step 1: Get the XML from the sitemap URL
url = "https://bookdash.org/books-sitemap.xml"
response = requests.get(url)
response.raise_for_status()  # Raises an error if the request failed

# Step 2: Parse the XML
root = ET.fromstring(response.content)

# Step 3: Extract all <loc> elements (which are like the rows of the sitemap)
namespaces = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}  # Define XML namespace
locs = root.findall('.//ns:loc', namespaces)

# Step 4: Print each URL (you could treat this like the "td" values)
book_urls = [loc.text for loc in locs]
print(json.dumps(book_urls))

Here is the list of the URLs:
https://gist.githubusercontent.com/mekarpeles/dc5e89708bbf7cae38b35f071021a12a/raw/99abd429075ae5246caa15b8878ad0c59456f38a/bookdashorg_urls.json

re: metadata:

• <picture> is the cover
• <h1> is the title
• ISBN: will be on the page before the ISBN
• contents in with href "/languages/" will be the language
• description: use meta og:description
• with href starting with "/themes" will be subjects
• authors: search for "https://bookdash.org/team-members/"
• publisher: bookdash

Problem / Opportunity

Write a basic python request scraper to respectfully curl each book page, extract the metadata in the right formats, as an openlibrary book in json

Proposal

Breakdown

Related files

Refer to this map of common Endpoints:
*

Requirements Checklist

Checklist of requirements that need to be satisfied in order for this issue to be closed:

• [ ]

Stakeholders

---

Instructions for Contributors

• Before creating a new branch or pushing up changes to a PR, please first run these commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

[mekarpeles]

This is approximately what the script should look like:

import requests
import xml.etree.ElementTree as ET
import json
from bs4 import BeautifulSoup
import re # For finding ISBN and Publication Date

# Step 1: Get the XML from the sitemap URL
sitemap_url = "https://bookdash.org/books-sitemap.xml"
try:
    response = requests.get(sitemap_url)
    response.raise_for_status()  # Raises an error if the request failed
except requests.exceptions.RequestException as e:
    print(f"Error fetching sitemap: {e}")
    exit()

# Step 2: Parse the XML
try:
    root = ET.fromstring(response.content)
except ET.ParseError as e:
    print(f"Error parsing sitemap XML: {e}")
    exit()

# Step 3: Extract all <loc> elements
namespaces = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
locs = root.findall('.//ns:loc', namespaces)
book_urls = [loc.text for loc in locs]

all_books_data = []

# Step 4: Iterate through each book URL and extract data
for book_url in book_urls[1:]:
    print(f"Processing: {book_url}")
    try:
        book_response = requests.get(book_url, timeout=10) # Added timeout
        book_response.raise_for_status()
        soup = BeautifulSoup(book_response.content, 'html.parser')

        book_data = {}

        # Cover image
        book_data['cover_image'] = None

        # Try to get the Open Graph image first (most reliable)
        meta_og_image = soup.find('meta', property='og:image')
        if meta_og_image and meta_og_image.get('content'):
            book_data['cover_image'] = meta_og_image['content'].strip()
        else:
            # Fallback to existing logic if og:image is not found
            picture_tag = soup.find('picture')
            if picture_tag:
                img_tag = picture_tag.find('img')
                if img_tag and img_tag.get('src') and not "preload" in img_tag['src']: # Avoid preload images
                    book_data['cover_image'] = img_tag['src']
                else:
                    # Check source tags within picture for srcset
                    source_tag = picture_tag.find('source')
                    if source_tag and source_tag.get('srcset'):
                        # Take the first URL from srcset, it's usually the smallest/default
                        # and ensure it's not a preload image
                        srcset_urls = [s.strip().split(' ')[0] for s in source_tag['srcset'].split(',') if "preload" not in s]
                        if srcset_urls:
                            book_data['cover_image'] = srcset_urls[0]
            # If still no image, look for a prominent <img> tag directly
            if not book_data['cover_image']:
                main_content_div = soup.find('div', class_='book-hero-image') # Assuming a class for the main image
                if main_content_div:
                    img_tag = main_content_div.find('img')
                    if img_tag and img_tag.get('src') and not "preload" in img_tag['src']:
                        book_data['cover_image'] = img_tag['src']

        # Title
        title_tag = soup.find('h1')
        book_data['title'] = title_tag.text.strip() if title_tag else None

        # ISBN
        # Search for "ISBN:" and get the text that follows.
        # This is a bit fragile and depends on consistent formatting.
        isbn = None
        isbn_element = soup.find(string=re.compile(r"ISBN:"))
        if isbn_element:
            # Try to get the text immediately following "ISBN:"
            # This might need adjustment based on actual HTML structure
            potential_isbn = isbn_element.split("ISBN:")[1].strip()
            # Basic validation for an ISBN-like structure (10 or 13 digits, possibly with hyphens)
            if re.match(r'^[\dX-]+$', potential_isbn):
                 isbn = potential_isbn.split()[0] # Take the first part if there are spaces after
            else: # If not directly after, check parent or siblings
                parent_text = isbn_element.parent.get_text(separator=' ', strip=True)
                match = re.search(r"ISBN:\s*([\d\-X]+)", parent_text)
                if match:
                    isbn = match.group(1)
        book_data['isbn'] = isbn

        # Language
        # Contents within the <a> tag where the href contains "/languages/"
        language_tag = soup.find('a', href=lambda href: href and "/languages/" in href)
        book_data['language'] = language_tag.text.strip() if language_tag else None

        # Description (meta og:description)
        meta_description_og = soup.find('meta', property='og:description')
        meta_description_name = soup.find('meta', attrs={'name': 'description'})
        if meta_description_og and meta_description_og.get('content'):
            book_data['description'] = meta_description_og['content'].strip()
        elif meta_description_name and meta_description_name.get('content'):
            book_data['description'] = meta_description_name['content'].strip()
        else:
            book_data['description'] = None


        # Subjects (contents of a tags with hrefs that start with "/themes")
        subject_tags = soup.find_all('a', href=lambda href: href and href.startswith("/themes"))
        book_data['subjects'] = [tag.text.strip() for tag in subject_tags] if subject_tags else []

        # Authors (author names will be in a tags where href starts with "https://bookdash.org/team-members/")
        author_tags = soup.find_all('a', href=lambda href: href and href.startswith("https://bookdash.org/team-members/"))
        book_data['authors'] = [tag.text.strip() for tag in author_tags] if author_tags else []

        # Publisher
        book_data['publisher'] = "BookDash"

        # Publication Date (will always be "20XX")
        # Attempt to find a year in the format 20XX in the page content.
        # This might be on a copyright line or similar.
        publication_date = None
        # Search for a 4-digit year starting with "20"
        year_match = re.search(r'\b(20\d{2})\b', soup.get_text())
        if year_match:
            publication_date = year_match.group(1)
        else:
            # Fallback based on user note if no specific year found
            publication_date = "20XX" # Or None, if "20XX" isn't desired as a literal fallback
        book_data['publication_date'] = publication_date

        book_data['url'] = book_url # Add the source URL for reference

        all_books_data.append(book_data)
        print(book_data)
    except requests.exceptions.RequestException as e:
        print(f"Error fetching book page {book_url}: {e}")
    except Exception as e:
        print(f"Error processing book page {book_url}: {e}")
        # Optionally, add a placeholder or skip the entry
        all_books_data.append({
            "url": book_url,
            "error": str(e),
            "title": "Error processing this book"
        })

[sai-krishna-kotha]

Hi @mekarpeles 👋

Using the code you provided above, here’s a sample of data I extracted from a Book Dash page:
https://bookdash.org/...

Sample output:

{
  "cover_image": "https://bookdash.org/wp-content/uploads/2019/05/babys-first-family-photo_en_20190506_cover.jpg",
  "title": "Baby’s First Family Photo",
  "isbn": "978-1-928497-28-8",
  "language": "English",
  "description": "Baby’s family is excited about their first family photo. What will Mama, Papa, Big Sister, Baby and Gogo wear?",
  "subjects": ["Family, friends, and home", "Let's learn!"],
  "authors": ["Megan Andrews (Illustrator)", "Nadene Kriel (Designer)", "Glynis Lloyd (Editor)", "Edem Torkornoo (Writer)"],
  "publisher": "BookDash",
  "publication_date": "2019",
  "url": "https://bookdash.org/books/babys-first-family-photo/"
}

Could you kindly provide a brief example of the expected JSON format for Open Library Trusted Provider imports? I want to confirm it matches before proceeding. Also, please assign me to this issue—I’m eager to work on it!