Skip to content

Commit b564c7e

Browse files
fix: Goodreads cover scraper fetches via system curl (Cloudflare bypass)
The v0.7.45-rc.1 Goodreads cover lookup used PHP's HTTP client (Guzzle), which Cloudflare answers with a 202 anti-bot page on real servers — so it returned NULL in production and only "worked" from a residential dev IP. Verified on a live server: PHP's curl-extension and Guzzle both get 202, the system `curl` binary gets 200. Open Library plugin 1.0.3: getGoodreadsCover now shells out to the system curl binary (found via a guarded lookup, skipped when shell_exec is disabled), which passes Cloudflare's TLS fingerprint check. The ISBN is validated to [0-9Xx]{10,13} and passed through escapeshellarg, so no shell injection is possible. Where exec is unavailable it degrades to no cover, exactly as before. Verified end-to-end from a real cPanel server AND a php:8.4-apache container (both return the og:image). Bumps version to 0.7.45-rc.2.
1 parent 9f2d0b5 commit b564c7e

5 files changed

Lines changed: 84 additions & 22 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ A catalogue-consistency and scraper-maintenance release.
4545
- **Catalogue availability facets now reflect real circulation state** — reserved, on-loan, available and non-circulating records are shown and counted separately; records without physical copies remain visible under the complete catalogue without being mislabelled as loans ([#303](https://github.com/fabiodalez-dev/Pinakes/issues/303)).
4646
- **Catalogue card rows stay aligned** — a card with a subtitle no longer pushes its author, publisher and "Details" out of line with neighbouring cards; the subtitle space is reserved per row only where a row actually needs it, and the layout re-aligns after AJAX filtering, on resize, and once web fonts settle ([#298](https://github.com/fabiodalez-dev/Pinakes/issues/298)).
4747
- **Language completion statistics are derived from the Italian source catalogue** — every locale (list and edit views alike) now uses the same live denominator, so translation percentages cannot drift from the shipped keys.
48-
- **Open Library plugin 1.0.2** — replaces the retired third-party Goodreads-cover service with a direct, timeout-bounded lookup of the public Goodreads ISBN page. Only HTTPS cover URLs from exact Goodreads/Amazon CDN domains or their subdomains are accepted; its manifest now correctly requires Pinakes 0.7.16+ and PHP 8.2+.
48+
- **Open Library plugin 1.0.3** — replaces the retired third-party Goodreads-cover service with a direct, timeout-bounded lookup of the public Goodreads ISBN page. Because Goodreads sits behind Cloudflare (which serves an anti-bot page to PHP's HTTP client but not to the system `curl`), the cover lookup shells out to the `curl` binary when the host allows process execution, and degrades gracefully to no cover where it doesn't. Only HTTPS cover URLs from exact Goodreads/Amazon CDN domains or their subdomains are accepted; its manifest requires Pinakes 0.7.16+ and PHP 8.2+.
4949
- **Email templates: `{{pickup_deadline}}` now resolves**, and every placeholder token carries a localised description tooltip in the template editor ([#304](https://github.com/fabiodalez-dev/Pinakes/issues/304)).
5050

5151
### Database Changes

storage/plugins/open-library/OpenLibraryPlugin.php

Lines changed: 74 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -419,25 +419,84 @@ private function getGoodreadsCover(string $isbn): ?string
419419
return null;
420420
}
421421

422-
try {
423-
$res = \App\Support\HttpClient::get(
424-
'https://www.goodreads.com/book/isbn/' . urlencode($clean),
425-
[
426-
'Accept' => 'text/html,application/xhtml+xml',
427-
'Accept-Language' => 'en-US,en;q=0.9',
428-
],
429-
['timeout' => 5, 'max_redirects' => 3, 'user_agent' => self::USER_AGENT]
430-
);
431-
432-
if (!$res['ok'] || $res['status'] !== 200 || empty($res['body'])) {
433-
return null;
434-
}
422+
$html = $this->fetchGoodreadsHtml($clean);
423+
if ($html === null) {
424+
return null;
425+
}
426+
return $this->extractGoodreadsCoverFromHtml($html);
427+
}
435428

436-
return $this->extractGoodreadsCoverFromHtml((string) $res['body']);
429+
/**
430+
* Fetch the Goodreads book page HTML.
431+
*
432+
* Goodreads sits behind Cloudflare, which serves a 202 anti-bot interstitial
433+
* to PHP's HTTP stack (Guzzle/libcurl present a TLS/JA3 fingerprint Cloudflare
434+
* challenges) — so an in-process HTTP client never gets the real page. The
435+
* system `curl` binary's fingerprint passes, so we shell out to it when the
436+
* host allows process execution; otherwise we give up gracefully (the caller
437+
* simply gets no cover). $isbn is already validated to [0-9Xx]{10,13} by the
438+
* caller and is passed through escapeshellarg, so no shell injection is
439+
* possible.
440+
*
441+
* @param string $isbn Validated ISBN-10/13 (digits and X only)
442+
* @return string|null Raw HTML, or null when unreachable / exec unavailable
443+
*/
444+
private function fetchGoodreadsHtml(string $isbn): ?string
445+
{
446+
$curl = $this->findCurlBinary();
447+
if ($curl === null) {
448+
return null;
449+
}
450+
$url = 'https://www.goodreads.com/book/isbn/' . $isbn;
451+
$cmd = sprintf(
452+
'%s -sL --max-time 8 -A %s %s 2>/dev/null',
453+
escapeshellarg($curl),
454+
escapeshellarg(self::USER_AGENT),
455+
escapeshellarg($url)
456+
);
457+
try {
458+
$out = shell_exec($cmd);
437459
} catch (\Throwable $e) {
438-
\App\Support\SecureLogger::warning('[OpenLibrary] Goodreads cover scrape error: ' . $e->getMessage());
460+
\App\Support\SecureLogger::warning('[OpenLibrary] Goodreads cover fetch error: ' . $e->getMessage());
439461
return null;
440462
}
463+
return (is_string($out) && $out !== '') ? $out : null;
464+
}
465+
466+
/**
467+
* Locate a usable system `curl` binary, or null when process execution is
468+
* unavailable (shell_exec disabled — common on hardened shared hosting) or
469+
* no curl is installed. Cached per instance.
470+
*
471+
* @return string|null Absolute path to curl, or null
472+
*/
473+
private function findCurlBinary(): ?string
474+
{
475+
static $resolved = false;
476+
static $path = null;
477+
if ($resolved) {
478+
return $path;
479+
}
480+
$resolved = true;
481+
482+
if (!function_exists('shell_exec')) {
483+
return $path = null;
484+
}
485+
$disabled = array_map('trim', explode(',', (string) ini_get('disable_functions')));
486+
if (in_array('shell_exec', $disabled, true)) {
487+
return $path = null;
488+
}
489+
foreach (['/usr/bin/curl', '/bin/curl', '/usr/local/bin/curl'] as $candidate) {
490+
if (@is_executable($candidate)) {
491+
return $path = $candidate;
492+
}
493+
}
494+
$found = @shell_exec('command -v curl 2>/dev/null');
495+
$found = is_string($found) ? trim($found) : '';
496+
if ($found !== '' && @is_executable($found)) {
497+
return $path = $found;
498+
}
499+
return $path = null;
441500
}
442501

443502
/**

storage/plugins/open-library/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"name": "open-library",
33
"display_name": "Open Library Scraper",
44
"description": "Integrazione con le API di Open Library (openlibrary.org) per lo scraping di metadati dei libri. Fornisce dati completi su edizioni, opere, autori e copertine ad alta risoluzione, con copertine di fallback recuperate dalla pagina pubblica di Goodreads.",
5-
"version": "1.0.2",
5+
"version": "1.0.3",
66
"author": "Fabiodalez",
77
"author_url": "",
88
"plugin_url": "https://openlibrary.org",

tests/plugin-goodreads-cover.unit.php

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,13 @@
55
* Unit test for the Goodreads cover scraper in OpenLibraryPlugin
66
* (replaces the retired bookcover.longitood.com service).
77
*
8-
* The parsing (og:image extraction + trusted-CDN host guard) is tested
9-
* deterministically against HTML fixtures — no network. A final best-effort
10-
* live check hits Goodreads only if reachable, and never fails the suite when
11-
* the network is unavailable.
8+
* The scraper fetches the Goodreads book page by shelling out to the system
9+
* `curl` binary — Cloudflare's anti-bot serves a 202 to PHP's HTTP client but
10+
* 200 to curl's TLS fingerprint — and parses the og:image with a trusted-CDN
11+
* host guard. The parsing is tested deterministically against HTML fixtures
12+
* (no network). A final best-effort live check hits Goodreads only when a curl
13+
* binary is available and the page is reachable, and never fails the suite when
14+
* exec is disabled or the network is unavailable.
1215
*
1316
* Run: php tests/plugin-goodreads-cover.unit.php (exit 0 iff all pass)
1417
*/

version.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
22
"name": "Pinakes",
3-
"version": "0.7.45-rc.1",
3+
"version": "0.7.45-rc.2",
44
"description": "Library Management System - Sistema di Gestione Bibliotecaria"
55
}

0 commit comments

Comments
 (0)