Crawler exit stuck

[benoit74]

Crawler version : 1.5.1 (will update "soon")

We have a situation where the crawler gets interrupted due to the time limit but never exits. I sent a SIGTERM to the crawler and this is the result (we have multiple occurences of the crawler getting stuck, and for two of them I sent the SIGTERM and got same result as shown below. What is weird is that we've probably been blacklisted because all pages before the time limit seems to be ending with Direct fetch of page URL timed out error.

{"timestamp":"2025-02-11T12:58:32.272Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907"}}
{"timestamp":"2025-02-11T12:58:32.274Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T12:58:32.271Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:BookSources\\/0953926907\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:00:02.305Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","workerid":0}}
{"timestamp":"2025-02-11T13:00:02.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland"}}
{"timestamp":"2025-02-11T13:00:02.326Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:00:02.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Talk:Ireland\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:01:32.357Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:01:32.380Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland"}}
{"timestamp":"2025-02-11T13:01:32.382Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:01:32.379Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:WhatLinksHere\\/Ireland\",\"added\":\"2025-02-11T11:04:42.934Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:03:02.413Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:03:02.418Z","logLevel":"info","context":"general","message":"Time threshold reached 7279.758 > 7200, stopping","details":{}}
{"timestamp":"2025-02-11T13:03:02.430Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-11T13:03:02.431Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:33:39.133Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:33:39.134Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Limerick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Fahan","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Galway","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Sligo","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Westport,_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle,_N_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Bangor","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Belfast_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Carrickfergus","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Donegal","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Foyle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Larne","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Portaferry","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Phennick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Strangford_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Template:Port","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/File:Transport_map.jpg","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.150Z","logLevel":"warn","context":"general","message":"Link Extraction failed in frame","details":{"page":"https://www.cruiserswiki.org/wiki/Kilrush","workerid":0,"type":"exception","message":"Protocol error (Runtime.callFunctionOn): Target closed","stack":"TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at Connection.dispose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:177:22)\n    at CdpBrowser.disconnect (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:226:26)\n    at CdpBrowser.close (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:222:20)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Browser.close (file:///app/dist/util/browser.js:241:13)\n    at async Crawler.serializeAndExit (file:///app/dist/crawler.js:923:13)\n    at async handleTerminate (file:///app/dist/main.js:30:13)"}}
{"timestamp":"2025-02-12T20:33:39.260Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

{"timestamp":"2025-02-12T07:35:08.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm"}}
{"timestamp":"2025-02-12T07:35:08.325Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:35:08.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Strasbourg.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:36:38.353Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","workerid":0}}
{"timestamp":"2025-02-12T07:36:38.384Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm"}}
{"timestamp":"2025-02-12T07:36:38.387Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:36:38.383Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Tallinn.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:38:08.417Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","workerid":0}}
{"timestamp":"2025-02-12T07:38:08.444Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm"}}
{"timestamp":"2025-02-12T07:38:08.446Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:38:08.443Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Toulouse.htm\",\"added\":\"2025-02-12T05:41:02.017Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:39:38.452Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","workerid":0}}
{"timestamp":"2025-02-12T07:39:38.480Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm"}}
{"timestamp":"2025-02-12T07:39:38.481Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:39:38.479Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Venice.htm\",\"added\":\"2025-02-12T05:41:02.018Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:41:08.509Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","workerid":0}}
{"timestamp":"2025-02-12T07:41:08.517Z","logLevel":"info","context":"general","message":"Time threshold reached 7213.174 > 7200, stopping","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:30:50.242Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:30:50.243Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:30:50.263Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-tickets-by-phone.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-booking-horizons.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-travel-from-uk-regions.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-reservations.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-passes-on-eurostar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/ferry-to-spain.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Algarve.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Madrid.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Amsterdam.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Athens.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Barcelona.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Switzerland.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Belgrade.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Berlin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bordeaux.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bruges.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bratislava.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Brussels.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bucharest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Budapest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Nice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cesky-Krumlov.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cinque-Terre.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Cologne.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Copenhagen.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dublin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dubrovnik.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Florence.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Frankfurt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Gibraltar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Hallstatt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Hamburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Helsinki.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Innsbruck.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Istanbul.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kyiv.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Krakow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Lisbon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Ljubljana.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Luxembourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kiev.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Lyon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Marseille.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Milan.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Montpellier.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Moscow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Munich.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Naples.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Oslo.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Paris.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Passau.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Pisa.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Porto.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Prague.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Riga.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Rome.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-St-Petersburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Salzburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-San-Sebastian.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Sofia.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Stockholm.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.283Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.320Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.566Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

I have another stuck task which is a bit different:

{"timestamp":"2025-02-12T08:56:22.845Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/goToSite/2800","workerid":0}}
{"timestamp":"2025-02-12T08:56:23.573Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://bananaguide.com/article/67290/you-love-jack-review"}}
{"timestamp":"2025-02-12T08:56:23.575Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":917,"total":4682,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-12T08:56:22.877Z\",\"extraHops\":0,\"url\":\"https:\\/\\/bananaguide.com\\/article\\/67290\\/you-love-jack-review\",\"added\":\"2025-02-12T07:48:33.215Z\",\"depth\":3}"]}}
{"timestamp":"2025-02-12T08:56:24.963Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:25.299Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://bananaguide.com/article/67290/you-love-jack-review","frameId":"77D42A85B5CCDF255253DE5BCC2CE7FA"}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://bananaguide.com/article/67290/you-love-jack-review"],"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.658Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.659Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.660Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.664Z","logLevel":"info","context":"general","message":"Size threshold reached 4307928621 >= 4294967296, stopping","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T08:57:24.804Z","logLevel":"warn","context":"recorder","message":"Failed to load response body","details":{"url":"https://www.lucasentertainment.com/tour/?nats=MTAwMDA0MC4yNy4yLjU1LjAuMC4wLjAuMA","networkId":"00904D649DE4D581E0F7CD7BF89BAAA8","type":"exception","message":"Protocol error (Fetch.getResponseBody): Target closed","stack":"TargetCloseError: Protocol error (Fetch.getResponseBody): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at WebSocket.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/node/NodeWebSocketTransport.js:43:30)\n    at callListener (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)\n    at WebSocket.onClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:220:9)\n    at WebSocket.emit (node:events:518:28)\n    at WebSocket.emitClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:272:10)\n    at Socket.socketOnClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1341:15)\n    at Socket.emit (node:events:518:28)","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T20:34:03.730Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:34:03.732Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:34:03.735Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

[benoit74]

Issue still present on 1.5.4 ; I strongly suspect this is somehow linked to new retry logic and the usage of sizeLimit / timeLimit. How can I help to further diagnose the problem?

[ikreymer]

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

[ikreymer]

Have not seen that before - and I don't think its related to retries, since all that happens after is:

    logger.info(`Exiting, Crawl status: ${status}`);

    await this.closeLog();

    if (this.crawlState && status) {
      await this.crawlState.setStatus(status);
    }
    process.exit(exitCode);

We could add a timeout to closeLog() and setStatus(), have not seen any issues stalling there before..

[benoit74]

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

No, sorry, look at timestamps, it got stuck at Failed to load response body. Then I sent a SIGTERM and it printed the last two lines. I kept them as reference in the logs to indicate that process was really still alive, waiting for something to happen

[benoit74]

And first task got stuck at Worker done, all tasks complete