confederations file is empty

[MarioPrieto6]

Hello, i am executing "scrapy crawl confederations > confederations.json" but my json file is empty
Could anyone help me?

I modified settings.py in tfmkt folder with ROBOTSTXT_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'

[dcaribou]

Hey @MarioPrieto6 . Similar issues were reported #74 and #30.
Could you kindly try those suggestions and follow-up in any of those other issues if you can't figure it out?

[DanielHermosilla]

Hey, thanks for all of your work!

I'm having the same problem. Tried following the instructions from the issues, but didn't get to anything. I don't know what could be happening...

Here is a debug statement:

❯ scrapy crawl confederations -L DEBUG

2025-06-15 22:48:13 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: tfmkt)
2025-06-15 22:48:13 [scrapy.utils.log] INFO: Versions:
{'lxml': '5.4.0',
 'libxml2': '2.13.8',
 'cssselect': '1.3.0',
 'parsel': '1.10.0',
 'w3lib': '2.3.1',
 'Twisted': '25.5.0',
 'Python': '3.11.9 (main, Jun 15 2025, 22:27:52) [Clang 20.1.7 ]',
 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)',
 'cryptography': '45.0.4',
 'Platform': 'macOS-14.0-x86_64-i386-64bit'}
2025-06-15 22:48:13 [scrapy.addons] INFO: Enabled addons:
[]
2025-06-15 22:48:13 [py.warnings] WARNING: /Users/daniel/.pyenv/versions/3.11.9/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting.
It will be removed in a future version of Scrapy.
  return cls(crawler)

2025-06-15 22:48:13 [asyncio] DEBUG: Using selector: KqueueSelector
2025-06-15 22:48:13 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-06-15 22:48:13 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-06-15 22:48:13 [scrapy.extensions.telnet] INFO: Telnet Password: 563435ef7d1e4a77
2025-06-15 22:48:13 [py.warnings] WARNING: /Users/daniel/.pyenv/versions/3.11.9/lib/python3.11/site-packages/scrapy/extensions/feedexport.py:455: ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details
  exporter = cls(crawler)

2025-06-15 22:48:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2025-06-15 22:48:14 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tfmkt',
 'HTTPCACHE_ENABLED': True,
 'NEWSPIDER_MODULE': 'tfmkt.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'ROBOTSTXT_OBEY': True,
 'ROBOTSTXT_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
                         'AppleWebKit/605.1.15 (KHTML, like Gecko) '
                         'Version/17.0 Safari/605.1.15',
 'SPIDER_MODULES': ['tfmkt.spiders']}
2025-06-15 22:48:14 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-06-15 22:48:14 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-15 22:48:14 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2025-06-15 22:48:14 [scrapy.core.engine] INFO: Spider opened
2025-06-15 22:48:14 [py.warnings] WARNING: /Users/daniel/.pyenv/versions/3.11.9/lib/python3.11/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: tfmkt.spiders.common.BaseSpider (inherited by tfmkt.spiders.confederations.ConfederationsSpider) defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html
  warn(

2025-06-15 22:48:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-06-15 22:48:14 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in /Users/daniel/transfermarkt-scraper/.scrapy/httpcache
2025-06-15 22:48:14 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2025-06-15 22:48:15 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.transfermarkt.co.uk/robots.txt> (referer: None)
2025-06-15 22:48:15 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.transfermarkt.co.uk> (referer: None)
2025-06-15 22:48:15 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.transfermarkt.co.uk>: HTTP status code is not handled or not allowed
2025-06-15 22:48:15 [scrapy.core.engine] INFO: Closing spider (finished)
2025-06-15 22:48:15 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: stdout:
2025-06-15 22:48:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 458,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 970,
 'downloader/response_count': 2,
 'downloader/response_status_count/403': 2,
 'elapsed_time_seconds': 1.535617,
 'feedexport/success_count/StdoutFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 6, 16, 2, 48, 15, 705710, tzinfo=datetime.timezone.utc),
 'httpcache/firsthand': 2,
 'httpcache/miss': 2,
 'httpcache/store': 2,
 'httpcompression/response_bytes': 324,
 'httpcompression/response_count': 2,
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/403': 1,
 'items_per_minute': 0.0,
 'log_count/DEBUG': 6,
 'log_count/INFO': 12,
 'log_count/WARNING': 3,
 'memusage/max': 60100608,
 'memusage/startup': 60100608,
 'response_received_count': 2,
 'responses_per_minute': 120.0,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/403': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2025, 6, 16, 2, 48, 14, 170093, tzinfo=datetime.timezone.utc)}
2025-06-15 22:48:15 [scrapy.core.engine] INFO: Spider closed (finished)