How to documentation (Cannot read property '_id' of null)

[almereyda]

When installing and running the CLI, we learn that this needs a Redis instance at hands, a torrent-hash and some configuration to be passed.

The hash is presumably an info hash, and can be retrieved for example with torrenttools show infohash sm_85100000-85199999.torrent.

podman run -d -p 6379:6379 -v /srv/redis/state:/data:z docker.io/library/redis:alpine redis-server --appendonly yes
torrenttools show infohash sm_85100000-85199999.torrent

export REDIS_PORT=6379
export REDIS_HOST=localhost
export MAX_AGE=10

Then the application can be used in multiple ways.

echo "ea18b2c1560e6b90f22c2d22a1a56e9c63af6335" | node src/scrape-cli.js --torrent-hashes-stdin
node src/scrape-cli.js --torrent-hash=ea18b2c1560e6b90f22c2d22a1a56e9c63af6335

Unfortunately both will error in the same way:

Scraping hash ea18b2c1560e6b90f22c2d22a1a56e9c63af6335
Redis client connected
Waiting for queue lock
Fetching queue contents
TypeError: Cannot read property '_id' of null
    at updateStats (/home/yala/src/github.com/phillmac/torrent-health-scraper/src/scrape-functions.js:138:20)
    at processTicksAndRejections (node:internal/process/task_queues:93:5)
    at async /home/yala/src/github.com/phillmac/torrent-health-scraper/src/scrape-cli.js:56:5
Finished

This means the Redis queue does not return any entry to be parsed.

• Is this the correct way to invoke the scraper?

Passing a Magnet link (including the trackers, which might be needed for peer discovery) also does not work:

torrenttools magnet sm_85100000-85199999.torrent | node src/scrape-cli.js --torrent-hashes-stdin

• Which is the correct way to set up and invoke this package?

[phillmac]

Ok, so. Long story short, scrape-cli.js expects the database to be pre-populated; it purely exists to scrape the trackers, not add to the db.
There's a few ways you can populate the db:

1.

If you have a bunch of torrent files in a web server directory you can use node src/add.js --torrent-url=TORRENT_URL --type=TYPE
Imo this is the easiest way, because you can use wget/curl and grep to crawl the dir listing and supply the --torrent-url arg.

Ex:

url="***Http torrents url here***"
collection_type="***Torrent collection type here***"
while read -r file_name
do
    echo "Adding ${file_name}"
    node --unhandled-rejections=strict src/add.js --torrent-url "${url}/${file_name}" --type "${collection_type}"
done < <(curl --silent -L "${url}" | grep -o "[a-zA-Z0-9./?=_%:-]*\.torrent" | sort -u)

2.

The other way is if you have a formatted json file that lists the torrents you wish to track in a big batch.
It needs to be formatted like this:

[
  {
    "_id": "torrent 1 info hash",
    "name": "torrent  1 name ",
    "size_bytes": torrent 1  size,
    "created_unix": torrent 1 unix timestamp,
    "link": "torrent 1 http link",
    "type": "books"
  },
  {
    "_id": "torrent 2 info hash",
    "name": "torrent  2 name ",
    "size_bytes": torrent 2  size,
    "created_unix": torrent 2 unix timestamp,
    "link": "torrent 2 http link",
    "type": "books"
  },
 ... etc. etc
]

First run:
node src/populate.js --torrents-file=torrents.json

And then run:
src/add-trackers.js --trackers-file=trackers.json
to populate in the tracker urls you wish to scrape.
The trackers.json file should contain just a json formatted array of the urls.

e.g.

[
  "tracker 1 url",
  "tracker 2 url",
  etc. etc.
]

3.

If you want to just holus bolus import the stats from an existing instance, the torrent-health-frontend/stats.php file will export a snapshot that can be used with node src/preload.js --preload-url=PRELOAD_URL

The advantage of this method is you have usable data instantly and no need to run the scraper worker for a lengthy amount of time to initially scrape every torrent.

The disadvantage is you end up with all of the collections with no fine grain control over only loading a specific collection 😟

[phillmac]

The MAX_AGE param should be set to a large value e.g. 10800
It defines how scrape result is marked as current for e.g. 10800 is 3 hours in seconds
The scraper will not repeat any tracker scrapes for a given torrent if it's still got a current stats.
If one tracker for a torrent yields an error and all the others return valid scrapes, the db will be updated with the valid results.
running the scraper with that same infohash will only try to scrape the tracker that previously had an error.

[phillmac]

To refresh the stale results in the db: node src/stale.js | node src/scrape-cli.js --torrent-hashes-stdin-ln

[phillmac]

Tomorrow I'll add some notes about automatically detecting when a tracker goes down / offline for a long period of time

[almereyda]

@phillmac Many thanks for the detailed explanations! I'll look into reproducing them within the next days and will report findings here. Eventually I'll also find a moment to sketch a PR introducing a README to this repository.