Scraper for basic information by radimkvet · Pull Request #1 · radimkvet/factorio-formus-scraper

radimkvet · 2026-02-17T13:25:10Z

Because I can not assign reviewers without you being collaborators first I tag you in this PR.

@oklinov @metalwarrior665
Was instructed by Ondro to also tag @JuanGalilea

radimkvet · 2026-02-17T13:40:20Z

I am sorry for notifying you so early, forgot there is a button to create the PR as Draft to begin with.
Will not happen next time.

metalwarrior665

Looks great, just a few small remarks.

metalwarrior665 · 2026-02-17T14:41:39Z

+                "type": "boolean"
+            }
+        },
+        "required": ["category", "title", "url", "author", "createdAt", "lastPostAt", "posts", "isAnnouncement"]


Having required fields in the dataset output is considered quite dangerous. If any of them is missing, the pushData will throw and you will loose the data. And websites change so one field might be broken. Generally, our approach is to rather push incomplete data and catch the issue in periodic tests/monitoring.

Using required fields is possible but we will only do it when it is validated with TypeScript (library to ensure 1:1 between TS and schema was recently released) and we can ensure these fields are always possible. E.g. currently it is usually not the case because we are pushing "error items" into the default dataset. We want to move to secondary dataset (also a new feature) soon

I forgot about partially pushing data, and it makes sense. Removed the required field from the schema ~~and added partial push of data on some routes~~. There is no case where partial data push makes sense to me, so that feature is not used.

metalwarrior665 · 2026-02-17T14:44:02Z

+                "useApifyProxy": true,
+                "apifyProxyGroups": []
+            }
+        },


In non-generic (that don't work on any website) public Actors, we don't expose the proxy to the user. Usually, we set either datacenter or residential depending which works. Users have no idea what proxies they should use.

Thank you, change it o that the proxies are not in the input

metalwarrior665 · 2026-02-17T14:47:14Z

+        const $views = $topic.find(selectors.topic.views).first();
+        const views = parseNumber($views.text().trim().split(' ')[0]) ?? 0;
+
+        const $lastPost = $topic.find(selectors.topic.lastPost).first();
+        const lastPostAt = $lastPost.find('time').attr('datetime') || $lastPost.text().trim();
+
+        const $author = $details.find(selectors.topic.author).first();
+        const author = $author.text().trim() || 'Unknown';
+
+        const $createdAt = $details.find(selectors.topic.createdAt).first();
+        const createdAt = $createdAt.attr('datetime') || $createdAt.text().trim();


Lot of these variables are not reused so unless the extraction is too long, I would try to inline them to the result object so that the reader doesn't need to carry the variables in their head checking if they will be used somewhere.

Where it made sense I moved it, but not everything because it made it less readable sometimes when there were a lot of function calls on one line because of the formatter.

metalwarrior665 · 2026-02-17T14:48:23Z

+            urls: [url],
+            label: Label.TOPIC_DETAIL,
+            userData,
+        });


First buffer all the URLs into an array and then enqueue them at once to only do 1 API call instead of many. The same applies for pushing data

Had to buffer all the Request objects instead of URLs because of each URL having different user data, but the push is now in bulk.

metalwarrior665 · 2026-02-17T14:49:31Z

+import type { UserData } from '../types/index.js';
+import { Label } from '../types/index.js';
+
+export const forumSearchStartHandler = async (ctx: CheerioCrawlingContext) => {


You are missing the userData type here

Added the missing typing

metalwarrior665 · 2026-02-17T14:59:18Z

+    await enqueueLinks({
+        urls: [nextPageUrl],
+        label: Label.TOPIC_DETAIL,
+        userData: topic,


Instead of aggregating the data and pushing at last page it might be better to push each small thing separately? Usually we try to push the smallest thing (e.g. one dataset item = one review) so we don't have to aggregate anything and it has simple schema. But in some more complex use-cases aggregation makes sense. Just be aware that single dataset item can be max 9MB.

I was thinking of one item being the topic itself. But here there might be issue with the size limitation which I forgot about. Might check it out if there are bigger file regularly and change it around.

metalwarrior665 · 2026-02-17T15:01:28Z

+    const normalized = value.replace(/[,-]/g, '');
+    const parsed = Number.parseInt(normalized, 10);
+    return Number.isNaN(parsed) ? null : parsed;
+};


I think your utils are small enough for them to be a single file. Many files can be harder to navigate.

Generally, I recommend starting with fewer files/folders and then refactoring out if the project gets bigger. Some projects never grow in features so there is no need to prepare "future-proof structure".

I found them distinct enough to have them in separate files, but I understand the problem this might present. Will merge them into one utils file.

metalwarrior665 · 2026-02-17T15:02:49Z

+    proxyConfiguration,
+    requestHandler: router,
+    // this is 2 requests a second, this should be fine and not overload the site since it is not so frequently used one
+    maxRequestsPerMinute: 120,


A bit of issue with this option is that (I think) it doesn't distribute the requests in the minute, it will just stop processing once done. So it could still lead to high bursts at the start (not sure how likely it is in practice). You could solve that by adding maxConcurrency limit as well.

yes I noticed that it just does huge burst at the beginning and then it slows down. Will try to do this with the maxConcurrency setting.

metalwarrior665 · 2026-02-17T15:06:37Z

+A topic is a page with posts provide by users.

-Start a new [web scraping](https://apify.com/web-scraping) project quickly and easily in TypeScript (Node.js) with our empty project template. It provides a basic structure for the Actor with [Apify SDK](https://docs.apify.com/sdk/js/) and allows you to easily add your own functionality.
+## Inputs


We generally prefer to link the Input page to the Actor Store page, which is derived from the inpjt schema. That way there is only single source of truth for the descriptions, you will often forget to change the readme.

I linked the input page statically in the README.

metalwarrior665

Thanks! Please don't force-push after a review, it makes seeing new changes harder.

radimkvet · 2026-02-19T14:21:22Z

Thanks! Please don't force-push after a review, it makes seeing new changes harder.

Oh I did not know GitHub removes the old lines from the comment (GitLab kept them from what I remember), will adjust workflow for next PR.

radimkvet · 2026-02-19T15:49:27Z

Thanks! Please don't force-push after a review, it makes seeing new changes harder.

Now I see you already mentioned it last week in another review. I did not notice it. Will change it.

radimkvet self-assigned this Feb 17, 2026

radimkvet marked this pull request as draft February 17, 2026 13:25

radimkvet force-pushed the feat/basic-features branch 3 times, most recently from 3cb2181 to 62367e8 Compare February 17, 2026 14:22

metalwarrior665 requested changes Feb 17, 2026

View reviewed changes

radimkvet force-pushed the feat/basic-features branch from 62367e8 to 3f93dfc Compare February 17, 2026 15:15

radimkvet marked this pull request as ready for review February 17, 2026 15:18

feat: define output schema for Actor

9b1bb7b

radimkvet force-pushed the feat/basic-features branch from 3f93dfc to 5b471f8 Compare February 19, 2026 13:28

radimkvet added 6 commits February 19, 2026 14:34

chore: update readme.md with description of the scraper

253dcfb

feat: define output and input types

65b8c7b

feat: define the userData types with labels

3667177

feat: create basic Cheerio crawler with separate router

11593dc

feat: create handler for forum index page

3756abe

feat: create a handler for forum category with pagination

55dfb7c

radimkvet force-pushed the feat/basic-features branch from 5b471f8 to e4f523e Compare February 19, 2026 13:34

radimkvet added 4 commits February 19, 2026 14:41

feat: create handler for topic site with pagination for replies

8797245

feat: add crawling statistics for limiting number of pages crawled

b81e128

chore: change the description and name in actor.json

642ef0f

feat: add proxy configuration and more sensible defaults

50a0de0

radimkvet force-pushed the feat/basic-features branch from e4f523e to 50a0de0 Compare February 19, 2026 13:42

radimkvet requested a review from metalwarrior665 February 19, 2026 13:45

metalwarrior665 approved these changes Feb 19, 2026

View reviewed changes

Conversation

radimkvet commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radimkvet commented Feb 17, 2026

Uh oh!

metalwarrior665 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radimkvet Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

metalwarrior665 left a comment

Choose a reason for hiding this comment

Uh oh!

radimkvet commented Feb 19, 2026

Uh oh!

radimkvet commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

radimkvet commented Feb 17, 2026 •

edited

Loading

radimkvet Feb 19, 2026 •

edited

Loading