Porting Crawlee to Dart

[egyleader]

Hey 👋🏻 ,

This is a thank you , a question and a poll

First of all I wanted to thank you for the great effort in porting this important library to dart , such famous libraries is important part of the ecosystem and one of the blockers of using dart as a backend in a wide range of products .

I was thinking about porting another scrapping-related Nodejs package https://crawlee.dev/ and I was wondering if it is worth the effort ?? so the question is how did you manage to port such a huge library to dart ?? is it just hardworking from scratch or there is another strategy that makes this job easier and faster ???

and the poll is , does anyone else need this scrapping automation library in dart ?

the benefits includes (as I understand them):

✅ Controlling multiple scrapper types with the same dart API
✅ Queuing , storage and data minpulation
✅ Helpful utils & configurations for scrpping
✅ Automatic scaling and proxy management

[xvrh]

Yeah to all the projects in Dart for backend development 👍

Regarding the Dart port of puppeteer, it was just a very boring line by line porting. It was a long time ago when puppeteer was still in JS (not TS).
A big part is also generated code from Chrome schemas, it helped a lot.

The most important aspect was to convert all the tests. This was very boring but super important to have a usable library.
You can still find some relics of one-off scripts I used to ease the conversion of each file (https://github.com/xvrh/puppeteer-dart/blob/master/tool/replace_for_test.dart)

[egyleader]

thanks for your feedback about the porting process , I didn't know that is was even Javascript not Typescript when you did it ,
and of course there where no chatGPT or LLMS to help with the code conversion , I guess these two parts of the puzzle plus code generation can hugely help in the process .. thanks again 🙏🏻