Firstversion

[Nehanth]

No description provided.

[jjasghar]

Firecrawl.dev is a paid service. In my opinion this goes against our open source core values.

[mairin]

@jjasghar Sorry! This was a miscommunication on my part. There is a codepath to using it as a local service and it is licensed AGPL. The intention was to enable folks to run it on their laptop to ingest website content and output as instructlab-able markdown; my bad for not communicating that clearly to @Nehanth - he will look at refactoring to run as a local service using open source components only and no external APIs.

[jjasghar]

Ah! Understood. Thanks for the clarification.

[Nehanth]

@jjasghar @mairin I looked into it and made the necessary changes. Instead of deleting the API version, I created a new version called scrape_website_local.py and updated the README with instructions on how to run it locally. Let me know if you'd like me to delete the previous version that uses the API.