Skip to content

README.md

Latest commit

 

History

History
126 lines (89 loc) · 4.39 KB

README.md

File metadata and controls

126 lines (89 loc) · 4.39 KB

KamiYomu Crawler Agent Core — Create Crawler Agents for KamiYomu

A small foundational library and set of conventions for building, extending, and integrating custom crawler agents within the KamiYomu ecosystem. It provides lifecycle abstractions, utilities, and optional integrations for scraping HTML and controlling headless browsers.

Key features

  • Agent lifecycle hooks for crawling and metadata extraction.
  • Integration-ready with the KamiYomu runtime.
  • Built-in examples and helpers for HtmlAgilityPack and PuppeteerSharp.
  • Compatible with .net-8.0 to maximize host compatibility (works from modern .NET SDKs, including .NET 8).

Quick start

  1. Create a class library project (target net-8.0 for widest compatibility):

    Create project:

     dotnet new classlib -n MyKamiYomuAgent -f net-8.0

  2. Add NuGet.Config in the solution folder to ensure standard feeds:

    NuGet.Config content (place next to your .sln):

     <?xml version="1.0" encoding="utf-8"?>
 <configuration>
   <packageSources>
     <clear />
     <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
   </packageSources>
 </configuration>

  3. Install the core package:

     dotnet add package KamiYomu.CrawlerAgents.Core

  4. Make your package discoverable by KamiYomu (add PackageTags to your .csproj):

    Add inside your .csproj:

     <PropertyGroup>
     <PackageTags>crawler-agents;manga-download</PackageTags>
 </PropertyGroup>

  5. Implement your agent

    • Create a class that implements ICrawlerAgent from the KamiYomu.CrawlerAgents.Core namespace.
    • Implement required lifecycle methods (crawl, extract metadata, etc.). The interface defines how KamiYomu will call your agent.

Validate and test locally

Packaging and publishing

  • Build a distributable package:

      dotnet pack -c Release

  • To automatically generate a NuGet package for Debug builds, add to your .csproj:

      <PropertyGroup Condition="'$(Configuration)' == 'Debug'">
      <GeneratePackageOnBuild>True</GeneratePackageOnBuild>
  </PropertyGroup>

  • Publish to a feed accessible by KamiYomu.Web (Nuget.org, GitHub Packages, Azure Artifacts, private feed, or local folder).

  • To test without a feed, upload the generated .nupkg directly into KamiYomu.Web.

Debugging an installed agent

  • Place the .pdb alongside the agent DLL inside the agent folder (e.g. /AppData/agents/{your.package}/lib/net8.0/) to enable source-level debugging when running inside KamiYomu.Web.

Packaging notes

  • Ensure your package includes necessary runtime assets and dependencies.
  • Keep the public API surface minimal and document required configuration and permissions.

Commands summary

  • Create project:

      dotnet new classlib -n MyKamiYomuAgent -f net8.0

  • Add package:

      dotnet add package KamiYomu.CrawlerAgents.Core

  • Build Release package:

      dotnet pack -c Release

  • Enable package on Debug build:

      Add `<GeneratePackageOnBuild>True</GeneratePackageOnBuild>` under Debug condition in `.csproj`

Dependencies

Package Version
HtmlAgilityPack 1.12.4
PuppeteerSharp 20.2.4

Contributing

  • Follow repository coding conventions and include unit tests for new behavior.
  • Use the validator repo above to confirm compliance before publishing.
  • Open issues or pull requests against the core repository with clear descriptions and reproducible examples.

License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0). See the LICENSE file for full terms.

Support / Contact

Changelog

  • See repository Releases for version-specific notes.

Copyright

© KamiYomu. Licensed under GPL-3.0.