Open
Description
Indexes all pages that are part of a Github Pages-based website. Should use the Github APIs directly, as we want to be able to index sites that are behind authentication / internal firewalls. Where possible, we should re-use functionality / common utilities present in the existing Github / Web connectors.
Checkout the connector creation README here for more details on the best way to add new connectors: https://github.com/danswer-ai/danswer/blob/main/backend/danswer/connectors/README.md.