Github connector: Add an include_code option and index code

[patrickfweston]

Description

Index the code in your Github repositories. Also, adds an include_code option to the settings screen.

Addresses feature request: #4025

How Has This Been Tested?

1. Add a new Github connector
2. Specify the details and configure the connector
3. When setting up the options, check the "Include code?" box
4. Let the connector run
5. Verify that code files are being included using the Explorer tool

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

Notes

• Documentation update: Update github.mdx to indicate code is indexed documentation#189

[vercel]

@patrickfweston is attempting to deploy a commit to the Danswer Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

PR Summary

This PR adds code indexing capabilities to the GitHub connector, allowing users to include repository code files alongside existing content types through a new include_code configuration option.

• Added include_code boolean field to GitHub connector configuration in web/src/lib/connectors/connectors.tsx with corresponding UI checkbox
• Implemented code file indexing logic in backend/onyx/connectors/github/connector.py with new methods _convert_code_file_to_document and _fetch_repo_code_files
• Added example configuration with include_code option in backend/scripts/add_connector_creation_script.py for reference implementation
• Handles GitHub API rate limiting and includes error handling for code file fetching
• Potential concern around timestamp handling and file content processing that needs review

_{💡 (1/5) You can manually trigger the bot by mentioning @greptileai in a comment!}

_{3 file(s) reviewed, 3 comment(s)
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

logic: Decoding binary content without checking file type could fail for non-text files. Need to handle or skip binary files

[patrickfweston]

I'm going to close this because it's a duplicate of #1650 and it was mentioned that the Onyx team is adding code support in a different way:

Hello! We're going to be building code search into the project in the feature (hopefully starting around end of year). The issue with ingesting code like with documentation or conversations is that code has dependencies across files that can't be captured well by a pure top n approach. We'll be building code ingestion through a different pipeline