✨ New Connector Idea: "GitHub Files" Source

[aaronsteers]

Desired Connector Spec

This new connector would allow users to get file data from a github (or generic git) repo, and apply our CDK parser logic (especially unstructured parser) along with glob patterns, to get data and content from a github repo into an Airbyte data pipeline.

[Ishankoradia]

Hi @aaronsteers , this is what me and @siddhant3030 were thinking, please let me know what you think.
Lets say a git repo has 10 files.

1. Connector takes input: git repo url and its credentials.
2. The connector should read all files in this repo and extract their content (via some repo content api).
3. We will generate records with schema [{content: "text content from the api in above step", file_name: "source.py"}]. Add more rows based on the glob patterns (this is a bit unclear; if you could explain this would be great)
4. So basically we will have on stream per repo

If you could shed light on the below questions would be great

1. What is unclear is why do we need a new connector ? Can't we add another stream in the existing github connector that reads content from the files.
2. Is the motivation behind this connector to give users ability to plug their git content data into an llm for retrieval ?
3. Does having one stream per file here makes sense ?
4. Could give an example of glob patterns you are referring to ?

[TLazarevic]

@aaronsteers
This one is interesting as well, but we would also appreciate the answer to the question above:

What is unclear is why do we need a new connector ? Can't we add another stream in the existing github connector that reads content from the files.

[aaronsteers]

@Ishankoradia - response inline:

1. Connector takes input: git repo url and its credentials.

👍

2. The connector should read all files in this repo and extract their content (via some repo content api).

👍

3. We will generate records with schema [{content: "text content from the api in above step", file_name: "source.py"}]. Add more rows based on the glob patterns (this is a bit unclear; if you could explain this would be great)

Yes, as you suggest would include content, (relative) file name - and hopefully also created_at and last_modified_at.

Since this is related to LLM use cases, it might make sense to try to match the schema of "unstructured" sources like source-google-drive.

If easy to include, some other fields like last_commit_sha could be helpful also.

4. So basically we will have on stream per repo

👍

If you could shed light on the below questions would be great

1. What is unclear is why do we need a new connector ? Can't we add another stream in the existing github connector that reads content from the files.

TL;DR: This part is your call. I don't feel strongly about it either way.

The thought process is that it was probably easier to not have to worry about impacts related to modifying the existing connector. And if created as a separate connector, we could always incorporate back into the GitHub source later on.

If adding to the existing GitHub connector, we can't change the auth or other options.

Another consideration here is that we may want to go all-in on the unstructured documents paradigm or files-type extractor paradigm. There is some existing prior art here that you could reuse - notably source-s3 and source-google-drive, for example. In those cases, the config is file-centric, allowing stream definition per glob and different globs/streams using different parsers.

2. Is the motivation behind this connector to give users ability to plug their git content data into an llm for retrieval ?

Yes! Goal here is to make git content available to LLMs.

3. Does having one stream per file here makes sense ?

I think one stream per repo (or else one stream per glob?) is probably the right call.

4. Could give an example of glob patterns you are referring to ?

Sorry for the confusion on this point. Here are some example glob patterns, keeping in mind the LLM use cases.

1. * (or **/*?) - get all files from the repo. Helpful if you just want all the files content.
2. *.md - get all markdown documentation pages from the repo.
3. *.py (or another language-specific filter) - gather code files for purposes of code analysis, code summaries, and/or AI-assisted code creation.

[Ishankoradia]

@aaronsteers thanks for getting back super quick. Appreciate all the answers.

Summarizing for our understanding

• Agreed, separate connector makes it easy for testing and we dont have to worry about OCP design making sure existing things doesn't break.
• Motivation is clear to us now.
• One stream per glob pattern makes more sense as you mentioned.
• Schema could have things like last_commit_sha. Eventually a record could look like
  [{content: "text content from the api in above step", relative_file_name: "src/source.py", last_modified_at: "2023-01-01", created_on: "2024-01-01", last_commit_sha: "asdasda..."}]

Have two more questions

1. Will this be a python CDK connector ?
2. In your initial comment what do you mean by CDK parser logic ?

Could you assign this to me ? Its very interesting.

[aaronsteers]

Have two more questions

1. Will this be a python CDK connector ?

Yes, if that works for you. 👍 We have a CDK "extra" for file-based connectors, which may be helpful.

2. In our initial comment what do you mean by CDK parser logic

There's a common CDK backend for sources such as Google Drive and S3, which I'll screenshot below for comparison.

Show/hide S3 example

Show/hide Google Drive example

Could you assign this to me ? Its very interesting.

Absolutely. You've got first dibs as first person to reply. If above sounds good to you, simply confirm and @marcosmarxm will assign to you.

[Ishankoradia]

Perfect, Sounds good to me @aaronsteers . Thanks for getting back quickly.
I will look into the resources you shared.
This should be exciting. You can assign it to me.

[aaronsteers]

@Ishankoradia - We are very excited also - thanks for taking this on. I've assigned it to you!

[avirajsingh7]

@Ishankoradia if you want to drop this issue, Can I work on this one?

[Ishankoradia]

Hey @avirajsingh7 i am still working on this. Deep into it, i am planning to submit a first draft of the PR in the coming week. Thanks

[aaronsteers]

Hi, @Ishankoradia - Any update or questions on this before end of event? I'll check in tomorrow (Saturday) in case I can help in any way. Thanks!