Support for remote info stores like, website, Confluence, Sharepoint etc,

[emanueol]

Or must all files exist locally ?

In real world of large enterprises, theres a confluence server, a jira server, sharepoint server that typically reside in a data center or as a SaaS cloud, and some on-prem custom html, excel files etc.

Would be great a ChatGPT type of ingest/compute searches across remote systems.. how feasible is this ?
Thanks

[dartpain]

Thats a good feature we can start working on, as loang as we can prep it in a neat and readable format we can ingest it all. But I do think we have to vectorise all this data first or summarise at least.

[terrafying]

I was able to pull confluence data by exporting the whole space as HTML (in a zip), extracting it to the docsgpt folder, changing the glob pattern in ingest_rst.py to point to the extracted HTML files, and using BeautifulSoup to pull out the text content inside the main

tags (soup.find_all('div', attrs={"class": "wiki-content group"})).
I couldn't get the API to download a whole space at once, but the manual method worked decently, as long as there isn't a huge number of spaces to deal with.

[emanueol]

I suppose theres 2 things:

initial export (Confluence allows exporting a space in couple different formats).

incremental updates (Jira supports webhooks , sending json via http post to some listener)

Or maybe easier to prep a clone so you gpt stuff doesn't interfer with users, but it all boils down to file formats.

im not specialist of under the hood of Confluence, Jira, etc.. but im part of those 0.001% that care centralizing knowledge for both tech guys (majority) and satisfy business/stakeholders.
Its a very interesting point of discussion "hiw to decide organization of information, but i suppose some basic high level stuff could be inserted on the source pages (tags, etc) to help with categorization, as least avoid showing code to business manager when he just looking for the 5 business rules agreed with Development in some project. etc.

[JohnRSim]

https://platform.openai.com/docs/tutorials/web-qa-embeddings

Could this be used?

[dartpain]

I think what needs to be built here is just a module for our parser. basically something that loads data and converts it into (.rst, .md, .pdf, .docx, .csv, .epub, .html) As DocsGPT already loads this files with easy. Its just method of scraping files that we need to implement here

[tardigrde]

This feature would be a killer. TBH I gave it a try and it's a more complex proeblem than I thought. OpenAI example would try to fetch every URL on the website and put every text in embeddings. The problem is that there is a lot of gibberish, non relvant text on these webpages.

We could start by something simple, like Wikipedia. There should be some good projects already doing webscraping on wikipedia in Python. But then again I think it's a big development effort. I think other projects like AutoGPT do something similar already.

[tardigrde]

FYI: This nice repo implemented loading text from YT videos as well as websites
https://github.com/embedchain/embedchain