The more data it is fed, the less are the answers useful/accurate

[smileBeda]

I notice that if I ingest a 60KB PDF and query it the answers are very accurate.
However, the more data I ingest (similar topics, just more data) the answers become more and more useless, up to the point where questions it had answers on a low data set, it suddenly has no more answers on a larger dataset and comes up with "I do not have an answer to the question"

Why might that be? This is quite "a showstopper" since the whole point of getting a broader "knowledge" ingested, is increasing the data.

[pseudotensor]

If it's related to number of docs and chunks, then this is something I solved in https://github.com/h2oai/h2ogpt .

h2oai/h2ogpt@58a17a8#diff-fb9fd206ee33cd1324b9732364e0ecdf19e8d693c0c11f6181fc1e8f1f96329eR550-R553

i.e. had to deal with chroma bug langchain-ai/langchain#1946

It's a simple work-around that can be applied to PrivateGPT or other such repos.

FAISS has no such issue, but is not persistent. I didn't try weaviate.

[smileBeda]

I updated to langchain 0.0.17, rebuilt, re-queried:
The results are indeed better.
This repo should probably require 0.0.17 instead of 0.0.166 as currently, going to PR that.

Although, there is still gibberish repeating in the answer

The ruler of all things mentioned in the document is the sorcerer from the darkness, his ruler and collector. 

He is the one who uses the power of words to control and manipulate people and situations. 

He is the one who is responsible for the creation of the world and the laws that govern it. 

He is the one who is responsible for the creation of the physical world and the laws that govern it. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern the physical world and the laws that govern the physical world. 

He is the one who is responsible for the creation of the laws that govern

My document does indeed describe a rule of all things, and the first sentence in above response is relatively accurate. Then, it starts repeating itself in several fashions until it blows up completely.

Another example is:
what is what does logos mean in greek (indeed my document explains it)

The word "logos" in Greek refers to the concept of 
the word, the sense, the reason, the sentence, the meaning, the understanding, the comprehension, the interpretation, the explanation, the communication, the translation, 
the interpretation, the explanation, the comprehension, the understanding, the translation, 
the interpretation, the explanation, the comprehension, the understanding, the translation, 
the interpretation, the explanation, the comprehension, the understanding, the translation, 
the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the interpretation, the explanation, the comprehension, the understanding, the translation, the...

Something is off :)

[andrewginns]

I updated to langchain 0.0.17, rebuilt, re-queried: The results are indeed better. This repo should probably require 0.0.17 instead of 0.0.166 as currently, going to PR that.

Although, there is still gibberish repeating in the answer

Something is off :)

The gibberish at the end could likely be constrained by limiting the number of tokens to output or changing the LLM utilised. I would assume using models with RLHF would specifically reduce this sort of repetitive output.

[imartinez]

LangChain dependency updated in 1590c58

Thanks for your advise!