Incorrect query text used when retrieving additional context

[adamdossa]

When retrieving additional context for the latest user query, the query string (used for KNN against the pinecone embeddings) includes the system prompt (as well as some of the users previous queries).

Whilst the users previous user queries might be useful (although we already have context for these in theory), the system prompt (AI instructions + initial context) doesn't seem relevant or useful here (querying for additional context) and seems likely to cause the additional context to not be a great match for the latest user query.

Looks like a bug to me, but perhaps I am missing something?

https://github.com/mtenenholtz/chat-twitter/blob/f889b688aa35d38a9823e0dde5c0cff94a6ae1fc/backend/main.py#LL168C17-L168C31

[adamdossa]

On a different domain, using:
docs = embedding_search(latest_query, k=2)
seems a reasonable fix.

IMO since we're already including context for the original query and any subsequent queries (up to the token limits) it doesn't make much sense to use the previous queries when doing a semantic match for new context in pinecone (and def. not the system_message which includes the priming prompt and initial 10 document context).

[audvin]

@adamdossa what were the specific code changes you did, I would like to try this and see if my experience improves as well.

[adamdossa]

In line:

chat-twitter/backend/main.py

Line 172 in f889b68

docs = embedding_search(query_text, k=2)

I changed it to:
docs = embedding_search(latest_query, k=2)