Feat: Add support for JSON endpoints in Conversational Search - #690
Merged
Conversation
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/buddy-plugins/test-conversational-intent.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 30 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
apt-get install jq -y > /dev/null; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 << EOF
CREATE TABLE docs (
id BIGINT,
content TEXT,
title TEXT,
category TEXT,
embedding_vector FLOAT_VECTOR
knn_type='hnsw'
hnsw_similarity='cosine'
model_name='sentence-transformers/all-MiniLM-L6-v2'
from='content'
api_key='\${OPENAI_API_KEY}'
) TYPE='rt';
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 << 'EOF'
INSERT INTO docs (id, content, title, category) VALUES
(1, 'Artificial intelligence is the capability of computational systems to perform tasks typically associated with human intelligence such as learning, reasoning, problem-solving, perception and decision-making. High-profile applications include advanced web search engines, recommendation systems, virtual assistants, autonomous vehicles and generative AI tools. The field was founded as an academic discipline in 1956 and has gone through multiple cycles of optimism and disappointment known as AI winters.', 'Artificial Intelligence', 'AI'),
(2, 'Machine learning is a field of study in artificial intelligence concerned with the development of statistical algorithms that can learn from data and generalize to unseen data. It finds application in natural language processing, computer vision, speech recognition, email filtering, agriculture and medicine. Deep learning, a subset of machine learning, has allowed neural networks to surpass many previous approaches in performance.', 'Machine Learning', 'AI'),
(3, 'Retrieval-augmented generation (RAG) is a technique that enables large language models to retrieve and incorporate new information. RAG improves LLMs by incorporating information retrieval before generating responses, pulling relevant text from databases, uploaded documents or web sources. This method helps reduce AI hallucinations and allows LLMs to include sources in their responses for greater transparency.', 'Retrieval-Augmented Generation', 'AI'),
(4, 'PHP is a general-purpose scripting language geared towards web development created by Rasmus Lerdorf in 1993. PHP code is usually processed on a web server by a PHP interpreter implemented as a module, a daemon or a Common Gateway Interface executable. The language has evolved without a written formal specification with the original implementation acting as the de facto standard.', 'PHP', 'Programming'),
(5, 'Python is a high-level general-purpose programming language designed by Guido van Rossum. Its design philosophy emphasizes code readability with the use of significant indentation. Python supports multiple programming paradigms including structured, object-oriented and functional programming. It consistently ranks as one of the most popular programming languages and has gained widespread use in the machine learning community.', 'Python', 'Programming'),
(6, 'Breaking Bad is an American neo-Western crime drama television series created by Vince Gilligan for AMC. The series follows Walter White, an underpaid high school chemistry teacher who turns to producing methamphetamine after being diagnosed with lung cancer. The show stars Bryan Cranston as Walter White, Aaron Paul as Jesse Pinkman, Anna Gunn as Skyler White, Dean Norris as Hank Schrader, and Bob Odenkirk as Saul Goodman. It premiered in 2008 and has been praised as one of the greatest television series of all time.', 'Breaking Bad', 'Entertainment'),
(7, 'The Mandalorian is an American space Western television series created by Jon Favreau for Disney Plus. It is the first live-action series in the Star Wars franchise and follows Din Djarin, a lone bounty hunter played by Pedro Pascal, who protects a Force-sensitive child named Grogu. The series premiered with the launch of Disney Plus on November 12, 2019 and has received largely positive reviews from critics.', 'The Mandalorian', 'Entertainment'),
(8, 'Stranger Things is an American television series created by the Duffer Brothers for Netflix. Set in the 1980s, the series centers on the residents of the fictional small town of Hawkins, Indiana, after a nearby facility opens a gateway to a hostile alternate dimension known as the Upside Down. The show stars Winona Ryder, David Harbour, Millie Bobby Brown and Finn Wolfhard.', 'Stranger Things', 'Entertainment'),
(9, 'Game of Thrones is an American fantasy drama television series created by David Benioff for HBO. It is an adaptation of A Song of Ice and Fire novels. Set on the fictional continents of Westeros and Essos, the show follows several story arcs throughout eight seasons including the Iron Throne succession and the threat from the North.', 'Game of Thrones', 'Entertainment'),
(10, 'The Office is an American mockumentary sitcom television series adapted for NBC by Greg Daniels. The show depicts the everyday work lives of office employees at the Scranton Pennsylvania branch of the fictional Dunder Mifflin Paper Company. It stars Steve Carell, Rainn Wilson, John Krasinski, Jenna Fischer and B.J. Novak.', 'The Office', 'Entertainment');
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "debug sleep 30;" > /dev/null 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE CHAT MODEL test_assistant (model='openai:gpt-4o-mini');"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('What is RAG?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_1.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_1.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([3])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Its boring', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('what about movies?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_3.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_3.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([8])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('I like Breaking Bad', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_4.txt
––– output –––
*************************** 1. row ***************************
#!/.*conversation_uuid: test-conv-intent-001/!#
#!/.*user_query: I like Breaking Bad/!#
#!/.*search_query: .*[Bb]reaking [Bb]ad.*/!#
- #!/.*response: .*[Bb]reaking [Bb]ad.*/!#
+ response: I don’t have enough information in the provided context to answer.
#!/.*sources: .*/!#
––– input –––
grep 'sources:' /tmp/output_4.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([6])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('show me more entertainment', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Tell me about Python', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_7.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_7.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([5])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP CHAT MODEL test_assistant;"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS docs;"
––– output –––
OK |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/mcl/auto-embeddings-jina-remote.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
local file1="$1" file2="$2"
awk '
NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
{
dot += a[FNR]*$1
sumb2 += $1*$1
}
END {
print dot / (sqrt(suma2) * sqrt(sumb2))
}' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: error adding table 'test_jina_remote': prealloc: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
- *************************** 1. row ***************************
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- Table: test_jina_remote
- Create Table: CREATE TABLE test_jina_remote (
- id bigint,
- title text,
- content text,
- description text,
- embedding float_vector knn_type='hnsw' hnsw_similarity='L2' model_name='jina/jina-embeddings-v4' from='title, content' api_timeout='10'
- )
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt
SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)
echo "Cosine similarity: $SIMILARITY"
RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
if (sim > 0.99)
print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
else
print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')
echo "$RESULT"
rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
+ Cosine similarity: -nan
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- single_field_md5: #!/[0-9a-f]{32}/!#
+ multi_field_md5: d41d8cd98f00b204e9800998ecf8427e
- SUCCESS: Different FROM specifications produce different vectors
+ single_field_md5: 292f8b7c5f455f3c88abfeb9918ac7a7
+ SUCCESS: Different FROM specifications produce different vectors
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK |
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 8b2a51a |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/buddy-plugins/test-conversational-intent.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 30 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
apt-get install jq -y > /dev/null; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 << EOF
CREATE TABLE docs (
id BIGINT,
content TEXT,
title TEXT,
category TEXT,
embedding_vector FLOAT_VECTOR
knn_type='hnsw'
hnsw_similarity='cosine'
model_name='sentence-transformers/all-MiniLM-L6-v2'
from='content'
api_key='\${OPENAI_API_KEY}'
) TYPE='rt';
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 << 'EOF'
INSERT INTO docs (id, content, title, category) VALUES
(1, 'Artificial intelligence is the capability of computational systems to perform tasks typically associated with human intelligence such as learning, reasoning, problem-solving, perception and decision-making. High-profile applications include advanced web search engines, recommendation systems, virtual assistants, autonomous vehicles and generative AI tools. The field was founded as an academic discipline in 1956 and has gone through multiple cycles of optimism and disappointment known as AI winters.', 'Artificial Intelligence', 'AI'),
(2, 'Machine learning is a field of study in artificial intelligence concerned with the development of statistical algorithms that can learn from data and generalize to unseen data. It finds application in natural language processing, computer vision, speech recognition, email filtering, agriculture and medicine. Deep learning, a subset of machine learning, has allowed neural networks to surpass many previous approaches in performance.', 'Machine Learning', 'AI'),
(3, 'Retrieval-augmented generation (RAG) is a technique that enables large language models to retrieve and incorporate new information. RAG improves LLMs by incorporating information retrieval before generating responses, pulling relevant text from databases, uploaded documents or web sources. This method helps reduce AI hallucinations and allows LLMs to include sources in their responses for greater transparency.', 'Retrieval-Augmented Generation', 'AI'),
(4, 'PHP is a general-purpose scripting language geared towards web development created by Rasmus Lerdorf in 1993. PHP code is usually processed on a web server by a PHP interpreter implemented as a module, a daemon or a Common Gateway Interface executable. The language has evolved without a written formal specification with the original implementation acting as the de facto standard.', 'PHP', 'Programming'),
(5, 'Python is a high-level general-purpose programming language designed by Guido van Rossum. Its design philosophy emphasizes code readability with the use of significant indentation. Python supports multiple programming paradigms including structured, object-oriented and functional programming. It consistently ranks as one of the most popular programming languages and has gained widespread use in the machine learning community.', 'Python', 'Programming'),
(6, 'Breaking Bad is an American neo-Western crime drama television series created by Vince Gilligan for AMC. The series follows Walter White, an underpaid high school chemistry teacher who turns to producing methamphetamine after being diagnosed with lung cancer. The show stars Bryan Cranston as Walter White, Aaron Paul as Jesse Pinkman, Anna Gunn as Skyler White, Dean Norris as Hank Schrader, and Bob Odenkirk as Saul Goodman. It premiered in 2008 and has been praised as one of the greatest television series of all time.', 'Breaking Bad', 'Entertainment'),
(7, 'The Mandalorian is an American space Western television series created by Jon Favreau for Disney Plus. It is the first live-action series in the Star Wars franchise and follows Din Djarin, a lone bounty hunter played by Pedro Pascal, who protects a Force-sensitive child named Grogu. The series premiered with the launch of Disney Plus on November 12, 2019 and has received largely positive reviews from critics.', 'The Mandalorian', 'Entertainment'),
(8, 'Stranger Things is an American television series created by the Duffer Brothers for Netflix. Set in the 1980s, the series centers on the residents of the fictional small town of Hawkins, Indiana, after a nearby facility opens a gateway to a hostile alternate dimension known as the Upside Down. The show stars Winona Ryder, David Harbour, Millie Bobby Brown and Finn Wolfhard.', 'Stranger Things', 'Entertainment'),
(9, 'Game of Thrones is an American fantasy drama television series created by David Benioff for HBO. It is an adaptation of A Song of Ice and Fire novels. Set on the fictional continents of Westeros and Essos, the show follows several story arcs throughout eight seasons including the Iron Throne succession and the threat from the North.', 'Game of Thrones', 'Entertainment'),
(10, 'The Office is an American mockumentary sitcom television series adapted for NBC by Greg Daniels. The show depicts the everyday work lives of office employees at the Scranton Pennsylvania branch of the fictional Dunder Mifflin Paper Company. It stars Steve Carell, Rainn Wilson, John Krasinski, Jenna Fischer and B.J. Novak.', 'The Office', 'Entertainment');
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "debug sleep 30;" > /dev/null 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE CHAT MODEL test_assistant (model='openai:gpt-4o-mini');"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('What is RAG?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_1.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_1.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([3])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Its boring', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('what about movies?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_3.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_3.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([8])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('I like Breaking Bad', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_4.txt
––– output –––
*************************** 1. row ***************************
#!/.*conversation_uuid: test-conv-intent-001/!#
#!/.*user_query: I like Breaking Bad/!#
#!/.*search_query: .*[Bb]reaking [Bb]ad.*/!#
- #!/.*response: .*[Bb]reaking [Bb]ad.*/!#
+ response: I don’t have enough information in the provided context to answer.
#!/.*sources: .*/!#
––– input –––
grep 'sources:' /tmp/output_4.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([6])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('show me more entertainment', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Tell me about Python', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_7.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_7.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([5])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP CHAT MODEL test_assistant;"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS docs;"
––– output –––
OK |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/mcl/auto-embeddings-jina-remote.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
local file1="$1" file2="$2"
awk '
NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
{
dot += a[FNR]*$1
sumb2 += $1*$1
}
END {
print dot / (sqrt(suma2) * sqrt(sumb2))
}' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
+--------------+
| record_count |
+--------------+
- | 1 |
+ | 0 |
+--------------+
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt
SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)
echo "Cosine similarity: $SIMILARITY"
RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
if (sim > 0.99)
print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
else
print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')
echo "$RESULT"
rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ Cosine similarity: -nan
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
- single_field_md5: #!/[0-9a-f]{32}/!#
+ multi_field_md5: d41d8cd98f00b204e9800998ecf8427e
- SUCCESS: Different FROM specifications produce different vectors
+ single_field_md5: d41d8cd98f00b204e9800998ecf8427e
+ INFO: FROM field comparison result
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK |
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 8b2a51a |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/buddy-plugins/test-conversational-intent.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 30 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
apt-get install jq -y > /dev/null; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 << EOF
CREATE TABLE docs (
id BIGINT,
content TEXT,
title TEXT,
category TEXT,
embedding_vector FLOAT_VECTOR
knn_type='hnsw'
hnsw_similarity='cosine'
model_name='sentence-transformers/all-MiniLM-L6-v2'
from='content'
api_key='\${OPENAI_API_KEY}'
) TYPE='rt';
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 << 'EOF'
INSERT INTO docs (id, content, title, category) VALUES
(1, 'Artificial intelligence is the capability of computational systems to perform tasks typically associated with human intelligence such as learning, reasoning, problem-solving, perception and decision-making. High-profile applications include advanced web search engines, recommendation systems, virtual assistants, autonomous vehicles and generative AI tools. The field was founded as an academic discipline in 1956 and has gone through multiple cycles of optimism and disappointment known as AI winters.', 'Artificial Intelligence', 'AI'),
(2, 'Machine learning is a field of study in artificial intelligence concerned with the development of statistical algorithms that can learn from data and generalize to unseen data. It finds application in natural language processing, computer vision, speech recognition, email filtering, agriculture and medicine. Deep learning, a subset of machine learning, has allowed neural networks to surpass many previous approaches in performance.', 'Machine Learning', 'AI'),
(3, 'Retrieval-augmented generation (RAG) is a technique that enables large language models to retrieve and incorporate new information. RAG improves LLMs by incorporating information retrieval before generating responses, pulling relevant text from databases, uploaded documents or web sources. This method helps reduce AI hallucinations and allows LLMs to include sources in their responses for greater transparency.', 'Retrieval-Augmented Generation', 'AI'),
(4, 'PHP is a general-purpose scripting language geared towards web development created by Rasmus Lerdorf in 1993. PHP code is usually processed on a web server by a PHP interpreter implemented as a module, a daemon or a Common Gateway Interface executable. The language has evolved without a written formal specification with the original implementation acting as the de facto standard.', 'PHP', 'Programming'),
(5, 'Python is a high-level general-purpose programming language designed by Guido van Rossum. Its design philosophy emphasizes code readability with the use of significant indentation. Python supports multiple programming paradigms including structured, object-oriented and functional programming. It consistently ranks as one of the most popular programming languages and has gained widespread use in the machine learning community.', 'Python', 'Programming'),
(6, 'Breaking Bad is an American neo-Western crime drama television series created by Vince Gilligan for AMC. The series follows Walter White, an underpaid high school chemistry teacher who turns to producing methamphetamine after being diagnosed with lung cancer. The show stars Bryan Cranston as Walter White, Aaron Paul as Jesse Pinkman, Anna Gunn as Skyler White, Dean Norris as Hank Schrader, and Bob Odenkirk as Saul Goodman. It premiered in 2008 and has been praised as one of the greatest television series of all time.', 'Breaking Bad', 'Entertainment'),
(7, 'The Mandalorian is an American space Western television series created by Jon Favreau for Disney Plus. It is the first live-action series in the Star Wars franchise and follows Din Djarin, a lone bounty hunter played by Pedro Pascal, who protects a Force-sensitive child named Grogu. The series premiered with the launch of Disney Plus on November 12, 2019 and has received largely positive reviews from critics.', 'The Mandalorian', 'Entertainment'),
(8, 'Stranger Things is an American television series created by the Duffer Brothers for Netflix. Set in the 1980s, the series centers on the residents of the fictional small town of Hawkins, Indiana, after a nearby facility opens a gateway to a hostile alternate dimension known as the Upside Down. The show stars Winona Ryder, David Harbour, Millie Bobby Brown and Finn Wolfhard.', 'Stranger Things', 'Entertainment'),
(9, 'Game of Thrones is an American fantasy drama television series created by David Benioff for HBO. It is an adaptation of A Song of Ice and Fire novels. Set on the fictional continents of Westeros and Essos, the show follows several story arcs throughout eight seasons including the Iron Throne succession and the threat from the North.', 'Game of Thrones', 'Entertainment'),
(10, 'The Office is an American mockumentary sitcom television series adapted for NBC by Greg Daniels. The show depicts the everyday work lives of office employees at the Scranton Pennsylvania branch of the fictional Dunder Mifflin Paper Company. It stars Steve Carell, Rainn Wilson, John Krasinski, Jenna Fischer and B.J. Novak.', 'The Office', 'Entertainment');
EOF
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "debug sleep 30;" > /dev/null 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE CHAT MODEL test_assistant (model='openai:gpt-4o-mini');"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('What is RAG?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_1.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_1.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([3])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Its boring', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('what about movies?', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_3.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_3.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([8])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('I like Breaking Bad', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_4.txt
––– output –––
*************************** 1. row ***************************
#!/.*conversation_uuid: test-conv-intent-001/!#
#!/.*user_query: I like Breaking Bad/!#
#!/.*search_query: .*[Bb]reaking [Bb]ad.*/!#
- #!/.*response: .*[Bb]reaking [Bb]ad.*/!#
+ response: I don’t have enough information in the provided context to answer.
#!/.*sources: .*/!#
––– input –––
grep 'sources:' /tmp/output_4.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([6])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('show me more entertainment', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }'
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CALL CHAT('Tell me about Python', 'docs', 'test_assistant', 'test-conv-intent-001')\G;" | awk '/^[[:space:]]*response: ?/ { if (in_field) print r; r=$0; in_field=1; next } in_field && /^[[:space:]]+sources: / { print r; print; in_field=0; next } in_field { r=r " " $0; next } { print } END { if (in_field) print r }' | tee /tmp/output_7.txt
––– output –––
OK
––– input –––
grep 'sources:' /tmp/output_7.txt | grep -o '\[.*\]' | head -1 | { read -r s; if echo "$s" | jq -e 'map(.id) | contains([5])' >/dev/null 2>&1; then echo "OK"; else echo "FAIL"; echo "Found: $(echo "$s" | jq -c 'map(.id)')"; echo "$s"; fi; }
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP CHAT MODEL test_assistant;"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS docs;"
––– output –––
OK |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/mcl/auto-embeddings-jina-remote.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
local file1="$1" file2="$2"
awk '
NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
{
dot += a[FNR]*$1
sumb2 += $1*$1
}
END {
print dot / (sqrt(suma2) * sqrt(sumb2))
}' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: error adding table 'test_jina_remote': prealloc: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
- *************************** 1. row ***************************
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- Table: test_jina_remote
- Create Table: CREATE TABLE test_jina_remote (
- id bigint,
- title text,
- content text,
- description text,
- embedding float_vector knn_type='hnsw' hnsw_similarity='L2' model_name='jina/jina-embeddings-v4' from='title, content' api_timeout='10'
- )
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt
SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)
echo "Cosine similarity: $SIMILARITY"
RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
if (sim > 0.99)
print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
else
print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')
echo "$RESULT"
rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
+ Cosine similarity: -nan
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- single_field_md5: #!/[0-9a-f]{32}/!#
+ multi_field_md5: d41d8cd98f00b204e9800998ecf8427e
- SUCCESS: Different FROM specifications produce different vectors
+ single_field_md5: 292f8b7c5f455f3c88abfeb9918ac7a7
+ SUCCESS: Different FROM specifications produce different vectors
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK |
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 8b2a51a |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/buddy/test-unserialize-error-absence-kafka-operations.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
seq 1 5 | xargs -I{} bash -c "mysql -h0 -P9306 -e 'DROP MATERIALIZED VIEW view_table;' || true; mysql -h0 -P9306 -e 'DROP SOURCE kafka;' || true; mysql -h0 -P9306 -e 'DROP TABLE kafka_test;' || true; mysql -h0 -P9306 -e \"CREATE SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type=\\\"kafka\\\" broker_list=\\\"kafka:9092\\\" topic_list=\\\"my-data\\\" consumer_group=\\\"manticore\\\" num_consumers=\\\"2\\\" batch=50; CREATE TABLE kafka_test (id bigint, name text, short_name text, received_at text, size multi); CREATE MATERIALIZED VIEW view_table TO kafka_test AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size FROM kafka; SHOW SOURCES; SHOW TABLES; SHOW MVS;\""
––– output –––
- ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'MATERIALIZED VIEW view_table'
- +-------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka'
- | name |
+ ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
- +-------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type="kafka" broker_list="kafka:9092" topic_list="my-data" consumer_group="manticore" num_consumers="2" batch=50'
- | kafka |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'MATERIALIZED VIEW view_table'
- +-------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka'
- +------------+------+
+ ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
- | Table | Type |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type="kafka" broker_list="kafka:9092" topic_list="my-data" consumer_group="manticore" num_consumers="2" batch=50'
- +------------+------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'MATERIALIZED VIEW view_table'
- | kafka_test | rt |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka'
- +------------+------+
+ ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
- +------------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type="kafka" broker_list="kafka:9092" topic_list="my-data" consumer_group="manticore" num_consumers="2" batch=50'
- | name |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'MATERIALIZED VIEW view_table'
- +------------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka'
- | view_table |
+ ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
- +------------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type="kafka" broker_list="kafka:9092" topic_list="my-data" consumer_group="manticore" num_consumers="2" batch=50'
- +-------+
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'MATERIALIZED VIEW view_table'
- | name |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka'
- +-------+
+ ERROR 1064 (42000) at line 1: DROP TABLE failed: unknown table 'kafka_test'
- | kafka |
+ ERROR 1064 (42000) at line 1: P01: syntax error, unexpected identifier, expecting USER near 'SOURCE kafka (id bigint, term text, abbrev text, GlossDef json) type="kafka" broker_list="kafka:9092" topic_list="my-data" consumer_group="manticore" num_consumers="2" batch=50'
- +-------+
- +------------+------+
- | Table | Type |
- +------------+------+
- | kafka_test | rt |
- +------------+------+
- +------------+
- | name |
- +------------+
- | view_table |
- +------------+
- +-------+
- | name |
- +-------+
- | kafka |
- +-------+
- +------------+------+
- | Table | Type |
- +------------+------+
- | kafka_test | rt |
- +------------+------+
- +------------+
- | name |
- +------------+
- | view_table |
- +------------+
- +-------+
- | name |
- +-------+
- | kafka |
- +-------+
- +------------+------+
- | Table | Type |
- +------------+------+
- | kafka_test | rt |
- +------------+------+
- +------------+
- | name |
- +------------+
- | view_table |
- +------------+
- +-------+
- | name |
- +-------+
- | kafka |
- +-------+
- +------------+------+
- | Table | Type |
- +------------+------+
- | kafka_test | rt |
- +------------+------+
- +------------+
- | name |
- +------------+
- | view_table |
- +------------+
––– input –––
tail -n 1000 /var/log/manticore/searchd.log | grep -i "unserialize"
––– output –––
OK |
clt❌ CLT tests in Failed tests:🔧 Edit failed tests in UI: test/clt-tests/mcl/auto-embeddings-jina-remote.rec––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
local file1="$1" file2="$2"
awk '
NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
{
dot += a[FNR]*$1
sumb2 += $1*$1
}
END {
print dot / (sqrt(suma2) * sqrt(sumb2))
}' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: error adding table 'test_jina_remote': prealloc: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
- *************************** 1. row ***************************
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- Table: test_jina_remote
- Create Table: CREATE TABLE test_jina_remote (
- id bigint,
- title text,
- content text,
- description text,
- embedding float_vector knn_type='hnsw' hnsw_similarity='L2' model_name='jina/jina-embeddings-v4' from='title, content' api_timeout='10'
- )
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt
mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
grep -v embedding | \
sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
grep -E '^[0-9]+(\.[0-9]+)?$' | \
awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt
SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)
echo "Cosine similarity: $SIMILARITY"
RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
if (sim > 0.99)
print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
else
print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')
echo "$RESULT"
rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
+ Cosine similarity: -nan
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: error adding table 'test_jina_title_only': prealloc: Failed to send request to remote model
- single_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: table test_jina_remote: parse error: unknown column: embedding
- SUCCESS: Different FROM specifications produce different vectors
+ ERROR 1064 (42000) at line 1: table test_jina_title_only: parse error: unknown column: embedding
+ multi_field_md5: d41d8cd98f00b204e9800998ecf8427e
+ single_field_md5: d41d8cd98f00b204e9800998ecf8427e
+ INFO: FROM field comparison result
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK |
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 4f40a20 |
1 similar comment
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 4f40a20 |
🐳 Docker Images PushedBuddy Test-Kit Images:
Base image used: These images contain the buddy code from this PR and can be used for testing. Built from commit: 64cb620 |
donhardman
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref: #683