Skip to content

Commit 5f8dff5

Browse files
committed
Add confidence score to question routing eval output
We want to use the confidence score in the evaluation app, so we'll need to output it here. Also changed the key for the label to "classification" as this matches the terminology used in the existing evaluation projects.
1 parent 0a7585f commit 5f8dff5

2 files changed

Lines changed: 8 additions & 3 deletions

File tree

lib/tasks/evaluation.rake

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,11 @@ namespace :evaluation do
110110

111111
raise "Error occurred generating answer: #{answer.error_message}" if answer.status =~ /^error/
112112

113-
puts({ question_routing_label: answer.question_routing_label }.to_json)
113+
result = {
114+
classification: answer.question_routing_label,
115+
confidence_score: answer.question_routing_confidence_score,
116+
}
117+
118+
puts(result.to_json)
114119
end
115120
end

spec/lib/tasks/evaluation_spec.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,10 +228,10 @@
228228

229229
it "outputs the response as JSON to stdout" do
230230
ClimateControl.modify(INPUT: input) do
231-
answer = build(:answer, question_routing_label: "genuine_rag")
231+
answer = build(:answer, question_routing_label: "genuine_rag", question_routing_confidence_score: 0.2)
232232
allow(AnswerComposition::PipelineRunner).to receive(:call).and_return(answer)
233233
expect { Rake::Task[task_name].invoke("openai") }
234-
.to output("{\"question_routing_label\":\"genuine_rag\"}\n").to_stdout
234+
.to output("{\"classification\":\"genuine_rag\",\"confidence_score\":0.2}\n").to_stdout
235235
end
236236
end
237237

0 commit comments

Comments
 (0)