Feat/prompt#18
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aligns the QA prompting format with the EuroEval-style templates, updates generation/post-processing behavior, and removes the legacy SelfCheckGPT script/prompts.
Changes:
- Updated all
qa_prompt_*.txttemplates to use a${text}-based prompt format (max 3 words), and updatedPromptUtils.format_context()accordingly. - Adjusted local HF generation to decode only newly generated tokens and added markdown marker stripping.
- Tweaked dataset splitting logic, hallucination detector model naming, and reduced generation/training lengths in config; removed SelfCheckGPT script and prompt templates.
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/scripts/selfcheckgpt.py |
Removed legacy SelfCheckGPT runner script. |
src/scripts/detect_hallucinations.py |
Adjusts Hugging Face detector path naming for synthetic-hallucination dataset. |
src/prompts/selfcheckgpt_prompt_en.txt |
Removed SelfCheckGPT prompt template (EN). |
src/prompts/selfcheckgpt_prompt_de.txt |
Removed SelfCheckGPT prompt template (DE). |
src/prompts/selfcheckgpt_prompt_da.txt |
Removed SelfCheckGPT prompt template (DA). |
src/prompts/qa_prompt_uk.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_sv.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_sr.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_sl.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_sk.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_ro.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_pt.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_pl.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_no.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_nl.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_lv.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_lt.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_it.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_is.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_hu.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_hr.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_fr.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_fo.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_fi.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_et.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_es.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_en.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_el.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_de.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_da.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_cs.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_ca.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_bs.txt |
Updated QA prompt to ${text} format. |
src/prompts/qa_prompt_bg.txt |
Updated QA prompt to ${text} format. |
src/factuality_eval/prompt_utils.py |
Updates prompt formatting to pass ${text} and removes passage-label formatting. |
src/factuality_eval/model_generation.py |
Improves HF generation decoding and strips markdown markers from outputs. |
src/factuality_eval/hallucination_detection.py |
Removes per-example exception handling around detector prediction. |
src/factuality_eval/dataset_generation.py |
Changes dataset split handling when only train is available; otherwise raises. |
config/hallucination_detection.yaml |
Reduces training.max_length and generation.max_new_tokens. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/alexandrainst/factuality_eval/sessions/045c333a-8816-4b8a-919f-217ac634e975 Co-authored-by: FrejaThoresen <13599833+FrejaThoresen@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This reverts commit 2247eba.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minor bugfixes.
Remove selfcheckgpt code.