Add FAIR4AI evaluation agent and documentation #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initial commit of the FAIR4AI evaluation agent, including core scripts, GUI, example usage, requirements, form checklist, template outputs, and sample evaluation results. Provides full documentation in README.md and an environment variable template for API configuration.
Member
|
Could you remove the |
Collaborator
Author
|
ah, thanks, will do! |
egrace479
reviewed
Jan 28, 2026
egrace479
reviewed
Jan 30, 2026
| @@ -0,0 +1,991 @@ | |||
| # FAIR4AI Automated Evaluation Agent | |||
|
|
|||
| An AI-powered system that automatically evaluates datasets against the FAIR4AI checklist using Large Language Models (LLMs). The agent analyzes metadata files or dataset landing pages to provide comprehensive FAIR (Findable, Accessible, Interoperable, Reusable) assessments. | |||
Member
There was a problem hiding this comment.
Suggested change
| An AI-powered system that automatically evaluates datasets against the FAIR4AI checklist using Large Language Models (LLMs). The agent analyzes metadata files or dataset landing pages to provide comprehensive FAIR (Findable, Accessible, Interoperable, Reusable) assessments. | |
| An AI-powered system that automatically evaluates datasets against the [FAIR4AI checklist form](https://forms.gle/P3MWmJJAi5vq248E8) using Large Language Models (LLMs). The agent checks metadata files or dataset landing pages for the information requested in the checklist to provide comprehensive FAIR (Findable, Accessible, Interoperable, Reusable) assessments. |
| - Generates structured outputs in JSON and CSV formats with FAIR score estimates | ||
|
|
||
| **Key Features:** | ||
| - ✅ Multiple LLM providers (OpenAI, Azure OpenAI, Anthropic Claude) |
Member
There was a problem hiding this comment.
Suggested change
| - ✅ Multiple LLM providers (OpenAI, Azure OpenAI, Anthropic Claude) | |
| - ✅ Multiple LLM providers supported (OpenAI, Azure OpenAI, Anthropic Claude) |
| **System Requirements:** | ||
| - Python 3.8 or higher | ||
| - API key for OpenAI, Azure OpenAI, or Anthropic | ||
| - Dependencies: pandas, openai, anthropic, requests, beautifulsoup4 |
Member
There was a problem hiding this comment.
Suggested change
| - Dependencies: pandas, openai, anthropic, requests, beautifulsoup4 |
This is covered by the install and doesn't need to be included in the README
Comment on lines
+35
to
+41
|
|
||
| Or install individually: | ||
| ```bash | ||
| pip install openai pandas requests beautifulsoup4 # For OpenAI | ||
| # OR | ||
| pip install anthropic pandas requests beautifulsoup4 # For Anthropic Claude | ||
| ``` |
Member
There was a problem hiding this comment.
Suggested change
| Or install individually: | |
| ```bash | |
| pip install openai pandas requests beautifulsoup4 # For OpenAI | |
| # OR | |
| pip install anthropic pandas requests beautifulsoup4 # For Anthropic Claude | |
| ``` |
This seems like an unnecessary complication -- I don't think there should be issues installing openai and anthropic with the requirements file.
| ## Quick Start | ||
|
|
||
| ### 1. Install Dependencies | ||
|
|
Member
There was a problem hiding this comment.
Suggested change
| We recommend creating a [virtual environment](https://imageomics.github.io/Collaborative-distributed-science-guide/wiki-guide/Virtual-Environments/) in which to install the requirements as described below. | |
| 5. Save all extracted metadata to output directory | ||
| 6. Run the evaluation using extracted metadata | ||
|
|
||
| Supported URLs include NEON, DataONE, Zenodo, Dryad, and other repositories with structured metadata. |
Member
There was a problem hiding this comment.
Suggested change
| Supported URLs include NEON, DataONE, Zenodo, Dryad, and other repositories with structured metadata. | |
| Supported URLs include those for NEON, DataONE, Zenodo, Dryad, Hugging Face Datasets, and other repositories with structured metadata. |
| - **Progress Bar**: Shows completion percentage | ||
| - **Log Window**: Real-time messages and status updates | ||
|
|
||
| #### Example Workflow: URL-Based Evaluation |
Member
There was a problem hiding this comment.
Could be cool to add a screenshot or two of the GUI, but that'd be for bigger docs. If this is more fleshed-out and public then we could use MkDocs to set something up (more user-friendly than a super long README).
| - `options`: Multiple choice options (pipe-separated) | ||
| - `required`: Whether required (TRUE/FALSE) | ||
|
|
||
| Default form: `form_ai_checklist_automated.csv` (135 questions across 9 sections) |
Member
There was a problem hiding this comment.
Suggested change
| Default form: `form_ai_checklist_automated.csv` (135 questions across 9 sections) | |
| Default form: `form_ai_checklist_automated.csv` (135 questions across 9 sections) | |
| The code used to create this file from the [FAIR4AI Checklist form](), is in the [checklist workflow directory](../checklist-workflow/00_README.md). |
| ├── {prefix}.json # FAIR4AI evaluation results | ||
| └── {prefix}.csv # FAIR4AI evaluation results | ||
| ``` | ||
|
|
Member
There was a problem hiding this comment.
Suggested change
| `prefix` is the name passed to the `--output` parameter. | |
also ignore Mac system files
egrace479
force-pushed
the
fair4ai-eval-agent
branch
from
a6f39cb to
f787705
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial commit of the FAIR4AI evaluation agent, including core scripts, GUI, example usage, requirements, form checklist, template outputs, and sample evaluation results. Provides full documentation in README.md and an environment variable template for API configuration.