From Unstructured Text to JSON-Structured Insights
After carefully analyzing the project requirements, I decided to implement a task-oriented approach for this information extraction project using OpenAI's LLM family. The goal was to develop both a notebook for interactive development and testing of different pipeline phases, and a main script for terminal-based execution (excluding evaluation). My approach can be divided into several key phases:
- Data Loading: Load articles from JSON format
- Articles Exploration: A small EDA implementation to evaluate basic statistics
- Preprocessing: Minimal but effective text cleaning focused on markdown removal while preserving semantic content. (I decided not to use it for the following steps due to time)
- LLM Extraction: Single-step extraction using OpenAI's structured output
- Validation: Pydantic schema validation for output consistency
- Evaluation: Precision/Recall/F1 metrics against ground truth
You can find more details about my decisions and project details in the notebook.
- Process all required information in one API call for cost efficiency
- Use OpenAI's
response_formatwith Pydantic schemas for JSON generation - Process only articles with corresponding ground truth annotations (200 articles)
I Chose gpt-4o-mini to process 200+ articles within budget constraints while maintaining high extraction quality. This decision was due to the choice of optimizing costs and avoiding possible risks of unexpected errors. In addition, the results obtained seemed valid enough to continue with the following phases.
My initial idea was to do a more completed preprocessing step, but in accordance with the tight schedule and evaluating the data quite clean and valid I preferred to continue the other phases of the project. Only the initial markdown is removed here.
git clone https://github.com/silvano315/Information-extractor-from-text-with-OpenAI.gitpip install -r requirements.txt- Create
.envfile:
OPENAI_API_KEY=your_openai_api_key_here- Create directory structure:
mkdir -p data/{raw,preprocessed,output}Test extraction on a single random article:
python main.pyYou can find evaluation results here or you can retrive them using the notebook.
| Metric | Description | Score |
|---|---|---|
| Entity F1 | Named entity extraction accuracy | 0.812 |
| Role F1 | Role assignment accuracy | 0.741 |
| Topic Accuracy | Topic classification accuracy | 0.975 |
| Subtopic Accuracy | Subtopic classification accuracy | 0.970 |
The final result I think is quite satisfactory in terms of both performance and realization. Interesting result for topics and subtopics that achieve very high accuracy. This result shows the strength of OpenAI models when a well-structured schema of possible options is offered.
Entity extraction (for entities and roles) shows a good F1 scores, despite the complexity for named entity recognition tasks. I used F1 score for these tasks because I learnt that it should be a standard metric for NLP and NER. The lower F1 compared to accuracy is expected because of open-vocabulary extraction. Role assignment scores slightly lower due to possible semantic variations.