Ready Feature: OpenAI Evaluation of Nightscout Data

[xannasavin]

This is not a feature request — the feature is already implemented and working.

Disclaimer: Since I’m not deeply familiar with the Nightscout project’s infrastructure and coding standards, I made use of an AI coding agent for the development. I have tested the feature thoroughly and it works as expected, but I cannot guarantee the code quality or compliance with project conventions.

---

What the feature does

I have added a new AI Evaluation tab to the Nightscout reporting view.

• The selected CGM report data (up to 14 days) is sent to an LLM (e.g., OpenAI GPT-4o) for evaluation.
• Each day is analyzed individually, and the AI then produces a clear and understandable summary of the whole period.
• Results are displayed directly in Nightscout, structured into daily interim evaluations and a final report.

---

Admin Settings

New settings in the Admin Tools allow users to:

• Configure the system and user prompts for AI calls.
• Control the model and endpoint used.
• View a cost overview: all tokens are converted into USD, logged, and aggregated monthly.
• See a cost preview for each current request directly in the frontend.

---

Key Features

• Configurable prompts for flexibility.
• Secure API key & Credentials handling (server-side only).
• Token usage & cost tracking with summaries per month.
• Debug mode to inspect prompts and AI responses.
• Automatic JSON repair if the AI returns invalid data.

---

Status

• Feature is implemented and running in a test branch.
• Supports OpenAI-compatible endpoints with configurable models.
• Admin integration for prompts and usage stats is included.
• Fully tested functionally, but code quality needs review against Nightscout standards.

---

Why it matters

This plugin allows Nightscout to go beyond raw graphs:

• It provides natural-language explanations of glucose patterns.
• It highlights fluctuations, stability issues, and potential improvements.
• It adds an intelligent summary layer to complement traditional reporting.

---

More information

https://github.com/xannasavin/cgm-remote-monitor/blob/feat/ai-report-plugin/ai_evaluation.md

---

Screenshots

Admin Prompt Settings

Usage Statistics

AI Evaluation Tab

During Sending

Results

Summary

Single Day

Usage Statistic for current run

---

I’d love to hear feedback from the community: is this something you’d like to see integrated into Nightscout?
If yes, what would be the next steps?
For sure someone would need to make a code review of this.

[psonnera]

Maybe you should post this also in Discord nightscout_developers channel, I will have a look at it for sure.

[psonnera]

@xannasavin i tried your code and modified it for Claude. I have mixed feelings about the results as it reported things that I don't see in my reports (too many lows, inconsistent TIR, ...), and suggested the same mistakes I had been doing (increasing post-meal basal instead of reducing I:C). But the concept is nice though. Probably working on the directives could help make it better but I'll wait for more testers. The code is functional (just last attempt 14 days crashed in the last day...).

[xannasavin]

thanks for your feedback @psonnera . The prompt needs to be adjusted for sure, I just added a very basic one in the code so something is there. the results were quite well on my end.
I think I have added a code that limits the number of days that can be requested, since the interaction with the LLM takes quite some time and the script can't handle too many days due to that fact.
I have tested with o4-mini only yet, probably using a model with bigger context capabilities like the new 5 might work even better.
Have you just exchanged the model or have you implemented a posibility to switch? That would be my next goal, so it becomes more model agnostic.

[psonnera]

@xannasavin I modified the code directly, let me PR in yours so that you can see the changes. It probably can be more flexible softcoding differences based on AI_LLM_API_URL. Sorry with the mess with # 59. Ignore it please.

[PieterGit]

@xannasavin and @psonnera nice idea, although I think the implementation and prompts need quite some work to make it more usefull. Some remarks:

• If you want it to be included in Nightscout, you need to open a PR to the dev branch and not to the master
• From my experience gpt-4o can be quite bad with doing math, so I wonder if that is solved with gpt-5. GPT-5 recommended GPT-5 mini and Confucius3-Math (open source, 14B parameters, designed for math reasoning).
• The screenshots do not give a lot of good advices. I wonder if you have seen patterns you didn't notice yourself so that the advice makes some sense.

Good luck. Unfortuately I don't have time to give it a try.

[xannasavin]

Hey @PieterGit , thanks for your feedback!
I did not want to open a PR yet, since I'm not sure if someone wants to include this into Nightscout. If yes I am happy to do so.
I agree on the gpt-4 remarks, the model is chosen via the environment settings, so any model can be chosen. Though I was quite OK with the results the 4o model returned.
The screenshots are not meant to give any insight on my data, just to give an idea what this is about. Of course one can scan through the graphs provided by nightscout and draw conclusions based on the basal, iob and cob - but I wanted to automate this so I might also get ideas I would not have seen. Just like a second pair of eyes.
Currently I'm not able to dive deeper into this, hope I will get the chance until end of this year (where my license for the coding agent is runing out ><)

[PieterGit]

Hi @xannasavin . I think this can be of help, but in order to be mergable it must be usable, and there must be people that want to maintain it. Also it will need some added documentation, but you started that in the UI. Currently I am not interested on working on this as a project, but maybe others are. The screenshots can give a good impression of what the AI component is capable is. The current screenshots didn't gave me that impression yet 😄 . Let me now if you and perhaps @psonnera want to bring this further.

PR's should be based on dev to make it possible to merge them, so working on master branch is not something I would suggest. Good luck. Note that having a AI coding agent, you and others still need to review the code for possible security issues and potential bugs.

[sulkaharo]

Having used LLMs extensively for a variety of purposes, I don't think this should be merged. LLMs give unpredictable outputs and we can't guarantee anything about what the users will see unless this is tested with every model and a large dataset, including all future models as part of maintenance. While it's entirely possible most of the outputs are sensible and valuable, it's highly likely some of the recommendations will be catastrophically wrong and put people at risk if they trust the outputs. This is basically not a use case for an LLM, even if we wanted this to be the case.