-
Notifications
You must be signed in to change notification settings - Fork 96
First Asserts tool #105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
First Asserts tool #105
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, a couple of comments on the time format.
I am trying to reproduce using ops and no luck so far.
Maybe adding a cloud test along with a couple unit tests could do it. Happy to help!
Not sure how to do the cloud test against |
tools/asserts.go
Outdated
EntityType string `json:"entityType" jsonschema:"description=The type of the entity to list"` | ||
EntityName string `json:"entityName" jsonschema:"description=The name of the entity to list"` | ||
Env string `json:"env" jsonschema:"description=The env of the entity to list"` | ||
Site string `json:"site" jsonschema:"description=The site of the entity to list"` | ||
Namespace string `json:"namespace" jsonschema:"description=The namespace of the entity to list"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for each of these string types, is the model expected to know the possible options in advance? If it's a closed set (e.g. entity type) we should provide a list of options in the description really; for the others, perhaps we should tell the LLM how to obtain valid values either in the field description or tool description.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can supply a list of know types. This will help LLM.
@xujiaxj I saw a comment about getting zero values for the time struct which was strange so tried pushing a separate commit to your branch, hope that's OK. It might be unnecessary though? Let me know what you think. |
That happened when I was using Cursor with MCP. Somehow it didn't happen with Claude desktop, so it seems okay for me. But for sure, the new commit looks good to me, and I will test it out. |
I think the reason it's flaky with Cursor is that I keep changing my MCP config to different Grafana instances. I should've restarted Cursor just like I restart Claude desktop whenever I change the MCP config. |
Just fixed another bug while testing against |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code-wise this LGTM now. I am wondering if we could have some cloud tests running against a cloud instance which has asserts installed & running (rather than using the mcptests instance, which doesn't). Do you think that sounds doable?
Added another test and tested it locally, but need help to put up the two variables in the setting on this page as I don't have access. cc: @csmarchbanks
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks again @xujiaxj 🙌
A couple of thoughts on the testing flow.
Let me know what you think :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
Adding the first Asserts tool for an API call. It's not quite important what this tool does. It will probably go through more iterations in the future and we will add more tools. This is just a simple PR as the first step.
Tested with Claude Desktop: