Skip to content

First Asserts tool #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Apr 30, 2025
Merged

First Asserts tool #105

merged 15 commits into from
Apr 30, 2025

Conversation

xujiaxj
Copy link
Contributor

@xujiaxj xujiaxj commented Apr 21, 2025

Adding the first Asserts tool for an API call. It's not quite important what this tool does. It will probably go through more iterations in the future and we will add more tools. This is just a simple PR as the first step.

Tested with Claude Desktop:

image

@xujiaxj xujiaxj requested a review from a team as a code owner April 21, 2025 21:07
Copy link
Contributor

@ioanarm ioanarm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a couple of comments on the time format.
I am trying to reproduce using ops and no luck so far.

Maybe adding a cloud test along with a couple unit tests could do it. Happy to help!

@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 22, 2025

Not sure how to do the cloud test against mcptests stack as it does not have any metric.

@xujiaxj xujiaxj requested review from sd2k and ioanarm April 23, 2025 17:31
@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 23, 2025

image

tools/asserts.go Outdated
Comment on lines 35 to 39
EntityType string `json:"entityType" jsonschema:"description=The type of the entity to list"`
EntityName string `json:"entityName" jsonschema:"description=The name of the entity to list"`
Env string `json:"env" jsonschema:"description=The env of the entity to list"`
Site string `json:"site" jsonschema:"description=The site of the entity to list"`
Namespace string `json:"namespace" jsonschema:"description=The namespace of the entity to list"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for each of these string types, is the model expected to know the possible options in advance? If it's a closed set (e.g. entity type) we should provide a list of options in the description really; for the others, perhaps we should tell the LLM how to obtain valid values either in the field description or tool description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can supply a list of know types. This will help LLM.

@sd2k
Copy link
Collaborator

sd2k commented Apr 25, 2025

@xujiaxj I saw a comment about getting zero values for the time struct which was strange so tried pushing a separate commit to your branch, hope that's OK. It might be unnecessary though? Let me know what you think.

@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 25, 2025

@xujiaxj I saw a comment about getting zero values for the time struct which was strange so tried pushing a separate commit to your branch, hope that's OK. It might be unnecessary though? Let me know what you think.

That happened when I was using Cursor with MCP. Somehow it didn't happen with Claude desktop, so it seems okay for me. But for sure, the new commit looks good to me, and I will test it out.

@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 25, 2025

@xujiaxj I saw a comment about getting zero values for the time struct which was strange so tried pushing a separate commit to your branch, hope that's OK. It might be unnecessary though? Let me know what you think.

That happened when I was using Cursor with MCP. Somehow it didn't happen with Claude desktop, so it seems okay for me. But for sure, the new commit looks good to me, and I will test it out.

I think the reason it's flaky with Cursor is that I keep changing my MCP config to different Grafana instances. I should've restarted Cursor just like I restart Claude desktop whenever I change the MCP config.

@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 25, 2025

Just fixed another bug while testing against ops. The scope attributes env/site/namespace is optional, so we shouldn't pass the empty string "" to the API.

@xujiaxj xujiaxj requested a review from sd2k April 25, 2025 16:55
@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 25, 2025

Latest test result in Cursor against ops seem fine.
image

Copy link
Collaborator

@sd2k sd2k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code-wise this LGTM now. I am wondering if we could have some cloud tests running against a cloud instance which has asserts installed & running (rather than using the mcptests instance, which doesn't). Do you think that sounds doable?

@xujiaxj
Copy link
Contributor Author

xujiaxj commented Apr 28, 2025

Added another test and tested it locally, but need help to put up the two variables in the setting on this page as I don't have access. cc: @csmarchbanks

ASSERTS_GRAFANA_URL="https://dev.grafana-dev.net"
ASSERTS_GRAFANA_API_KEY=[the token for service account "asserts-mcp-cloud-test" on the instance]

Monosnap Image 2025-04-28 14-49-40

Copy link
Contributor

@ioanarm ioanarm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again @xujiaxj 🙌

A couple of thoughts on the testing flow.
Let me know what you think :)

@xujiaxj xujiaxj requested review from sd2k and ioanarm April 29, 2025 16:29
Copy link
Contributor

@ioanarm ioanarm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@xujiaxj xujiaxj merged commit 4b87710 into main Apr 30, 2025
5 checks passed
@xujiaxj xujiaxj deleted the jiaxu/draft/asserts-tools branch April 30, 2025 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants