Skip to content

Implement sample verification loop#12298

Closed
maorleger wants to merge 2 commits into
mainfrom
maorleger/sample-generation-tool
Closed

Implement sample verification loop#12298
maorleger wants to merge 2 commits into
mainfrom
maorleger/sample-generation-tool

Conversation

@maorleger
Copy link
Copy Markdown
Member

@maorleger maorleger commented Oct 1, 2025

This PR implements typechecking and sample verification loop for generated samples. Starting with TypeScript, with the rest to follow.

Typechecking infrastructure

  • Introduced the ILanguageTypechecker interface to provide a consistent abstraction for language-specific typecheckers, enabling easier extension to additional languages. (Azure.Sdk.Tools.Cli/SampleGeneration/ILanguageTypechecker.cs)
  • Refactored LanguageSupport to include a CreateTypechecker factory method for instantiating the appropriate typechecker based on language, and added GetTypecheckingInstructions to supply language-specific guidance for fixing type errors. (Azure.Sdk.Tools.Cli/SampleGeneration/LanguageSupport.cs)

Dependency and namespace organization

  • Updated LanguageSupport.cs to include necessary service and language namespaces for new typechecker and verification logic. (Azure.Sdk.Tools.Cli/SampleGeneration/LanguageSupport.cs)

@github-actions github-actions Bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Oct 1, 2025
Comment thread tools/azsdk-cli/Azure.Sdk.Tools.Cli/Program.cs
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch from 386eb70 to 4a2581b Compare October 3, 2025 17:30
@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch 3 times, most recently from 3ff295f to d79f309 Compare October 3, 2025 22:58
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch 3 times, most recently from 42edaf4 to dd3a7a2 Compare October 6, 2025 17:43

var fixedCode = await microagentService.RunAgentToCompletion(microagent, ct);

if (string.IsNullOrEmpty(fixedCode))
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we allow this or throw?

using Microsoft.Extensions.Logging.Abstractions;
using Moq;

namespace Azure.Sdk.Tools.Cli.Tests.Services.Languages
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: move this

using System.Text.Json;
using System.Text.RegularExpressions;

// TODO: how do we better organize these? Ideally a language folder with concrete implementations of each cross-language interface
Copy link
Copy Markdown
Member Author

@maorleger maorleger Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something to consider, we now have Languages under SampleGeneration (which I admittedly added) as well as Languages under Services. Ideally we can provide a single place for organization of language-specific implementations.

It may be something like:

Services/Languages/<language>Service.cs

Which is registered as a service, or maybe if needed:

Services/<scenario>/Languages/<language><scenario>Service // or
Services/Languages/<language>/<language><scenario>Service

etc.

Each concrete implementation can implement multiple interfaces based on a scenario like TypeScriptLanguageService : ISampleVerificationService, ILanguageSpecificChecksService etc

The Q I am asking is how to best organize language-specific implementation of common services / interfaces

@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch from d79f309 to da4f6d8 Compare October 6, 2025 18:49
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch 3 times, most recently from b0277b1 to 941891b Compare October 6, 2025 19:39
@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch from da4f6d8 to 03223e8 Compare October 6, 2025 22:06
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch from 941891b to 4965b34 Compare October 6, 2025 22:20
@maorleger maorleger marked this pull request as ready for review October 6, 2025 22:22
@maorleger maorleger requested a review from a team as a code owner October 6, 2025 22:22
@maorleger maorleger requested a review from deyaaeldeen October 6, 2025 22:22
@maorleger maorleger changed the title wip: verification flow Implement sample verification loop Oct 6, 2025
@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch from 03223e8 to 8e2454c Compare October 7, 2025 01:58
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch from 4965b34 to 36f566c Compare October 7, 2025 17:02
@maorleger maorleger force-pushed the maorleger/sample-generation-tool branch from f4cb61c to d72c699 Compare October 7, 2025 21:42
@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch 3 times, most recently from 0a107ae to ab60cd9 Compare October 8, 2025 19:03
@deyaaeldeen deyaaeldeen force-pushed the sample-generation-tool branch 2 times, most recently from d7426c3 to e8cb366 Compare October 8, 2025 19:35
Base automatically changed from sample-generation-tool to main October 27, 2025 23:45
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jan 2, 2026

Hi @maorleger. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

@github-actions github-actions Bot added the no-recent-activity There has been no recent activity on this issue. label Jan 2, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jan 9, 2026

Hi @maorleger. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing /reopen if you'd like to continue working on these changes. Please be sure to use the command to reopen or remove the no-recent-activity label; otherwise, this is likely to be closed again with the next cleanup pass.

@github-actions github-actions Bot closed this Jan 9, 2026
@ronniegeraghty ronniegeraghty added the AzSDK Tools Agent Issue related to the AzSDK Tools Agent. label May 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AzSDK Tools Agent Issue related to the AzSDK Tools Agent. azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli no-recent-activity There has been no recent activity on this issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants