Skip to content

Commit ea025c7

Browse files
ZachhKZachary King
ZachhK
and
Zachary King
authored
adding image js samples (Azure#33986)
### Packages impacted by this PR ### Issues associated with this PR ### Describe the problem that is addressed by this PR image input samples generated in js ### What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen? ### Are there test cases added in this PR? _(If not, why?)_ ### Provide a list of related PRs _(if any)_ ### Command used to generate this PR:**_(Applicable only to SDK release request PRs)_ ### Checklists - [ ] Added impacted package name to the issue description - [ ] Does this PR needs any fixes in the SDK Generator?** _(If so, create an Issue in the [Autorest/typescript](https://github.com/Azure/autorest.typescript) repository and link it here)_ - [ ] Added a changelog (if necessary) Co-authored-by: Zachary King <[email protected]>
1 parent fab1496 commit ea025c7

File tree

7 files changed

+344
-3
lines changed

7 files changed

+344
-3
lines changed

‎sdk/ai/ai-projects/samples-dev/agents/agentsImageInputWithBase64.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
/**
55
* @summary This sample demonstrates how to use basic agent operations with image input (base64 encoded) for the Azure Agents service.
66
*
7-
* @azsdk-skip-javascript
87
*/
98

109
import { AIProjectsClient } from "@azure/ai-projects";

‎sdk/ai/ai-projects/samples-dev/agents/agentsImageInputWithFile.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
* @summary This sample demonstrates how to use basic agent operations using image file input for the
66
* Azure Agents service.
77
*
8-
* @azsdk-skip-javascript
98
*/
109

1110
import { AIProjectsClient } from "@azure/ai-projects";

‎sdk/ai/ai-projects/samples-dev/agents/agentsImageInputWithUrl.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
* @summary This sample demonstrates how to use basic agent operations using image url input for the
66
* Azure Agents service.
77
*
8-
* @azsdk-skip-javascript
98
*/
109

1110
import { AIProjectsClient } from "@azure/ai-projects";

‎sdk/ai/ai-projects/samples/v1-beta/javascript/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ These sample programs show how to use the JavaScript client libraries for Azure
2323
| [agents/agentsBingGrounding.js][agents_agentsbinggrounding] | demonstrates how to use agent operations with the Grounding with Bing Search tool. |
2424
| [agents/agentsBingGroundingWithStreaming.js][agents_agentsbinggroundingwithstreaming] | demonstrates how to use agent operations with the Grounding with Bing Search tool using streaming. |
2525
| [agents/agentsFabric.js][agents_agentsfabric] | demonstrates how to use agent operations with the Microsoft Fabric tool. |
26+
| [agents/agentsImageInputWithBase64.js][agents_agentsimageinputwithbase64] | This sample demonstrates how to use basic agent operations with image input (base64 encoded) for the Azure Agents service. |
27+
| [agents/agentsImageInputWithFile.js][agents_agentsimageinputwithfile] | This sample demonstrates how to use basic agent operations using image file input for the Azure Agents service. |
28+
| [agents/agentsImageInputWithUrl.js][agents_agentsimageinputwithurl] | This sample demonstrates how to use basic agent operations using image url input for the Azure Agents service. |
2629
| [agents/agentsSharepoint.js][agents_agentssharepoint] | demonstrates how to use agent operations with the Sharepoint tool. |
2730
| [agents/agentsWithFunctionTool.js][agents_agentswithfunctiontool] | demonstrates how to use basic agent operations using function tool. |
2831
| [agents/agentsWithOpenApi.js][agents_agentswithopenapi] | demonstrates how to use agent operations with an OpenApi tool. |
@@ -92,6 +95,9 @@ Take a look at our [API Documentation][apiref] for more information about the AP
9295
[agents_agentsbinggrounding]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsBingGrounding.js
9396
[agents_agentsbinggroundingwithstreaming]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsBingGroundingWithStreaming.js
9497
[agents_agentsfabric]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsFabric.js
98+
[agents_agentsimageinputwithbase64]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithBase64.js
99+
[agents_agentsimageinputwithfile]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithFile.js
100+
[agents_agentsimageinputwithurl]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithUrl.js
95101
[agents_agentssharepoint]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsSharepoint.js
96102
[agents_agentswithfunctiontool]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsWithFunctionTool.js
97103
[agents_agentswithopenapi]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsWithOpenApi.js

‎sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithBase64.js‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
/**
5+
* @summary This sample demonstrates how to use basic agent operations with image input (base64 encoded) for the Azure Agents service.
6+
*/
7+
8+
const { AIProjectsClient } = require("@azure/ai-projects");
9+
const { DefaultAzureCredential } = require("@azure/identity");
10+
const fs = require("fs");
11+
12+
// Load environment variables
13+
const connectionString = process.env.AZURE_AI_PROJECTS_CONNECTION_STRING || "<connection-string>";
14+
const modelDeployment = process.env.MODEL_DEPLOYMENT_NAME || "<model-deployment-name>";
15+
const filePath = "./data/image_file.png";
16+
17+
/**
18+
* Convert an image file to a Base64-encoded data URL.
19+
* @param imagePath - The path to the image file
20+
* @param mimeType - The MIME type of the image (e.g., 'image/png', 'image/jpeg')
21+
* @returns A data URL with the Base64-encoded image
22+
*/
23+
function imageToBase64DataUrl(imagePath, mimeType) {
24+
try {
25+
// Read the image file as binary
26+
const imageBuffer = fs.readFileSync(imagePath);
27+
// Convert to base64
28+
const base64Data = imageBuffer.toString("base64");
29+
// Format as a data URL
30+
return `data:${mimeType};base64,${base64Data}`;
31+
} catch (error) {
32+
console.error(`Error reading image file at ${imagePath}:`, error);
33+
throw error;
34+
}
35+
}
36+
37+
async function main() {
38+
console.log("== AI Projects Agent with Base64 Image Input Sample ==");
39+
40+
// Create the client
41+
const client = AIProjectsClient.fromConnectionString(
42+
connectionString,
43+
new DefaultAzureCredential(),
44+
);
45+
46+
// Create an agent
47+
console.log(`Creating agent with model ${modelDeployment}...`);
48+
const agent = await client.agents.createAgent(modelDeployment, {
49+
name: "my-agent",
50+
instructions: "You are helpful agent",
51+
});
52+
console.log(`Created agent, agent ID: ${agent.id}`);
53+
54+
// Create a thread
55+
console.log("Creating thread...");
56+
const thread = await client.agents.createThread();
57+
console.log(`Created thread, thread ID: ${thread.id}`);
58+
59+
// Create a message with text and image content using base64
60+
console.log("Converting image to base64...");
61+
const imageDataUrl = imageToBase64DataUrl(filePath, "image/png");
62+
63+
// Create a message with both text and image content
64+
console.log("Creating message with image content...");
65+
const inputMessage = "Hello, what is in the image?";
66+
const content = [
67+
{
68+
type: "text",
69+
text: inputMessage,
70+
},
71+
{
72+
type: "image_url",
73+
image_url: {
74+
url: imageDataUrl,
75+
detail: "high",
76+
},
77+
},
78+
];
79+
80+
const message = await client.agents.createMessage(thread.id, {
81+
role: "user",
82+
content: content,
83+
});
84+
console.log(`Created message, message ID: ${message.id}`);
85+
86+
// Create and poll a run
87+
console.log("Creating run...");
88+
let run = await client.agents.createRun(thread.id, agent.id);
89+
90+
// Poll the run as long as run status is queued or in progress
91+
while (
92+
run.status === "queued" ||
93+
run.status === "in_progress" ||
94+
run.status === "requires_action"
95+
) {
96+
// Wait for a second
97+
console.log(`Run status: ${run.status}, waiting...`);
98+
await new Promise((resolve) => setTimeout(resolve, 1000));
99+
run = await client.agents.getRun(thread.id, run.id);
100+
}
101+
console.log(`Run complete with status: ${run.status}`);
102+
103+
// Delete the agent
104+
await client.agents.deleteAgent(agent.id);
105+
console.log(`Deleted agent, agent ID: ${agent.id}`);
106+
107+
// List messages
108+
const messages = await client.agents.listMessages(thread.id, {
109+
order: "asc",
110+
});
111+
112+
for (const dataPoint of messages.data) {
113+
const textContent = dataPoint.content.find((item) => item.type === "text");
114+
if (textContent && "text" in textContent) {
115+
console.log(`${dataPoint.role}: ${textContent.text.value}`);
116+
}
117+
}
118+
119+
console.log("Messages: ", messages);
120+
}
121+
122+
main().catch((error) => {
123+
console.error("An error occurred:", error);
124+
});
125+
126+
module.exports = { main };

‎sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithFile.js‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
/**
5+
* @summary This sample demonstrates how to use basic agent operations using image file input for the
6+
* Azure Agents service.
7+
*/
8+
9+
const { AIProjectsClient } = require("@azure/ai-projects");
10+
const { DefaultAzureCredential } = require("@azure/identity");
11+
const fs = require("fs");
12+
13+
// Load environment variables
14+
const connectionString = process.env.AZURE_AI_PROJECTS_CONNECTION_STRING || "<connection-string>";
15+
const modelDeployment = process.env.MODEL_DEPLOYMENT_NAME || "<model-deployment-name>";
16+
const imagePath = "./data/image_file.png";
17+
18+
async function main() {
19+
console.log("== AI Projects Agent with Image Input Sample ==");
20+
21+
// Create the client
22+
const client = AIProjectsClient.fromConnectionString(
23+
connectionString,
24+
new DefaultAzureCredential(),
25+
);
26+
27+
// Create an agent
28+
console.log(`Creating agent with model ${modelDeployment}...`);
29+
const agent = await client.agents.createAgent(modelDeployment, {
30+
name: "my-agent",
31+
instructions: "You are helpful agent",
32+
});
33+
console.log(`Created agent, agent ID: ${agent.id}`);
34+
35+
// Create a thread
36+
console.log("Creating thread...");
37+
const thread = await client.agents.createThread();
38+
console.log(`Created thread, thread ID: ${thread.id}`);
39+
40+
// Upload an image file
41+
console.log("Uploading image file...");
42+
const fileStream = fs.createReadStream(imagePath);
43+
const imageFile = await client.agents.uploadFile(fileStream, "assistants", {
44+
fileName: "image_file.png",
45+
});
46+
console.log(`Uploaded file, file ID: ${imageFile.id}`);
47+
48+
// Create a message with both text and image content
49+
console.log("Creating message with image content...");
50+
const inputMessage = "Hello, what is in the image?";
51+
const content = [
52+
{
53+
type: "text",
54+
text: inputMessage,
55+
},
56+
{
57+
type: "image_file",
58+
image_file: {
59+
file_id: imageFile.id,
60+
detail: "high",
61+
},
62+
},
63+
];
64+
const message = await client.agents.createMessage(thread.id, {
65+
role: "user",
66+
content: content,
67+
});
68+
console.log(`Created message, message ID: ${message.id}`);
69+
70+
// Create and poll a run
71+
console.log("Creating run...");
72+
let run = await client.agents.createRun(thread.id, agent.id);
73+
74+
// Poll the run as long as run status is queued or in progress
75+
while (
76+
run.status === "queued" ||
77+
run.status === "in_progress" ||
78+
run.status === "requires_action"
79+
) {
80+
// Wait for a second
81+
console.log(`Run status: ${run.status}, waiting...`);
82+
await new Promise((resolve) => setTimeout(resolve, 1000));
83+
run = await client.agents.getRun(thread.id, run.id);
84+
}
85+
console.log(`Run complete with status: ${run.status}`);
86+
87+
// Delete the agent
88+
await client.agents.deleteAgent(agent.id);
89+
console.log(`Deleted agent, agent ID: ${agent.id}`);
90+
91+
// List messages
92+
const messages = await client.agents.listMessages(thread.id, {
93+
order: "asc",
94+
});
95+
96+
for (const dataPoint of messages.data) {
97+
const textContent = dataPoint.content.find((item) => item.type === "text");
98+
if (textContent && "text" in textContent) {
99+
console.log(`${dataPoint.role}: ${textContent.text.value}`);
100+
}
101+
}
102+
103+
console.log("Messages: ", messages);
104+
}
105+
106+
main().catch((error) => {
107+
console.error("An error occurred:", error);
108+
});
109+
110+
module.exports = { main };

‎sdk/ai/ai-projects/samples/v1-beta/javascript/agents/agentsImageInputWithUrl.js‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
/**
5+
* @summary This sample demonstrates how to use basic agent operations using image url input for the
6+
* Azure Agents service.
7+
*/
8+
9+
const { AIProjectsClient } = require("@azure/ai-projects");
10+
const { DefaultAzureCredential } = require("@azure/identity");
11+
12+
// Load environment variables
13+
const connectionString = process.env.AZURE_AI_PROJECTS_CONNECTION_STRING || "<connection-string>";
14+
const modelDeployment = process.env.MODEL_DEPLOYMENT_NAME || "<model-deployment-name>";
15+
const imageUrl =
16+
"https://github.com/Azure/azure-sdk-for-js/blob/0aa88ceb18d865726d423f73b8393134e783aea6/sdk/ai/ai-projects/data/image_file.png?raw=true";
17+
18+
async function main() {
19+
console.log("== AI Projects Agent with Image Input Sample ==");
20+
21+
// Create the client
22+
const client = AIProjectsClient.fromConnectionString(
23+
connectionString,
24+
new DefaultAzureCredential(),
25+
);
26+
27+
// Create an agent
28+
console.log(`Creating agent with model ${modelDeployment}...`);
29+
const agent = await client.agents.createAgent(modelDeployment, {
30+
name: "my-agent",
31+
instructions: "You are helpful agent",
32+
});
33+
console.log(`Created agent, agent ID: ${agent.id}`);
34+
35+
// Create a thread
36+
console.log("Creating thread...");
37+
const thread = await client.agents.createThread();
38+
console.log(`Created thread, thread ID: ${thread.id}`);
39+
40+
// Create a message with both text and image content
41+
console.log("Creating message with image content...");
42+
const inputMessage = "Hello, what is in the image?";
43+
const content = [
44+
{
45+
type: "text",
46+
text: inputMessage,
47+
},
48+
{
49+
type: "image_url",
50+
image_url: {
51+
url: imageUrl,
52+
detail: "high",
53+
},
54+
},
55+
];
56+
const message = await client.agents.createMessage(thread.id, {
57+
role: "user",
58+
content: content,
59+
});
60+
console.log(`Created message, message ID: ${message.id}`);
61+
62+
// Create and poll a run
63+
console.log("Creating run...");
64+
let run = await client.agents.createRun(thread.id, agent.id);
65+
66+
// Poll the run as long as run status is queued or in progress
67+
while (
68+
run.status === "queued" ||
69+
run.status === "in_progress" ||
70+
run.status === "requires_action"
71+
) {
72+
// Wait for a second
73+
console.log(`Run status: ${run.status}, waiting...`);
74+
await new Promise((resolve) => setTimeout(resolve, 1000));
75+
run = await client.agents.getRun(thread.id, run.id);
76+
}
77+
console.log(`Run complete with status: ${run.status}`);
78+
79+
// Delete the agent
80+
await client.agents.deleteAgent(agent.id);
81+
console.log(`Deleted agent, agent ID: ${agent.id}`);
82+
83+
// List messages
84+
const messages = await client.agents.listMessages(thread.id, {
85+
order: "asc",
86+
});
87+
88+
for (const dataPoint of messages.data) {
89+
const textContent = dataPoint.content.find((item) => item.type === "text");
90+
if (textContent && "text" in textContent) {
91+
console.log(`${dataPoint.role}: ${textContent.text.value}`);
92+
}
93+
}
94+
95+
console.log("Messages: ", messages);
96+
}
97+
98+
main().catch((error) => {
99+
console.error("An error occurred:", error);
100+
});
101+
102+
module.exports = { main };

0 commit comments

Comments
 (0)