Skip to content

[FDC] Generate schema during init dataconnect flow #8596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 46 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
d2f9774
Generate schema in fdc init
fredzqm May 15, 2025
f8cdc0f
tested
fredzqm May 15, 2025
ae7e61e
Merge remote-tracking branch 'origin/master' into fz/gen-schema-init
fredzqm May 15, 2025
9239426
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
6208153
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
091706d
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
909ef53
Merge remote-tracking branch 'origin/master' into fz/gen-schema-init
fredzqm May 15, 2025
c3c08c9
tos
fredzqm May 15, 2025
e9eec2a
tweaks
fredzqm May 15, 2025
68ea00f
m
fredzqm May 15, 2025
ac584ce
m
fredzqm May 15, 2025
7290205
vaib code a test
fredzqm May 15, 2025
1f84dc3
CHANGELOG
fredzqm May 15, 2025
bc2647e
m
fredzqm May 15, 2025
1beed8d
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
74d8710
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
ae6868d
m
fredzqm May 15, 2025
a59900f
Merge remote-tracking branch 'origin/master' into fz/gen-schema-init
fredzqm May 15, 2025
adfa8b8
Update src/init/features/dataconnect/index.ts
fredzqm May 15, 2025
1923130
Update src/init/features/dataconnect/index.ts
fredzqm May 15, 2025
d41bd43
format
fredzqm May 15, 2025
220a79d
m
fredzqm May 15, 2025
9bead7f
clean
fredzqm May 15, 2025
ee4c8c0
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
345d54a
Update ensureApis.ts
fredzqm May 15, 2025
5a82f27
Merge branch 'master' into fz/gen-schema-init
fredzqm May 15, 2025
eefd4ce
Merge branch 'master' into fz/gen-schema-init
fredzqm May 16, 2025
8f452f5
Merge branch 'master' into fz/gen-schema-init
fredzqm May 16, 2025
46967ed
Show GiF ToS in firebase login
fredzqm May 22, 2025
d5e0621
Merge branch 'master' into fz/show-gif-tos
fredzqm May 22, 2025
2fa36c0
tweaks
fredzqm May 22, 2025
d1d1100
Merge remote-tracking branch 'origin/master' into fz/show-gif-tos
fredzqm May 22, 2025
b0f3bc5
Merge remote-tracking branch 'origin/master' into fz/show-gif-tos
fredzqm May 22, 2025
1ef575f
Merge branch 'master' into fz/show-gif-tos
fredzqm May 23, 2025
3db2767
Merge remote-tracking branch 'origin/master' into fz/show-gif-tos
fredzqm May 23, 2025
dd092dd
tweaks
fredzqm May 24, 2025
a1f20db
Merge remote-tracking branch 'origin/fz/show-gif-tos' into fz/show-gi…
fredzqm May 24, 2025
6b7a8da
Merge branch 'master' into fz/show-gif-tos
fredzqm May 27, 2025
b19986d
Merge remote-tracking branch 'origin/master' into fz/show-gif-tos
fredzqm May 29, 2025
129c382
Merge branch 'master' into fz/show-gif-tos
fredzqm May 29, 2025
f884056
Merge remote-tracking branch 'origin/fz/show-gif-tos' into fz/show-gi…
fredzqm May 29, 2025
55d87d0
Merge branch 'fz/show-gif-tos' into fz/gen-schema-init
fredzqm May 29, 2025
2e09e7e
merge
fredzqm Jun 6, 2025
cd14029
m
fredzqm Jun 6, 2025
04b1c42
merge
fredzqm Jun 6, 2025
2403de2
adjust question
fredzqm Jun 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- Fixed multi tenancy support for SSO users in the auth emulator (#8544)
- Add SDK autoinit capabilities to App Hosting emulator. (#8582)
- Provision App Hosting compute service account during init flow. (#8580)
- `firebase init dataconnect` is now integrated with Gemini in Firebase API to generate Schema based on description. (#8596)
- Updated the Firebase Data Connect local toolkit to v2.6.1, which includes the following changes: (#8598)
- Fixed a bug where `@transaction` continued to execute after errors and produces more confusing errors.
- Fixed a bug where rolled-back fields in `@transaction` returned bogus data that had been invalidated by the rollback. They now correctly return null with a corresponding error "(rolled back)".
Expand Down
2 changes: 0 additions & 2 deletions src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,32 @@

let commandScopes = new Set<string>();

export const authProxyOrigin = () =>

Check warning on line 8 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("FIREBASE_AUTHPROXY_URL", "https://auth.firebase.tools");
// "In this context, the client secret is obviously not treated as a secret"
// https://developers.google.com/identity/protocols/OAuth2InstalledApp
export const clientId = () =>

Check warning on line 12 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride(
"FIREBASE_CLIENT_ID",
"563584335869-fgrhgmd47bqnekij5i8b5pr03ho849e6.apps.googleusercontent.com",
);
export const clientSecret = () =>

Check warning on line 17 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("FIREBASE_CLIENT_SECRET", "j9iVZfS8kkCEFUPaAeJV0sAi");
export const cloudbillingOrigin = () =>

Check warning on line 19 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("FIREBASE_CLOUDBILLING_URL", "https://cloudbilling.googleapis.com");
export const cloudloggingOrigin = () =>

Check warning on line 21 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("FIREBASE_CLOUDLOGGING_URL", "https://logging.googleapis.com");
export const cloudMonitoringOrigin = () =>

Check warning on line 23 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("CLOUD_MONITORING_URL", "https://monitoring.googleapis.com");
export const containerRegistryDomain = () =>

Check warning on line 25 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("CONTAINER_REGISTRY_DOMAIN", "gcr.io");

export const developerConnectOrigin = () =>

Check warning on line 28 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("DEVELOPERCONNECT_URL", "https://developerconnect.googleapis.com");
export const developerConnectP4SADomain = () =>

Check warning on line 30 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("DEVELOPERCONNECT_P4SA_DOMAIN", "gcp-sa-devconnect.iam.gserviceaccount.com");

export const artifactRegistryDomain = () =>

Check warning on line 33 in src/api.ts

View workflow job for this annotation

GitHub Actions / lint (20)

Missing return type on function
utils.envOverride("ARTIFACT_REGISTRY_DOMAIN", "https://artifactregistry.googleapis.com");
export const appDistributionOrigin = () =>
utils.envOverride(
Expand Down Expand Up @@ -91,8 +91,6 @@

export const cloudbuildOrigin = () =>
utils.envOverride("FIREBASE_CLOUDBUILD_URL", "https://cloudbuild.googleapis.com");
export const cloudCompanionOrigin = () =>
utils.envOverride("CLOUD_COMPANION_URL", "https://cloudaicompanion.googleapis.com");
export const cloudschedulerOrigin = () =>
utils.envOverride("FIREBASE_CLOUDSCHEDULER_URL", "https://cloudscheduler.googleapis.com");
export const cloudTasksOrigin = () =>
Expand Down
8 changes: 6 additions & 2 deletions src/dataconnect/ensureApis.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,19 @@
import * as api from "../api";
import { ensure } from "../ensureApiEnabled";

const prefix = "dataconnect";

export async function ensureApis(projectId: string): Promise<void> {
const prefix = "dataconnect";
await ensure(projectId, api.dataconnectOrigin(), prefix);
await ensure(projectId, api.cloudSQLAdminOrigin(), prefix);
await ensure(projectId, api.computeOrigin(), prefix);
}

export async function ensureSparkApis(projectId: string): Promise<void> {
const prefix = "dataconnect";
// These are the APIs that can be enabled without a billing account.
await ensure(projectId, api.cloudSQLAdminOrigin(), prefix);
}

export async function ensureGIFApis(projectId: string): Promise<void> {
await ensure(projectId, api.cloudAiCompanionOrigin(), prefix);
}
67 changes: 67 additions & 0 deletions src/gif/fdcExperience.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import { expect } from "chai";
import { extractCodeBlock } from "./fdcExperience";

describe("extractCodeBlock", () => {
it("should extract a basic GraphQL query block", () => {
const text =
'Here is a GraphQL query:\n```graphql\nquery GetUser { user(id: "1") { name email } }\n```\nThanks!';
const expected = 'query GetUser { user(id: "1") { name email } }';
expect(extractCodeBlock(text)).to.eq(expected);
});

it("should extract a multi-line GraphQL mutation block", () => {
const text = `
Some preamble.
\`\`\`graphql
mutation CreatePost($title: String!, $content: String!) {
createPost(title: $title, content: $content) {
id
title
}
}
\`\`\`
Followed by some description.
`;
const expected = `mutation CreatePost($title: String!, $content: String!) {
createPost(title: $title, content: $content) {
id
title
}
}`;
expect(extractCodeBlock(text)).to.eq(expected);
});

it("should extract a GraphQL fragment block", () => {
const text = "```graphql\nfragment UserFields on User { id name }\n```";
const expected = "fragment UserFields on User { id name }";
expect(extractCodeBlock(text)).to.eq(expected);
});

it("should extract an empty GraphQL code block", () => {
const text = "```graphql\n\n```";
const expected = "";
expect(extractCodeBlock(text)).to.eq(expected);
});

it("should extract a GraphQL schema definition block", () => {
const text = `
\`\`\`graphql
type Query {
hello: String
}

schema {
query: Query
}
\`\`\`
`;
const expected = `type Query {
hello: String
}

schema {
query: Query
}`;
expect(extractCodeBlock(text)).to.eq(expected);
});
});
19 changes: 17 additions & 2 deletions src/gif/fdcExperience.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { Client } from "../apiv2";
import { cloudCompanionOrigin } from "../api";
import { cloudAiCompanionOrigin } from "../api";
import { FirebaseError } from "../error";

const apiClient = new Client({ urlPrefix: cloudCompanionOrigin(), auth: true });
const apiClient = new Client({ urlPrefix: cloudAiCompanionOrigin(), auth: true });
const schemaGeneratorExperience = "/appeco/firebase/fdc-schema-generator";
const geminiInFirebaseChatExperience = "/appeco/firebase/firebase-chat/free";
const operationGeneratorExperience = "/appeco/firebase/fdc-query-generator";
Expand Down Expand Up @@ -128,3 +129,17 @@ export async function generateOperation(
);
return res.body.output.messages[0].content;
}

/**
* extractCodeBlock extracts the code block from the generated response.
* @param text the generated response from the service.
* @return the code block from the generated response.
*/
export function extractCodeBlock(text: string): string {
const regex = /```(?:[a-z]+\n)?([\s\S]*?)```/m;
const match = text.match(regex);
if (match && match[1]) {
return match[1].trim();
}
throw new FirebaseError(`No code block found in the generated response: ${text}`);
}
32 changes: 27 additions & 5 deletions src/init/features/dataconnect/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { Setup } from "../..";
import { provisionCloudSql } from "../../../dataconnect/provisionCloudSql";
import { checkFreeTrialInstanceUsed, upgradeInstructions } from "../../../dataconnect/freeTrial";
import * as cloudsql from "../../../gcp/cloudsql/cloudsqladmin";
import { ensureApis, ensureSparkApis } from "../../../dataconnect/ensureApis";
import { ensureApis, ensureGIFApis, ensureSparkApis } from "../../../dataconnect/ensureApis";
import {
listLocations,
listAllServices,
Expand All @@ -19,10 +19,11 @@ import { Schema, Service, File, Platform } from "../../../dataconnect/types";
import { parseCloudSQLInstanceName, parseServiceName } from "../../../dataconnect/names";
import { logger } from "../../../logger";
import { readTemplateSync } from "../../../templates";
import { logBullet, envOverride } from "../../../utils";
import { logBullet, envOverride, promiseWithSpinner } from "../../../utils";
import { isBillingEnabled } from "../../../gcp/cloudbilling";
import * as sdk from "./sdk";
import { getPlatformFromFolder } from "../../../dataconnect/fileUtils";
import { extractCodeBlock, generateSchema } from "../../../gif/fdcExperience";

const DATACONNECT_YAML_TEMPLATE = readTemplateSync("init/dataconnect/dataconnect.yaml");
const CONNECTOR_YAML_TEMPLATE = readTemplateSync("init/dataconnect/connector.yaml");
Expand Down Expand Up @@ -110,8 +111,7 @@ export async function askQuestions(setup: Setup): Promise<void> {
default: true,
}));
if (shouldConfigureBackend) {
// TODO: Prompt for app idea and use GiF backend to generate them.
info = await promptForService(info);
info = await promptForSchema(setup, info);
info = await promptForCloudSQL(setup, info);

info.shouldProvisionCSQL = !!(
Expand Down Expand Up @@ -425,12 +425,34 @@ async function promptForCloudSQL(setup: Setup, info: RequiredInfo): Promise<Requ
return info;
}

async function promptForService(info: RequiredInfo): Promise<RequiredInfo> {
async function promptForSchema(setup: Setup, info: RequiredInfo): Promise<RequiredInfo> {
if (info.serviceId === "") {
info.serviceId = await input({
message: "What ID would you like to use for this service?",
default: basename(process.cwd()),
});
if (
setup.projectId &&
(await confirm({
message: `Do you want Gemini to help generate a schema for your service?`,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Do you want Gemini in Firebase to help generate a schema for your service"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done~

default: false,
}))
) {
logBullet(
`Check out the terms of service of Gemini in Firebase https://firebase.google.com/docs/gemini-in-firebase/set-up-gemini#required-permissions`,
);
await ensureGIFApis(setup.projectId);
const prompt = await input({
message: "Describe the app you are building:",
default: "movie rating app",
});
const schema = await promiseWithSpinner(
() => generateSchema(prompt, setup.projectId!),
"Generating the Data Connect Schema...",
);
info.schemaGql = [{ path: "schema.gql", content: extractCodeBlock(schema) }];
info.connectors = [emptyConnector];
}
}
return info;
}
Expand Down
Loading