Skip to content

Commit 9879e02

Browse files
prestoncabeprestoncabe
authored
Merge pull request #260 from CodeForPhilly/147-refactor-library-metadata-sync
library-api: refactor metadata sync to work in dev
2 parents 75c184f + 413da99 commit 9879e02

File tree

11 files changed

+310
-16
lines changed

11 files changed

+310
-16
lines changed

‎.env.example‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,8 @@ QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE="http://127.0.0.1:9199"
44
GCS_BUCKET_NAME="demo-bdt-dev.appspot.com"
55
FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
66

7+
# Library API Configuration
8+
# Defaults to http://localhost:8083 (no config needed for dev)
9+
# For production sync, set:
10+
# LIBRARY_API_BASE_URL=https://library-api-1034049717668.us-central1.run.app
11+

‎.github/workflows/load-library-metadata.yml‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,17 @@ jobs:
1616
with:
1717
python-version: "3.11"
1818

19-
- name: Install dependencies
20-
working-directory: scripts
21-
run: pip install -r requirements.txt
22-
2319
- name: Create GCP credentials file
2420
run: |
25-
echo '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}' > scripts/gcp-key.json
21+
echo '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}' > bin/library/gcp-key.json
2622
27-
- name: Run script
28-
working-directory: scripts
23+
- name: Run sync
24+
working-directory: bin/library
2925
env:
3026
GOOGLE_APPLICATION_CREDENTIALS: gcp-key.json
27+
LIBRARY_API_BASE_URL: https://library-api-1034049717668.us-central1.run.app
3128
run: |
32-
python load-library-metadata.py
29+
./sync-metadata
3330
3431
- name: Cleanup credentials
35-
run: rm scripts/gcp-key.json
32+
run: rm bin/library/gcp-key.json

‎CLAUDE.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,57 @@ Admin → builder-frontend → builder-api → Firebase (Firestore + Storage)
167167
- Learn DMN basics: https://learn-dmn-in-15-minutes.com/
168168
- Access raw XML: Right-click → "Reopen with Text Editor"
169169

170+
### Library Check Metadata Sync
171+
172+
The builder-api needs metadata about available library checks (from library-api). This metadata is stored in Firebase Storage and referenced from Firestore.
173+
174+
**Automatic Sync (Development)**:
175+
- Runs automatically when you start services via `devbox services up`
176+
- Syncs from local library-api (http://localhost:8083) to Firebase emulators
177+
- Happens after library-api starts, before builder-api starts
178+
- Library checks will then be visible in the builder UI!
179+
180+
**Manual Sync (Development)**:
181+
```bash
182+
# Re-sync after making library-api changes
183+
./scripts/sync-library-metadata.sh
184+
185+
# Then restart builder-api to pick up new metadata
186+
# (In process-compose UI, restart the builder-api process)
187+
```
188+
189+
**Production Sync** (maintainers only):
190+
```bash
191+
# Set production library-api URL and unset emulator variables
192+
export LIBRARY_API_BASE_URL=https://library-api-1034049717668.us-central1.run.app
193+
unset FIRESTORE_EMULATOR_HOST
194+
unset GCS_BUCKET_NAME
195+
unset QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE
196+
197+
# Authenticate with GCP
198+
gcloud auth application-default login
199+
200+
# Run sync
201+
./scripts/sync-library-metadata.sh
202+
```
203+
204+
**How It Works**:
205+
1. Fetches OpenAPI spec from library-api
206+
2. Extracts check metadata (inputs, outputs, versions)
207+
3. Uploads JSON to Firebase Storage
208+
4. Updates Firestore `system/config` with storage path
209+
5. builder-api reads this metadata on startup
210+
211+
**Environment Configuration**:
212+
- **Default**: `http://localhost:8083` (development mode - no config needed)
213+
- **Production**: Set `LIBRARY_API_BASE_URL` to production Cloud Run URL
214+
- Environment is inferred from URL pattern (localhost = dev, else = prod)
215+
216+
**Troubleshooting**:
217+
- **"Firebase Storage emulator not responding"**: Start emulators first (`firebase emulators:start`)
218+
- **"library-api not responding"**: Start library-api (`cd library-api && quarkus dev`)
219+
- **Stale metadata in builder-api**: Restart builder-api (reads metadata on startup)
220+
170221
### Firebase Emulators
171222

172223
The project uses Firebase emulators for local development:

scripts/load-library-metadata.py renamed to bin/library/load-library-metadata.py

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,51 @@
44
from firebase_admin import credentials, storage, firestore
55
import json
66
from datetime import datetime
7+
import os
8+
9+
# -----------------------------------
10+
# CONFIGURATION
11+
# -----------------------------------
12+
13+
# Default to localhost for developer-friendly setup
14+
DEFAULT_LIBRARY_API_URL = "http://localhost:8083"
15+
LIBRARY_API_BASE_URL = os.getenv("LIBRARY_API_BASE_URL", DEFAULT_LIBRARY_API_URL)
16+
17+
# Infer production mode from URL (for versioned URL logic)
18+
IS_PRODUCTION = not ("localhost" in LIBRARY_API_BASE_URL or "127.0.0.1" in LIBRARY_API_BASE_URL)
19+
20+
# Storage bucket defaults - use dev bucket by default
21+
DEFAULT_DEV_BUCKET = "demo-bdt-dev.appspot.com"
22+
DEFAULT_PROD_BUCKET = "benefit-decision-toolkit-play.firebasestorage.app"
23+
STORAGE_BUCKET = os.getenv("GCS_BUCKET_NAME",
24+
DEFAULT_PROD_BUCKET if IS_PRODUCTION else DEFAULT_DEV_BUCKET)
25+
26+
# Log configuration
27+
print(f"========================================")
28+
print(f"Library Metadata Sync Configuration")
29+
print(f"========================================")
30+
print(f"Mode: {'production' if IS_PRODUCTION else 'development'}")
31+
print(f"Library API URL: {LIBRARY_API_BASE_URL}")
32+
print(f"Storage Bucket: {STORAGE_BUCKET}")
33+
print(f"========================================\n")
734

835
# -----------------------------------
936
# INIT FIREBASE
1037
# -----------------------------------
1138

39+
# Point google-cloud-storage SDK at the emulator using the existing Quarkus config variable
40+
storage_host_override = os.getenv("QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE")
41+
if storage_host_override:
42+
os.environ["STORAGE_EMULATOR_HOST"] = storage_host_override
43+
1244
cred = credentials.ApplicationDefault()
1345

14-
firebase_admin.initialize_app(cred, {
15-
"storageBucket": "benefit-decision-toolkit-play.firebasestorage.app"
16-
})
46+
firebase_options = {"storageBucket": STORAGE_BUCKET}
47+
if not IS_PRODUCTION:
48+
# Emulators need an explicit project ID; production gets it from credentials
49+
firebase_options["projectId"] = os.getenv("QUARKUS_GOOGLE_CLOUD_PROJECT_ID", "demo-bdt-dev")
50+
51+
firebase_admin.initialize_app(cred, firebase_options)
1752

1853
db = firestore.client()
1954
bucket = storage.bucket()
@@ -292,7 +327,9 @@ def save_json_to_storage_and_update_firestore(json_string, firestore_doc_path):
292327
# --------------------------------------------
293328
if __name__ == "__main__":
294329

295-
url = "https://library-api-1034049717668.us-central1.run.app/q/openapi.json"
330+
url = f"{LIBRARY_API_BASE_URL}/q/openapi.json"
331+
332+
print(f"Fetching OpenAPI spec from: {url}")
296333

297334
# Send a GET request
298335
response = requests.get(url)

scripts/requirements.txt renamed to bin/library/requirements.txt

File renamed without changes.

‎bin/library/sync-metadata‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
6+
7+
# Load environment from root .env if it exists
8+
if [ -f "$PROJECT_ROOT/.env" ]; then
9+
set -a
10+
source "$PROJECT_ROOT/.env"
11+
set +a
12+
fi
13+
14+
# Determine library API URL (defaults to localhost)
15+
LIBRARY_API_URL="${LIBRARY_API_BASE_URL:-http://localhost:8083}"
16+
17+
# Infer mode from URL
18+
if [[ "$LIBRARY_API_URL" == *"localhost"* ]] || [[ "$LIBRARY_API_URL" == *"127.0.0.1"* ]]; then
19+
MODE="development"
20+
else
21+
MODE="production"
22+
fi
23+
24+
echo "=========================================="
25+
echo "Library Metadata Sync"
26+
echo "=========================================="
27+
echo "Mode: $MODE"
28+
echo "Library API: $LIBRARY_API_URL"
29+
30+
if [ "$MODE" = "development" ]; then
31+
echo "Target: Firebase Emulators (Firestore + Storage)"
32+
echo ""
33+
echo "Prerequisites:"
34+
echo " 1. Firebase emulators must be running"
35+
echo " 2. library-api must be running (quarkus dev)"
36+
echo ""
37+
38+
# Check if Firebase Storage emulator is running
39+
if ! curl -s http://localhost:9199 >/dev/null 2>&1; then
40+
echo "ERROR: Firebase Storage emulator not responding at localhost:9199"
41+
echo "Start emulators with: firebase emulators:start --project demo-bdt-dev --only auth,storage,firestore"
42+
exit 1
43+
fi
44+
45+
# Check if library-api is running
46+
if ! curl -s "${LIBRARY_API_URL}/q/health" >/dev/null 2>&1; then
47+
echo "ERROR: library-api not responding at ${LIBRARY_API_URL}"
48+
echo "Start library-api with: cd library-api && quarkus dev"
49+
exit 1
50+
fi
51+
52+
# Check if $VENV_DIR is set and activate it
53+
if [[ "$VENV_DIR" != "" ]]; then
54+
if [ -f "$VENV_DIR/bin/activate" ]; then
55+
source "$VENV_DIR/bin/activate"
56+
echo "✓ Activated virtual environment at $VENV_DIR"
57+
else
58+
echo "WARNING: Virtual environment not found at $VENV_DIR"
59+
fi
60+
fi
61+
62+
echo "✓ Firebase emulators are running"
63+
echo "✓ library-api is running"
64+
echo ""
65+
else
66+
echo "Target: Production Firebase"
67+
echo ""
68+
echo "Prerequisites:"
69+
echo " - Valid Google Cloud credentials (Application Default Credentials)"
70+
echo " - Deployed library-api at: $LIBRARY_API_URL"
71+
echo ""
72+
73+
# Check if we have GCP credentials
74+
if ! gcloud auth application-default print-access-token >/dev/null 2>&1; then
75+
echo "ERROR: No valid Google Cloud credentials found"
76+
echo "Authenticate with: gcloud auth application-default login --project benefit-decision-toolkit-play"
77+
exit 1
78+
fi
79+
80+
echo "✓ Google Cloud credentials found"
81+
echo ""
82+
fi
83+
84+
# Run the Python script
85+
echo "Running metadata sync..."
86+
pip install -q -r $SCRIPT_DIR/requirements.txt
87+
python3 "$SCRIPT_DIR/load-library-metadata.py"
88+
89+
echo ""
90+
echo "=========================================="
91+
echo "Metadata sync complete!"
92+
echo "=========================================="

‎builder-api/src/main/java/org/acme/service/LibraryApiService.java‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
import org.acme.model.domain.EligibilityCheck;
1313
import org.acme.persistence.StorageService;
1414
import org.acme.persistence.FirestoreUtils;
15+
import org.eclipse.microprofile.config.inject.ConfigProperty;
1516

1617
import java.net.URI;
1718
import java.net.http.HttpClient;
@@ -25,14 +26,36 @@
2526

2627
@ApplicationScoped
2728
public class LibraryApiService {
29+
private static final String DEFAULT_LIBRARY_API_URL = "http://localhost:8083";
30+
2831
@Inject
2932
private StorageService storageService;
3033

34+
@ConfigProperty(name = "library-api.base-url")
35+
Optional<String> libraryApiBaseUrl;
36+
3137
private List<EligibilityCheck> checks;
38+
private String effectiveBaseUrl;
39+
private boolean useVersionedUrls;
3240

3341
@PostConstruct
3442
void init() {
3543
try {
44+
// Determine effective base URL
45+
effectiveBaseUrl = libraryApiBaseUrl.orElse(DEFAULT_LIBRARY_API_URL);
46+
47+
// Infer environment from URL - localhost = development, else = production
48+
boolean isProduction = !(effectiveBaseUrl.contains("localhost") || effectiveBaseUrl.contains("127.0.0.1"));
49+
useVersionedUrls = isProduction;
50+
51+
Log.info("========================================");
52+
Log.info("Library API Configuration");
53+
Log.info("========================================");
54+
Log.info("Base URL: " + effectiveBaseUrl);
55+
Log.info("Mode: " + (isProduction ? "production" : "development"));
56+
Log.info("Versioned URLs: " + (useVersionedUrls ? "enabled" : "disabled"));
57+
Log.info("========================================");
58+
3659
// Get path of most recent library schema json document
3760
Optional<Map<String, Object>> configOpt = FirestoreUtils.getFirestoreDocById("system", "config");
3861
if (configOpt.isEmpty()){
@@ -51,6 +74,7 @@ void init() {
5174
ObjectMapper mapper = new ObjectMapper();
5275

5376
checks = mapper.readValue(apiSchemaJson, new TypeReference<List<EligibilityCheck>>() {});
77+
Log.info("Loaded " + checks.size() + " library checks");
5478
} catch (Exception e) {
5579
throw new RuntimeException("Failed to load library api metadata", e);
5680
}
@@ -87,8 +111,20 @@ public EvaluationResult evaluateCheck(CheckConfig checkConfig, Map<String, Objec
87111
String bodyJson = mapper.writeValueAsString(data);
88112

89113
HttpClient client = HttpClient.newHttpClient();
90-
String urlEncodedVersion = checkConfig.getCheckVersion().replace('.', '-');
91-
String baseUrl = String.format("https://library-api-v%s---library-api-cnsoqyluna-uc.a.run.app", urlEncodedVersion);
114+
115+
// Determine base URL based on configuration
116+
String baseUrl;
117+
if (useVersionedUrls) {
118+
// Production: Use versioned Cloud Run URLs
119+
String urlEncodedVersion = checkConfig.getCheckVersion().replace('.', '-');
120+
baseUrl = String.format("https://library-api-v%s---library-api-cnsoqyluna-uc.a.run.app", urlEncodedVersion);
121+
Log.debug("Using versioned URL: " + baseUrl);
122+
} else {
123+
// Development: Use configured base URL directly
124+
baseUrl = effectiveBaseUrl;
125+
Log.debug("Using base URL: " + baseUrl);
126+
}
127+
92128
HttpRequest request = HttpRequest.newBuilder()
93129
.uri(URI.create(baseUrl + checkConfig.getEvaluationUrl()))
94130
.header("Content-Type", "application/json")

‎builder-api/src/main/resources/application.properties‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,8 @@ quarkus.http.auth.permission.secured.policy=authenticated
1414

1515
quarkus.http.auth.permission.public.paths=/api/published/*
1616
quarkus.http.auth.permission.public.policy=permit
17+
18+
# Library API Configuration
19+
# Defaults to localhost:8083 (development mode)
20+
# For production, set LIBRARY_API_BASE_URL to production Cloud Run URL
21+
library-api.base-url=${LIBRARY_API_BASE_URL:http://localhost:8083}

‎devbox.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88
"google-cloud-sdk@latest",
99
"nodejs@22",
1010
"bruno-cli@latest",
11-
"process-compose@latest"
11+
"process-compose@latest",
12+
"python@latest"
1213
],
1314
"env_from": ".env",
1415
"shell": {

0 commit comments

Comments
 (0)