Skip to content

PR request for ef13f1ba62b95d30bcc13300292a2848f778603f #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
25dff76
remove unused imports: json
sviola-aws Mar 6, 2024
01b8ff4
moving import of uuid to imports at the beginning of file
sviola-aws Mar 6, 2024
fc4760c
add spacing between functions
sviola-aws Mar 6, 2024
1f5b237
move import time to other imports
sviola-aws Mar 6, 2024
a4a6d94
add stable uuid
wolanlu Mar 6, 2024
068e9c5
make question asking more generic
wolanlu Mar 6, 2024
3f63d22
add logging and replace prints
sviola-aws Mar 6, 2024
39a9889
remove duplicate of import os
sviola-aws Mar 6, 2024
a6d1af0
spacing for vars corrected
sviola-aws Mar 6, 2024
db87ae4
add saving to S3
sviola-aws Mar 7, 2024
df01477
rename variables repo_url and ssh_url from outer scope
sviola-aws Mar 7, 2024
5036f75
pass s3_bucket env var from Lambda to AWS Batch job
sviola-aws Mar 7, 2024
0da9886
fixing documentation filepath pattern
sviola-aws Mar 7, 2024
dbd74c6
upload answers to s3
wolanlu Mar 7, 2024
c65f394
externalize prompt config to parameter store
wolanlu Mar 8, 2024
cc4e57c
Merge remote-tracking branch 'wiola/main' into feature/external-promp…
wolanlu Mar 8, 2024
b87c373
use file_path to have exact structure on s3 like in repo
wolanlu Mar 8, 2024
fee9003
Merge pull request #1 from WStobieniecka/feature/external-prompt-config
WStobieniecka Mar 11, 2024
adf5dd2
add RestApiConstruct
wolanlu Mar 13, 2024
554e9ba
basic pushEvent validation
wolanlu Mar 13, 2024
b07a016
extract attributes from custom resource
wolanlu Mar 13, 2024
a85017a
pass qAppId and qAppIndexId to rest api lambda
wolanlu Mar 13, 2024
e378744
trigger batch job on code push
wolanlu Mar 13, 2024
f8aceb3
adding KeyCloak server
wolanlu Mar 18, 2024
8d2958b
script for adding webhook
wolanlu Mar 18, 2024
d61085c
added commit processing
sviola-aws Mar 18, 2024
7639ce8
update env variables
sviola-aws Mar 18, 2024
ad55089
replace ssh with access token
sviola-aws Mar 18, 2024
1735e7a
add managed policy for ec2 to enable SSM management and always restar…
wolanlu Mar 19, 2024
6b103a9
Merge branch 'feature/api_gateway' into uc_1
wolanlu Mar 19, 2024
ee03351
Merge branch 'feature/create-webhook-script' into uc_1
wolanlu Mar 19, 2024
3527ec4
handle ping from github
wolanlu Mar 19, 2024
d430dfb
add prompt for commit diff doc geenration to Parameter Store & Amazon…
sviola-aws Mar 19, 2024
f656e25
merged from main
sviola-aws Mar 19, 2024
81eb7f4
relaced GitPython with PyGitHub package
sviola-aws Mar 19, 2024
0082c85
Merge pull request #1 from WStobieniecka/feature/commit_processing_wo…
wolanlu Mar 20, 2024
b9eac0a
add sdk call to create web experience (in the same way as using AWS C…
wolanlu Mar 20, 2024
f4cc09e
Merge pull request #2 from WStobieniecka/feature/create-web-experience
WStobieniecka Mar 20, 2024
303b925
fix constructs
sviola-aws Mar 20, 2024
389336a
fix missing access_token_name arg
sviola-aws Mar 21, 2024
32b2200
fixed arg name when passing to submit_job
sviola-aws Mar 21, 2024
ebfe676
fixed arg name in logger
sviola-aws Mar 21, 2024
7bab084
add extraction of branch name
sviola-aws Mar 21, 2024
ef13f1b
add access_token_name param to restAPI
wolanlu Mar 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions assets/create-webhook.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
############################################################
# Help #
############################################################
Help()
{
# Display Help
echo "Add webhook to repo."
echo
echo "Syntax: create-webhook.sh [-h|-o org -r repo -u url -t token]"
echo "options:"
echo "-o Enter the org name."
echo "-h Print this Help."
echo "-r Enter the repo name"
echo "-t Enter the repo auth token e.g. \$(gh auth token)"
echo "-u Enter the webhook url"
echo
}

Webhook()
{
curl "https://api.github.com/repos/$org/$repo/hooks" \
-H "Authorization: Token $token" \
-d @- << EOF
{
"name": "web",
"active": true,
"events": [
"push"
],
"config": {
"url": "$url",
"content_type": "json"
}
}
EOF
}

############################################################
############################################################
# Main program #
############################################################
############################################################

# Set variables
org=""
repo=""
token=""
url=""

############################################################
# Process the input options. Add options as needed. #
############################################################
# Get the options
while getopts ":h:o:r:t:u:" option; do
case $option in
h) # display Help
Help
exit;;
o) # Enter a name
org=$OPTARG;;
r) # Enter a name
repo=$OPTARG;;
t) # Enter a name
token=$OPTARG;;
u) # Enter a name
url=$OPTARG;;
\?) # Invalid option
echo "Error: Invalid option"
exit;;
esac
done

echo "Org: $org, Repo: $repo, Token: $token, Url: $url"
if [ -z $org ] || [ -z $repo ] || [ -z $token ] || [ -z $url ]; then
Help
else
Webhook
fi
148 changes: 81 additions & 67 deletions cdk/lib/assets/lambdas/amazon_q_app/amazon_q_app_resource.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,88 +3,90 @@
import time
import json
import datetime
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import botocore.session

amazon_q = boto3.client('qbusiness')


def on_event(event, context):
q_app_name = os.environ['Q_APP_NAME']
q_app_role_arn = os.environ['Q_APP_ROLE_ARN']
# physical_id = "PhysicalIdAmazonQCodeAnalysisApp"

print(json.dumps(event))
request_type = event['RequestType']
if request_type == 'Create': return on_create(event,
q_app_name=q_app_name,
q_app_role_arn=q_app_role_arn,
)
physical_id = event['PhysicalResourceId']
if request_type == 'Update': return on_update(event, physical_id=physical_id)
if request_type == 'Delete': return on_delete(event, physical_id=physical_id)
raise Exception("Invalid request type: %s" % request_type)
q_app_name = os.environ['Q_APP_NAME']
q_app_role_arn = os.environ['Q_APP_ROLE_ARN']
# physical_id = "PhysicalIdAmazonQCodeAnalysisApp"

print(json.dumps(event))
request_type = event['RequestType']
if request_type == 'Create': return on_create(event,
q_app_name=q_app_name,
q_app_role_arn=q_app_role_arn,
)
physical_id = event['PhysicalResourceId']
if request_type == 'Update': return on_update(event, physical_id=physical_id)
if request_type == 'Delete': return on_delete(event, physical_id=physical_id)
raise Exception("Invalid request type: %s" % request_type)


def on_create(event, q_app_name, q_app_role_arn):
props = event["ResourceProperties"]
print("create new resource with props %s" % props)

# Create Amazon Q App
amazon_q_app_id = create_q_app(q_app_name=q_app_name,
role_arn=q_app_role_arn)

# Create Q Index
amazon_q_index_id = create_q_index(q_app_name=q_app_name,
amazon_q_app_id=amazon_q_app_id)

# Create Q Retriever
amazon_q_retriever_id = create_q_retriever(
q_app_name=q_app_name,
amazon_q_app_id=amazon_q_app_id,
amazon_q_index_id=amazon_q_index_id,
q_app_role_arn=q_app_role_arn,
)

return {
'PhysicalResourceId': amazon_q_app_id,
'Data': {
'AmazonQAppId': amazon_q_app_id,
'AmazonQIndexId': amazon_q_index_id,
'AmazonQRetrieverId': amazon_q_retriever_id
}
}
props = event["ResourceProperties"]
print("create new resource with props %s" % props)

# Create Amazon Q App
amazon_q_app_id = create_q_app(q_app_name=q_app_name,
role_arn=q_app_role_arn)

# Create Q Index
amazon_q_index_id = create_q_index(q_app_name=q_app_name,
amazon_q_app_id=amazon_q_app_id)

# Create Q Retriever
amazon_q_retriever_id = create_q_retriever(
q_app_name=q_app_name,
amazon_q_app_id=amazon_q_app_id,
amazon_q_index_id=amazon_q_index_id,
q_app_role_arn=q_app_role_arn,
)

web_experience_id = create_q_web_experience(amazon_q_app_id=amazon_q_app_id, q_app_name=q_app_name)

return {
'PhysicalResourceId': amazon_q_app_id,
'Data': {
'AmazonQAppId': amazon_q_app_id,
'AmazonQIndexId': amazon_q_index_id,
'AmazonQRetrieverId': amazon_q_retriever_id,
'AmazonQWebExperienceId': web_experience_id
}
}


def on_update(event, physical_id):
# physical_id = event["PhysicalResourceId"]
props = event["ResourceProperties"]
print("update resource %s with props %s" % (physical_id, props))
# physical_id = event["PhysicalResourceId"]
props = event["ResourceProperties"]
print("update resource %s with props %s" % (physical_id, props))

return { 'PhysicalResourceId': physical_id }
return {'PhysicalResourceId': physical_id}


def on_delete(event, physical_id):
# physical_id = event["PhysicalResourceId"]
print("delete resource %s" % physical_id)
delete_amazon_q_app(q_app_id=physical_id)
# physical_id = event["PhysicalResourceId"]
print("delete resource %s" % physical_id)
delete_amazon_q_app(q_app_id=physical_id)

return { 'PhysicalResourceId': physical_id }
return {'PhysicalResourceId': physical_id}


def create_q_app(q_app_name, role_arn):
import datetime
response = amazon_q.create_application(
import datetime
response = amazon_q.create_application(
attachmentsConfiguration={
'attachmentsControlMode': 'ENABLED'
},
displayName=q_app_name,
description=f"{q_app_name} created with Cloudformation on {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
roleArn=role_arn,
)
amazon_q_app_id = response["applicationId"]
amazon_q_app_id = response["applicationId"]

return amazon_q_app_id

return amazon_q_app_id

def create_q_index(q_app_name, amazon_q_app_id):
response = amazon_q.create_index(
Expand All @@ -98,18 +100,19 @@ def create_q_index(q_app_name, amazon_q_app_id):
index_id = response["indexId"]

while True:
response = amazon_q.get_index(
applicationId=amazon_q_app_id,
indexId=index_id,
)
status = response.get('status')
print(f"Creat index status {status}")
if status == 'ACTIVE':
break
time.sleep(10)
response = amazon_q.get_index(
applicationId=amazon_q_app_id,
indexId=index_id,
)
status = response.get('status')
print(f"Creat index status {status}")
if status == 'ACTIVE':
break
time.sleep(10)

return index_id


def create_q_retriever(q_app_name, amazon_q_app_id, amazon_q_index_id, q_app_role_arn):
response = amazon_q.create_retriever(
applicationId=amazon_q_app_id,
Expand All @@ -126,9 +129,20 @@ def create_q_retriever(q_app_name, amazon_q_app_id, amazon_q_index_id, q_app_rol

return retriever_id


def delete_amazon_q_app(q_app_id):
response = amazon_q.delete_application(
response = amazon_q.delete_application(
applicationId=q_app_id
)

return response
return response


def create_q_web_experience(amazon_q_app_id, q_app_name):
response = amazon_q.create_web_experience(
applicationId=amazon_q_app_id,
title=q_app_name
)
web_experience_id = response["webExperienceId"]

return web_experience_id
66 changes: 38 additions & 28 deletions cdk/lib/assets/lambdas/batch_lambdas/submit_batch_job.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,39 +34,49 @@ def on_create(event, physical_id):
q_app_user_id = os.environ.get("Q_APP_USER_ID")
ssh_url = os.environ.get("SSH_URL")
ssh_key_name = os.environ.get("SSH_KEY_NAME")
prompt_config_param_name = os.environ.get("PROMPT_CONFIG_SSM_PARAM_NAME")
print("Getting AP id and index...")
q_app_id = get_q_app_id(q_app_name)
q_app_index = get_q_app_index(q_app_name, q_app_id)

container_overrides = {
"environment": [{
"name": "REPO_URL",
"value": repo_url
},
{
"name": "SSH_URL",
"value": ssh_url
},
{
"name": "SSH_KEY_NAME",
"value": ssh_key_name
},
{
"name": "AMAZON_Q_APP_ID",
"value": q_app_id
},
{
"name": "AMAZON_Q_USER_ID",
"value": q_app_user_id
},
{
"name": "Q_APP_INDEX",
"value": q_app_index
},
{
"name": "Q_APP_ROLE_ARN",
"value": q_app_role_arn
}],
"environment": [
{
"name": "REPO_URL",
"value": repo_url
},
{
"name": "SSH_URL",
"value": ssh_url
},
{
"name": "SSH_KEY_NAME",
"value": ssh_key_name
},
{
"name": "AMAZON_Q_APP_ID",
"value": q_app_id
},
{
"name": "AMAZON_Q_USER_ID",
"value": q_app_user_id
},
{
"name": "Q_APP_INDEX",
"value": q_app_index
},
{
"name": "Q_APP_ROLE_ARN",
"value": q_app_role_arn
},
{
"name": "S3_BUCKET",
"value": s3_bucket
},
{
"name": "PROMPT_CONFIG_SSM_PARAM_NAME",
"value": prompt_config_param_name
}],
"command": [
"sh","-c",f"yum -y install python-pip git && pip install boto3 awscli GitPython && aws s3 cp s3://{s3_bucket}/code-processing/generate_documentation_and_ingest_code.py . && python3 generate_documentation_and_ingest_code.py"
]
Expand Down
Loading