fix: Allow github organization repo creation (#374)

* fix: Allow github organization repo creation

* fix: validation script fixes

Author: weklund

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### **Added**
 
 ### **Changed**
+- updated GitHub repo creation in `sagemaker-templates` module to support GitHub organizations
 
 ## v2.1.0
 

examples/manifests/sagemaker-templates-modules-github.yaml

@@ -5,7 +5,7 @@ parameters:
   - name: repository-type
     value: GitHub
   - name: repository-owner
-    value: <github_account_owner>
+    value: <github_account_owner>  # Can be a personal account username or organization name
   - name: access-token-secret-name
     value: github_token
   - name: aws-codeconnection-arn

modules/sagemaker/sagemaker-templates/README.md

@@ -43,8 +43,10 @@ The template contains an example CI/CD pipeline to deploy the model endpoints to
 
 The template is based on basic multi-account template from [AWS Enterprise MLOps Framework](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/README.md#sagemaker-project-stack).
 
-#### Third-party Code Repository Integration 
-SageMaker templates support third party code repository (GitHub) integration along with default AWS CodeCommit. As part of integration, SageMaker templates will be able to manage (create, delete) repositories. As an example, if `sagemaker-templates-service-catalog` template configured to use GitHub as repository type then it would create code repository directly into GitHub account provided with manifest configuration. Repository will be named after SageMaker project name in AWS account `{sagemaker-project}-deploy`. For example, if SageMaker project name is `aiops-abalone-model` then GitHub repository would be created with name `aiops-abalone-model-deploy`.
+#### Third-party Code Repository Integration
+SageMaker templates support third party code repository (GitHub) integration along with default AWS CodeCommit. As part of integration, SageMaker templates will be able to manage (create, delete) repositories in both **personal GitHub accounts** and **GitHub organizations**. The template automatically detects whether the `repository-owner` parameter refers to an organization or personal account and uses the appropriate GitHub API endpoint.
+
+As an example, if `sagemaker-templates-service-catalog` template configured to use GitHub as repository type then it would create code repository directly into the GitHub account or organization specified in manifest configuration. Repository will be named after SageMaker project name in AWS account `{sagemaker-project}-deploy`. For example, if SageMaker project name is `aiops-abalone-model` then GitHub repository would be created with name `aiops-abalone-model-deploy`.
 
 
 ## Prerequesites:
@@ -53,9 +55,19 @@ SageMaker templates support third party code repository (GitHub) integration alo
 > [!IMPORTANT] 
 > It is important to note AWS CodeCommit is no longer available to new customers. Existing customers of AWS CodeCommit can continue to use the service as normal. 
 ### GitHub repository integration
-- Target AWS account should contain AWS Secret Manager secret that contains GitHub personal access token with required permissions to manage repository. Refer guide [Creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token) in order to create access token.
+- Target AWS account should contain AWS Secret Manager secret that contains GitHub personal access token with required permissions to manage repositories. Refer guide [Creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token) in order to create access token.
+  - **For personal accounts**: Token requires `repo` scope to create private repositories
+  - **For organizations**: Token requires `repo` scope AND the user must be a member of the organization with permissions to create repositories. For organization SSO, ensure the token is authorized for SSO access.
 - Template also requires AWS CodeConnection created for GitHub provider in order to integrated GitHub repositories AWS CodeBuild and AWS CodePipeline. Refer guide [Create a connection to GitHub](https://docs.aws.amazon.com/dtconsole/latest/userguide/connections-create-github.html) in order to create connection with GitHub.
 
+> [!IMPORTANT]
+> **For GitHub Organizations**: AWS CodeConnections have a **1:1 mapping** with GitHub organizations. Each organization requires:
+> - A separate GitHub App installation in that organization (must be installed by an organization owner)
+> - A dedicated CodeConnection ARN for that organization
+> - The `repository-owner` parameter must match the organization where the GitHub App is installed
+>
+> **Common Issue**: If your CodeConnection was created for a personal account (e.g., `username`) but you specify an organization name (e.g., `MyOrg`) in the `repository-owner` parameter, deployments will fail with "Repository not found" errors. Ensure the GitHub App is installed in the correct organization. See [How connections work with organizations](https://docs.aws.amazon.com/dtconsole/latest/userguide/welcome-connections-how-it-works-github-organizations.html) for details.
+
 ## Inputs and outputs:
 
 ### Required inputs:
@@ -65,7 +77,7 @@ SageMaker templates support third party code repository (GitHub) integration alo
 
 ### Common Optional Inputs:
 - `repository-type` - type of repository to be integrated with SageMaker template source code. Options: `CodeCommit` (default), `GitHub`, `GitHub Enterprise`
-- `repository-owner` - owner or organisation of project code repository (required for GitHub)
+- `repository-owner` - GitHub username (for personal accounts) or organization name (for organizations) where repositories will be created (required for GitHub)
 - `access-token-secret-name` - AWS Secret Manager secret name where access token is stored (default: `github_token`)
 - `aws-codeconnection-arn` - AWS CodeConnection ARN for repository provider
 - `sagemaker-domain-id` - SageMaker domain id
@@ -184,4 +196,57 @@ parameters:
 ```
 ### Sample manifest example for source repository options
 [sagemaker-templates-modules-github.yaml](/examples/manifests/sagemaker-templates-modules-github.yaml)
-[sagemaker-templates-modules-codecommit.yaml](/examples/manifests/sagemaker-templates-modules-codecommit.yaml)
+[sagemaker-templates-modules-codecommit.yaml](/examples/manifests/sagemaker-templates-modules-codecommit.yaml)
+
+## Troubleshooting GitHub Integration
+
+### Issue: "Repository not found" error during deployment
+
+**Symptoms:**
+- Lambda logs show: `Repository created: https://github.com/username/repo-name.git`
+- Followed by: `remote: Repository not found` when trying to push
+- Repository was created under a personal account instead of the intended organization
+
+**Root Cause:**
+This occurs when the `repository-owner` parameter specifies an organization name (e.g., `MyOrganization`), but either:
+1. The AWS CodeConnection was created for a personal account instead of the organization
+2. The GitHub App is not installed in the organization
+
+**Solution:**
+1. Verify the CodeConnection is properly configured:
+   - Go to AWS Developer Tools Console → Connections
+   - Check which GitHub account/organization the connection is associated with
+   - The connection must have the GitHub App installed in the target organization
+2. For organizations, ensure:
+   - An organization owner installed the GitHub App in the organization
+   - The CodeConnection ARN corresponds to that organization's installation
+   - The `repository-owner` parameter matches the organization name exactly
+3. Clean up failed deployment:
+   - Delete any repositories created under the wrong account
+   - Delete or rollback the failed CloudFormation stack
+4. Redeploy with the correct CodeConnection ARN
+
+### Issue: Permission denied when creating repositories in organization
+
+**Symptoms:**
+- Lambda logs show HTTP 403 or 422 errors
+- Error message mentions permissions or organization access
+
+**Solution:**
+1. Verify the GitHub Personal Access Token:
+   - Token must have `repo` scope
+   - User must be a member of the organization with repository creation permissions
+   - For organizations with SSO: Token must be authorized for SSO access (check in GitHub Settings → Personal Access Tokens → Configure SSO)
+2. Verify organization settings:
+   - Check organization member permissions in GitHub
+   - Ensure the user has rights to create repositories
+
+### Issue: CodePipeline or CodeBuild cannot access GitHub repository
+
+**Symptoms:**
+- Repository is created successfully
+- CodePipeline or CodeBuild fails to clone/access the repository
+
+**Solution:**
+1. Ensure the CodeConnection ARN is correct for the organization
+2. For organizations: Confirm the GitHub App has the necessary repository permissions in the organization settings

modules/sagemaker/sagemaker-templates/common/code_repo_construct.py

@@ -112,6 +112,27 @@ def extract_zip_file(zip_file_path, extract_dir):
         print(f"Error extracting zip file: {str(e)}")
         raise e
 
+def is_organization(github_owner, headers):
+    # Check if github_owner is a GitHub organization (not a personal user account).
+    # Returns True if it's an organization, False otherwise.
+    print(f"Checking if '{github_owner}' is a GitHub organization...")
+    req = urllib.request.Request(f'https://api.github.com/orgs/{github_owner}', headers=headers)
+    try:
+        with urllib.request.urlopen(req) as response:
+            if response.getcode() == 200:
+                print(f"'{github_owner}' is a GitHub organization")
+                return True
+            else:
+                print(f"Unexpected status code {response.getcode()} when checking organization")
+                return False
+    except urllib.error.HTTPError as e:
+        if e.code == 404:
+            print(f"'{github_owner}' is not an organization (404 - not found)")
+            return False
+        else:
+            print(f"Error checking if '{github_owner}' is an organization: {e.code} - {e.reason}")
+            # For other errors (403, etc.), assume it might be an org and let the create call fail with proper error
+            return True
 
 def lambda_handler(event, context):
     print(f"Received event: {json.dumps(event)}")
@@ -142,19 +163,29 @@ def lambda_handler(event, context):
             'private': True
         }).encode('utf-8')
 
-        req = urllib.request.Request('https://api.github.com/user/repos', data=data, headers=headers, method='POST')
+        # Determine if github_owner is an organization or personal user account
+        if is_organization(github_owner, headers):
+            # Use organization endpoint
+            api_url = f'https://api.github.com/orgs/{github_owner}/repos'
+            print(f"Creating repository under organization '{github_owner}' using {api_url}")
+        else:
+            # Use user endpoint
+            api_url = 'https://api.github.com/user/repos'
+            print(f"Creating repository under user account using {api_url}")
+
+        req = urllib.request.Request(api_url, data=data, headers=headers, method='POST')
 
         try:
             with urllib.request.urlopen(req) as response:
                 repo_data = json.loads(response.read().decode())
                 repo_url = repo_data['clone_url']
+                print(f"Repository created successfully: {repo_url}")
         except urllib.error.HTTPError as e:
-            print(f"HTTP Error: {e.code} - {e.reason}")
-            print(e.read().decode())
+            print(f"HTTP Error creating repository: {e.code} - {e.reason}")
+            error_body = e.read().decode()
+            print(f"Error details: {error_body}")
             raise
 
-        print(f"Repository created: {repo_url}")
-
         # Download entire S3 bucket
         s3 = boto3.client('s3')
         with tempfile.TemporaryDirectory() as tmp_dir:

modules/sagemaker/sagemaker-templates/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.ruff]
 line-length = 120
-target-version = "py38"
+target-version = "py310"
 extend-include = ["*.ipynb"]
 
 exclude = [
@@ -24,7 +24,7 @@ select = ["F", "I", "E", "W"]
 fixable = ["ALL"]
 
 [tool.mypy]
-python_version = "3.8"
+python_version = "3.10"
 strict = true
 ignore_missing_imports = true
 disallow_untyped_decorators = false

modules/sagemaker/sagemaker-templates/templates/finetune_llm_evaluation/pipeline_constructs/build_pipeline_construct.py

@@ -228,8 +228,7 @@ def __init__(
                     "sagemaker:ListTags",
                 ],
                 resources=[
-                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
-                    f"{sagemaker_pipeline_name}",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/{sagemaker_pipeline_name}",
                     f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
                     f"{sagemaker_pipeline_name}/execution/*",
                 ],

modules/sagemaker/sagemaker-templates/templates/finetune_llm_evaluation/seed_code/build_app/source_scripts/evaluate.py

@@ -77,7 +77,7 @@ def evaluate_model(args: Any) -> None:
         os.makedirs(args.model_dir, exist_ok=True)
         tar_file.extractall(args.model_dir)
 
-    logger.info(f"Decompressed Model assets: {os.listdir( args.model_dir )}")
+    logger.info(f"Decompressed Model assets: {os.listdir(args.model_dir)}")
 
     # Load test dataset
     test_dataset = load_from_disk(args.test_data_dir)
@@ -88,7 +88,7 @@ def evaluate_model(args: Any) -> None:
     # ensure that we do not have any trailing/leading whitespaces, this can lead to issues during inference
     test_data = test_dataset.map(lambda sample: {"prompt": sample["prompt"].strip()})
     logger.info("Loading test dataset")
-    logger.info(f"Test dataset has {len( test_data )} samples")
+    logger.info(f"Test dataset has {len(test_data)} samples")
 
     model = AutoModelForCausalLM.from_pretrained(args.model_dir, device_map="auto", torch_dtype=torch.float16)
 

modules/sagemaker/sagemaker-templates/templates/finetune_llm_evaluation/seed_code/build_app/source_scripts/train.py

@@ -71,7 +71,7 @@ def create_peft_model(model: Any, gradient_checkpointing: bool = True, bf16: boo
 
     # get lora target modules
     modules = find_all_linear_names(model)
-    logger.info(f"Found {len( modules )} modules to quantize: {modules}")
+    logger.info(f"Found {len(modules)} modules to quantize: {modules}")
 
     peft_config = LoraConfig(
         r=64,

modules/sagemaker/sagemaker-templates/templates/model_deploy/product_stack.py

@@ -111,8 +111,7 @@ def __init__(
         dev_region: str = Aws.REGION
 
         model_package_arn = (
-            f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package/"
-            f"{model_package_group_name}/*"
+            f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package/{model_package_group_name}/*"
         )
         model_package_group_arn = (
             f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package-group/"

modules/sagemaker/sagemaker-templates/templates/xgboost_abalone/pipeline_constructs/build_pipeline_construct.py

@@ -262,8 +262,7 @@ def __init__(
                     "sagemaker:ListTags",
                 ],
                 resources=[
-                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
-                    f"{sagemaker_pipeline_name}",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/{sagemaker_pipeline_name}",
                     f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
                     f"{sagemaker_pipeline_name}/execution/*",
                 ],