AI Skills Assessor Requirements

[chinaexpert1]

Overview

We need to identify and list out the requirements for the AI Skills Assessor in order to facilitate building the tool

Action Items

• Draft requirements document
• Get requirements doc signed off
• Create a 1Password vault for this project
• Decide what you want the GitHub Bot to be called (try not to make it sound juicy to hackers)
  • Hack for LA Admin will set up the email for you to use
  • Create a new GitHub bot account using the email
  • Add the information to 1Password
  • Add the information to issue Create repo labels using API #3
• Gather resources and answer questions
  • ChatGPT API key (paid)
  • List of GitHub handles for the sample/test runs
  • Repository to check for users issues
  • Approve me access to the skills labels sheet linked in the Epic
  • Where will the script be run? Locally or in the cloud?
    • in GitHub with secrets if possible
    • in our AWS account if not
  • I met with Will Gillis and informed him of the project. He is now an advisor and is very interested in the project. Got him added to the repo and sending him the requirements doc so he can get up to speed.
  • Needed: to prefix the skills labels with a "z" so they show up at the end of the issue on the kanban board...or they will crowd out other labels.

Resources/Instructions

• Google Drive: Data Science, Folder: AI-Skills-Assessor
• Ai.Skills.Requirements - Google Doc
• Ai Skills Requirements.pdf

[chinaexpert1]

Update

1. Progress: Finished Requirements analysis, need approval of requirements doc
2. Blockers:

NEEDED FROM BONNIE (same as above)

• APPROVE requirements doc
• ChatGPT API key (paid)
• Need Github API key
• List of usernames for the sample/test runs
• Need Bonnie to show me how to make the branch protection rule
• Approve me access to the skills labels sheet linked in the Epic

3. Availability: I should put in 6 hours and start generating/testing code this week
4. ETA. This issue should be completed Monday, and new issue opened after CoP meeting.
5. Pictures: None, but here is the requirements doc:

Ai Skills Requirements.pdf

[Rabia2219]

List of GitHub handles for the sample/test runs

• JackRichman
• JasonUranta
• mgodoy2023
• Zak234
• anonymousanemone

Repo to check their issues

https://github.com/hackforla/website

[ExperimentsInHonesty]

example of what it will look like when this project will have a github bot

[chinaexpert1]

example of what it will look like when this project will have a github bot

That looks amazing. A lot of it wasn't in my requirements document so I'll have to update it....

[chinaexpert1]

Update:

Still waiting for below

Blockers:

NEEDED FROM BONNIE (same as above)

• ChatGPT API key (paid)
• Need Github API key
• List of usernames for the sample/test runs
• Need Bonnie to show me how to make the branch protection rule
• Approve me access to the skills labels sheet linked in the Epic

Availability: I'm going to pick this up after we come back in August

ETA:

September

To Do:

Update requirements document with features indicated in Bonnie's screenshot

[chinaexpert1]

Update:

I was sick so no work was done on this issue.

The next steps are to update the requirements doc with functionality indicated in Bonnie's screenshot and then present that doc to Bonnie for approval when we get back from break.

Blockers:

NEEDED FROM BONNIE (same as above)

• ChatGPT API key (paid)
• Need Github API key
• List of usernames for the sample/test runs
• Need Bonnie to show me how to make the branch protection rule
• Skills list for labeling

Availability: I'm going to pick this up after we come back in August

ETA:

September

[chinaexpert1]

UPDATE TO REQUIREMENTS NEEDED:

Note: Github bots are just dummy accounts that we create to run code, with an email the Hack for LA owns.

[ExperimentsInHonesty]

Requirements document draft as of 2025-07-21

Details

Requirements Document: AI-Enabled Skills and Contribution Assessor
Project Overview
This project aims to create an AI-enabled system leveraging the GitHub Issue API and ChatGPT to summarize intern contributions and classify skills gained per issue in the Hack for LA organization. It generates resume-ready bullet points and labels skill sets from a predefined CSV taxonomy.
Functional Requirements
User Input and Interaction
• Users specify contributors one at a time through text entry.
• No authentication or verification is required.
Data Processing
• Fetch comments per issue for each specified user via GitHub Issue API.
• Extract context from issue overview and action items.
• Submit combined issue context and user comments to ChatGPT.

Output Generation

Bullet Points:
• Formal style with software development concepts.
• Maximum of 50 words per bullet point.
• Explicitly state task status (completed or ongoing).
• Publicly posted as a comment to the original GitHub issue.

Skill Classification:
• Based on provided CSV file ( HfLA_skills.csv ).

       • Output multiple skill labels relevant to each user's contributions.

Non-Functional Requirements
Performance
• The entire operation per issue per user must complete within 1 minute.

Reliability and Error Handling
• Graceful error handling with public posting of error code and message to the corresponding GitHub issue upon API failure (GitHub or ChatGPT).

Security
• No specific encryption or anonymization required.
• No anticipated privacy concerns.
Deployment and Environment
• Currently no existing deployment infrastructure.
• Designed for scheduled daily execution across all open issues within Hack for LA organization,
incrementing through issues sequentially.
• Codebase will reside in GitHub repo: hackforla/ai-skills-assessor.
• Data persistence will use GitHub as the database (folder: /Data ).
Scalability
• Designed for low concurrent load given daily, sequential batch processing.
Usability
• No graphical user interface; automated execution through daily time-triggered scripts.
Data and API Requirements APIs
• GitHub Issue API:
• Used to retrieve user comments and issue context.
• Respect GitHub API rate limits (to be researched and managed in implementation).

     • ChatGPT API:

    • Premium ChatGPT API key to be acquired.
    • Usage within API limits to be researched and respected.  

Data
• Skill taxonomy sourced from provided CSV file (HfLA_skills.csv), structured as comma
separated skill labels without newlines.
• All relevant issues and comments pre-existing within GitHub.

Feasibility Considerations
• Technical Feasibility:
• Ensure API response times and limits from GitHub and OpenAI are manageable.
• Initial prototyping to validate response times and API usage efficiency.

• Financial Feasibility:

        •  Consider costs associated with premium ChatGPT API usage.

Development Stack
• JavaScript development building on existing codebase.
• Leverage existing true-github-contributors package, modifying it for AI-enabled summarization and
skill classification tasks.

This requirements document serves as the foundation for subsequent project development, providing clear guidelines for implementation, validation, and future enhancements.

Notes from the review of the requirements doc

rewrite document to be POC requirements as cost will help us to understand and shape new requirements for Pilot.

Project Overview
This project aims to create an AI-enabled system leveraging the GitHub Issue API and ChatGPT to summarize intern contributions and classify skills gained per issue in the Hack for LA organization. It generates resume-ready bullet points and labels skill sets from a predefined CSV taxonomy.

• update "and labels skill sets from a predefined CSV taxonomy. " to say "and add skill set labels from a predefined CSV taxonomy. "

User Input and Interaction
• Users specify contributors one at a time through text entry. - we decided for the POC to use a defined list

Output Generation

Bullet Points:
• Formal style with software development concepts. - should be revised to also contain star method, but definitely retaining the SDC is important.
• Maximum of 50 words per bullet point. - update to max 30 words
• Explicitly state task status (completed or ongoing). - in POC all the issues are closed already. This will change in the pilot

Reliability and Error Handling
• Graceful error handling with public posting of error code and message to the corresponding GitHub issue upon API failure (GitHub or ChatGPT). - This lets us know that this issue failed the process, and it will need to either be done manually the process restarted... also, an opportunity to see maybe why it failed. Does the issue have something specific that would cause the script to fail every time same condition is found. This information will also need to be written to a log.

Deployment and Environment
• Data persistence will use GitHub as the database (folder: /Data ) for the POC, please specify the file type. CSV, JSON

---

We will need to define the schema at some point, not sure if it should be in the requirements doc, other than the need to and what the schema should accomplish.

• That we are converting the true github contributors repo to Python and using python for our dev language.

[ExperimentsInHonesty]

Andrew had concerns on the following

• GitHub restriction on being spammy - which we anticipate no problem with because the bot will only operate on repos where it has permissions. Hfla website team has a bot that posts on all open assigned issues that have not been updated, and that runs at a specific time on a lot of issues
• personal access token - according to our documentation we have the access token set to never expire
  • https://github.com/hackforla/website/wiki/Resources
  • HfLA-Website-Admin: How to update Hackforlabot token
• Standardized formatting in all comments should include
  • opening message (describes briefly the purpose of the comment, and has a link to more detail about the automation)
• Formatting for the resume bullet comments
  • after opening message, at messaging all the people in the following way for the POC, in pilot we can message them separately if we desire.

    @experimentsinhonesty
    - Resume Bullet 1
    - Resume Bullet 2
    @chinaexpert1 
    - Resume Bullet 1
    - Resume Bullet 2
    

• Formatting for the label comment
  • after opening message, list all labels with check boxes, so that we can check off which ones we agree with.
• Error messages should also be written to an issue as well as a error log file

[chinaexpert1]

AISA Requirements Doc v2.docx

ChatGPT AISA Requirements Chat Link

Details

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

1. Progress: Requirements doc updated with the items from the meeting, GPT chat link shared above too
2. Blockers: Ready for action items
3. Availability: I have some time this week I can take a task
4. ETA: I thought we were coming back from break in August?
5. Pictures (if necessary):

Notes:

My to do list includes setting up my github AISA local repo with a pull, converting the existing code to Python, and pushing it to a new folder in the AISA repo, also getting GPT to add comments to the Python version

Follow up:

I got that done. The main.py file is here AISA Python Folder