straight2work-gates-datathon (aka "Suffering from Success")

Setting up Local Environment

For Mac OS

• Install pyenv from homebrew

Install Appropriate Python Version (if not already installed)

• pyenv install 3.11

Create the environment

• pyenv virtualenv 3.11 gates-datathon

Activate the environment

• To activate (on Mac) source ~/.pyenv/versions/gates-datathon/bin/activate
• Install dependencies pip install -r requirements.txt

Configure to Run

• Place the needed keys in a file called .keys in the root folder. It should have the following format:

[configuration]
OPEN_AI_KEY=open-ai-key

Local DB Setup SQLite

python src/db/setup_vendor_db.py --db src/db/vendor_inventory.db python src/db/setup_vendor_db.py --db src/db/vendor_inventory.db --import-dir src/data/vendor_info

Run Local Web UI

streamlit run src/app.py

Example Vendor CSV File

CSV Schema

Column	Required	Description
Vendor	✅	Name of the system or platform (e.g., EAB Navigate, Canvas).
Unit Admin	✅	Department or unit responsible for managing the contract or data flow.
Link	✅	Public link to the vendor’s main product page or documentation.
Additional Details	✅	Description of what the system does and how it’s used on campus.
Data Point	✅	Name of each data field that flows into or out of the system.
Direction	✅	Inbound (data sent to the vendor) or Outbound (data generated by the vendor).
Data Point Description	✅	Concise, one-sentence explanation of what that data represents or why it’s used.

Example CSV for Single Vendor

Vendor,Unit Admin,Link,Additional Details,Data Point,Direction,Data Point Description
EAB Navigate,Academic Success & Undergraduate Advising,https://eab.com/solutions/navigate360/,"Cal
 State Hypothetical uses EAB Navigate as a student success platform for students, faculty, and staff to manage advising, scheduling, and communication.",Student ID,Inbound,Unique identifier assigned by the university for each student.
,,,,Email,Inbound,Primary email used for authentication and notifications.
,,,,Sex,Inbound,Gender field from student records used for demographic reporting.
,,,,IPEDS Ethnicity,Inbound,Standardized ethnicity code used for reporting and analytics.
,,,,URM Flag,Inbound,Indicates underrepresented minority status.
,,,,Economic Status Description (Pell-Grant),Inbound,Flag indicating Pell Grant eligibility.
,,,,First Generation Status,Inbound,Indicator that neither parent holds a bachelor’s degree.
,,,,Term GPA,Outbound,Term grade point average calculated and imported from SIS.
,,,,Cumulative GPA,Outbound,Cumulative GPA generated and synced from SIS.
,,,,Advising Notes,Outbound,Text notes created by advisors during appointments.
,,,,Appointment Attendance,Outbound,Record of whether a student attended an advising session.
,,,,Major Code,Inbound,Primary major used to route students to appropriate advisors.
,,,,Full Time/Part Time Status,Inbound,Enrollment load used for scheduling and cohort identification

Good Sample Questions

• "How many different systems are producing the same datapoints?"
• "List all outbound data fields from Qualtrics."
• "Do we have any software that will work well as a learner management systems?"
• "Which systems produce or export GPA data?"
• "Do we have any software that connects students to alumni?"

Additional documentation

Inside the /docs folder.

• Database schema photo + database markup language
• Tool calling explanatory diagram

Future Improvements

• Augmenting vendors with externally sourced data through webscraping.
• Introducing cost data along with vendor data to help identify the potential saving associated with addressing redundancies.
• Continue to iterate with the prompting for the SQL tool, perhaps introducing enumerations for values like "school admin" so that the LLM can do less guessing about WHERE clause filters for some of the columns with narrow sets of possible values.