hack.coms special

Inspiration

Our inspiration for CANT stemmed from firsthand experiences as a note taker for NTID students. Witnessing the struggles of both students and note takers highlighted the need for a more efficient way to capture and summarize lecture content. The challenge of following an ASL interpreter while also focusing on the lecturer made it clear that a solution was necessary to improve accessibility and retention of information.

What it does

CANT (Content Aware Note Taker) is an intelligent note-taking system designed to enhance classroom learning by integrating lecture materials with real-time transcripts. It allows lecturers to input presentation slides (PDFs) and listens to class discussions, generating structured notes in PDF format after the session. Additionally, CANT enables users to interact through a chat interface, where they can ask questions about assignments, midterms, and class topics, leveraging the context from both the transcript and the lecture materials.

How we built it

We built CANT using a tech stack that includes Node.js for the backend, React.js for the frontend, and MongoDB for data storage. We incorporated speech-to-text web kits for real-time transcription and leveraged large language models (LLMs) for content summarization. To address challenges with token limits and costs, we implemented an iterative summarization technique, offloading tasks like PDF reading and note generation to our Node.js application.

Challenges we ran into

We faced several challenges during development, particularly with combining information from PDFs and real-time transcripts. Managing token limits imposed by LLMs was a significant hurdle, as was ensuring cost-effectiveness while maintaining performance. Additionally, integrating speech-to-text capabilities and ensuring accurate summarization posed technical difficulties.

Accomplishments that we're proud of

We are proud to have successfully implemented an iterative summarization technique that effectively overcomes token size limitations. Our application allows for seamless reading and generation of PDFs, ensuring that users receive comprehensive and coherent notes. The chat feature is another highlight, as it enhances user interaction and facilitates quick access to class-related information.

What we learned

Throughout the development process, we learned the importance of user feedback and iterative design. Balancing performance and cost was a crucial aspect, teaching us to prioritize efficiency in our tech stack. We also gained insights into the complexities of working with LLMs and the significance of context in generating meaningful summaries.

What's next for CANT

Looking ahead, we plan to expand CANT's capabilities by training our model with diverse types of notes to better understand and meet the expectations for various course formats. We aim to enhance the chat interface with more sophisticated querying capabilities and explore integrations with other educational tools. Additionally, we want to refine the user experience based on feedback from both students and lecturers to ensure that CANT remains a valuable asset in educational settings.

What's Changed

• Master by @sanath1599 in #1

New Contributors

• @sanath1599 made their first contribution in #1

Full Changelog: https://github.com/sanath1599/content-aware-note-taker/commits/0.1.4