-
Notifications
You must be signed in to change notification settings - Fork 1
Project Proposal
- 2021-18067 Reeahn Kim
- 2021-11692 Solji Kim
- 2021-17051 Eugene Sohn
- 2022-11321 Shinwon Lee
- 2020-19722 Taeho Han
: StoryBridge
: Multicultural children who have difficulty understanding Korean, as well as their parents and educators.
In Korean kindergartens, nearly all physical picture books and classroom reading materials are available only in Korean. StoryBridge addresses this limitation by transforming physical Korean picture books into interactive translations in each child’s preferred non-Korean language. This direction is intentional: it ensures that multicultural children can access the same classroom books in a language that supports their comprehension and engagement, making storytime inclusive.
Beyond inclusivity, StoryBridge also supports multicultural children’s linguistic and cognitive growth. Studies show that a strong home-language foundation supports the learning of the majority language, with children in dual-language programs often achieving comparable or superior literacy to monolingual peers. Research also highlights that early exposure to both majority language and the home language is critical for cognitive, linguistic, and socio-emotional development.
StoryBridge sets itself apart by transforming a child's physical picture books into interactive experiences, a stark contrast to services limited to a fixed library of pre-selected digital content. While platforms like the National Library for Children and Young Adults’ ‘XR Book Play’ and ‘Epic!’ offer a limited selection of pre-produced stories, StoryBridge empowers users to utilize any Korean book.
The service further distinguishes itself with its advanced Emotional Text-to-Speech (TTS). Unlike generic translation apps that simply overlay text (e.g., Papago), StoryBridge analyzes the story's emotional context to deliver a deeply engaging and expressive narration. This technology ensures the reading experience is not just functional but also emotionally resonant, making it far more far more immersive than a mechanical voice or a simple sound effect.
-
Translation:
- Automatic Capture: Since children cannot easily hold a book and phone together and press a capture button, the system captures each page automatically using the CameraX API with edge or blur detection.
- OCR: OCR extracts text and bounding boxes from the captured page, enabling translation boxes to be placed near the original Korean text. gpt-4o is employed for its strong vision and multimodal capability. Another model can be used to get accurate bounding boxes.
- Translation: For translation, gpt-4o is also chosen for its multilingual contextual strength and integration consistency with OCR. A lightweight stopword filter will be added to ensure child-safe translation.
-
Emotional Text-to-Speech (TTS): Our system uses a prompt-based approach for expressive narration. gpt-4o-mini is employed to analyze the translated text sentence-by-sentence to infer dynamic emotional context, rather than applying one static emotion. This analysis generates fine-grained natural language prompts (e.g. "whisper suspensefully") which are passed to gpt-4o-mini-tts engine to vocalize the story with the appropriate feeling.
To make a child's experience with the story more dynamic and engaging, StoryBridge will offer both in-story interaction—such as Interacting with illustrations or viewing word definitions and pronunciations—and after-reading activities, including writing a letter to a character, describing what happens after the story's ending, or creating a simple song about the story using words from both languages.
two Samsung Galaxy S23 devices (tablets if available)
We will test our application with multicultural families and their children by providing Korean picture books and having them use StoryBridge to read in their preferred language. Feedback will focus on translation accuracy, emotional TTS quality, and the child's overall experience and level of interaction with the service. For the final demonstration, we will demonstrate these key features. We will point the device at a book to show the instant text recognition and translation, play the expressive audio narration, and showcase the additional learning activities.