Skip to content

Requirements and Specifications

Jump to bottom
Woohyuk Jang edited this page Dec 7, 2025 · 22 revisions

Project: Foodigram

Table of Contents

  1. Document Revision History
  2. Project Abstract
  3. System Design
    3.1 Architecture
    3.1.1 High-Level Architecture
    3.2 Data Model
    3.2.1 Core Entities
    3.2.2 Key Features
    3.2.3 DB Model
    3.3 Testing
  4. Customer
  5. Competitive Landscape
  6. Functional Requirements
    6.1 Use Cases
    6.2 User Stories
    6.2.1 Authentication
    6.2.2 Personalization
    6.2.3 Restaurant Discovery & Recommendation
    6.2.4 Food Record & AI-Based Analysis
    6.3 User Acceptance Test
  7. Non-functional Requirements
  8. User Interface Requirements
    8.1 Workflow
    8.2 Views
  9. Class Diagram

1. Document Revision History

Version Date Description
1.0 2025-10-02 Initial draft created for Foodigram project
1.1 2025-10-04 Added Project Abstract and Functional Requirements
1.2 2025-10-05 Completed Use Cases, UI, and Class Diagram
1.3 2025-10-19 Added System Design, updated data model and technical specifications
1.4 2025-11-02 Gallery Import Flow, Recommendation Engine details, and Testing section
1.5 2025-11-16 Updated to reflect current implementation: PostgreSQL database, ML Kit Image Labeling, enhanced UserGalleryImage model, current API endpoints
1.6 2025-11-30 Added scrap-based recommendation, RL scoring, NLP intent extraction, explanation generation
1.7 2025-11-30 Updated to reflect actual implementation: Django session authentication, streaming NDJSON APIs, enhanced gallery features, logout confirmation system, persistent scrap storage
1.8 2025-12-07 Fixes after iteration 5 added: parallel scoring, batch embeddings, Naver Map integration, EAS production build

2. Project Abstract

Choosing what to eat and where to dine is a recurring challenge in everyday life. While there are many restaurant and review platforms available, most fail to understand the personal context of the user — their unique taste preferences and eating history.

Foodigram addresses this gap by offering a personalized, context-aware restaurant and menu recommendation service powered by AI. The system analyzes food photos imported from the user's gallery to understand their past meals to augment individual taste profiles. Based on this analysis, the app provides recommendations tailored to each user's preferences, considering contextual factors such as location, time, and weather.

Users can navigate easily through two main tabs — Recommendation and Profile — making food discovery faster and more enjoyable. The former focuses on providing personalized restaurant and menu recommendation, while the latter serves to manage the food history - both the gallery image and images liked/bookmarked by user.

By combining AI analysis for contextual understanding, Foodigram transforms the process of choosing a meal into a smart, personalized, and engaging experience — promoting both convenience and healthier dietary habits.

3. System Design

3.1 Architecture

3.1.1 High-Level Architecture

Frontend:

  • Framework: React Native (Expo)
  • State Management: MobX State Tree
  • UI Components: Custom component library with theme system
  • Platform: Android

Backend:

  • Framework: Django 5.2.6+ + Django REST Framework
  • Database: PostgreSQL (with Docker setup for local development)
  • AI/ML:
    • Vector Search: ChromaDB + SentenceTransformers (jhgan/ko-sbert-sts)
    • On-device Classification: ML Kit Image Labeling (@infinitered/react-native-mlkit-image-labeling)
    • Model: Custom TensorFlow Lite model (foodNonFood) for food/non-food classification
    • NLP: Intent extraction from natural language queries
    • Explanation: GPT-based recommendation reason generation
  • API: RESTful endpoints + Streaming NDJSON with session-based authentication (Django sessions)
  • Package Management: UV for Python dependency management

External Services:

  • AWS S3: Image storage for food photos
  • Django Sessions: User authentication (AWS Cognito integration planned)
  • Location API: Distance calculation and mapping
  • Naver Place API: Restaurant data scraping
  • Naver Map: Deep linking for restaurant navigation (nmap:// scheme)

Recommendation Engine:

  • Vector Database: ChromaDB for semantic search
  • Embedding Model: jhgan/ko-sbert-sts (Korean sentence transformers)
  • Algorithm: Hybrid scoring (text similarity + popularity + distance + price) with MMR reranking
  • On-device Analysis: TensorFlow Lite for food photo classification
  • Performance: Parallel scoring with ThreadPoolExecutor, batch embedding processing
  • Streaming: Phase 1/Phase 2 architecture for progressive recommendation delivery
  • Personalization Enhancements:
    • Scrap-based Category Derivation: Analyzes user's scrapped restaurants to extract preferred categories via scrap_category_service.py
    • Gallery Category Preferences: Computes category preference scores from gallery images via gallery_category_service.py
    • Gallery-based Exploration: Calculates category diversity entropy from user's food photos
    • RL Scoring: Per-user weight vectors learned from interactions (click, scrap, hide)
    • NLP Intent Extraction: Extracts structured intents from natural language queries
    • Explanation Generation: GPT-based human-readable recommendation reasons in Korean
  • Performance Optimizations:
    • Parallel Menu Scoring: ThreadPoolExecutor-based concurrent scoring for improved response times
    • Batch Embedding Processing: Batch processing of menu embeddings (batch_size=32) for optimization
    • Enhanced Streaming: Phase 1/Phase 2 split architecture for progressive recommendation delivery

Gallery Import & Analysis Flow:

  1. Permission Request: Request photo library access via Expo MediaLibrary with granular permissions
  2. Album Scanning: Scan all device albums including smart albums
  3. On-device Classification: ML Kit Image Labeling classifies each image as food/non-food using custom TensorFlow Lite model (foodNonFood)
  4. Selective Upload: Only food images are uploaded to AWS S3 (protected/{identityId}/album/{albumTitle}/{filename} path)
  5. Server Registration: Image URLs sent to backend via POST /api/v1/photos/ endpoint with image bytes
  6. Server-side Analysis: Server uses CLIP + foodlist.json matching to generate primary label, alternatives, and confidence scores
  7. Embedding Generation: Server generates 768-dim embeddings for semantic search using ko-sbert-sts
  8. Profile Update: User's food history automatically enriched with AI labels and embeddings for better recommendations
  9. Category Preference Calculation: Gallery category service analyzes images to compute category preferences and exploration preference
  10. Recommendation Integration: Category preferences automatically integrated into recommendation scoring algorithm

3.2 Data Model

3.2.1 Core Entities

  • User Management: User, Profile, UserPreference, UserGalleryImage, UserScrap, UserRemoteScrap, UserInteraction, RLWeightHistory
  • Restaurant & Menu: Restaurant, Menu, MenuCandidate, RestaurantMenu
  • Recommendation: RecommendationResult, RecommendationRanking, EmbeddingCache, MenuReasonFeatures, MenuExternalMapping, RestaurantExternalMapping

3.2.2 Key Features

  • Vector embeddings stored as JSON for semantic food search
  • Flexible JSON-based schema for user preferences and recommendation contexts
  • Multi-layer caching (embedding cache + query result cache)
  • Strategic indexing on frequently queried fields
  • Photo upload and storage integration with AWS S3
  • RL weight vectors for personalized scoring per user
  • Interaction logging for reward-based learning
  • Reason feature storage for explainable recommendations

3.2.3 DB Model

E-R Diagram a

Database Schema Overview

Our database architecture is designed to support a personalized food recommendation system. The schema consists of four main domains:

1. User Management Domain

  • User: Extends Django's AbstractUser with additional fields (nickname, email uniqueness)
  • Profile: Stores user bio and general preferences as JSON
  • UserPreference: Captures taste preferences and personalization data:
    • spicy_level, sweet_level, salty_level: 0-10 scale
    • exploration_preference: 0-5 scale (0=familiar foods, 5=adventurous)
    • allergies: JSON array (e.g. ["peanut", "shellfish"])
    • disliked_ingredients: JSON array (e.g. ["cilantro", "mushroom"])
    • favorite_cuisines: JSON array (e.g. ["korean", "italian"])
    • rl_weight_vector: JSON array of 7 RL-optimized weights for personalized scoring
      • Weights for: text_similarity, popularity, distance, price, freshness, query_similarity, taste_alignment
  • UserGalleryImage: Stores AI-analyzed food images from user's gallery with embeddings for similarity search. Includes:
    • Primary AI label (from CLIP + foodlist.json matching)
    • Alternative labels (top 5 suggestions with confidence scores)
    • Label confidence score (0.0-1.0)
    • Manual edit tracking (label_manually_edited, original_ai_label)
    • 768-dimensional embeddings (ko-sbert-sts) for semantic similarity search
    • Category preferences automatically computed via gallery_category_service.py
  • UserScrap: Manages user bookmarks/saves for restaurants
    • Category preferences automatically computed via scrap_category_service.py
    • Preferences sync with recommendation system
    • Unique constraint: one user can only scrap same restaurant once
  • UserRemoteScrap: Stores complete scrap data for remote sync (cross-device restoration)
    • Contains complete menu data (menu_id, menu_name, place_name, price, category, location, rating, review_count, image_url, coordinates)
    • Tracks original scrap time (scrapped_at) and sync time (synced_at)
    • Unique constraint: one user can only have one scrap per menu_id
    • Used for cross-device scrap synchronization via AWS storage
  • UserInteraction: Logs user interactions for RL-based recommendation tuning:
    • Interaction types: scrap, click, view, hide, expand, allergic_reaction
    • Reward values for each interaction (+1.0 scrap, +0.5 click, -1.0 hide, etc.)
    • Context query for search-based interactions
  • RLWeightHistory: Stores RL weight vector history for each user:
    • Weight vector snapshots for analysis
    • Update cycle number for tracking

2. Restaurant & Menu Domain

  • Restaurant: Core restaurant information with geolocation data (latitude/longitude) and external source tracking
  • Menu: Normalized menu items with vector embeddings for semantic search and recommendation
  • MenuCandidate: Staging table for scraped menu data before merging into the main Menu table
  • RestaurantMenu: Junction table managing many-to-many relationships between restaurants and menus

3. Recommendation Domain

  • RecommendationResult: Logs recommendation queries and results for analytics and caching
  • RecommendationRanking: Detailed ranking information for each recommendation
  • EmbeddingCache: Optimizes performance by caching computed vector embeddings for users, menus, images, and restaurants
  • MenuReasonFeatures: Stores computed reason features for each menu recommendation:
    • Semantic similarity, image similarity, category match score
    • Taste alignment, query alignment, temporal fit score
    • Distance score, popularity score
    • Allergy/dislike penalties
    • GPT-generated explanation text and top 3 reason keys
    • Final recommendation score
  • MenuExternalMapping: Junction table linking Django Menu IDs to external PostgreSQL UUIDs
  • RestaurantExternalMapping: Junction table linking Django Restaurant IDs to external PostgreSQL UUIDs

4. External Database Domain

  • DbRestaurant: Restaurant model from external PostgreSQL database
    • UUID primary key (not Django auto-increment)
    • External ID tracking (external_id)
    • Rich location data (address, road_address, group1/2/3 for hierarchical location)
    • PostGIS geometry field (geom) for spatial queries
    • Category information (category, category_code, category_code_list, category_normalized)
    • Rating and review data (avg_rating, review_count)
    • Image arrays (place_images)
    • Embedding vectors for semantic search (embedding_vector)
    • Menu inference (inferred_menu)
    • Timestamps (created_at, updated_at)
  • DbMenu: Menu model from external PostgreSQL database
    • UUID primary key linked to DbRestaurant
    • External ID tracking (external_id)
    • Price, description, images array
    • Taste profile and allergen info as JSON
    • Embedding vectors for semantic search (embedding_vector)
    • Recommendation flag (recommend)
    • Index within restaurant (index_in_rest)
    • Cleaned name field (name_clean)
    • Timestamps (created_at, updated_at)
  • External Mapping: MenuExternalMapping and RestaurantExternalMapping link Django models to external UUIDs for data synchronization

5. Key Design Decisions

  • Vector Embeddings: JSON fields store high-dimensional vectors for semantic similarity search using ChromaDB and Korean sentence transformers
  • Flexible Schema: JSON fields for preferences, embeddings, and recommendation contexts allow for easy iteration without schema migrations
  • Performance Optimization: Strategic indexes on frequently queried fields (user_id, created_at, name) and unique constraints to prevent data duplication
  • External Integration: source field in Restaurant table tracks external data sources (e.g., Naver Place IDs) for data synchronization
  • PostgreSQL Setup: Docker-based PostgreSQL with PostGIS extension for spatial queries (optional, currently using standard PostgreSQL)
  • RL Learning: UserInteraction logs enable reward-based weight optimization for personalized scoring

3.3 Testing

Test Coverage:

  • Overall backend coverage: 84% (maintained above 80% target)
  • Comprehensive test suites for all major modules
  • Frontend testing with Jest and React Native Testing Library

Test Structure:

Recommendation System Tests (110 test cases across 20 test classes):

  • Data class validation (TastePreference, UserOnboardingData, GalleryAnalysisResult)
  • User profile generation and management
  • Scoring algorithms (HybridScorer, MMRReranker)
  • Search context and filtering
  • API endpoint testing (menu/place recommendations)
  • Document builders and embedding generation
  • Integration tests for end-to-end recommendation flow
  • RL scoring and weight vector tests
  • NLP intent extraction tests
  • Explanation generation tests

Users App Tests (57 test cases):

  • Authentication (register, login, logout, CSRF)
  • Permission classes (IsOwnerOrReadOnly)
  • User service layer (profile management, photo upload)
  • ViewSet operations (Me, User, Scrap, Onboarding, Photo)
  • Interaction logging (click, scrap, hide)

Image Classification Tests (frontend):

  • ML Kit classifier initialization and readiness
  • Food vs non-food classification accuracy
  • Error handling for classification failures
  • Batch processing of multiple images
  • S3 upload success/failure scenarios
  • API integration for photo URL registration

Test Configuration:

  • Framework: Django TestCase with Django REST Framework's APIClient
  • Mocking: unittest.mock for external dependencies
  • Coverage tool: coverage.py with .coveragerc configuration
  • Excluded from coverage: migrations, auto-generated files (wsgi.py, asgi.py, manage.py)

Test Execution:

# Run all tests
python manage.py test

# Run with coverage
coverage run manage.py test
coverage report

4. Customer

Foodigram is designed for anyone who faces the daily challenge of deciding what to eat or where to go.

Our main customers include:

  • People who want personalized restaurant and menu recommendations that reflect their taste and eating habits.
  • Users who are aware of allergen information when searching for new cuisine.
  • Individuals who seek to maintain healthier diets while still enjoying diverse dining experiences.
  • Travelers exploring new areas who want location-based, context-aware restaurant suggestions.

5. Competitive Landscape

While many restaurant recommendation apps (e.g., MangoPlate, Naver Map, and Kakao Map) provide a list of venues,
they generally rely on aggregated reviews by other people rather than the user.

Foodigram differentiates itself by focusing on AI-driven personalization — analyzing the user's own food history, preferences, and context.
Unlike conventional services that suggest popular places, Foodigram tailors its recommendations for each individual.

Its core strength lies in the integration of:

  • AI-based food photo analysis
  • Contextual awareness (time, weather, location)
  • Scrap-based preference learning — recommendations improve based on what users save
  • Natural language query understanding — users can describe what they want in plain Korean
  • Explainable recommendations — each suggestion comes with a human-readable reason

Technical Differentiation:

  • Self-hosted recommendation engine using ChromaDB and Korean language models (jhgan/ko-sbert-sts)
  • On-device photo analysis using TensorFlow Lite for privacy and speed
  • Hybrid scoring algorithm combining multiple factors (text similarity, popularity, distance, price) instead of just ratings
  • Gallery import feature for instant personalization without manual uploads (unique to Foodigram)
  • Vector embeddings for semantic search enabling "show me something like this" recommendations
  • Reinforcement Learning-based weight optimization — scoring weights adapt to each user's behavior
  • NLP intent extraction — understands queries like "얼큰한 순대국밥" or "가성비 좋은 점심"
  • GPT-powered explanations — provides reasons like "매운맛을 좋아하시는 분께 딱 맞아요!"

This combination provides a recommendation experience that feels personal, adaptive, and data-driven, rather than generic.

6. Functional Requirements

6.1 Use Cases

Goal Actor Pre-conditions Main Scenario Extensions (Failure) Variations
Get restaurant recommendations Registered user Logged in, location enabled 1. System analyzes history and context
2. AI generates top n menu options
3. User surfs through recommended menus		 1a. No data available → fallback message 3a. User refreshes to get a new list
Filter restaurants by category Registered user On recommendation page 1. User selects preferred cuisine filter
2. System updates list		 2a. No results match filter —
Filter restaurants by allergen Registered user On recommendation page 1. User selects allergen
2. System updates list		 2a. No results match filter —
Import gallery photos Registered user Gallery permission granted 1. User taps "Import from Gallery"
2. ML Kit scans and classifies images on-device
3. Food images uploaded to S3
4. Server generates embeddings
5. Gallery enriched		 2a. Classification fails → skip photo
3a. Upload fails → retry
4a. Embedding generation fails → log error		 1a. User can import from specific albums
1b. User views import progress in real-time
Search with natural language Registered user Logged in, location enabled 1. User enters query like "얼큰한 국밥"
2. System extracts intent (categories, tastes)
3. Enhanced recommendations returned
4. Each result shows explanation		 2a. Intent extraction fails → use raw query 3a. User can view why each was recommended
Save restaurant to scraps Registered user Viewing recommendation 1. User taps scrap button
2. Restaurant saved to scraps
3. Interaction logged for RL		 3a. Save fails → retry 1a. User can unsave (toggle)

6.2 User Stories

6.2.1 Authentication

  • As a new user,

    • I can register and log in using my email and password to access Foodigram.

  • As a returning user,

    • I am automatically logged in for convenience but can log out anytime to protect my privacy.
    • If I decide to leave the service, I can delete my account, which permanently removes all my data from the system.

Acceptance Criteria:

  • When a new user provides valid email and password, the system creates a new account and grants access to the main interface.
  • When a registered user enters valid credentials, the system authenticates them and loads their personalized environment.
  • When the user revisits the app after logging in once, the system automatically authenticates them unless they have explicitly logged out.
  • Upon selecting "Log Out," the user is redirected to the login screen, and local session data is securely cleared.
  • When "Delete Account" is confirmed, all personal data, credentials, and stored records are permanently erased from the database, with a confirmation prompt shown before deletion.
  • Authentication errors (invalid credentials, duplicate email, network failure) are handled gracefully with clear feedback messages.
  • Logout confirmation modal provides Korean localized interface with scrap preservation notification.
  • Session persistence maintains scraped menu data across logout/login cycles.
  • Authentication supports both username and email-based login.

6.2.2 Personalization

  • As a new user,

    • I can specify my taste preferences (e.g., spicy/sweet/salty levels), allergens, and disliked ingredients during onboarding so that I receive personalized restaurant and menu recommendations from the beginning.

  • As a returning user,

    • I can update my profile photo, taste preferences, and allergy settings anytime to ensure my recommendations remain aligned with my latest habits and health needs.
    • My recommendations automatically improve over time as the system learns from my scraps, clicks, and hides.

Acceptance Criteria:

  • After successful registration, the user is guided to an onboarding screen where they can select or skip:
    • taste intensity levels (spicy/sweet/salty),
    • allergens (e.g., soy, nuts, sesame),
    • disliked ingredients (e.g., coriander, onions).
  • When preferences are set, the recommendation engine immediately integrates them into the user's profile for subsequent suggestions.
  • Users can revisit and edit these preferences in their Profile Settings at any time.
  • Updating profile details (photo, preferences, allergies) reflects instantly in their data model and recommendation results.
  • If the user skips onboarding, default neutral preferences are applied until the user manually updates them later.
  • All changes are saved persistently to the cloud database and retrieved on next login.
  • User interactions (scrap, click, hide) are logged and used to optimize personal scoring weights via RL.
  • Exploration preference can be auto-calculated from gallery image category diversity.

6.2.3 Restaurant Discovery and Recommendation

  • As a user,
    • I can receive personalized restaurant and menu recommendations based on my taste profile, location, time, and weather, helping me find suitable meals with minimal effort.
    • I can apply filters such as "Korean," "Japanese," or "Healthy" to refine my search results, and also exclude restaurants or menus containing allergens like "Soy" or "Sesame."
    • When I turn on the application, the system presents various meal options according to the context, and the provided options can be renewed by refresh button.
    • If I'm traveling or visiting a new area, the app automatically adjusts recommendations based on my current location, showing nearby restaurants with summarized reviews for quick decision-making.
    • I can describe what I want in natural language (e.g., "얼큰한 순대국밥", "가성비 좋은 점심") and receive relevant recommendations.
    • I can see why each menu was recommended with a human-readable explanation.
    • My recommendations improve over time as I scrap, click, or hide items.

Acceptance Criteria:

  • The recommendation engine references the user's profile (preferences, allergies, and past meal history) when generating results.
  • The recommendation engine uses streaming NDJSON delivery for real-time results.
  • First recommendations appear within 1 second via streaming response.
  • When the user toggles refresh button, a new streaming list of recommendations is provided.
  • When category filters (e.g., "Japanese") are applied, only matching restaurants remain visible.
  • When allergen filters are selected, all menus containing or likely containing those allergens are excluded.
  • When location access is granted, nearby restaurant recommendations update dynamically.
  • For travelers, each recommendation includes summarized reviews, distance, and average rating.
  • Natural language queries are processed by NLP intent extractor to extract categories, tastes, and textures.
  • Natural language queries return enhanced results with GPT-generated explanations.
  • Each recommendation includes a GPT-generated explanation in Korean.
  • Scrapped restaurants influence future recommendations and are preserved across sessions.

Technical Implementation:

  • The recommendation engine uses a multi-stage pipeline:

    1. User Profile Generation: Combines UserPreference data (taste levels, allergies, dislikes, favorite cuisines), UserGalleryImage embeddings (food history), and optional behavior data into a personalized text profile.
    2. Scrap-based Category Derivation (scrap_category_service.py): Analyzes user's scrapped restaurants to extract top-k preferred categories from category_normalized field. Category preferences automatically sync with recommendation scoring.
    3. Gallery Category Preferences (gallery_category_service.py): Computes category preference scores from gallery images with recency weighting (90-day decay). Category preferences integrated into recommendation algorithm.
    4. Gallery-based Exploration: Calculates category diversity entropy from user's gallery images to determine exploration preference (0-5 scale).
    5. Query Intent Extraction: NLP module extracts structured intents from natural language queries:
      • Categories (e.g., ['한식', '국밥'])
      • Preferred tastes (e.g., ['매운맛', '얼큰한'])
      • Avoid tastes (e.g., ['느끼한'])
      • Texture preferences (e.g., ['국물'])
    6. Vector Search: ChromaDB searches menu/place embeddings using cosine similarity against the user profile, retrieving top k*2 candidates.
    7. Batch Embedding Processing: Menu embeddings processed in batches (batch_size=32) for optimization.
    8. Parallel Scoring: ThreadPoolExecutor-based concurrent menu scoring for improved performance.
    9. Hybrid Scoring & RL Weights: Strategy pattern applies scoring with per-user RL weight vectors:
      • Text similarity (default 65%)
      • Popularity (default 20%)
      • Distance (default 10%)
      • Price (default 5%)
      • Freshness, query similarity, taste alignment (learned weights)
      • Penalties: Allergen/dislike detection in menu items
    10. MMR Reranking: Maximal Marginal Relevance algorithm ensures diversity by balancing relevance (λ=0.7) and dissimilarity to already selected items.
    11. Explanation Generation: GPT generates human-readable Korean explanations based on MenuReasonFeatures (semantic similarity, category match, taste alignment, etc.)
    12. Interaction Logging: User interactions (click, scrap, hide) are logged to UserInteraction model for RL weight updates.
    13. Streaming Delivery: Phase 1 delivers menu recommendations immediately, Phase 2 generates explanations progressively via NDJSON streaming.

  • API Endpoints:

    • POST /api/v1/recommendation/recommend/menu/: Returns personalized menu recommendations (streaming NDJSON with Phase 1/Phase 2 architecture)
    • POST /api/v1/recommendation/recommend/place/: Returns personalized restaurant recommendations
    • GET /api/v1/recommendation/health/: Health check endpoint
    • POST /api/v1/interactions/log_click/: Log menu click interaction
    • POST /api/v1/interactions/log_scrap/: Log scrap interaction
    • POST /api/v1/interactions/log_hide/: Log hide interaction

  • Request requires: user_location (lat/lon), optional query_text, max_results (default 20)

  • Response includes: menu/place details, score (0-1), and human-readable reason for recommendation

  • Streaming: Phase 1 delivers menu recommendations immediately, Phase 2 generates explanations progressively

  • Performance: Parallel scoring and batch embedding processing for faster response times

6.2.4 Food Record & AI-Based Analysis

  • As a user who wants to track my eating history,
    • I can import existing food photos from my device's gallery, and the app automatically filters food images using on-device ML Kit classification powered by a TensorFlow Lite model.
    • Only photos identified as food are uploaded to AWS S3 and sent to the server, protecting my privacy and saving bandwidth.
    • The system generates semantic embeddings (768-dimensional vectors using ko-sbert-sts) for each food photo to enable similarity-based search and personalized recommendations.
    • All of my meal records are stored chronologically with their embeddings, allowing the recommendation engine to understand my eating patterns and preferences.
    • I can view my food gallery in the Profile tab, organized by date with AI-generated tags.
    • My gallery's category diversity automatically influences my exploration preference for recommendations.

Acceptance Criteria:

  • When the user selects "Import from Gallery," the app requests photo library permission using Expo MediaLibrary with granular permissions.
  • The app scans all albums (including smart albums) and processes images in batches.
  • ML Kit Image Labeling's on-device classifier (using custom TensorFlow Lite model "foodNonFood") determines if each image contains food.
  • Only images classified as food are uploaded to AWS S3 under path: protected/{identityId}/album/{albumTitle}/{filename}
  • Successfully uploaded image URLs are sent to backend API endpoint POST /api/v1/photos/ with image bytes, user authentication, and local URI.
  • The server uses CLIP + foodlist.json matching to generate:
    • Primary AI label (ai_label)
    • Top 5 alternative labels with confidence scores (label_alternatives)
    • Confidence score (label_confidence)
  • The server generates 768-dimensional embeddings using the Korean sentence transformer model (jhgan/ko-sbert-sts).
  • Embeddings are stored in the UserGalleryImage model with fields: image_url, ai_label, category_tag, label_alternatives, label_confidence, embedding, created_at, local_uri.
  • Users can manually edit labels, which is tracked via label_manually_edited and original_ai_label fields.
  • All processing must occur asynchronously without blocking the UI, with progress feedback shown to the user.
  • Classification errors are logged but do not halt the batch process; failed images are skipped.
  • The imported food history is immediately available to the recommendation engine for personalized suggestions.
  • Gallery category diversity is calculated using entropy formula to derive exploration_preference (0-5 scale).
  • Gallery category preferences are automatically computed via gallery_category_service.py and integrated into recommendation scoring.
  • Gallery import includes real-time progress tracking with photo count updates.
  • Permission management provides graceful UX for denied access scenarios.
  • Batch processing handles up to 100+ images concurrently with progress feedback.
  • Image classification errors are handled gracefully without stopping the import process.
  • Users receive visual feedback for each step: scanning → classifying → uploading → processing.

6.2.5 User Experience Enhancements

  • As a user,
    • When I log out, I receive clear confirmation with information about data preservation, ensuring I understand what happens to my scraped restaurants.
    • I can see streaming recommendations load progressively, providing immediate feedback while the full results load.
    • I receive toast notifications for successful actions (scraping, account changes) for clear interaction feedback.
    • My scraped restaurants display rich metadata including category badges, ratings, and scrapped dates for easy identification.

Acceptance Criteria:

  • Logout confirmation modal displays in Korean with explanation of scrap preservation
  • Streaming recommendations show loading states and progressive result population
  • Toast notifications appear for all major user actions with appropriate messaging
  • Scraped menu cards display category badges, ratings, prices, and temporal information
  • All modal interfaces follow consistent design patterns with proper Korean localization

6.3 User Acceptance Test

We will use 5 following user stories for the user acceptance test:

  • Register & Authenticate
  • Onboarding (Taste Profile Setup)
  • Integrating Album Image
  • Food Recommending
  • Menu Scraping

7. Non-functional Requirements

Category Description
Performance Streaming recommendations deliver first results within 4-5 seconds via Phase 1, complete results within 10-15 seconds on average. Parallel scoring and batch embedding processing improve response times.
Mobile Experience Full-screen card interface optimized for mobile interaction patterns. Progressive loading with smooth animations. Loading indicators provide user feedback.
Session Management Scraped menu persistence across logout/login cycles. Smart data clearing preserves user scraps while clearing sensitive data. Exit confirmation modal for app termination.
AI Performance Food photo classification completed within 3-5 seconds per image using on-device ML Kit with TensorFlow Lite model. Vector similarity search (768-dim embeddings) returns results within 1-2 seconds using ChromaDB. Gallery import processes images in batches. NLP intent extraction completes within 500ms. Batch embedding processing (batch_size=32) optimizes menu embedding generation.
Recommendation Quality Hybrid scoring combines 7 factors with RL-optimized weights. MMR reranking ensures diversity (λ=0.7). Explanations generated for each recommendation. Gallery and scrap category preferences automatically integrated into scoring.
Scalability Backend designed to handle multiple concurrent users. Performance optimizations (parallel scoring, batch processing) support reasonable load.
Reliability System designed for stable operation during development and testing phases. Error handling and graceful degradation implemented.
Usability Main functions accessible within 2-3 taps. Natural language search available.
Security HTTPS communication, Django session-based authentication, secure password hashing.
Data Privacy Gallery photos processed locally on device. Only embeddings and metadata sent to server.
Maintainability Modular code architecture by domain (auth, feed, recommendation). Strategy pattern for scoring algorithms.
Portability Compatible with Android. EAS production builds configured for Android deployment.
Offline Support Basic profile viewing works with cached data. Full functionality requires internet connection.
Testing Backend test coverage maintained above 80%. Comprehensive unit and integration tests for critical paths.
Personalization RL weight vectors adapt to user behavior over time. Recommendations improve with usage.

8. User Interface Requirements

8.1 Workflow

Foodigram uses a 2-tab bottom navigation after login:

  1. Recommendation Tab – personalized restaurant suggestions
  2. Profile Tab – user's preferences, bookmarks, and history

Workflow Steps

  1. Launch app → Login or Register
  2. Onboarding: Set taste preferences and allergens (new users only)
  3. Main screen with bottom toolbar appears
  4. Navigate between tabs for discovery and personal management
  5. Access restaurant or record detail pages from any tab
  6. Settings and logout accessible from Profile tab

8.2 Views

Initialize

  • Splash screen checks login token
  • Redirects to Main or Login page

Login / Register

  • Email and password fields
  • "Sign up" option for new users
  • Auto-login toggle

Onboarding (New Users)

  • Step 1: Taste preferences (spicy/sweet/salty levels 0-10)
  • Step 2: Select allergens from predefined list
  • Step 3: Select disliked ingredients
  • Step 4: Choose favorite cuisines
  • Progress bar showing 4 steps
  • Skip option available

Recommendation Tab (Foodigram Screen)

  • Top: context summary with location status
  • Search bar: Natural language query input with modal interface (e.g., "얼큰한 순대국밥")
  • Main: Full-screen card interface with streaming recommendations
  • Loading: Progressive result display as data streams in with enhanced loading spinners
  • Buttons: "Filter," "Regenerate"
  • Gesture: swipe to skip (logged as "hide"), tap for details (logged as "click")
  • Scrap button with toast notification feedback
  • Menu items clickable to open restaurant location in Naver Map

Profile Tab

  • Profile picture, username, short bio
  • Sections: Gallery History (food photos with AI labels), Bookmarks (Scraps), Preferences
  • Gallery images display with AI labels; users can edit labels
  • Scraps section shows saved restaurants with quick access
  • Buttons: Edit Profile / Edit Preferences / Logout / Delete Account

Gallery Import View

  • Header: "Import Food Photos from Gallery"
  • Permission status indicator
  • Album list with photo counts
  • Progress bar during scanning (shows "Analyzing X/Y photos")
  • Real-time counter: "Found N food photos"
  • Preview thumbnails of detected food images
  • Actions: Start Import / Cancel / View Results
  • Background processing indicator
  • Success/error notifications

Record Detail View

  • Photos, tags, captions, timestamps
  • Options to edit or delete

Restaurant Detail View

  • Info: menu, category, distance
  • AI-generated recommendation reason displayed
  • Actions: Add Bookmark (Scrap) / View on Map

Scrap List View

  • Grid layout of saved restaurants with enhanced metadata
  • Each card shows: restaurant name, category badge, price, rating, scraped date
  • Category badges sized appropriately for mobile interface
  • Quick actions: Remove from scraps, View details
  • Persistent storage across app sessions and logout cycles
  • Menu items clickable to open restaurant location in Naver Map
  • Empty state with prompt to explore recommendations

9. Class Diagram

9.1 Overview

The Foodigram data model connects users, restaurants, and recommendation logic through modular and relational entities.

9.2 Class Specifications

User / Profile

  • User: Basic account information (extends Django AbstractUser).
    • Additional fields: email (unique), nickname, created_at
  • Profile: Stores user bio and general preferences.
    • preferences: JSON (e.g. { "bio": "Food lover", "theme": "dark" })
  • Relations: One-to-one (User ↔ Profile), One-to-many (User ↔ Record, Bookmark, GalleryImage, Interaction)

UserPreference

  • Separate from Profile, stores structured onboarding data:
    • spicy_level, sweet_level, salty_level: 0-10 scale (PositiveSmallIntegerField)
    • exploration_preference: 0-5 scale (FloatField, 0=familiar, 5=adventurous)
    • allergies: JSON array (e.g. ["peanut", "shellfish"])
    • disliked_ingredients: JSON array (e.g. ["cilantro", "mushroom"])
    • favorite_cuisines: JSON array (e.g. ["korean", "italian"])
    • rl_weight_vector: JSON array of 7 RL-optimized weights for personalized scoring
      • Format: [text_sim, popularity, distance, price, freshness, query_sim, taste_align]
      • Default: [0.65, 0.20, 0.10, 0.05, 0.10, 0.0, 0.0]
  • Relations: One-to-one with User

UserGalleryImage

  • Stores imported gallery photos with AI analysis and embeddings
    • image_url: AWS S3 URL (path: protected/{identityId}/album/{albumTitle}/{filename})
    • local_uri: Local file path/URI for reference
    • ai_label: Primary classification result from CLIP + foodlist.json matching (e.g., "치킨", "단팥빵")
    • original_ai_label: Original AI-inferred label before user edits
    • category_tag: Broader category for grouping (e.g., "noodles", "dessert")
    • label_alternatives: JSON array of top 5 alternative labels with confidence scores
      • Format: [{"name": "치킨", "confidence": 0.95}, ...]
    • label_confidence: Confidence score (0.0-1.0) of the primary label
    • label_manually_edited: Boolean flag indicating if user manually edited the label
    • label_edited_at: Timestamp when label was manually edited
    • embedding: JSON array of 768-dimensional vector from ko-sbert-sts model
    • created_at: Timestamp for chronological ordering
  • Processing Flow:
    1. ML Kit Image Labeling classifies image on-device using custom TensorFlow Lite model (foodNonFood)
    2. If classified as food, upload to AWS S3
    3. Server receives image URL, local URI, and image bytes via API
    4. Server uses CLIP + foodlist.json to generate primary label, alternatives, and confidence
    5. Server generates 768-dim embedding using ko-sbert-sts
    6. All data stored in UserGalleryImage model for semantic search
  • Relations: Many-to-one with User
  • Indexes:
    • Composite index on (user_id, created_at) for efficient chronological queries
    • Composite index on (user_id, category_tag) for category-based filtering
  • Usage:
    • Recommendation engine queries user's gallery embeddings to understand food preferences
    • Category diversity entropy calculation for exploration_preference
    • Users can view and edit labels in Profile tab

UserScrap

  • User-saved/bookmarked restaurants
    • Unique constraint: one user can only scrap same restaurant once
  • Relations: Many-to-one with User and Restaurant
  • Indexes: Composite index on (user_id, restaurant_id)
  • Usage: Scraps are analyzed to derive preferred categories for recommendations

UserRemoteScrap

  • Stores complete scrap data for remote sync (cross-device restoration)
    • menu_id: External menu ID (string)
    • menu_name: Name of the menu item
    • place_name: Name of the restaurant
    • price: Menu price (nullable)
    • category: Food category
    • location: Location string
    • rating: Restaurant rating (nullable)
    • review_count: Number of reviews (nullable)
    • image_url: Menu image URL
    • coordinates: JSON array of [latitude, longitude]
    • scrapped_at: Original scrap time from client
    • synced_at: When synced to remote server
  • Relations: Many-to-one with User
  • Constraints: Unique constraint on (user_id, menu_id)
  • Indexes:
    • Composite index on (user_id, synced_at)
    • Composite index on (user_id, menu_id)
  • Usage: Enables cross-device scrap synchronization via AWS storage

UserInteraction

  • NEW: Logs user interactions for RL-based recommendation tuning
    • user: ForeignKey to User
    • menu_id: Integer ID of menu from recommendation system
    • restaurant_id: Integer ID of restaurant
    • interaction_type: TextChoices enum
      • scrap: User saved/scrapped (+1.0 reward)
      • click: User clicked/opened (+0.5 reward)
      • view: User viewed (+0.1 reward)
      • hide: User swiped to hide (-1.0 reward)
      • expand: User expanded detail page (+0.3 reward)
      • allergic_reaction: User reported negative reaction (-2.0 reward)
    • reward_value: Float reward signal for RL
    • context_query: Text of natural language query that led to interaction
    • timestamp: Auto-generated timestamp
  • Relations: Many-to-one with User
  • Indexes:
    • Composite index on (user_id, timestamp)
    • Composite index on (user_id, interaction_type)
    • Index on timestamp

RLWeightHistory

  • NEW: Stores RL weight vector history for analysis
    • user: ForeignKey to User
    • weights: JSON array of 7 weights
    • update_cycle: Integer cycle number (0, 1, 2, ...)
    • created_at: Auto-generated timestamp
  • Relations: Many-to-one with User
  • Usage: Track weight evolution for debugging and analysis

Restaurant

  • Core restaurant DB with attributes like name, address, latitude, longitude
    • phone: Contact information
    • image_url: Restaurant photo
    • source: Unique external ID (e.g. Naver Place ID "12345678")
  • Relations: One-to-many with MenuCandidate, UserScrap, Many-to-many with Menu through RestaurantMenu
  • Indexes: Index on name, unique constraint on source

Menu

  • Normalized menu items
    • category: Menu category (e.g. "main", "dessert", "beverage")
    • description: Menu description
    • image_url: URL to menu item photo
    • embedding: Vector embedding for semantic search (JSON array)
    • tag_name: Classification tags
  • Relations: Many-to-many with Restaurant through RestaurantMenu
  • Indexes: Composite index on (name, category)

MenuCandidate

  • Staging table for scraped menu data before normalization
    • price: Decimal field for pricing
    • embedding: Vector for recommendation engine
    • created_at: Import timestamp
  • Relations: Many-to-one with Restaurant
  • Indexes: Composite index on (restaurant_id, name)

RestaurantMenu

  • Junction table for Restaurant-Menu many-to-many relationship
  • Constraints: Unique constraint on (restaurant_id, menu_id) pairs
  • Indexes: Composite index on (restaurant_id, menu_id)

Record & Photo

  • Record: User's meal logs (restaurant, time, category).
  • Photo: Multiple per record, AI auto-tags with food type.

RecommendationResult

  • Stores generated recommendations with contextual data
    • user: ForeignKey to User
    • query_text: User's search query (optional)
    • input_context: JSON with context data
      • Example: {"weather": "rainy", "time": "lunch", "location": [37.5, 126.9]}
    • recommended_menu_ids: JSON array of Menu IDs
    • scores: JSON array of corresponding scores
    • created_at: Timestamp for caching
  • Relations: Many-to-one with User, One-to-many with RecommendationRanking
  • Indexes: Composite index on (user_id, created_at)

RecommendationRanking

  • Detailed ranking information for each recommendation
    • recommendation: ForeignKey to RecommendationResult
    • menu: ForeignKey to Menu
    • rank: Position in recommendation list (1, 2, 3...)
  • Constraints: Unique constraint on (recommendation_id, menu_id)
  • Indexes: Composite index on (recommendation_id, rank)

EmbeddingCache

  • Performance optimization layer
    • Caches computed embeddings for users, menus, images, restaurants
    • entity_type: TextChoices enum (user/menu/image/restaurant)
    • entity_id: BigInteger ID of the entity
    • embedding: JSON array of vector
  • Constraints: Unique constraint on (entity_type, entity_id)
  • Indexes: Composite index on (entity_type, entity_id)
  • Relations: None (independent cache layer)

MenuReasonFeatures

  • NEW: Stores computed reason features for explainable recommendations
    • menu: ForeignKey to Menu
    • restaurant: ForeignKey to Restaurant
    • user: ForeignKey to User
    • Feature scores (0.0-1.0):
      • semantic_similarity: Text similarity score
      • image_similarity: Visual similarity score
      • category_match_score: Category preference match
      • taste_alignment: User taste preference alignment
      • query_alignment: Alignment with natural language query
      • temporal_fit_score: Time-based fit (breakfast/lunch/dinner)
      • distance_score: Distance appropriateness
      • popularity_score: Restaurant popularity/rating
    • Penalties:
      • allergy_penalty: Allergy conflict penalty
      • dislike_penalty: Disliked ingredient penalty
    • Generated content:
      • explanation: GPT-generated Korean explanation
      • explanation_reason_keys: JSON array of top 3 contributing reason keys
    • final_score: Final recommendation score
    • query_context: User's natural language query
    • created_at: Timestamp
  • Relations: Many-to-one with Menu, Restaurant, User
  • Indexes:
    • Composite index on (user_id, created_at)
    • Composite index on (menu_id, user_id)
    • Composite index on (restaurant_id, user_id)

MenuExternalMapping

  • NEW: Links Django Menu to external PostgreSQL UUID
    • menu: OneToOne with Menu
    • external_uuid: Unique 36-char UUID string
    • created_at: Timestamp
  • Constraints: Unique on external_uuid
  • Indexes: Index on external_uuid

RestaurantExternalMapping

  • NEW: Links Django Restaurant to external PostgreSQL UUID
    • restaurant: OneToOne with Restaurant
    • external_uuid: Unique 36-char UUID string
    • created_at: Timestamp
  • Constraints: Unique on external_uuid
  • Indexes: Index on external_uuid

DbRestaurant

  • External PostgreSQL database restaurant model (separate from Django Restaurant)
    • id: UUID primary key (not Django auto-increment)
    • external_id: Unique external identifier
    • name: Restaurant name (text field)
    • category: Restaurant category
    • phone: Contact phone number
    • address: Full address
    • road_address: Road-based address
    • group1, group2, group3: Hierarchical location grouping
    • category_code: Category code
    • category_code_list: Array of category codes
    • geom: PostGIS geometry field (stored as text in Django)
    • place_images: Array of image URLs
    • avg_rating: Average rating (decimal)
    • review_count: Number of reviews
    • category_normalized: Normalized category name
    • meaningful_name: Boolean flag
    • inferred_menu: Inferred menu information
    • embedding_vector: Array of floats for semantic search
    • created_at, updated_at: Timestamps
  • Relations: One-to-many with DbMenu
  • Usage: Represents restaurant data from external PostgreSQL database

DbMenu

  • External PostgreSQL database menu model (separate from Django Menu)
    • id: UUID primary key (not Django auto-increment)
    • external_id: External identifier
    • name: Menu name (text field)
    • price: Menu price (integer)
    • description: Menu description
    • images: Array of image URLs
    • recommend: Boolean recommendation flag
    • index_in_rest: Index within restaurant
    • name_clean: Cleaned menu name
    • taste_profile: JSON taste profile data
    • allergen_info: JSON allergen information
    • embedding_vector: Array of floats for semantic search
    • restaurant_id: ForeignKey to DbRestaurant (UUID)
    • created_at, updated_at: Timestamps
  • Relations: Many-to-one with DbRestaurant
  • Usage: Represents menu data from external PostgreSQL database

9.3 Integration Summary

Functionality Involved Classes Description
Recommendation System RecommendationResult, UserPreference, UserGalleryImage, EmbeddingCache, Menu, Restaurant, MenuReasonFeatures, UserInteraction Generates personalized suggestions using vector embeddings, user preferences, gallery analysis, RL weights, and interaction history.
Profile View Profile, UserPreference, UserScrap, UserRemoteScrap, Record Combines user info, preferences, and activity history.
Record Management Record, Photo, UserGalleryImage Manages uploaded meal history and visual data.
Gallery Analysis UserGalleryImage, EmbeddingCache AI-powered food photo analysis and embedding generation.
Restaurant Discovery Restaurant, Menu, MenuCandidate, RestaurantMenu Structured data for restaurant and menu information.
Explainable Recommendations MenuReasonFeatures, UserPreference Stores reason features and generates GPT explanations.
RL-based Personalization UserInteraction, RLWeightHistory, UserPreference Logs interactions, tracks weight history, stores learned weights.
External Data Sync MenuExternalMapping, RestaurantExternalMapping, DbRestaurant, DbMenu Links internal IDs to external PostgreSQL UUIDs and manages external database models.
Cross-Device Sync UserRemoteScrap Enables scrap data synchronization across devices via AWS storage.

9.4 Diagram

alt

Clone this wiki locally