Food analyzer app

Food analyzer app is a personalized GenAI nutritional webapp for your shopping and cooking recipes built with serverless architecture and generative AI capabilities. It was first created as the winner of the AWS Hackathon France 2024 and then introduced at several AWS Summit.

• Customers use their cell phone to scan a bar code of a product to get the explanations of the ingredients and nutritional information of a grocery product personalized with your allergies and diet.
• Customers can also take a picture of food products and discover 3 personalized recipes based on their food preferences.

The back-end of the app is made using AWS services such as:

• Amazon Cognito
• AWS LambdaURL
• Amazon DynamoDB
• Amazon S3
• Amazon Bedrock

The app is designed to have minimal code, be extensible, scalable, and cost-efficient. It uses Lazy Loading to reduce cost and ensure the best user experience.

We developed this exhibit to create an interactive serverless application using generative AI services.

Features overview

• Personalized product information: Curious about what is in a product and if it is good for you? Just scan the barcode with the app for an explained list of ingredients/allergens and a personalized summary based on your preferences, health goals, and dietary restrictions. The app provides direct allergen detection and quantitative nutritional analysis using data from Open Food Facts.

• Personalized recipe generator: Capture multiple photos of ingredients in your fridge and pantry, and the app will generate recipes based on your preferences using those ingredients.

Demo

Table of Contents

• Architecture
• Features
  • Personalized product information
  • Personalized recipe generator
• Key Functional Highlights
• Key Technical Features
• License

Architecture

The architecture of the application can be split in 4 blocks:

• Frontend and Auth
• Product Management
• Product Summary
• Recipe Generator

Frontend and Auth:

• Implementation: Using Amazon CloudFront, Amazon S3 for the static website, AWS IAM, Amazon Cognito for authentication.
• Lambda URL Functions: AWS Lambda ensured serverless computing, optimizing costs and enhancing overall performance.
• To protect the LambdaURLs, we used Amazon CloudFront secured with Lambda@Edge to authenticate with Amazon Cognito and signed requests.

Product Management:

• Implementation: Using AWS Lambda for server-side logic and a database from Open Food Facts accessed through APIs.
• Data Integration: The app retrieves allergen tags and nutritional data (calories, sugars, fats, proteins, etc.) from Open Food Facts API for accurate, data-driven recommendations.
• Product Thumbnails: Real product images are displayed as 60x60px thumbnails next to product names, sourced directly from Open Food Facts database for visual product identification.
• Safety Features: Direct allergen detection from API data ensures reliable allergen warnings without relying solely on ingredient text parsing.

Product Summary and Generative Recipe:

• Implementation: Using AWS Lambda for server-side logic, Amazon Bedrock as a generative artificial intelligence (GenAI) building platform, Anthropic Claude as Large Language Models (LLM) and Amazon Nova Canvas for generating images.

• AI Model Development: Choosing the LLM model had an impact on response quality and latency. Ultimately, we chose Anthropic Claude 3 Haiku as a good ratio between latency and quality.

• AI-Generated Images: Prompting for an image is very sensitive and was a challenge to generate an image that truly highlights the nutritive features of products. To craft the prompt, we used a first LLM to generate the prompt based on product nutritive features. This technique is similar to a self-querying for vector databases. Using multi-shot prompt-engineering also helped a lot to improve the quality of the prompt. Images are generated using Amazon Nova Canvas.

Features

Personalized product information

Bar code scanning

• Strategy: "Do not reinvent the wheel"

• Implementation: Bar code scanners have been there for a long time, we picked an open source library that was compatible with our stack. With time constraint, we did not spend much time on library comparison, focusing on having a working prototype rather than finding the perfect one. "Perfect is the enemy of good".

Summary

• Strategy: Acknowledging the diversity of user preferences and dietary needs, our app incorporates a robust personalization feature. Beyond providing raw data, the app aims to educate users about the nutritional implications of their choices.

• Implementation: Users feel a sense of ownership and connection as the app tailors its insights to align with their individual health goals and dietary constraints. Incorporating concise and informative content within the app ensures that users understand the significance of various nutritional components. This educational aspect transforms the app into a learning tool, fostering a deeper connection with users seeking to enhance their nutritional literacy.

AI-Generated Image as visual cues

• Strategy: To captivate users' attention and communicate key nutritional information effectively, our app employs AI-generated images.

• Implementation: Amazon Bedrock offers an out-of-the-box developer experience in generating visually striking representations of scanned products. If a product contains excessive sugar, for example, the AI image surrounds it with a visual depiction of sugar, serving as an engaging and memorable visual cue.

Personalized recipe generator

Food aliment detection

• Strategy: Extract ingredients from multiple images, works well on fruits and vegetables. Users can capture their entire fridge and pantry to enable comprehensive meal planning.

• Implementation: We use Anthropic Claude 3 Sonnet on Amazon Bedrock with its vision capabilities to extract only food elements from the images. This allows us to focus on the food elements and ignore the background or other elements in the images. Claude 3 is a multi-modal model that can handle both text and images. The output is a list of ingredients present across all captured images. The backend processes multiple images via the list_images_base64 array parameter.

• Prompt Engineering: To exploit the full potential of the model, we use a system prompt. A system prompt is a way to provide context, instructions, and guidelines to Claude before presenting it with a question or task. By using a system prompt, you can set the stage for the conversation, specifying Claude's role, personality, tone, or any other relevant information that will help it to better understand and respond to the user's input.

  system_prompt="You have perfect vision and pay great attention to ingredients in each picture, you are very good at detecting food ingredients on images"

Recipe proposal

• Strategy: Generate 3 recipes from ingredients matched in the pictures:

  • Easy recipe with few ingredients and made in less than 15 minutes
  • Medium recipe with more ingredients and intermediate complexity
  • Hard recipe with even more ingredients and high complexity. Each recipe must also respect the user's dietary restrictions and allergies.

• Implementation: We use Claude 3 Sonnet to generate the 3 recipes. Each recipe contains the following JSON information:

{
      "recipe_title": "Succulent Grilled Cheese Sandwich",
      "description": "A classic comforting and flavorful dish, perfect for a quick meal",
      "difficulty": "easy",
      "ingredients": ["bread", "cheese", "butter"],
      "additional_ingredients": ["ham", "tomato"],
      "preparation_time": 5,
      "cooking_time": 6
}

• Image Generation: Once we have the recipe title and description, we use Amazon Nova Canvas to generate the image of the recipe. We generate images in parallel to reduce the latency of the response.

Step by step recipe

• Strategy: Generate a step by step recipe for the user to follow.

• Implementation: We use Anthropic Claude 3 Haiku on Amazon Bedrock to generate the step by step recipe. The steps are streamed to the user to reduce the latency of the response, we leverage lambda URL streaming. This method ensures smoother access to text content, improving user experience and interaction.

The output format is a Markdown file to faciliate the display of the recipe on the website. It also greatly simplifies the parsing of the recipe by the front-end when using streaming mode.

Key Functional Highlights

Picking the Right AI Model

• Challenge: The selection of the Language Model (LM) significantly influenced both response latency and quality, posing a critical decision point.

• Solution: Following a comprehensive assessment of various models, we've chosen the following Anthropic Claude models for different components within the app:

  • Barcode scanning image generation: Utilizing Amazon Nova Canvas.
  • Recipe fridge photograph food aliment detection: Utilizing Anthropic Claude 3 Sonnet to extract food ingredients from images.
  • Recipe proposals, Product ingredients: Leveraging Anthropic Claude 3 Sonnet. Sonnet was selected for cases where the output needed to be parsed for other tasks or displayed using HTML components.
  • Product summary, and recipe steps: Utilizing Anthropic Claude 3 Haiku was favored for enhancing user experience by displaying results in streaming mode, and because the output was solely intended for display, enabling us to designate the output type as markdown.

Use Lazy Loading to reduce cost/bandwidth

• Challenge: Generative AI is costly in price and bandwidth. We wanted to be frugal and efficient when running in packed events.

• Solution: Using Lazy loading with the key being the hash of the prompt allow us to reduce cost and deliver response faster.

Prompt Engineering for AI Image Generation

• Challenge: Crafting a prompt for generating AI images that effectively highlighted nutritional features posed a nuanced challenge.

• Solution: Leveraging a two-step approach, we utilized a first LLM to generate a prompt based on product nutritional features. This self-querying technique (akin to a self-querying process for a vector database) was complemented by incorporating multi-shot prompting. This method significantly improved the quality and relevance of the generated images, ensuring that users received visually compelling representations of product features.

User Personalization Complexity

• Challenge: Integrating personalized dietary preferences and restrictions into the model presented complexity.

• Solution: To enhance the LLM's understanding, we dynamically incorporated a header in our prompt containing personalized allergy and diet inputs. This approach significantly improved the accuracy and relevance of the LLM's responses, ensuring a tailored experience for users. Personalized prompts became a cornerstone in delivering precise and relevant information based on individual preferences.

Multi language support

• Challenge: Present the application in multiple languages

• Solution: The same prompt is utilized, but the LLM is instructed to generate the output in a specific language, catering to the user's language preference (English, French, Spanish, Italian, Arabic). The UI includes comprehensive translations for all features including allergen warnings, ingredient descriptions, and nutritional information.

Direct Allergen Detection and Nutritional Analysis

• Challenge: Ensuring accurate allergen warnings and providing quantitative nutritional recommendations based on user health goals.

• Solution: Integrated Open Food Facts API to retrieve allergens_tags and nutriments fields. The app filters key nutritional data (calories, sugars, fats, proteins, salt, fiber) and stores them in DynamoDB. Product summaries now include:

  • Direct allergen detection with prominent warnings
  • Specific nutritional values (e.g., "539 kcal/100g", "56.3g sugars")
  • Health goal-specific recommendations (weight loss, muscle gain, etc.)
  • Dietary preference compatibility (keto, low carb, low sodium)

  A custom DecimalEncoder handles DynamoDB Decimal type serialization to JSON, ensuring proper data formatting in API responses.

Dietary Labels and Religious Requirements

• Challenge: Accurately identifying vegan, vegetarian, halal, and kosher products without relying solely on ingredient text parsing.

• Solution: Integrated labels_tags and categories fields from Open Food Facts API. The app now provides:

  • Direct vegan/vegetarian detection from product labels
  • Halal and kosher certification identification
  • Category-based product context for better recommendations
  • Religious requirement matching with clear certification status
  • Fallback to ingredient analysis when labels are unavailable

  This reduces LLM hallucination and provides more confident dietary and religious compatibility assessments.

Key Technical Features

AWS Lambda URL & Amazon CloudFront

• Challenge: Calling an LLM to execute some tasks in a request response mode can be slow.

• Solution: To address the 30-second timeout of an API Gateway endpoint, the selected approach involves utilizing an AWS Lambda URL via Amazon CloudFront. Within Amazon CloudFront, a Lambda@Edge function is triggered for each request, responsible for verifying user authentication against Amazon Cognito. If authentication is successful, the Lambda@Edge function signs the request to the Lambda URL (which utilizes AWS_IAM as the authentication method). While utilizing a Lambda URL offers a feasible solution, it's essential to acknowledge that opting for AWS AppSync presents an alternative offering additional benefits. AWS AppSync offers an alternative solution, featuring built-in authentication and authorization mechanisms that seamlessly meet these needs. However, for this demo app, we opted to use Lambda URL instead.

Amazon Bedrock synchronous/asynchronous response

• Challenge: Obtaining the response from Amazon Bedrock can occur either in a request/response mode or in a streaming mode, wherein the Lambda initiates streaming the response instead of awaiting the entire response to be generated.

• Solution: Some sections of the application operate in request/response mode (such as Product ingredients description or retrieving the three recipe proposals), while another part (Product summary, Getting the Step-by-Step Recipe) employs streaming mode to demonstrate both implementation methods.

Illustrated Use Cases of the GenAi Application

• Text generation - Amazon Bedrock utilizes Anthropic Claude 3 Haiku to generate the product summary.
• Text to image - Amazon Bedrock utilizes Amazon Nova Canvas to generate an image of the product.
• Image to text - Anthropic Claude 3 Sonnet is leveraged by Amazon Bedrock to identify food elements in the image.

Installation

npm install

Deploy

• For simplicity, deploy the stack in the us-east-1 region.

cdk deploy

Try it

Create a User

Before accessing the application, ensure you've established a user account in Amazon Cognito. To accomplish this, navigate to the AWS Console, then Amazon Cognito, and locate a user pool named something similar to AuthenticationFoodAnalyzerUserPoolXXX.

Access the App

Check the stack outputs for a URL resembling Food analyzer app.domainName. Paste this URL into your browser, log in with the previously created user, and start enjoying the app.

Running locally

You can run this vite react app locally following these steps.

1. Deploy infrastructure to AWS

Follow instructions above to deploy the cdk app.

2. Obtain environment configuration

Grab the aws-exports.json from the Amazon CloudFront distribution endpoint you obtained from the CDK Output, and save it into ./resources/ui/public/ folder.

The URL is something like:

https://dxxxxxxxxxxxx.cloudfront.net/aws-exports.json

3. Run local dev server

cd resources/ui
npm run dev

Requirements

• Node.js 18+ must be installed on the deployment machine. (Instructions)

• AWS CLI 2+ must be installed on the deployment machine. (Instructions)

Runtime Versions

The application uses the following AWS Lambda runtimes:

• Node.js 24.x - For TypeScript Lambda functions (product summary, recipe steps, authentication)
• Python 3.12 - For Python Lambda functions (barcode scanning, image analysis, recipe generation)

The Node.js functions were upgraded to Node.js 24 in December 2025 to leverage the latest features including improved HTTP performance (Undici 7), explicit resource management, and enhanced async context handling. See NODEJS_24_UPGRADE.md for details.

Resources

• A Journey into Crafting an Award-Winning Gen-AI Serverless App with AWS
• Building a personalized GenAI webapp | Serverless Office Hours YouTube | Twitch | LinkedIn

License

This project is licensed under the MIT-0 License. See the LICENSE file.