Presentation generator images

[dean2727]

Description

This PR introduces image generation capabilities into the existing (updated) presentation generator. This feature enhances the presentations generated for the teachers by reducing the need to find relevant and effective visuals for slides.

See Loom demo here: https://www.loom.com/share/7ebebeb92f2f41f9a6fdc5665e1fe8b5?sid=c9a0dc8c-aa1e-4742-a1dc-3d5212814215

Type of Change

Please select the type(s) of change that apply and delete those that do not.

• Bug fix: A non-breaking change that fixes an issue.
• New feature: A non-breaking change that adds functionality.
• Breaking change: A change that causes existing functionality to not work as expected.
• Documentation update: Changes or updates to documentation.
• Code style update: Changes that do not affect the meaning of the code (e.g., formatting).
• Refactoring: A code change that neither fixes a bug nor adds a feature.
• Performance improvement: A change that improves performance.
• Test enhancement: Adding or updating tests; no production code change.
• Chore: Changes to the build process or auxiliary tools; no production code change.
• Other: (please describe)

Proposed Solution

A new folder, image_generator, was created under presentation_generator_updated, which contains the new functionality (note: no new route was added to router.py, since the functionality is intended to be called in SlideGenerator. Hence, the image generator's executor exists only for tests).

On a given generated slide, 2 new attributes, needs_image and image_url, are determined. First, an LLM (Google Gemini 1.5 flash) determines needs_image, and if true, the image_url for the slide, which is a publicly accessible Google Cloud Storage link, is created on a successful image creation. The prompt for the image model is also dictated by an LLM (Google Gemini 1.5 flash) , using slide title and content. If image generation fails, it tries again 1 more time, and if the second time fails, a fallback image/URL is returned (note: this URL has yet to be determined). Errors and logs are handled gracefully and thoroughly on a given generate_slide_image() call to the new ImageGenerator.

There are 2 possible image generation models to use which can be configured in the code (though not passed in by the frontend): Flux 1.1 (default) and Imagen 3. Images are generated for slides synchronously, with around 3-3.3 seconds needed for each image (Flux model).

Explanation of design choices

Task	Technology	Reason
Image Generation	Imagen 3, Flux 1.1 (default)	Flux generation speeds are faster than imagen and displays text better
Backend API	FastAPI (Python)	Fast and scalable API framework
Image generation prompt (from slide content)	LLM	Covers the most amount of cases and avoids hardcoding values
Image generation classification	LLM	Circumvents the need to train a classification model, which requires many training examples and would be less accurate
Storage	Google Cloud Storage	Cheaper, scales better, serves files faster, and integrates with CDNs (compared to Firestore)

How to Test

To test the new unit tests:

1. cd into image_generator/tests/
2. Run pytest test_core.py

Unit Tests

Slide generator tests

test_executor() - Updated to handle new response format with "data" wrapper
test_executor_missing_inputs() - Unchanged
test_executor_loader_error() - Updated to expect ToolExecutorError
test_executor_unexpected_error() - Unchanged
test_validate_slides_content() - Unchanged
test_validate_slides_content_with_garbage() - Unchanged
test_validate_slides_content_empty_slides() - Unchanged
test_slide_generator_compile_context() - Updated to handle new chain tuple return
test_slide_model() - Updated to include needs_image and image_url fields
test_slide_presentation_model() - Updated to include needs_image and image_url fields
test_image_determination() - NEW - Tests the new image determination functionality

Image generator tests

Prompt Construction Tests:
test_construct_image_generation_prompt_string_content() - Tests prompt construction with string content
test_construct_image_generation_prompt_list_content() - Tests prompt construction with list content
test_construct_image_generation_prompt_dict_content() - Tests prompt construction with dictionary content
test_construct_image_generation_prompt_error_handling() - Tests error handling in prompt generation

Image Generation Tests:
test_generate_slide_image_flux() - Tests image generation with flux model
test_generate_slide_image_imagen() - Tests image generation with imagen model
test_generate_slide_image_retry() - Tests retry mechanism on failure
test_generate_slide_image_all_attempts_fail() - Tests fallback to placeholder when all attempts fail

Executor Function Tests:
test_executor_successful() - Tests successful execution
test_executor_missing_inputs() - Tests handling of missing required inputs
test_executor_loader_error() - Tests handling of LoaderError
test_executor_general_exception() - Tests handling of general exceptions

Documentation Updates

Indicate whether documentation needs to be updated due to this PR.

• [] Yes
• No

Checklist

• I have performed a self-review of my code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• Any dependent changes have been merged and published in downstream modules.

Additional Information

• A new environment variable, REPLICATE_API_TOKEN, must be set prior to testing (see here for token API page)
• Take note of the TODO comment, which mentions that, if this PR is approved, the fallback image for failed image generation would need to be added to GCS, with URL updated

[buriihenry]

Superb work. I know you are preloading the Json payload locally in your code for generating slide images. Could you test it on FastAPI via Submit tool?. This would be the best approach give how we have been testing other tools.

[dean2727]

Thank you, Henry! It has been tested and works.

[buriihenry]

Please share Json payload for testing?

[buriihenry]

Share the Json object for testing

[dean2727]

See the following payload:

{
    "user": {
        "id": "string",
        "fullName": "string",
        "email": "string"
    },
    "type": "tool",
    "tool_data": {
        "tool_id": "slide-generator",
        "inputs": [
            {
                "name": "instructional_level",
                "value": "intermediate"
            },
            {
                "name": "topic",
                "value": "Science"
            },
            {
                "name": "slides_titles",
                "value": ["Introduction: The Ever-Cycling Water", "Evaporation and Transpiration: Waters Journey Up", "Condensation and Precipitation: Waters Descent", "Collection and Storage: Where Water Resides", "The Water Cycle in Action: Real-World Examples (Interactive)", "Summary: The Importance of the Water Cycle"]
            },
            {
                "name": "lang",
                "value": "en"
            }
        ]
    }
}

[stevenrayhinojosa-gmail-com]

I've thoroughly tested your PR for the presentation generator image integration, and I'm impressed with the quality and robustness of your implementation. The code is well-structured, thoroughly tested, and includes excellent error handling.

Strengths
Comprehensive Implementation: Your image generator provides a complete solution with support for both Flux and Imagen models, giving flexibility in image generation approaches.
Excellent Error Handling: The implementation includes robust error handling with a retry mechanism and fallback to placeholder images when generation fails.
Well-Designed Architecture: The separation of concerns between prompt construction and image generation is clean and maintainable.
Thorough Test Coverage: The test suite is comprehensive, covering all aspects of the implementation including error cases and edge conditions.
Smart Prompt Engineering: Using Gemini to generate image prompts based on slide content is a clever approach that should produce better results than direct prompting.
Cloud Storage Integration: The integration with Google Cloud Storage for image hosting is well-implemented and includes proper error handling.
Detailed Documentation: The code is well-documented with clear docstrings and comments explaining the purpose and behavior of each component.
Areas for Improvement
Missing Metadata: The metadata.json file is empty, which would need to be completed before this can be properly registered as a tool.
Not Registered in tools_config.json: The image generator is not yet registered in the tools_config.json file.
No Integration with Slide Generator: While the image generator works well as a standalone tool, it's not yet integrated with the slide generator to automatically add images to slides.
Environment Variables: The implementation relies on environment variables (GCP_PROJECT_ID) that may not be set in all environments.
Recommendations
Complete Metadata: Add the required fields to the metadata.json file to properly register this as a tool.
Register in tools_config.json: Add an entry for the image generator in the tools_config.json file.
Integrate with Slide Generator: Consider adding code to the slide generator to automatically call the image generator for each slide.
Environment Variable Fallbacks: Add fallbacks or clear error messages when required environment variables are missing.
Add Documentation: Consider adding a README.md file explaining how to use the image generator and how it integrates with the presentation generator.
Conclusion
This is a high-quality PR that adds valuable functionality to the presentation generator. The image generation implementation is robust, well-tested, and should provide excellent results. With a few minor additions to complete the integration, this PR would be ready for merging.

Great work on implementing this feature! The attention to detail in error handling and the comprehensive test suite are particularly impressive.