feat: transform raw image responses to json #1067
Conversation
There was a problem hiding this comment.
@unsync I have a few items in mind wrt this PR
- we should be handling transforms only for specific routes and not based on response, so if segmind has a
/v1/images/generationsroute, we can do these changes for that route - I'm not entirely keen we should load the entire image file into memory and convert it to json, but that's okay
overall I'm saying this PR could only include changes to provider/segmind/imageGenerate.ts and provider/segmind/api.ts and provider/segmind/index.ts
|
@narengogi if i get this correctly, you would prefer to scope the changes to the providers handlers. Unless i'm missing something, that would not work without a change in the reponseHandler because of this: gateway/src/handlers/responseHandlers.ts Lines 137 to 139 in de327f6 This prevents to reach the code that handles the transformations: gateway/src/handlers/responseHandlers.ts Lines 159 to 166 in de327f6 And before you hit gateway/src/handlers/streamHandler.ts Lines 248 to 250 in de327f6 Since the provider returns a content type Let me know if i missed something here 🙏 And if that doesn't makes it's way to |
|
I spent some time on this, I was initially misreading what the PR is doing, this fix makes sense for the problem, one change we could do is to convert to json only if routes are in the unified routes like I'm copying @VisargD on this for a review of the approach |
@narengogi @VisargD i made a last commit to the PR to, imho, limit the scope of the behaviour in a more elegant way: This ensures that :
I also added a try/catch the the transformer wrapper, so that if the transformation fails we return the original response to avoid any failures due to this logic |
unsync
changed the title
|
Summary By MatterAI
🔄 What ChangedThis PR adds functionality to transform raw image responses from providers (specifically Segmind) into JSON format. It introduces a new handler for converting image responses to base64-encoded JSON, enhancing the API's response consistency. The implementation includes fallback mechanisms to return the original response if transformation fails. 🔍 Impact of the ChangeThis change improves client experience by providing consistent JSON responses instead of raw binary image data. It makes it easier for clients to handle image generation responses without needing special handling for binary data. The transformation is specifically targeted at the imageGenerate route and only applies when appropriate conditions are met. 📁 Total Files Changed
🧪 Test AddedN/A - No tests were added in this PR. 🔒Security VulnerabilitiesN/A - No security vulnerabilities were identified. Type of Change
How Has This Been Tested?
Checklist
Sequence DiagramsequenceDiagram
participant Client
participant Gateway
participant ResponseHandler
participant ImageToJsonHandler
participant SegmindProvider
participant ExternalAPI
Client->>Gateway: Request image generation
Gateway->>ExternalAPI: Forward request to provider API
ExternalAPI-->>Gateway: Return raw image response
Gateway->>ResponseHandler: Process response
Note over ResponseHandler: Check if response is for imageGenerate
Note over ResponseHandler: Check if content-type is image
Note over ResponseHandler: Check if provider has imageToJson transformer
ResponseHandler->>SegmindProvider: Call imageToJson transformer
SegmindProvider->>ImageToJsonHandler: Call getTransformedResponse()
Note over ImageToJsonHandler: Convert image to base64
Note over ImageToJsonHandler: Transform to JSON format
ImageToJsonHandler-->>SegmindProvider: Return JSON Response
SegmindProvider-->>ResponseHandler: Return transformed response
ResponseHandler-->>Gateway: Return final response
Gateway-->>Client: Return JSON response instead of raw image
Note over Gateway: If transformation fails, original response is returned
|
Implements provider-specific response formatting for image data. Adds helper functions to identify image providers and format responses correctly based on the provider's requirements, ensuring proper handling of base64 image responses. Improves the robustness of the image handling logic to adapt to different provider responses.
Streamlines the image response transformation process by consolidating functions for image transformation into `getImageTransformer`. Removes unnecessary constants and simplifies the logic to parse image data, resulting in improved readability and maintainability. Enhances logging for better traceability during retry attempts by providing detailed response status information. No functionality changes are introduced; the focus is on code clarity and structure.
unsync
force-pushed
the
feature/transform-images-to-json
branch
from
7777f37 to
14b014f
Compare
|
DescriptionSummary By MatterAI
🔄 What ChangedThis pull request introduces a new feature to transform raw image responses from providers into a JSON format. A new handler 🔍 Impact of the ChangeThis change allows clients to receive image generation responses in a standardized JSON format, where the image data is base64 encoded. This is particularly useful for environments that prefer or require JSON payloads, simplifying client-side parsing and handling of image responses. It enhances the flexibility of the API by providing an alternative response format for image generation. 📁 Total Files Changed4 files were changed:
🧪 Test AddedNo explicit tests were added or mentioned in the pull request description. The PR is marked as "base for discussion/comments", suggesting it might be a work in progress or requires further testing. 🔒Security VulnerabilitiesNo security vulnerabilities were detected in the provided code changes. MotivationTo transform raw image responses to JSON format, embedding the image as a base64 string, for better client-side handling and standardization. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related Issues. Tip Quality Recommendations
Sequence DiagramsequenceDiagram
participant User
participant API_Endpoint
participant ResponseHandler
participant ProviderTransformers
participant SegmindImageToJsonResponseTransform
participant GetTransformedResponse
User->>API_Endpoint: Request Image Generation (e.g., POST /v1/image/generate)
API_Endpoint->>ResponseHandler: Call responseHandler(response: Response, provider: Object, ...)
ResponseHandler->>ResponseHandler: Check if responseTransformer is 'imageGenerate'
ResponseHandler->>ResponseHandler: Check if providerTransformers.imageToJson exists
ResponseHandler->>ResponseHandler: Check if response.headers['content-type'] starts with 'image/'
alt Image Transformation Conditions Met
ResponseHandler->>ProviderTransformers: Call providerTransformers.imageToJson(originalResponse: Response)
ProviderTransformers->>SegmindImageToJsonResponseTransform: Call SegmindImageToJsonResponseTransform(originalResponse: Response)
SegmindImageToJsonResponseTransform->>GetTransformedResponse: Call getTransformedResponse({response: originalResponse, transformer: (base64Image: string) => ({ image: base64Image })})
GetTransformedResponse->>GetTransformedResponse: Read originalResponse.arrayBuffer()
GetTransformedResponse->>GetTransformedResponse: Convert ArrayBuffer to base64 string (via Uint8Array, String.fromCharCode, btoa)
GetTransformedResponse->>GetTransformedResponse: Apply transformer function to base64 string
GetTransformedResponse-->>SegmindImageToJsonResponseTransform: Return new Response (JSON, status, headers)
SegmindImageToJsonResponseTransform-->>ResponseHandler: Return transformed JSON Response
else No Image Transformation
ResponseHandler->>ResponseHandler: Continue with original Response
end
ResponseHandler->>ResponseHandler: Read final response.headers['content-type']
ResponseHandler-->>API_Endpoint: Return {response: finalResponse, ...}
API_Endpoint-->>User: Return Final Response (JSON or original format)
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
Summary By MatterAI
🔄 What Changed
This PR adds functionality to transform image content type responses from Segmind to JSON format. It introduces a new handler for converting image responses to base64-encoded JSON, specifically targeting the imageGenerate route. The implementation includes error handling to fall back to the original response if transformation fails.
🔍 Impact of the Change
This change improves the handling of image responses by standardizing them to JSON format, making it easier for clients to process the responses. It's designed as a non-breaking change that enhances the existing functionality without disrupting current behavior.
📁 Total Files Changed
5 files changed with 67 additions and 1 deletion:
🧪 Test Added
No tests have been added in this PR.
🔒 Security Vulnerabilities
No security vulnerabilities detected.
Type of Change
How Has This Been Tested?
Screenshots (if applicable)
N/A
Checklist
Related Issues
N/A
Quality Recommendations
Add unit tests for the new image transformation functionality
Add more detailed error logging in imageToJsonHandler.ts with specific error types
Consider adding a content-type validation function to avoid repetitive checks
Add documentation for the new image transformation feature
Sequence Diagram
sequenceDiagram participant Client participant ResponseHandler participant RetryHandler participant ImageToJsonHandler participant SegmindProvider Client->>ResponseHandler: Request image generation ResponseHandler->>RetryHandler: Process request with retry logic RetryHandler->>SegmindProvider: Call Segmind API SegmindProvider-->>RetryHandler: Return image response RetryHandler-->>ResponseHandler: Return response Note over ResponseHandler: Check if response needs transformation alt responseTransformer === 'imageGenerate' && has imageToJson transformer ResponseHandler->>ImageToJsonHandler: getTransformedResponse(response, transformer) Note over ImageToJsonHandler: Convert image to base64 ImageToJsonHandler-->>ResponseHandler: Return JSON response end ResponseHandler-->>Client: Return final response