File System Abstraction

[ManishMadan2882]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
  - Adding abstraction for uploading, retrieving, processing, deleting files
  - Option for S3 and localStorage
  - Updated file attachment uploads
• Why was this change needed? (You can also link to an open issue here)
  - ease of swapping between storage(local/cloud)

[vercel]

@ManishMadan2882 is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

[codecov]

Codecov Report

Attention: Patch coverage is 28.66894% with 209 lines in your changes missing coverage. Please review.

Project coverage is 28.82%. Comparing base (3e1ec23) to head (3cd9a72).
Report is 27 commits behind head on main.

Files with missing lines	Patch %	Lines
application/worker.py	15.38%	55 Missing ⚠️
application/storage/s3.py	24.59%	46 Missing ⚠️
application/storage/local.py	29.16%	34 Missing ⚠️
application/api/user/routes.py	0.00%	29 Missing ⚠️
application/vectorstore/faiss.py	9.52%	19 Missing ⚠️
application/llm/openai.py	27.27%	8 Missing ⚠️
application/api/internal/routes.py	33.33%	6 Missing ⚠️
application/storage/base.py	71.42%	6 Missing ⚠️
application/llm/google_ai.py	25.00%	3 Missing ⚠️
application/agents/llm_handler.py	0.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1742      +/-   ##
==========================================
+ Coverage   28.35%   28.82%   +0.46%     
==========================================
  Files          85       89       +4     
  Lines        5273     5444     +171     
==========================================
+ Hits         1495     1569      +74     
- Misses       3778     3875      +97     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Charlesnorris509]

Hey @manish It seems like you are done yet with this, but I came across some issues that I think you should take into consideration, The cuurent S3 connection script is implementing Boto3.client() but if you added a config section you would be able to ensure that transmission use SSL for security and in transit encryption. You should also try to overwrite the current boto3 retry components in the process_files() portion to limit unnecessary file downloads

[dartpain]

Hey @Charlesnorris509
I thought boto3 uses SSL by default.

Maybe im mistaken, can you provide some links to documentation about it?

[Charlesnorris509]

@ManishMadan2882, the current version of Boto use SSL by default but since you've specified which version of the API you'll use, it might actually route it to an insecure endpoint. Here's the link for the documentation https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

The proper Implementation should like this 'from botocore.config import Config

self.s3 = boto3.client(
's3',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
config=Config(
signature_version='s3v4',
s3={'addressing_style': 'path'}
)
)

[ManishMadan2882]

@Charlesnorris509 ​In boto3 version 1.35.97, the default signature version used for signing AWS service requests is Signature Version 4 (s3v4).
https://boto3.amazonaws.com/v1/documentation/api/1.35.97/guide/configuration.html

[Charlesnorris509]

Then sorry, i should have checked the version, and i think you should be fine then

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-gpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 23, 2025 10:15pm