[DRAFT] feat(uploadFiles): add endpoint to get presigned upload url #14943

v-tarasevich-blitz-brain · 2025-10-07T14:57:28Z

This PR includes:

Brings S3Utils from Saas to OSS
Adds additional method in S3Util to generate presigned upload url
Adds endpoint to get presigned upload url

codecov · 2025-10-09T13:14:30Z

Bundle Report

Changes will increase total bundle size by 38 bytes (0.0%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.59MB	38 bytes (0.0%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	38 bytes	18.93MB	0.0%

codecov · 2025-10-09T13:17:39Z

Codecov Report

❌ Patch coverage is 83.04094% with 29 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...java/com/linkedin/datahub/graphql/util/S3Util.java	84.46%	10 Missing and 6 partials ⚠️
...com/linkedin/datahub/graphql/GmsGraphQLEngine.java	0.00%	11 Missing ⚠️
...ava/com/linkedin/gms/factory/s3/S3UtilFactory.java	80.00%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

anshbansal · 2025-10-13T12:06:29Z

...rc/main/java/com/linkedin/datahub/graphql/resolvers/files/GetPresignedUploadUrlResolver.java

+      new HashSet<>(
+          Arrays.asList(
+              "pdf", "jpeg", "jpg", "png", "pptx", "docx", "xls", "xml", "ppt", "gif", "xlsx",
+              "bmp", "doc", "rtf", "gz", "zip", "mp4", "mp3", "wmv", "tiff", "txt", "md", "csv"));


comma separated config in application.yaml would make it easier to change this for anyone

Thanks, I created a followup issue for that - https://linear.app/acryl-data/issue/CH-844/move-list-of-available-file-extensions-to-applicationyml

chriscollins3456

awesome stuff so far

chriscollins3456 · 2025-10-13T19:59:01Z

...rc/main/java/com/linkedin/datahub/graphql/resolvers/files/GetPresignedUploadUrlResolver.java

+    if (scenario == UploadDownloadScenario.ASSET_DOCUMENTATION) {
+      return String.format("%s/product-assets/%s", bucketName, fileId);
+    } else {
+      throw new IllegalArgumentException("Unsupported upload scenario: " + scenario);
+    }


yeah and we'll come back to this later in order to support other scenarios like home page modules and the like.

i'm not positive we need to prefix with the bucket name here, but with local stack i'm seeing that's needed. it may be something we need to update once we deploy on a real instance with a real connection to s3

chriscollins3456 · 2025-10-13T20:02:00Z

datahub-graphql-core/src/main/resources/files.graphql

+  ASSET_DOCUMENTATION
+}
+
+type GetPresignedUploadUrl {


could rename to GetPresignedUploadUrlResponse

chriscollins3456 · 2025-10-13T20:02:16Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

+  private void configureFilesResolver(final RuntimeWiring.Builder builder) {
+    builder.type(
+        "GetPresignedUploadUrlResponse",
+        typeWiring ->
+            typeWiring
+                .dataFetcher(
+                    "latestVersion",
+                    new EntityTypeResolver(
+                        entityTypes, (env) -> ((VersionSet) env.getSource()).getLatestVersion()))
+                .dataFetcher(
+                    "versionsSearch",
+                    new VersionsSearchResolver(this.entityClient, this.viewService)));
+  }


i think this isn't necessary right?

chriscollins3456 · 2025-10-13T20:04:43Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/util/S3Util.java

+    this(s3Client, entityClient, null);
+  }
+
+  public S3Util(


let's double check @anshbansal has reviewed this file since he was the original implementer

chriscollins3456 · 2025-10-13T20:05:29Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/util/S3Util.java

+   * @param expirationSeconds The expiration time in seconds
+   * @return The pre-signed URL
+   */
+  public String generatePresignedDownloadUrl(


okay nice so you brought this over from my WIP branch

chriscollins3456 · 2025-10-13T20:06:18Z

docker/build.gradle

+            'quickstartDebugAws': [
+                    profile: 'debug-aws',
+                    modules: python_services_modules + backend_profile_modules + [':datahub-frontend', ':datahub-actions'],
+                    isDebug: true,
+                    additionalEnv: [
+                            DATAHUB_LOCAL_ACTIONS_ENV: "${rootProject.project(':smoke-test').projectDir}/test_resources/actions/actions.env"
+                    ]
+            ],


we want to keep this right? or not? i know you added it yourself for local dev purposes obviously

chriscollins3456 · 2025-10-13T20:07:13Z

metadata-service/factories/build.gradle

+  // AWS SDK for S3 file serving
+  implementation platform('software.amazon.awssdk:bom:2.23.6')
+  implementation 'software.amazon.awssdk:regions'
+  implementation 'software.amazon.awssdk:sts'
+  implementation 'software.amazon.awssdk:s3'


we will want to double check with at least one platform person whether this makes sense. i believe this was added for the S2UtilsFactory and we need that factory so we can us S3Utils in our new REST endpoint that will return a redirect to a download presigned url given a file ID. that will be in another PR.

github-actions bot added product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Oct 7, 2025

vercel bot deployed to Preview October 7, 2025 15:08 View deployment

vercel bot deployed to Preview October 7, 2025 17:00 View deployment

feat(uploadFiles): add endpoint to get presigned upload url

629229b

v-tarasevich-blitz-brain force-pushed the vt--ch-820-upload-files--add-upload-method branch from 3d2e071 to 629229b Compare October 9, 2025 13:01

vercel bot deployed to Preview October 9, 2025 13:20 View deployment

fix tests

1810481

vercel bot deployed to Preview October 9, 2025 16:21 View deployment

fix tests

26fb897

vercel bot deployed to Preview October 10, 2025 08:55 View deployment

v-tarasevich-blitz-brain added 4 commits October 10, 2025 12:28

add fileName, validate files extension

c698217

add some keys to docker and add fileId to response

a97f364

move new graphql things into separated file

0d3df07

add s3utilfactory

7e3e042

vercel bot deployed to Preview October 10, 2025 16:10 View deployment

anshbansal reviewed Oct 13, 2025

View reviewed changes

fix tests

e4993ee

vercel bot deployed to Preview October 13, 2025 12:52 View deployment

v-tarasevich-blitz-brain added 2 commits October 13, 2025 16:08

pass bucket name from config

618c378

fix and bump tests

cd5a976

vercel bot deployed to Preview October 13, 2025 15:35 View deployment

v-tarasevich-blitz-brain marked this pull request as ready for review October 13, 2025 17:13

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Oct 13, 2025

chriscollins3456 reviewed Oct 13, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Oct 13, 2025

[DRAFT] feat(uploadFiles): add endpoint to get presigned upload url #14943

Are you sure you want to change the base?

[DRAFT] feat(uploadFiles): add endpoint to get presigned upload url #14943

Uh oh!

Conversation

v-tarasevich-blitz-brain commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

codecov bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-tarasevich-blitz-brain Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chriscollins3456 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

v-tarasevich-blitz-brain commented Oct 7, 2025 •

edited

Loading

codecov bot commented Oct 9, 2025 •

edited

Loading

codecov bot commented Oct 9, 2025 •

edited

Loading

v-tarasevich-blitz-brain Oct 13, 2025 •

edited

Loading