feat: add support for processing of upload_v2 job type by slave #5796
+1,331
−477
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Master
loadfiles.go
loadfiles.go
for the integration tests to passstaging_file_id
to query load files. Created a new functiongetLoadFilesMetadataQuery
to query based on either upload_id or staging_file_id based on a env flagNotifier
JobTypeUploadV2
a valid job typeSlave
processStagingFile
andprocessMultiStagingFiles
for v1 and v2 jobs respectively. Both internally callprocessSingleStagingFile
whose job is to download the stagingfile and read it to create load files.payload
andpayload
sharingbasePayload
for common fieldsjobRun
struct can have multiple staging files, removedstagingFilePath
andstagingFileReader
and replaced it with a list of staging file processorsstagingFileProcessors
tableWriterMutexes
to prevent concurrent writes to the same tableoutputFileWritersMap
andtableEventCountMap
since they can now be accessed concurrentlymaxConcurrentStagingFiles
to control number of staging files being processed concurrentlySecurity