Skip to content

feat: add support for processing of upload_v2 job type by slave #5796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

shekhar-rudder
Copy link
Member

@shekhar-rudder shekhar-rudder commented Apr 30, 2025

Description

Master

  • Added some logs in loadfiles.go
  • Now setting status for staging files to succeeded in loadfiles.go for the integration tests to pass
  • I had missed one SQL query which was using staging_file_id to query load files. Created a new function getLoadFilesMetadataQuery to query based on either upload_id or staging_file_id based on a env flag

Notifier

  • Made JobTypeUploadV2 a valid job type

Slave

  • We now have 2 methods - processStagingFile and processMultiStagingFiles for v1 and v2 jobs respectively. Both internally call processSingleStagingFile whose job is to download the stagingfile and read it to create load files.
  • There are two payload types - payload and payload sharing basePayload for common fields
  • Since a jobRun struct can have multiple staging files, removed stagingFilePath and stagingFileReader and replaced it with a list of staging file processors stagingFileProcessors
  • Added tableWriterMutexes to prevent concurrent writes to the same table
  • Added mutex for outputFileWritersMap and tableEventCountMap since they can now be accessed concurrently
  • Now that a job can have multiple staging files, introduce a config maxConcurrentStagingFiles to control number of staging files being processed concurrently

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@shekhar-rudder shekhar-rudder force-pushed the war-463-slave-uploadv2 branch 2 times, most recently from ab5c56a to 0001588 Compare April 30, 2025 12:32
@shekhar-rudder shekhar-rudder force-pushed the war-463-slave-uploadv2 branch from 0001588 to b0bc42c Compare May 2, 2025 08:41
@shekhar-rudder shekhar-rudder force-pushed the war-463-slave-uploadv2 branch from b0bc42c to bb4e0cd Compare May 2, 2025 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant