[pull] master from rudderlabs:master #44
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
lvrach
force-pushed
the
master
branch
2 times, most recently
from
4155fa2 to
b124ce4
Compare
* fix: reporting type status detail * refactor: reporting_types_test
…ng to reporting (#5326)
….12.1 in the go-deps group (#5349)
…5316) * fix: bq partitioning for additional columns (#5293) * chore: release 1.38.1 (#5295) * fix: klaviyo bulk upload and BingAds OC (#5305) * fix: bing ads audience (#5299) * chore: handle 500kb limit, refactor to service impl (#5302) * chore: handle 500kb limit, refactor to service impl * chore: fix mock * chore: fix mock newline * chore: fix apiservice newline * chore: move error --------- Co-authored-by: Sudip Paul <[email protected]> * chore: release 1.38.2 (#5306) * fix: allow only enabled dest in backendSubscriber, fix klaviyo bulk (#5309) * chore: release 1.38.3 (#5311) * fix: json marshal l errors when parsing poll response from klaviyo * fix: payload size limit issue * chore: add rate limiter in http client for Klaviyo Bulk Upload async destination (#5353) * fix: allow only enabled dest in backendSubscriber, fix klaviyo bulk * chore: add rate limited http client for klaviyo bulk * refactor: add rate limiter klaviyo bulk upload --------- Co-authored-by: Dilip Kola <[email protected]> --------- Co-authored-by: Akash Chetty <[email protected]> Co-authored-by: devops-github-rudderstack <[email protected]> Co-authored-by: Yashasvi Bajpai <[email protected]> Co-authored-by: Sudip Paul <[email protected]>
# Description ## Motivation To introduce a cost attribution metric for all reporting services. I've noticed coupling and discrepancies between different client implementations, making it hard to test the changes and requiring redundant effort. First, I attempted a more [aggressive](#5529) refactor effort. However, after discussing this with the PipeOps team, we decided to take a more cynical approach. ## Plan * Introduce a common client for all communication with the reporting service * Support switching to the new common client via a single feature flag * Intention to remove old client/send method in the next release. It was also decided * To break the backward compatibility with metrics exposed by old clients. This might break some dashboards, but no alert should be impacted. ## Changes ### Metric names New metrics: ```go const ( StatRequestTotalBytes = "reporting_client_http_request_total_bytes" StatTotalDurationsSeconds = "reporting_client_http_total_durations_seconds" StatRequestLatency = "reporting_client_http_request_latency" StatHttpRequest = "reporting_client_http_request" ) ``` Deprecated metrics -- will no longer work after switching to the common client: ```go f.reqLatency = f.stats.NewTaggedStat("reporting_flusher_http_request_duration_seconds", stats.TimerType, tags) f.reqCount = f.stats.NewTaggedStat("reporting_flusher_http_requests_total", stats.CountType, tags) f.sentBytes = f.stats.NewTaggedStat("reporting_flusher_sent_bytes", stats.HistogramType, tags) edr.stats.NewTaggedStat("error_detail_reporting_http_request", stats.CountType, httpStatTags).Increment() edr.edReportingRequestLatency = edr.stats.NewTaggedStat("error_detail_reporting_request_latency", stats.TimerType, tags) ``` ### Metric labels Changes: * `workspaceID` -- REMOVED will no longer be captured by the client, as it is not relevant when multiple workspaces are batch to a single call * `clientName` -- RENAMED --> `module` * `endpoint` --> ADDED: reporting URL * `path` --> ADDED: reporting path/service ### `region` query param Region query parameter, used only by some reporting clients **has been removed**. Based on team feedback this query param was used temporary for multi region transition and is no longer needed. ### `REPORTING_URL` env var REPORTING_URL now always defaults to `"https://reporting.dev.rudderlabs.com"` Some of the existing clients defaulted to `https://reporting.rudderstack.com/` ## To Be Discussed * Should we use a default value for `REPORTING_URL`, would we rather fail if `REPORTING_URL` is not configured ? It could be a privacy risk for EU customers if reporting us endpoint is configured accidentally. * Should we introduce `.Ping()` method and corresponding endpoint ? ## Next steps 1. Rollout `useCommonClient` flag in production 2. Update Grafana dashboards as need 3. Remove all deprecated code (most of it is marked with commends` 4. Start the work for unifying reporting logic ## Linear Ticket Resolves PRI-168 https://linear.app/rudderstack/issue/PRI-168/introduce-reporting-client-metrics-in-rudder-server-for-cost ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
Remove logs that can expose event payloads.
…e frequent group (#5734)
# Description It is difficult to triage the issue from timeout-alerts without clearly knowing the type of destination the regulation worker is facing issues with. This PR is to add that. ## Linear Ticket Resolves INT-3436 https://linear.app/rudderstack/issue/INT-3436/research-regulation-worker-alerts ## Security - [ ] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description - Using reloadable limiters so that we can adjust limits at runtime - Tuning some router configuration - process limiter increased from 200 to 1024 - http client's max idle connections following the number of workers by default ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
Change jobsdb column type default to text. This change ensures - when not configured new datasets created have a `text` event_payload column. Left the config and related handling for now so that it can phased out at a later point so it leaves some time for jobsdb to compact and cleanup older jsonb column containing datasets.
# Description transformer compaction will now be enabled by default, as long as transformer service supports it ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description `TrackingPlanValidationEvent` should only include `message` and `metadata` ## Additional items - Remove `Processor.maxConcurrency` since concurrency can be controlled through `Transformer.Client.maxHTTPConnections` ## Linear Ticket resolves PIPE-2011 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description - `tx.Commit` for redshift getting stuck in some cases when loading data into table. In case of append mode, we can directly load file instead of doing it in transaction - Since we are now skipping creating staging table if append mode is preferred. In case of `loadUserTables`, it relies on indentifies staging table because `users` table will be deduped/merged even when append mode is preferred. So, creating a staging table for `identifies` is now handled by `loadUserTables` ## Linear Ticket Part of WAR-517, WAR-515 ## Security - [ ] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description Syncing patch release v1.47.2 to main branch **↓↓ Please review and edit commit overrides before merging ↓↓** BEGIN_COMMIT_OVERRIDE fix: race condition in table schema updates causing incorrect schema to be saved (#5756) fix: warehouse extracting messageId and receivedAt from event payload (#5757) END_COMMIT_OVERRIDE --------- Co-authored-by: Akash Chetty <[email protected]> Co-authored-by: shekhar-rudder <[email protected]>
# Description This PR adds poulating bytesPerTable in batch router so that it can be used by warehouse for batching of files ## Linear Ticket https://linear.app/rudderstack/issue/WAR-501/populate-bytespertable-in-rudder-server ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description This PR introduces support for creation of upload_v2 notifier jobs required for batching of staging files **Key Changes:** - Files are grouped into v1 and v2 based on the presence of BytesPerTable and a feature flag. - Extracted common logic for v1 and v2 paths into separate methods - For now, v2 notifier jobs include only one staging file per job, similar to v1. This is temporary — batching logic will be implemented in the next PR. - We will no longer track the processing status (success/failure) of staging files in v2. - v2 jobs will always force creation of load files (see state_generate_load_files.go). ## Linear Ticket https://linear.app/rudderstack/issue/WAR-461/enable-creation-of-upload-v2-notifier-jobs ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description disabling transformer request payload compaction ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
) # Description - `jobMinRowsLeftMigrateThreshold` is now the only configuration option that controls whether a table is small or not, i.e. the migration logic will expect two small tables to exists for an actual migration to happen (a.k.a ds compaction). - Including equality check during `jobDoneMigrateThres` comparison, so that it is easier to disable this option by setting it to 1. It is now recommended to turn off `jobDoneMigrateThres`, so that datasets only get migrated whenever a ds compaction is possible. Migrating an incomplete dataset requires time and acquires a lock (stops-the-world), thus if this is to lead to the same number of datasets, it doesn't provide much benefit. ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description Adding feature flags for: * enabling/disabling UT mirroring in the processor * doing "sampling sanity checks" while mirroring * mutually exclusive with "fire & forget" mode * fire & forget mode to forward all requests to an UT mirror * mutually exclusive with "sampling sanity checks" mode ## Linear Ticket < [Processor UT Mirroring](https://linear.app/rudderstack/issue/PIPE-2006/processor-ut-mirroring) > ## Dependencies * rudderlabs/rudder-go-kit#777 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description
- respecting `MaxConnsPerHost` & `MaxIdleConnsPerHost` in `httplb`
transport
- removed `recycled` client as it can be emulated using `httplb`
- `httplb` client should only use `WithRoundTripperMaxLifetime` in case
`{DEST,USER}_TRANSFORM_URL_IS_HEADLESS=false` or
`Transformer.Client.recycle: true`. If none is present,
`WithRoundTripperMaxLifetime` will not be used
## Security
- [x] The code changed/added as part of this pull request won't create
any security issues with how the software is being used.
…aceConfig (#5753) # Description Updated config processing in backend-config module to attach account details to the destination ## Linear Ticket https://linear.app/rudderstack/issue/INT-3447/integrating-workspace-config-to-rudder-server ## Security - [ ] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
…onfig (#5783) # Description - remove `brt_process` limiter which is no longer used after latest changes - increase `brt_upload` limiter to 200, since we'll be using multiple goroutines per worker now - rename `batchrouter_partition_worker` limiter to `brt_work` to adhere to existing naming conventions - use same options in `brt_work` limiter that we are using in other limiters - rename `batchrouter_partition_worker_add_job_delay` to `brt_partition_worker_add_job_delay` and include `destType` in stats tags ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
🤖 I have created a release *beep* *boop* --- ## [1.48.0](v1.47.0...v1.48.0) (2025-04-28) ### Features * introduce common http client for reporting ([#5669](#5669)) ([594bdce](594bdce)) * populate bytesPerTable for warehouse process API ([#5737](#5737)) ([fd46379](fd46379)) * query load files using upload_id instead of staging_file_id ([#5700](#5700)) ([3749ba9](3749ba9)) * support for creating upload v2 notifier jobs ([#5729](#5729)) ([dbc830f](dbc830f)) * use header augmenter for router transformer client ([#5640](#5640)) ([3749ba9](3749ba9)) ### Bug Fixes * httplb client doesn't respect connection limits ([#5773](#5773)) ([ef0a15e](ef0a15e)) * jobsdb payload limit used compressed column sizes instead of actual payload length ([#5723](#5723)) ([c711b25](c711b25)) * race condition in table schema updates causing incorrect schema to be saved ([#5756](#5756)) ([f5ab51a](f5ab51a)) * use copy to load table in append mode for redshift([#5743](#5743)) ([877da25](877da25)) * use warehouse transformer event instead of using processor event ([#5764](#5764)) ([f51e260](f51e260)) * version downgrade for databricks-sql-go to 1.6.1 ([#5741](#5741)) ([c711b25](c711b25)) * warehouse extracting messageId and receivedAt from event payload ([#5757](#5757)) ([f5ab51a](f5ab51a)) ### Miscellaneous * add destination type in metrics ([#5727](#5727)) ([fa38f34](fa38f34)) * **batchrouter:** limiter cleanup adjust stats and use harmonized config ([#5783](#5783)) ([96aba9d](96aba9d)) * change jobsdb payload column type default to text ([#5746](#5746)) ([0f436e7](0f436e7)) * configure pubsub publish settings ([#5755](#5755)) ([e481992](e481992)) * **deps:** bump google.golang.org/api from 0.228.0 to 0.229.0 in the frequent group ([#5734](#5734)) ([32854cc](32854cc)) * **deps:** bump the frequent group with 3 updates ([#5769](#5769)) ([12b7d96](12b7d96)) * disable transformer compaction by default ([#5766](#5766)) ([c504aa0](c504aa0)) * granular configuration for adaptive rate limiting ([#5739](#5739)) ([c711b25](c711b25)) * improve BRT performance ([#5730](#5730)) ([61642c8](61642c8)) * **jobsdb:** remove jobMinRowsMigrateThres configuration option ([#5763](#5763)) ([5d2fae9](5d2fae9)) * processor UT mirroring ([#5710](#5710)) ([f4f056c](f4f056c)) * remove payload logging ([#5447](#5447)) ([8e08c11](8e08c11)) * trim down tracking plan validation payload ([#5749](#5749)) ([e711579](e711579)) * use reloadable limiters ([#5744](#5744)) ([d344d40](d344d40)) * use transformer compaction by default ([#5740](#5740)) ([53792f2](53792f2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
# Description Syncing release v1.48.0 to main branch **WARNING: Do NOT rewrite git history and ALWAYS use a "Merge Commit" for merging!** **↓↓ Please review and edit commit overrides before merging ↓↓** BEGIN_COMMIT_OVERRIDE feat: update config to add account details with destination in workspaceConfig (#5753) chore: migrate from denisenkom/go-mssqldb to microsoft/go-mssqldb (#5776) END_COMMIT_OVERRIDE
….13.3 in the go_modules group (#5787)
# Description - Use `(varchar|nvarchar)(max)` when creating staging table in synapse for character types(`varchar`, `nvarchar`, `char`, `nchar`). This change is needed to support `COPY` command. Creating staging table with max length allows us to load data without any errors and truncate to desired length when loading into the final table. ## Linear Ticket - Part of WAR-524 ## Security - [ ] The code changed/added as part of this pull request won't create any security issues with how the software is being used.
# Description Syncing patch release v1.48.1 to main branch **↓↓ Please review and edit commit overrides before merging ↓↓** BEGIN_COMMIT_OVERRIDE fix(jobsdb): completed datasets don't get deleted without a pair (#5793) END_COMMIT_OVERRIDE --------- Co-authored-by: Aris Tzoumas <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )