Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shard integration tests, parallelise attw #4258

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Shard integration tests, parallelise attw #4258

wants to merge 18 commits into from

Conversation

dankochetov
Copy link
Contributor

No description provided.

Sukairo-02 and others added 11 commits March 13, 2025 15:48
@Sukairo-02 @AndriiSherman
RQB v2 (#3974)
e30ede4 
* Fixed `isConfig` checks using types of wrong fields

* Added changes from `relations-api-v2` branch, moved RQB v1 to `_query` field, RQB v1 entities to `_relations.ts`, added template to SQLWrapper for return type inferrence, switched RQB-related local tests & type tests to use `._query`

* Temporarily removed `db.query` type errors on empty relations for `withReplicas()` compatibility (to be reworked), switched tests to use `_query` method for RQB v1 tests

* Removed unused export

* Updated imports for RQBv1

* Fixed types in AWS Data Api, updated Session constructor in kit

* Fixed misplaced execute and executeRqbV2

* Updated `query` to `_query` in `js-tests`

* Fixed lack of file formats in imports

* Bump `better-sqlite3` version to `9.3.0` for jsonb functions support, fixed, simplified query builders, added JSON parsing for top level SQLite response mapper, fixed aggregated relation count sql

* Fixed RQB types breaking on different table db and ts names

* Reverted to using tsNames as `table._.name` & `column._.tableName` in relational configs, resolved potential type overlaps with similar table names from different schemas on query results, resolved relation builders allowing usage of columns from table with same name from different schema

* Updated database instance generics with new fields

* Fixed sqlite rqb v2 query not working properly with sync drivers

* Fixed extras inferrence with `SQL.Aliased`, returned `columns` argument relations' `where.$raw` callback

* Fixed self-referencing relations, switched dbConfig callbacks table types to tables instead of column records, relational tests v2 for postgresql, updated relations .where field names, added error on nothing selected from table in relational queries v2

* Restored pulled _relations changes

* Implemented rqb-v2 changes to new drivers, updated tests, fixed neon-http `$withAuth` rqbv2 tests

* Fixed drizzle-seed test imports

* Returned changes lost during merging

* Updated types

* Fixed broken new test cases

* Improved types, allowed true+false column selection, updated RQB v2 entity kinds, new `orderBy` api

* Fixed SQLite RQB V2 self-relations, added `better-sqlite3` tests for RQB V2, removed unnecessary argument from `WithContainer` generics

* Improved & fixed RQB filters, added runtime error on nulls in non-optional relation results, added forced relation optionality for queries with additional filters, fixed libsql rqb pre-mapping and transaction result mapping, added libsql RQBv2 tests, added new RQB test cases related to aforementioned changes

* MySQL2 tests for RQBv2, workarounds for MySQL limitations on lateral joins and `json_arrayagg()`'s `order by` order preservation, removed unused code, added error throw to incomplete RQBv2 feature `.through()`

* Switched MySQL RQBv2 builders to lateral left joins, fixed nested aggregated relations

* Removed unused code from planetscale session's `executeRqbV2()`, added planetscale tests for RQB v2, added common RQB tests for pg dialect drivers

* Experimental `.through()` implementation, removed errors on nulls in non-optional relations, defaulted relations to optional, allowed full booleans in RQB query config's `with` section, moved `coalesce` in pg RQB for consistency

* New test cases for RQB, SQLite and MySQL RQB tests per-driver, mysql-proxy session RQB fixes, fixed missing `TRelations` arguments in generics

* Fixed RQB v2 for `sqlite-proxy`, fixed out of scope sqlite tests, simplified RQB column builder for `MySQL`

* RQBv2 schema fixes, fixed sql builder adding schemas on table aliases, RQBv2 schema tests

* Tests for `where` and `orderBy` RQBv2 builders, fixed `where.raw` callback getting columns passed to first argument instead of full table

* Updated seeder test case schemae

* Version bump

* Version bump

* Fixed and returned `neon-http-batch` tests, `sql-js` rqb v2 tests, switched `sql-js` prepare logic to query-builder-side preparation instead of db-side because of the need to free connections manually

* Reverted latest changes for SQLite RQBv1 prepare logic

* View support for RQBv2, related tests, extended `View` class with `Table` class' symbols, supported `View` in `getTableUniqueName(table)`, fixed `like`-type operators not accepting `SQLWrapper`, fixed aliased `View`-s getting prefixed with schema name in `sql` builder, fixed RQBv2 issue occuring when root-level table's name matches with queried relation name by aliasing root-level table as well

* Fixed D1 batch for RQBv2, fixed RQBv2 replicas, added related tests, added AWS Data Api tests for RQBv2

* Fixed lockfile after merge

* RQBv2 for `bun-sql`, related tests, updated seeder's tests, added type-error to incompatible with rqb views on relations builder \& relations helper, removed `.aggs` from relational query builder for rework

* Fixed lack of file format on import

* Filters by relations, related tests

* Switched relational filter joins on junctions to inner joins for consistency

* Fixed relational filters in `NOT`\`OR`, fixed RQBv2 ignoring `casing` in some places [tests TBA]

* `sql` for `RAW`, `AND` operators for rqb v2 `where`

* Change kit introspect logic

* Fixed `pnpm-lock` after merge

* Improved types, retargeted relation definition's `where` to `targetTable`, updated `extras` api, removed `offset`, `orderBy` from `one` type relations, fixed data loss on postgres bigints in rqb, experimental implementation of V2 for `Gel` dialect, fixed broken `where` for tables without relations, `defineRelations(schema)` call variant for relationless RQBv2 usage, allowed uneven amount of columns in `from`/`to` of relation definition when `.through(...)` is present, type-level ensuring `.through(...)` is called for all columns

* Dprint?

* Import fix

* Fixed `View`+`casing`-related crashes in RQBv2, simplified type for relational filters to prevent Typescript server hangs, new test cases, casing fixes, edited relational test schemae for casing testing, additional test cases

* Added missed changes

* Fixed MySQL binary, varbinary mapping, fixed mysql varchar definition allowing config without `length`, improved RQBv2 definition errors, tested + fixed relations with `.through` with uneven column count, tested + fixed all supported types in RQBv2, disabled built-in parsers of `node-postgres`-like drivers, `postgres-js`, `pglite` for types: `date[]`, `timestamp[]`, `interval[]`, `date[]`, supported `blob`  in `sqlite` for RQBv2, fixed pg `numeric` sometimes being returned as a `number` when `string` is expected

* Fixed incorrect test

* Add release notes and api version between kit and orm

* Fix orm api version

---------

Co-authored-by: AndriiSherman <[email protected]>
@Sukairo-02
Fixed db names in RQBv2 error messages, replaced jsonb operators wi…
fd5d1e8 
…th `json` operators for `D1` sqlite driver in RQBv2
@Sukairo-02
Fixed PostgesJsTransaction type-level incompatibility with itself, …
af4c434 
…fixed errors when using `ExtractTablesWithRelations` on relations object
@dankochetov
Shard integration tests, parallelise attw
e11f885
@dankochetov
Thank you pnpm 10
e48f3e2
@dankochetov
Split int tests by provider
50d24f0
@dankochetov
Update Neon config, fix bash
8db2dae
@dankochetov
Restore webSocketConstructor
a68d3b9
@dankochetov
Use Docker for Neon
ac17c60
@dankochetov
Use docker-compose for Neon
1a42a13
@dankochetov
Fix env var
a61369a
github-advanced-security[bot]
github-advanced-security bot found potential problems Mar 18, 2025
View reviewed changes
drizzle-orm/src/relations.ts
tablesConfig[key]!.primaryKey.push(column);
}
}
if (parseJson) row[selectionItem.key] = JSON.parse(row[selectionItem.key] as string);

Check warning

Code scanning / CodeQL

Prototype-polluting assignment Medium

This assignment may alter Object.prototype if a malicious '__proto__' string is injected from
library input
Loading
.

Copilot Autofix AI about 18 hours ago

To fix the problem, we need to ensure that the keys used to access or assign properties on the row object cannot be used to modify Object.prototype. One effective way to achieve this is by checking if the key is a dangerous value like __proto__, constructor, or prototype before performing the assignment. If the key matches any of these values, we should skip the assignment.

Suggested changeset 1
drizzle-orm/src/relations.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/drizzle-orm/src/relations.ts b/drizzle-orm/src/relations.ts
--- a/drizzle-orm/src/relations.ts
+++ b/drizzle-orm/src/relations.ts
@@ -747,3 +747,8 @@
 
-			if (parseJson) row[selectionItem.key] = JSON.parse(row[selectionItem.key] as string);
+			if (parseJson) {
+				if (selectionItem.key === '__proto__' || selectionItem.key === 'constructor' || selectionItem.key === 'prototype') {
+					continue;
+				}
+				row[selectionItem.key] = JSON.parse(row[selectionItem.key] as string);
+			}
 
@@ -788,2 +793,5 @@
 		}
+		if (selectionItem.key === '__proto__' || selectionItem.key === 'constructor' || selectionItem.key === 'prototype') {
+			continue;
+		}
 		row[selectionItem.key] = decoder.mapFromDriverValue(value);
EOF
@@ -747,3 +747,8 @@

if (parseJson) row[selectionItem.key] = JSON.parse(row[selectionItem.key] as string);
if (parseJson) {
if (selectionItem.key === '__proto__' || selectionItem.key === 'constructor' || selectionItem.key === 'prototype') {
continue;
}
row[selectionItem.key] = JSON.parse(row[selectionItem.key] as string);
}

@@ -788,2 +793,5 @@
}
if (selectionItem.key === '__proto__' || selectionItem.key === 'constructor' || selectionItem.key === 'prototype') {
continue;
}
row[selectionItem.key] = decoder.mapFromDriverValue(value);
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
drizzle-orm/src/relations.ts
} else {
decoder = field.getSQL().decoder;
}
row[selectionItem.key] = decoder.mapFromDriverValue(value);

Check warning

Code scanning / CodeQL

Prototype-polluting assignment Medium

This assignment may alter Object.prototype if a malicious '__proto__' string is injected from
library input
Loading
.

Copilot Autofix AI about 18 hours ago

To fix the prototype pollution issue, we need to ensure that keys like __proto__, constructor, and prototype are not used as property names in the row object. This can be achieved by adding a check before assigning values to the row object. If the key matches any of these dangerous values, we should skip the assignment.

Suggested changeset 1
drizzle-orm/src/relations.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/drizzle-orm/src/relations.ts b/drizzle-orm/src/relations.ts
--- a/drizzle-orm/src/relations.ts
+++ b/drizzle-orm/src/relations.ts
@@ -788,3 +788,5 @@
 		}
-		row[selectionItem.key] = decoder.mapFromDriverValue(value);
+		if (selectionItem.key !== '__proto__' && selectionItem.key !== 'constructor' && selectionItem.key !== 'prototype') {
+			row[selectionItem.key] = decoder.mapFromDriverValue(value);
+		}
 	}
EOF
@@ -788,3 +788,5 @@
}
row[selectionItem.key] = decoder.mapFromDriverValue(value);
if (selectionItem.key !== '__proto__' && selectionItem.key !== 'constructor' && selectionItem.key !== 'prototype') {
row[selectionItem.key] = decoder.mapFromDriverValue(value);
}
}
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dankochetov @Sukairo-02