PLU-289: Tiles v2 #1051
Merged
PLU-289: Tiles v2 #1051
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pregnantboy
force-pushed
the
tiles-v2/trunk
branch
from
dd7d890 to
753f3fd
Compare
Datadog ReportBranch report: ✅ 0 Failed, 970 Passed, 0 Skipped, 3m 10.84s Total Time |
pregnantboy
force-pushed
the
tiles-v2/trunk
branch
2 times, most recently
from
16cd657 to
fc6950b
Compare
…#972) ### Changes Added a separate Postgres database for tiles functionality with corresponding configuration. ### What changed? - Added environment variables for tiles Postgres configuration in `.env-example` - Added a new `tiles-postgres` service in `docker-compose.dev.yml` running on port 5431 - Created a new volume `plumber-tiles-postgres` for persistent storage - Extended the app configuration in `src/config/app.ts` to include tiles Postgres settings - Created a new file `src/config/tiles-database.ts` that initializes a Knex client for the tiles database ### How to test? 1. Update your local `.env` file with the new tiles Postgres variables 2. Run `docker-compose -f docker-compose.dev.yml up -d` to start the new tiles-postgres service 3. Verify the tiles-postgres container is running and healthy
### Changes Added a new `db` column to the `table_metadata` table to distinguish between PostgreSQL and DynamoDB tables. ### What changed? - Created a new migration file `20250506071427_add_table_metadata_db_column.ts` that adds a `db` column to the `table_metadata` table with a default value of 'ddb' and not nullable constraint - Updated the `TableMetadata` model to include the new `db` property with type 'pg' | 'ddb' ### How to test? 1. Run the migration 2. Verify the column exists in the database schema 3. Check that existing records have the default 'ddb' value
…operations (#974) ### Changes Added PostgreSQL support for Tiles with streaming capabilities and ULID for row IDs. ### What changed? - Added new dependencies: `pg-query-stream` and `ulid` to support PostgreSQL streaming and monotonic ID generation - Created new PostgreSQL-specific implementation files for table operations: - `table-functions.ts` - For creating tables - `table-column-functions.ts` - For managing table columns - `table-row-functions.ts` - For CRUD operations on table rows - Implemented types for table operations in `types.ts` - Introduced ULID (Universally Unique Lexicographically Sortable Identifier) for row IDs ### How to test? 1. Install the new dependencies with `npm install` 2. Test the PostgreSQL implementation by running the existing integration tests
### Changes Reorganized table row models by moving them from `/models/dynamodb/table-row` to `/models/tiles/dynamodb/table-row` and separated types into a dedicated file. ### What changed? - Moved table row models and functions from `/models/dynamodb/table-row` to `/models/tiles/dynamodb/table-row` - Extracted types from `/models/dynamodb/table-row/types.ts` to `/models/tiles/types.ts` - Updated all imports across the codebase to reference the new file locations - Moved helper functions from `/models/dynamodb/helpers.ts` to `/models/tiles/dynamodb/helpers.ts` - Updated all test files to use the new import paths ### How to test? 1. Run the existing test suite to ensure all functionality works as expected 2. Verify that all table operations (create, read, update, delete) still function correctly 3. Check that the integration tests pass with the new import paths
### Changes Added a factory pattern for database operations with support for PostgreSQL and DynamoDB. ### What changed? - Created a factory function `getTableOperations` that returns the appropriate database implementation based on the database type - Implemented `DynamoDBTableOperations` class with table row operations and no-op functions for unsupported operations - Implemented `PostgresTableOperations` class with full support for table, column, and row operations - Both implementations conform to the `TableOperations` interface
…etAllRows payload (#977) ### Changes Added support for multiple database types in the table system with a factory pattern for database operations. ### What changed? - Implemented a database type factory pattern with `getTableOperations()` to support multiple database backends - Added a `databaseType` parameter to the `CreateTableInput` GraphQL type - Updated all table operations (create, read, update, delete) to use the factory pattern - Modified row data format to use CSV for network transfer to reduce payload size - Added conversion logic in the frontend to transform CSV data back to object format - Fixed table relationship definitions in `TableColumnMetadata` - Improved database connection handling in tests ### How to test? 1. Create a new table with the DynamoDB backend: `createTable({ name: "Test Table", isBlank: false, databaseType: "ddb" })` 2. Create a new table with the PostgreSQL backend (default): `createTable({ name: "Test Table", isBlank: false })` 3. Verify that operations (adding/updating/deleting rows and columns) work correctly on both table types 4. Check that row data is properly displayed in the frontend despite the CSV transfer format
### Changes Implemented a database abstraction layer for Tiles to support both DynamoDB and PostgreSQL backends. - Added a factory pattern with `getTableOperations()` to abstract database operations - Updated Tiles actions (create-row, find-single-row, update-row) to use the new factory pattern - Changed default database type from 'pg' to 'ddb' in create-table mutation - Removed the static `getColumns` method from TableColumnMetadata in favor of using relations - Enhanced TableMetadata to properly fetch columns with ordering - Improved row formatting in PostgreSQL implementation
### TL;DR Added Tiles Postgres database environment variables to ECS configuration. ### What changed? Added five new environment variables to the ECS environment configuration: - `TILES_POSTGRES_HOST` - `TILES_POSTGRES_DATABASE` - `TILES_POSTGRES_USERNAME` - `TILES_POSTGRES_PASSWORD` - `TILES_POSTGRES_PORT` Each variable references corresponding parameter store values with the `plumber-<ENVIRONMENT>-tiles-rds-*` pattern.
### TL;DR Removed the explicit database type selection when creating tables, instead using LaunchDarkly flags to determine the database type. ### What changed? - Removed `databaseType` from the `CreateTableInput` in GraphQL schema - Added LaunchDarkly integration to determine database type when creating tables - Added `databaseType` field to `TableMetadata` resolver to expose the database type to clients - Updated tests to mock LaunchDarkly flag values - Enhanced CSV import functionality with database-specific limits: - Increased max file size to 5MB for PostgreSQL (vs 2MB for DynamoDB) - Increased chunk size to 1000 rows for PostgreSQL (vs 100 for DynamoDB) - Added pagination support for table rows with `stringifiedCursor` parameter ### How to test? 1. Create a new table and verify it uses the database type determined by LaunchDarkly 2. Import CSV files of different sizes to verify the database-specific limits 3. Test pagination by loading large tables and checking that rows load correctly
### TL;DR
Added load testing scripts for Postgres database operations with k6 extensions.
### What changed?
- Added a `.env-example` file with Postgres connection string
- Updated the README with instructions for building k6 with extensions (postgres, dotenv, dashboard)
- Updated the k6 binary to support these extensions
- Added several new load testing scripts for Postgres operations:
- `seed_table.js` - Creates test tables
- `bulk_write.js` - Tests bulk insert operations
- `bulk_read.js` - Tests bulk read operations
- `read_action.js` - Tests single read operations
- `write_action.js` - Tests single write operations
- `update_action.js` - Tests update operations
- `test.js` - Simple connection test
### How to test?
1. Copy `.env-example` to `.env` and update with your Postgres connection details
2. Build k6 with extensions using the Docker command in the README:
```
docker run --rm -it -e GOOS=darwin -u "$(id -u):$(id -g)" -v "${PWD}:/xk6" \
grafana/xk6 build \
--with github.com/grafana/xk6-sql@latest \
--with github.com/grafana/xk6-sql-driver-postgres@latest \
--with github.com/grafana/xk6-dashboard@latest\
--with github.com/szkiba/xk6-dotenv@latest
```
3. Run the test scripts, for example:
```
./k6 run tiles/test.js --out dashboard
```
# Tests for Tiles v2 ## Changes 1. Added a PostgreSQL container for integration tests 2. Extending existing mutation tests to work with both DynamoDB and PostgreSQL 3. Adding unit tests for PostgreSQL table functions 4. Implementing proper database type switching in mutations
pregnantboy
force-pushed
the
tiles-v2/trunk
branch
from
fc6950b to
b977f26
Compare
pregnantboy
force-pushed
the
tiles-v2/trunk
branch
from
ba94b08 to
214dbf0
Compare
# Small tiles improvments - Add Table icon in breadcrumb to differentiate v2 from v1 tiles  - Improve cell editing functionality: - Allow adding line breaks with Alt+Enter - Change Shift+Enter to navigate upward - Prevent cell editing when data is being fetched - Fix cell selection behaviour to allow selecting substrings in cel
### TL;DR This change addresses a compatibility issue between Tiles v1 and v2. In v1, empty cells were represented as empty strings or by omitting the key entirely, while in v2, empty cells return null values. This update ensures the application can properly handle data from both versions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tiles v2
Staging
Prod