-
Notifications
You must be signed in to change notification settings - Fork 122
Add support for querying datetime fields #711
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 26 commits
Commits
Show all changes
52 commits
Select commit
Hold shift + click to select a range
589668b
Fix timestamp handling
abrookins a728492
Refactor datetime conversion exception handling
abrookins 1afdd24
Fix spellcheck issues in migration documentation
abrookins 801554f
Revert datetime conversion refactoring to fix timezone issues
abrookins de0a3d3
Fix spellcheck by adding migration-related words to wordlist
abrookins 94b6d4b
Address Copilot code review feedback
abrookins 7372fce
Address Copilot review feedback
abrookins 815567e
Expand migrations CLI to support create/run/rollback/status
abrookins d8dbc8b
Fix sync CLI commands and test failures
abrookins ded5c29
Fix Python 3.9 compatibility in CLI type annotations
abrookins a35c6f0
Fix spellcheck errors in migration documentation
abrookins ab337df
Fix MyPy errors for _meta attribute access
abrookins 865ef35
Fix CLI async/sync function call issues causing test failures
abrookins 63287ed
Fix CLI async/sync transformation and execution pattern
abrookins a83b591
Fix missing newline at end of CLI migrate file
abrookins 31cf4b0
Fix CLI sync/async transformation issues
abrookins 3e9fdfb
Trigger CI rebuild after network timeout
abrookins 2929b7c
Fix trailing whitespace in CLI docstring
abrookins 88a5ab7
Fix async/sync CLI transformation issues
abrookins 06ab091
Fix schema migration rollback logic bug
abrookins 929a22c
Fix test isolation for parallel execution
abrookins 5fd01cf
Improve test worker isolation by overriding APPLIED_MIGRATIONS_KEY
abrookins 9662cda
Separate legacy and new migration CLIs
abrookins 9e667a7
Remove test migration file and update docs
abrookins a3720fc
Fix linting issues in legacy migrate CLI
abrookins d752422
Apply final code formatting fixes
abrookins 9eaf012
Update aredis_om/cli/main.py
abrookins 0c055c6
Fix datetime field handling in NUMERIC queries
abrookins 9b9ceb3
Fix typo
abrookins ca85af7
Improve TAG field sortability error message and documentation
abrookins 148c6e7
Improve CLI error handling for Redis connection failures
abrookins d8acb43
Enhance datetime migration with production-ready features
abrookins b0998b1
Fix CI linting and spelling issues
abrookins a2422ac
Fix syntax error in datetime migration
abrookins a8a83ee
Add type ignore comments to fix MyPy linting errors
abrookins 35d5915
Exclude migration files from MyPy checking
abrookins 98708cc
Fix Makefile syntax for MyPy exclude patterns
abrookins c14bd95
Fix Pydantic v1 compatibility issues
abrookins 784a759
Temporarily exclude model directories from MyPy checking
abrookins 863afdc
Add automatic datetime field schema mismatch detection
abrookins 9e49895
Fix f-string linting error
abrookins 104670b
Fix bandit security warnings
abrookins 6c2c5cd
Reorganize migrations module into domain-driven structure
abrookins 0ad2fae
Fix linting issues: remove trailing whitespace and add missing newline
abrookins e8ee7d7
Fix import paths in data migration modules and update gitignore
abrookins 4e57467
Merge branch 'main' into fix/datetime-field-indexing-467
abrookins 6f33516
Consolidate migration docs and create 0.x to 1.0 migration guide
abrookins a7daec2
Remove production deployment checklist
abrookins 3972d7d
Update documentation to use model-level indexing syntax
abrookins ba712df
Fix codespell configuration to exclude dependency directories
abrookins d7ad230
Add missing words to spellcheck wordlist
abrookins 72b3284
Allow TAG fields to be sortable
abrookins File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| [codespell] | ||
| skip = .git,poetry.lock,*.pyc,__pycache__ | ||
| ignore-words-list = redis,migrator,datetime,timestamp,asyncio,redisearch,pydantic,ulid,hnsw |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -70,4 +70,8 @@ unix | |
| utf | ||
| validator | ||
| validators | ||
| virtualenv | ||
| virtualenv | ||
| datetime | ||
| Datetime | ||
| reindex | ||
| schemas | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| repos: | ||
| - repo: https://github.com/codespell-project/codespell | ||
| rev: v2.2.6 | ||
| hooks: | ||
| - id: codespell | ||
| args: [--write-changes] | ||
| exclude: ^(poetry\.lock|\.git/|docs/.*\.md)$ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # CLI package |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| """ | ||
| Redis OMCLI - Main entry point for the async 'om' command. | ||
| """ | ||
|
|
||
| import click | ||
|
|
||
| from ..model.cli.migrate import migrate | ||
| from ..model.cli.migrate_data import migrate_data | ||
|
|
||
|
|
||
| @click.group() | ||
| @click.version_option() | ||
| def om(): | ||
| """Redis OM Python CLI - Object mapping and migrations for Redis.""" | ||
| pass | ||
|
|
||
|
|
||
| # Add subcommands | ||
| om.add_command(migrate) | ||
| om.add_command(migrate_data, name="migrate-data") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| om() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| import asyncio | ||
| import os | ||
| import warnings | ||
| from typing import Optional | ||
|
|
||
| import click | ||
|
|
||
| from ...settings import get_root_migrations_dir | ||
| from ..migrations.migrator import Migrator | ||
|
|
||
|
|
||
| def run_async(coro): | ||
| """Run an async coroutine in an isolated event loop to avoid interfering with pytest loops.""" | ||
| import concurrent.futures | ||
|
|
||
| with concurrent.futures.ThreadPoolExecutor() as executor: | ||
| future = executor.submit(asyncio.run, coro) | ||
| return future.result() | ||
|
|
||
|
|
||
| def show_deprecation_warning(): | ||
| """Show deprecation warning for the legacy migrate command.""" | ||
| warnings.warn( | ||
| "The 'migrate' command is deprecated. Please use 'om migrate' for the new file-based migration system with rollback support.", | ||
| DeprecationWarning, | ||
| stacklevel=3, | ||
| ) | ||
| click.echo( | ||
| click.style( | ||
| "⚠️ DEPRECATED: The 'migrate' command uses automatic migrations. " | ||
| "Use 'om migrate' for the new file-based system with rollback support.", | ||
| fg="yellow", | ||
| ), | ||
| err=True, | ||
| ) | ||
|
|
||
|
|
||
| @click.group() | ||
| def migrate(): | ||
| """[DEPRECATED] Automatic schema migrations for Redis OM models. Use 'om migrate' instead.""" | ||
| show_deprecation_warning() | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.option("--module", help="Python module to scan for models") | ||
| def status(module: Optional[str]): | ||
| """Show pending automatic migrations (no file-based tracking).""" | ||
| migrator = Migrator(module=module) | ||
|
|
||
| async def _status(): | ||
| await migrator.detect_migrations() | ||
| return migrator.migrations | ||
|
|
||
| migrations = run_async(_status()) | ||
|
|
||
| if not migrations: | ||
| click.echo("No pending automatic migrations detected.") | ||
| return | ||
|
|
||
| click.echo("Pending Automatic Migrations:") | ||
| for migration in migrations: | ||
| action = "CREATE" if migration.action.name == "CREATE" else "DROP" | ||
| click.echo( | ||
| f" {action}: {migration.index_name} (model: {migration.model_name})" | ||
| ) | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.option("--module", help="Python module to scan for models") | ||
| @click.option( | ||
| "--dry-run", is_flag=True, help="Show what would be done without applying changes" | ||
| ) | ||
| @click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") | ||
| @click.option( | ||
| "--yes", | ||
| "-y", | ||
| is_flag=True, | ||
| help="Skip confirmation prompt to run automatic migrations", | ||
| ) | ||
| def run( | ||
| module: Optional[str], | ||
| dry_run: bool, | ||
| verbose: bool, | ||
| yes: bool, | ||
| ): | ||
| """Run automatic schema migrations (immediate DROP+CREATE).""" | ||
| migrator = Migrator(module=module) | ||
|
|
||
| async def _run(): | ||
| await migrator.detect_migrations() | ||
| if not migrator.migrations: | ||
| if verbose: | ||
| click.echo("No pending automatic migrations found.") | ||
| return 0 | ||
|
|
||
| if dry_run: | ||
| click.echo(f"Would run {len(migrator.migrations)} automatic migration(s):") | ||
| for migration in migrator.migrations: | ||
| action = "CREATE" if migration.action.name == "CREATE" else "DROP" | ||
| click.echo(f" {action}: {migration.index_name}") | ||
| return len(migrator.migrations) | ||
|
|
||
| if not yes: | ||
| operations = [] | ||
| for migration in migrator.migrations: | ||
| action = "CREATE" if migration.action.name == "CREATE" else "DROP" | ||
| operations.append(f" {action}: {migration.index_name}") | ||
|
|
||
| if not click.confirm( | ||
| f"Run {len(migrator.migrations)} automatic migration(s)?\n" | ||
| + "\n".join(operations) | ||
| ): | ||
| click.echo("Aborted.") | ||
| return 0 | ||
|
|
||
| await migrator.run() | ||
| if verbose: | ||
| click.echo( | ||
| f"Successfully applied {len(migrator.migrations)} automatic migration(s)." | ||
| ) | ||
| return len(migrator.migrations) | ||
|
|
||
| run_async(_run()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,18 +1,183 @@ | ||
| import asyncio | ||
| import os | ||
| from typing import Optional | ||
|
|
||
| import click | ||
|
|
||
| from aredis_om.model.migrations.migrator import Migrator | ||
| from ...settings import get_root_migrations_dir | ||
| from ..migrations.schema_migrator import SchemaMigrator | ||
|
|
||
|
|
||
| def run_async(coro): | ||
| """Run an async coroutine in an isolated event loop to avoid interfering with pytest loops.""" | ||
| import concurrent.futures | ||
|
|
||
| with concurrent.futures.ThreadPoolExecutor() as executor: | ||
| future = executor.submit(asyncio.run, coro) | ||
| return future.result() | ||
|
|
||
|
|
||
| @click.group() | ||
| def migrate(): | ||
| """Manage schema migrations for Redis OM models.""" | ||
| pass | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.option("--migrations-dir", help="Directory containing schema migration files") | ||
| def status(migrations_dir: Optional[str]): | ||
| """Show current schema migration status from files.""" | ||
| dir_path = migrations_dir or os.path.join( | ||
| get_root_migrations_dir(), "schema-migrations" | ||
| ) | ||
| migrator = SchemaMigrator(migrations_dir=dir_path) | ||
| status_info = run_async(migrator.status()) | ||
|
|
||
| click.echo("Schema Migration Status:") | ||
| click.echo(f" Total migrations: {status_info['total_migrations']}") | ||
| click.echo(f" Applied: {status_info['applied_count']}") | ||
| click.echo(f" Pending: {status_info['pending_count']}") | ||
|
|
||
| if status_info["pending_migrations"]: | ||
| click.echo("\nPending migrations:") | ||
| for migration_id in status_info["pending_migrations"]: | ||
| click.echo(f"- {migration_id}") | ||
|
|
||
| if status_info["applied_migrations"]: | ||
| click.echo("\nApplied migrations:") | ||
| for migration_id in status_info["applied_migrations"]: | ||
| click.echo(f"- {migration_id}") | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.option("--migrations-dir", help="Directory containing schema migration files") | ||
| @click.option( | ||
| "--dry-run", is_flag=True, help="Show what would be done without applying changes" | ||
| ) | ||
| @click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") | ||
| @click.option("--limit", type=int, help="Limit number of migrations to run") | ||
| @click.option( | ||
| "--yes", | ||
| "-y", | ||
| is_flag=True, | ||
| help="Skip confirmation prompt to create directory or run", | ||
| ) | ||
| def run( | ||
| migrations_dir: Optional[str], | ||
| dry_run: bool, | ||
| verbose: bool, | ||
| limit: Optional[int], | ||
| yes: bool, | ||
| ): | ||
| """Run pending schema migrations from files.""" | ||
| dir_path = migrations_dir or os.path.join( | ||
| get_root_migrations_dir(), "schema-migrations" | ||
| ) | ||
|
|
||
| if not os.path.exists(dir_path): | ||
| if yes or click.confirm(f"Create schema migrations directory at '{dir_path}'?"): | ||
| os.makedirs(dir_path, exist_ok=True) | ||
| else: | ||
| click.echo("Aborted.") | ||
| return | ||
|
|
||
| migrator = SchemaMigrator(migrations_dir=dir_path) | ||
|
|
||
| # Show list for confirmation | ||
| if not dry_run and not yes: | ||
| status_info = run_async(migrator.status()) | ||
| if status_info["pending_migrations"]: | ||
| listing = "\n".join( | ||
| f"- {m}" | ||
| for m in status_info["pending_migrations"][ | ||
| : (limit or len(status_info["pending_migrations"])) | ||
| ] | ||
| ) | ||
| if not click.confirm( | ||
| f"Run {min(limit or len(status_info['pending_migrations']), len(status_info['pending_migrations']))} migration(s)?\n{listing}" | ||
| ): | ||
| click.echo("Aborted.") | ||
| return | ||
|
|
||
| count = run_async(migrator.run(dry_run=dry_run, limit=limit, verbose=verbose)) | ||
| if verbose and not dry_run: | ||
| click.echo(f"Successfully applied {count} migration(s).") | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.argument("name") | ||
| @click.option("--migrations-dir", help="Directory to create migration in") | ||
| @click.option( | ||
| "--yes", "-y", is_flag=True, help="Skip confirmation prompt to create directory" | ||
| ) | ||
| def create(name: str, migrations_dir: Optional[str], yes: bool): | ||
| """Create a new schema migration snapshot file from current pending operations.""" | ||
| dir_path = migrations_dir or os.path.join( | ||
| get_root_migrations_dir(), "schema-migrations" | ||
| ) | ||
|
|
||
| if not os.path.exists(dir_path): | ||
| if yes or click.confirm(f"Create schema migrations directory at '{dir_path}'?"): | ||
| os.makedirs(dir_path, exist_ok=True) | ||
| else: | ||
| click.echo("Aborted.") | ||
| return | ||
|
|
||
| migrator = SchemaMigrator(migrations_dir=dir_path) | ||
| filepath = run_async(migrator.create_migration_file(name)) | ||
| if filepath: | ||
| click.echo(f"Created migration: {filepath}") | ||
| else: | ||
| click.echo("No pending schema changes detected. Nothing to snapshot.") | ||
|
|
||
|
|
||
| @migrate.command() | ||
| @click.argument("migration_id") | ||
| @click.option("--migrations-dir", help="Directory containing schema migration files") | ||
| @click.option( | ||
| "--dry-run", is_flag=True, help="Show what would be done without applying changes" | ||
| ) | ||
| @click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") | ||
| @click.option( | ||
| "--yes", | ||
| "-y", | ||
| is_flag=True, | ||
| help="Skip confirmation prompt to create directory or run", | ||
| ) | ||
| def rollback( | ||
| migration_id: str, | ||
| migrations_dir: Optional[str], | ||
| dry_run: bool, | ||
| verbose: bool, | ||
| yes: bool, | ||
| ): | ||
| """Rollback a specific schema migration by ID.""" | ||
| dir_path = migrations_dir or os.path.join( | ||
| get_root_migrations_dir(), "schema-migrations" | ||
| ) | ||
|
|
||
| if not os.path.exists(dir_path): | ||
| if yes or click.confirm(f"Create schema migrations directory at '{dir_path}'?"): | ||
| os.makedirs(dir_path, exist_ok=True) | ||
| else: | ||
| click.echo("Aborted.") | ||
| return | ||
|
|
||
| @click.command() | ||
| @click.option("--module", default="aredis_om") | ||
| def migrate(module: str): | ||
| migrator = Migrator(module) | ||
| migrator.detect_migrations() | ||
| migrator = SchemaMigrator(migrations_dir=dir_path) | ||
|
|
||
| if migrator.migrations: | ||
| print("Pending migrations:") | ||
| for migration in migrator.migrations: | ||
| print(migration) | ||
| if not yes and not dry_run: | ||
| if not click.confirm(f"Rollback migration '{migration_id}'?"): | ||
| click.echo("Aborted.") | ||
| return | ||
|
|
||
| if input("Run migrations? (y/n) ") == "y": | ||
| migrator.run() | ||
| success = run_async( | ||
| migrator.rollback(migration_id, dry_run=dry_run, verbose=verbose) | ||
| ) | ||
| if success: | ||
| if verbose: | ||
| click.echo(f"Successfully rolled back migration: {migration_id}") | ||
| else: | ||
| click.echo( | ||
| f"Migration '{migration_id}' does not support rollback or is not applied.", | ||
| err=True, | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.