Skip to content

Unasyncify Codegen Reference Implementation #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 139 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
3ef0564
Fixed #35629. Async DB Connection and Cursor
fcurella Jul 11, 2024
c84c2e4
Fix typos
fcurella Oct 22, 2024
083209c
add connection type (async or non-async) in logs
fcurella Oct 22, 2024
259905f
add `versionadded` note in docs
fcurella Oct 22, 2024
99817ec
fix redundant call to `get_database_version`
fcurella Oct 22, 2024
3813983
validate thread sharing
fcurella Oct 22, 2024
d77f89a
Remove unused method
fcurella Oct 22, 2024
acbb8c5
await `_astart_transaction_under_autocommit` method if implemented
fcurella Oct 22, 2024
d72b4f9
update comment
fcurella Oct 22, 2024
f809227
only call `aget_autocommit` if `commit` is truthy
fcurella Oct 23, 2024
20ed0bc
Remove outdated comment
fcurella Oct 23, 2024
1dbee13
refactor isolation level in postgres
fcurella Oct 23, 2024
69ee23d
Revert "Remove unused method"
fcurella Oct 24, 2024
463cf28
ensure connection when entering the async context manager
fcurella Oct 24, 2024
50d813b
test connection state
fcurella Oct 24, 2024
bb1834d
Add guard against non-async databases on `new_connection`
fcurella Oct 25, 2024
cd96f15
rename `execute` and `execute_many` to `aexecute` and `aexecute_many`
fcurella Oct 25, 2024
a40bf4a
add `afetchone()`, `afetchmany()`, `afetchall()`
fcurella Oct 25, 2024
a323c8d
add `acopy`, `astream` and `ascroll` to async cursors
fcurella Oct 25, 2024
83634e7
remove unused sentinel
fcurella Nov 6, 2024
9c0aa75
await pooled connections methods
fcurella Nov 8, 2024
51815b9
Update docs/topics/db/sql.txt
fcurella Nov 8, 2024
8010206
Update docs/releases/5.2.txt
fcurella Nov 8, 2024
8ae8e75
fix test skipping
fcurella Nov 8, 2024
c4255e1
fix label
fcurella Nov 8, 2024
86c4c9c
remove duplicated test
fcurella Nov 8, 2024
887a3f4
add codemod
rtpg Oct 25, 2024
eeef81f
Add unasync codegen helpers
rtpg Oct 25, 2024
6666152
Support asyunc_unsafe optionality in codegen
rtpg Oct 26, 2024
da364b5
generate_unasynced_codegen -> generate_unasynced
rtpg Oct 26, 2024
4a374ae
async for/with -> for/with
rtpg Oct 29, 2024
5ccbc73
generate_unasynced
rtpg Nov 11, 2024
73b154e
async for/width on the right transformer
rtpg Nov 11, 2024
a824930
add the names properly
rtpg Nov 11, 2024
330e70f
Add test_postgresql
rtpg Nov 11, 2024
8920cdb
add some signatures for my own benefit
rtpg Nov 13, 2024
3df74ac
codegen more helpers
rtpg Nov 13, 2024
3cb50b8
handle for comprehensions in code gen
rtpg Nov 13, 2024
b7169b5
more codegen
rtpg Jan 26, 2025
e4d9341
check not in atomic block
rtpg Nov 13, 2024
ce3d00a
even more codegen
rtpg Nov 13, 2024
55b6834
Fix available_apps check
rtpg Nov 13, 2024
f7adeeb
wrap db connections for debugging purposes for now
rtpg Nov 14, 2024
fef950e
figure out comingling case
rtpg Nov 14, 2024
771d268
Way more nonsense
rtpg Nov 18, 2024
69a55b9
add test_pg_async settings
rtpg Nov 18, 2024
e18ed53
Fixed #35918 -- Refactored execute_sql to reduce cursor management
rtpg Nov 19, 2024
c9a3351
remove attr and meth doc usage
rtpg Nov 19, 2024
f0e7abb
if -> elif
rtpg Nov 19, 2024
1a55ad2
don't rollback in testss
rtpg Nov 19, 2024
6c1e49d
rollback code
rtpg Nov 20, 2024
4d6fc79
cleanup some test nonsense
rtpg Nov 20, 2024
b8be7ae
patch up variations
rtpg Nov 20, 2024
7d29da0
use LEAK_CURSOR
rtpg Nov 20, 2024
93f1661
use_async_connections
rtpg Nov 20, 2024
e8e2fad
fix queryset proxying label
rtpg Nov 20, 2024
4e8519d
test runner helper
rtpg Nov 20, 2024
f602f1a
catch any non-rollbacking cxns
rtpg Nov 20, 2024
363dfee
pdb support for my partial running
rtpg Nov 20, 2024
082fbfd
fix up async
rtpg Nov 20, 2024
55ba900
use_async_connections some more
rtpg Nov 20, 2024
bc3fb7e
Try to fetch more
rtpg Nov 20, 2024
4a286c2
Fix TestCase.use_async_connections
rtpg Nov 20, 2024
da397c0
Fix test runner failure on Ctrl-C
rtpg Nov 21, 2024
f6553f6
Async generator
rtpg Nov 21, 2024
88110b8
Add sync op blocker utility for testing
rtpg Nov 21, 2024
17fcc4a
Have aget fall back to its sync variants when not in new_connection
rtpg Nov 23, 2024
9e9912a
more is_commit_allowed nonsense
rtpg Nov 27, 2024
50a2cbb
adding pq for otel instrumentation
rtpg Nov 27, 2024
bedf598
queryset aiter should use afetch_all
rtpg Nov 27, 2024
72b894b
Make sure that we actually get the "right" connection back
rtpg Nov 27, 2024
e777afe
Make async queryset methods have more coverage
rtpg Nov 28, 2024
3d3e6a4
Add some extra coverage around the edges
rtpg Nov 28, 2024
52e5162
acount
rtpg Nov 28, 2024
4f6dd71
acount (part2)
rtpg Nov 28, 2024
1bad0e2
acreate
rtpg Nov 28, 2024
6030bb1
aget
rtpg Nov 28, 2024
7e29a3f
transaction.atomic async support
rtpg Nov 29, 2024
fb19b11
aupdate_or_create
rtpg Nov 29, 2024
c6af21f
aas_sql
rtpg Nov 29, 2024
5bc38dd
aas_sql (continued)
rtpg Nov 29, 2024
2aa2628
aerliest/alatest
rtpg Nov 29, 2024
0f06a03
first/last
rtpg Nov 29, 2024
6596979
More async variants on delete
rtpg Nov 29, 2024
f7eeaeb
mark_for_rollback
rtpg Nov 29, 2024
ec77a8e
even more coverage of unknown stuff
rtpg Nov 29, 2024
e44e5c7
complete up all public APIs on queryset
rtpg Nov 29, 2024
8994a50
fix issues around boolean checks
rtpg Nov 29, 2024
e48b5f5
LEAK_CURSOR -> CURSOR
rtpg Dec 4, 2024
7e9c1fe
Increase statement timeout window for testing
rtpg Dec 4, 2024
41e8a03
Add testing of sync model overrides
rtpg Dec 4, 2024
4d0ab9b
Add contexts to coverage reports
rtpg Dec 4, 2024
884c8ad
Add tests for batch size exception
rtpg Dec 4, 2024
3be4ce0
improve xor coverage
rtpg Dec 4, 2024
f82a62c
make sure the model override properly labels async
rtpg Dec 4, 2024
c425455
simplify ASYNC_TRUTH_MARKER
rtpg Dec 4, 2024
accecfc
fix up annotation location
rtpg Dec 4, 2024
9346581
remove DB tracing
rtpg Dec 4, 2024
dc40ecf
allow/deny_async_db_commits
rtpg Dec 4, 2024
ca7b18e
remove some tracking code
rtpg Dec 4, 2024
4ce0993
remove a type signature
rtpg Dec 4, 2024
44d7a6c
typo cleanup
rtpg Dec 4, 2024
9206408
annotating some names as experimentation
rtpg Dec 4, 2024
db4423e
private: hide some local files
rtpg Jan 26, 2025
0eed4b5
private: add sync scripts
rtpg Jan 26, 2025
008fd50
private: file mode
rtpg Jan 26, 2025
586accd
private: .gitignore
rtpg Jan 26, 2025
a8a8341
private: more file ignores
rtpg Jan 26, 2025
f0a0636
private: file ignore
rtpg Jan 26, 2025
3797b06
Remove Django 60 deprecation issues
rtpg Jan 26, 2025
b9a9609
revert changeup on available_apps for now
rtpg Jan 26, 2025
e2fd9ed
Remove spurious tests
rtpg Jan 26, 2025
7afc27b
rtpg Jan 26, 2025
1e87960
private: update codegen script
rtpg Jan 26, 2025
0cc757c
Add async generic relation test
rtpg Jan 26, 2025
420078f
Name
rtpg Jan 26, 2025
dcedd07
backport save changes to asave
rtpg Jan 26, 2025
719d0c8
Backport _save_table changes to _asave_table
rtpg Jan 26, 2025
d09df67
Backport changes to new async canonical model
rtpg Jan 26, 2025
5aea537
Backport changes to django/db/models/sql/compiler.py
rtpg Jan 26, 2025
1f5f77c
Backport changes to django/db/models/sql/query.py
rtpg Jan 26, 2025
64b830f
Allow for codegenning of tests as well
rtpg Jan 26, 2025
5320798
make new_connection no-op in sync mode for tests
rtpg Jan 26, 2025
d8586ce
Remove reference to old connection alias
rtpg Jan 26, 2025
d7b6e08
don't transform all to ll
rtpg Jan 26, 2025
dc5394f
Codegen coverage for bulk_create
rtpg Jan 26, 2025
37bfab6
private: run_async_qs should generate contexts
rtpg Jan 26, 2025
2097907
Add generic_relation test
rtpg Jan 26, 2025
35c0fd9
_aprefetch_related_objects on QuerySet
rtpg Jan 26, 2025
80e13d0
Add helper utility to get a list from an async iterator
rtpg Jan 26, 2025
51c3a24
Move testdata-y tests over to TransactionTestCase
rtpg Jan 26, 2025
d2f9a52
private: add some notes
rtpg Jan 26, 2025
72580c2
some bulk create tests
rtpg Jan 26, 2025
d6b88eb
Some more coverage
rtpg Jan 26, 2025
6083449
private: notes
rtpg Jan 26, 2025
b74263c
more test coverage
rtpg Jan 26, 2025
04bb8ac
more random coverage
rtpg Jan 26, 2025
05cd5c8
More coverage on test returning
rtpg Jan 26, 2025
097e59c
coverage for earliest
rtpg Jan 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,13 @@ tests/.coverage*
build/
tests/report/
tests/screenshots/

.direnv

*.sqlite3
passed.tests

.coverage
.envrc
uv.lock
*.cobp
17 changes: 17 additions & 0 deletions .libcst.codemod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# String that LibCST should look for in code which indicates that the
# module is generated code.
generated_code_marker: '@generated'
# Command line and arguments for invoking a code formatter. Anything
# specified here must be capable of taking code via stdin and returning
# formatted code via stdout.
formatter: ['black', '-']
# List of regex patterns which LibCST will evaluate against filenames to
# determine if the module should be touched.
blacklist_patterns: []
# List of modules that contain codemods inside of them.
modules:
- 'django.utils.codegen'
# Absolute or relative path of the repository root, used for providing
# full-repo metadata. Relative paths should be specified with this file
# location as the base.
repo_root: '.'
123 changes: 123 additions & 0 deletions django/db/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
from contextlib import contextmanager

Check failure on line 1 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / isort

Imports are incorrectly sorted and/or formatted.
import os

Check failure on line 2 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / flake8

'os' imported but unused
from asgiref.local import Local

from django.core import signals
from django.db.utils import (
DEFAULT_DB_ALIAS,
DJANGO_VERSION_PICKLE_KEY,
AsyncConnectionHandler,
ConnectionHandler,
ConnectionRouter,
DatabaseError,
Expand Down Expand Up @@ -36,6 +41,124 @@
]

connections = ConnectionHandler()
async_connections = AsyncConnectionHandler()

new_connection_block_depth = Local()
new_connection_block_depth.value = 0


def modify_cxn_depth(f):
try:
existing_value = new_connection_block_depth.value
except AttributeError:
existing_value = 0
new_connection_block_depth.value = f(existing_value)


def should_use_sync_fallback(async_variant):
return async_variant and (new_connection_block_depth.value == 0)


commit_allowed = Local()
commit_allowed.value = False

from contextlib import contextmanager

Check failure on line 65 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / flake8

redefinition of unused 'contextmanager' from line 1

Check failure on line 65 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / flake8

module level import not at top of file


@contextmanager
def set_async_db_commit_permission(perm):
old_value = getattr(commit_allowed, "value", True)
commit_allowed.value = perm
try:
yield
finally:
commit_allowed.value = old_value


@contextmanager
def allow_async_db_commits():
with set_async_db_commit_permission(True):
yield


@contextmanager
def block_async_db_commits():
with set_async_db_commit_permission(False):
yield


def is_commit_allowed():
try:
return commit_allowed.value
except:

Check failure on line 93 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / flake8

do not use bare 'except'
# XXX making sure its set
commit_allowed.value = True
return True


class new_connection:
"""
Asynchronous context manager to instantiate new async connections.

"""

def __init__(self, using=DEFAULT_DB_ALIAS, force_rollback=False):
self.using = using
if not force_rollback and not is_commit_allowed():
# this is for just figuring everything out
raise ValueError(
"Commits are currently blocked, use allow_async_db_commits to unblock"
)
self.force_rollback = force_rollback

def __enter__(self):
# XXX I need to fix up the codegen, for now this is going to no-op
if self.force_rollback:
# XXX IN TEST CONTEXT!
return
else:
raise NotSupportedError("new_connection doesn't support a sync context")

def __exit__(self, exc_type, exc_value, traceback):
# XXX another thing to remove
return

async def __aenter__(self):
# XXX stupid nonsense
modify_cxn_depth(lambda v: v + 1)
conn = connections.create_connection(self.using)
if conn.supports_async is False:
raise NotSupportedError(
"The database backend does not support asynchronous execution."
)

if conn.in_atomic_block:
raise NotSupportedError(
"Can't open an async connection while inside of a synchronous transaction block"

Check failure on line 137 in django/db/__init__.py

View workflow job for this annotation

GitHub Actions / flake8

line too long (96 > 88 characters)
)
self.conn = conn

async_connections.add_connection(self.using, self.conn)

await self.conn.aensure_connection()
if self.force_rollback is True:
await self.conn.aset_autocommit(False)

return self.conn

async def __aexit__(self, exc_type, exc_value, traceback):
# silly nonsense (again)
modify_cxn_depth(lambda v: v - 1)
autocommit = await self.conn.aget_autocommit()
if autocommit is False:
if exc_type is None and self.force_rollback is False:
await self.conn.acommit()
else:
await self.conn.arollback()
await self.conn.aclose()

async_connections.pop_connection(self.using)


router = ConnectionRouter()

Expand Down
Loading
Loading