Add strict loading and preloading of associations

docs/en/index.rst

@@ -71,6 +71,8 @@ Advanced Topics
 * :doc:`NamingStrategy <reference/namingstrategy>`
 * :doc:`TypedFieldMapper <reference/typedfieldmapper>`
 * :doc:`Improving Performance <reference/improving-performance>`
+* :doc:`Preloading Associations <reference/preloading>`
+* :doc:`Strict Loading <reference/strict-loading>`
 * :doc:`Caching <reference/caching>`
 * :doc:`Partial Hydration <reference/partial-hydration>`
 * :doc:`Partial Objects <reference/partial-objects>`

docs/en/reference/preloading.rst

@@ -0,0 +1,187 @@
+Preloading Associations
+=======================
+
+.. note::
+
+    ``EntityManager::preload()`` is a proposal for Doctrine ORM 4, discussed
+    together with :doc:`strict loading <strict-loading>` in
+    `discussion #10931 <https://github.com/doctrine/orm/discussions/10931>`_.
+
+A fetch join or an eager fetch mode loads associations while a query is being
+hydrated. That does not help for entities that are already in memory - from a
+repository method, a paginator, the second level cache, or a previous preload.
+Iterating over those and touching a lazy association is the N+1 problem, and
+rewriting the original query is not always an option.
+
+``preload()`` loads an association for many entities at once:
+
+.. code-block:: php
+
+    <?php
+    $users = $entityManager->getRepository(User::class)->findAll();
+
+    $entityManager->preload($users, ['articles']);
+
+    foreach ($users as $user) {
+        foreach ($user->getArticles() as $article) {
+            // no queries here
+        }
+    }
+
+One query is issued for all the collections together, in batches of
+``Configuration::setEagerFetchBatchSize()`` entities (100 by default). This is
+what Rails calls
+`preload <https://guides.rubyonrails.org/active_record_querying.html#preload>`_,
+as opposed to
+`eager_load <https://guides.rubyonrails.org/active_record_querying.html#eager-load>`_,
+which is a fetch join.
+
+Fetching and preloading in one call
+-----------------------------------
+
+The repository finders take the paths directly, so the common case does not
+need a second statement:
+
+.. code-block:: php
+
+    <?php
+    $users = $repository->findAll(['articles']);
+    $users = $repository->findBy(['active' => true], preload: ['articles.comments']);
+    $user  = $repository->find($id, preload: ['articles']);
+    $user  = $repository->findOneBy(['email' => $email], preload: ['articles']);
+
+The same on a query, for the associations that a fetch join cannot cover - a
+second collection next to one that is already joined, for instance:
+
+.. code-block:: php
+
+    <?php
+    $users = $entityManager->createQuery(
+        'SELECT u, a FROM App\Entity\User u LEFT JOIN u.articles a',
+    )->preload(['address', 'articles.comments'])->getResult();
+
+and on the query builder, which hands the paths to the query it creates:
+
+.. code-block:: php
+
+    <?php
+    $users = $repository->createQueryBuilder('u')
+        ->where('u.active = true')
+        ->preload(['articles'])
+        ->getQuery()
+        ->getResult();
+
+The paths are preloaded after hydration, so a query preload costs the same
+queries as calling ``preload()`` on the result would. ``toIterable()`` throws
+when paths are set: it hydrates one row at a time, so there is nothing to load
+the associations for in one query.
+
+``EntityManager::preload()`` and ``EntityRepository::preload()`` stay the way to
+preload entities you already hold - from a paginator, an event listener, or a
+previous preload - where there is no finder call to pass paths to.
+
+Paths
+-----
+
+A path may walk several associations. Every step is batched over everything the
+previous step loaded, so the number of queries depends on the depth of the path,
+not on the number of entities:
+
+.. code-block:: php
+
+    <?php
+    $entityManager->preload($users, ['articles.comments.author']);
+
+Several paths can be preloaded at once:
+
+.. code-block:: php
+
+    <?php
+    $entityManager->preload($users, ['articles.comments', 'address']);
+
+Passing no path at all initializes the given entities themselves, which turns a
+list of references into a single query:
+
+.. code-block:: php
+
+    <?php
+    $references = array_map(
+        static fn (int $id): User => $entityManager->getReference(User::class, $id),
+        $ids,
+    );
+
+    $entityManager->preload($references);
+
+What is loaded
+--------------
+
+============================  ==========================================
+Association                   Queries per batch
+============================  ==========================================
+``ManyToOne``, owning
+``OneToOne``                  1
+``OneToMany``                 1
+``ManyToMany``                2 (the join table, then the targets)
+Inverse ``OneToOne``          0 - the ORM already loads it while
+                              hydrating the owner
+============================  ==========================================
+
+A preload always loads the **whole** association: it marks collections as
+initialized, so a partially filled collection would be indistinguishable from a
+complete one. Use
+:ref:`matching() <filtering-collections>` when a filtered subset is what you need
+- it returns a separate result and leaves the collection alone.
+
+Skipped silently:
+
+-  associations that are already loaded, including collections filled by a fetch
+   join, and collections that were preloaded before;
+-  collections with unflushed changes - loading them would take a snapshot that
+   contradicts those changes, so they initialize on their own later;
+-  entities that are not managed by this ``EntityManager``.
+
+Preloading an association that does not exist throws
+``ORMInvalidArgumentException``, naming the class and the path: a typo must not
+silently bring the N+1 back.
+
+Ordering and indexing behave exactly as they do for a lazy load: the
+association's ``orderBy`` is applied in SQL, and ``indexBy`` keys the collection.
+
+Strict loading
+--------------
+
+``preload()`` is the way out of a
+:doc:`strict loading <strict-loading>` violation for entities you already hold.
+Fetch first, then forbid loading:
+
+.. code-block:: php
+
+    <?php
+    $users = $repository->findAll(['articles.comments']);
+
+    $strictLoading->setMode(StrictLoadingMode::All);
+
+    return $this->render('users.html.twig', ['users' => $users]);
+
+Preloading itself never triggers a violation.
+
+Batching without ``preload()``
+------------------------------
+
+The same batching runs while a query is hydrated, for associations mapped with
+``fetch: 'EAGER'`` or overridden per query:
+
+.. code-block:: php
+
+    <?php
+    $query = $entityManager->createQuery('SELECT u FROM App\Entity\User u')
+        ->setFetchMode(User::class, 'articles', ClassMetadata::FETCH_EAGER);
+
+Batching is skipped, and the association is loaded one owner at a time, when:
+
+-  the query is iterated with ``toIterable()`` - rows are hydrated one by one, so
+   there is nothing to batch them with;
+-  the result comes from the second level cache;
+-  the association has a composite key on the side that has to be filtered;
+-  ``indexBy`` names a column rather than a mapped field, because only the SQL
+   result set can resolve that.

docs/en/reference/strict-loading.rst

@@ -0,0 +1,192 @@
+Strict Loading
+==============
+
+.. note::
+
+    Strict loading is a proposal for Doctrine ORM 4 and is being discussed in
+    `discussion #10931 <https://github.com/doctrine/orm/discussions/10931>`_.
+
+Lazy loading is convenient, but it hides database access behind ordinary
+property reads. The result is the N+1 query problem: a loop over 100 entities
+that touches a lazy association issues 101 queries, and nothing in the code
+shows that this happens.
+
+Strict loading turns that implicit database access into a reported violation.
+The idea comes from Ruby on Rails, which has had
+`strict loading <https://guides.rubyonrails.org/active_record_querying.html#strict-loading>`_
+since 6.1 (`ActiveRecord::Base#strict_loading!
+<https://api.rubyonrails.org/classes/ActiveRecord/Core.html#method-i-strict_loading-21>`_).
+The typical setup is to fetch everything a request needs up front - with fetch
+joins, ``Query::setFetchMode()`` or an eager fetch mode - and then forbid
+further loading for the rest of the request, for example while rendering a
+template or serializing a response.
+
+Enabling strict loading
+-----------------------
+
+Strict loading is configured per ``EntityManager`` and is disabled by default:
+
+.. code-block:: php
+
+    <?php
+    use Doctrine\ORM\StrictLoading\StrictLoadingMode;
+
+    $strictLoading = $entityManager->getConfiguration()->getStrictLoading();
+    $strictLoading->setMode(StrictLoadingMode::All);
+
+From now on, loading an uninitialized entity or collection throws
+``Doctrine\ORM\Exception\StrictLoadingViolation``:
+
+.. code-block:: php
+
+    <?php
+    $article = $entityManager->find(Article::class, 1);
+
+    echo $article->getAuthor()->getName();
+    // Strict loading violation: lazily loading entity App\Entity\User(42) is not
+    // allowed here. Fetch the data explicitly (fetch join, Query::setFetchMode()
+    // or an eager fetch mode), or wrap the offending code in
+    // Doctrine\ORM\StrictLoading\StrictLoading::allow().
+
+Fetching the association explicitly makes the code work again:
+
+.. code-block:: php
+
+    <?php
+    $article = $entityManager->createQuery(
+        'SELECT a, u FROM App\Entity\Article a JOIN a.author u WHERE a.id = :id',
+    )->setParameter('id', 1)->getSingleResult();
+
+    echo $article->getAuthor()->getName(); // no lazy load, no violation
+
+Modes
+-----
+
+``StrictLoadingMode::Disabled``
+    Lazy loading is allowed. This is the default and the historical behavior.
+
+``StrictLoadingMode::NPlusOneOnly``
+    Only lazy loads that repeat are reported. The first lazy load of a given
+    association - or of a given entity class, for to-one references - is
+    allowed, every following one is a violation. This is the mode to start with
+    in an existing application, because it only complains about loads that
+    actually degrade into an N+1 query.
+
+``StrictLoadingMode::All``
+    Every lazy load is reported. All data has to be fetched explicitly.
+
+The two active modes are the ones Rails offers as ``:n_plus_one_only`` and
+``:all``.
+
+Limiting strict loading to part of a request
+--------------------------------------------
+
+The mode can be changed at any point, which is how you separate "fetching" from
+"rendering":
+
+.. code-block:: php
+
+    <?php
+    // Controller: fetch everything the view needs.
+    $users = $repository->findAll(['articles']);
+
+    // View: no database access allowed from here on.
+    $strictLoading->setMode(StrictLoadingMode::All);
+
+    return $this->render('users.html.twig', ['users' => $users]);
+
+Reset the mode once per request - in a ``kernel.request`` listener, or wherever
+the application sets up its unit of work - so that a request that switched to
+``All`` does not affect the next one.
+
+``allow()`` is the escape hatch for code that knowingly lazy loads. It takes a
+callback because the previous state has to be restored even when the callback
+throws:
+
+.. code-block:: php
+
+    <?php
+    $author = $strictLoading->allow(static fn (): User => $article->getAuthor());
+
+Reporting instead of throwing
+-----------------------------
+
+By default a violation is thrown. Passing a different
+``StrictLoadingViolationHandler`` lets the load happen and only records it,
+which is useful to introduce strict loading into an existing application, or to
+run in ``NPlusOneOnly`` mode in production while the test suite throws. Rails
+makes the same distinction with
+`config.active_record.action_on_strict_loading_violation
+<https://guides.rubyonrails.org/configuring.html#config-active-record-action-on-strict-loading-violation>`_,
+which is either ``:raise`` or ``:log``:
+
+.. code-block:: php
+
+    <?php
+    use Doctrine\ORM\StrictLoading\LogViolation;
+    use Doctrine\ORM\StrictLoading\StrictLoading;
+    use Doctrine\ORM\StrictLoading\StrictLoadingMode;
+
+    $entityManager->getConfiguration()->setStrictLoading(new StrictLoading(
+        StrictLoadingMode::NPlusOneOnly,
+        new LogViolation($logger),
+    ));
+
+A custom handler receives a ``LazyLoad`` describing what was about to be
+loaded, and may for instance add the violation to a profiler collector:
+
+.. code-block:: php
+
+    <?php
+    use Doctrine\ORM\StrictLoading\LazyLoad;
+    use Doctrine\ORM\StrictLoading\StrictLoadingMode;
+    use Doctrine\ORM\StrictLoading\StrictLoadingViolationHandler;
+
+    final class CollectViolations implements StrictLoadingViolationHandler
+    {
+        /** @var list<LazyLoad> */
+        public array $violations = [];
+
+        public function violation(LazyLoad $lazyLoad, StrictLoadingMode $mode): void
+        {
+            $this->violations[] = $lazyLoad;
+        }
+    }
+
+What counts as a violation
+--------------------------
+
+Reported:
+
+-  initializing an entity reference (a proxy), including references created by
+   ``EntityManager::getReference()``;
+-  loading an uninitialized collection;
+-  a query that an uninitialized ``EXTRA_LAZY`` collection runs on its own -
+   ``count()``, ``contains()``, ``containsKey()``, ``get()``, ``first()``,
+   ``slice()`` and ``matching()``.
+
+Never reported, because the ORM loads on purpose there:
+
+-  everything that happens during ``flush()``, including change set
+   computation, orphan removal and cascades;
+-  ``persist()``, ``remove()``, ``refresh()`` and ``lock()``;
+-  explicit initialization through ``EntityManager::initializeObject()``.
+
+Not reported, because no lazy loading is involved: fetch joins, eager fetch
+modes (``fetch: 'EAGER'`` and ``Query::setFetchMode()``, which load in batches),
+``EntityManager::find()`` and DQL queries. Inverse to-one associations are
+loaded while hydrating the owning entity, not lazily, and are therefore not
+reported either - use an eager fetch mode or a fetch join to avoid the query
+per row.
+
+Caveats
+-------
+
+Entity references are shared through the identity map, so a violation names the
+entity that was about to be loaded, not the property that was read. Collection
+violations name the owning class and the field.
+
+In ``NPlusOneOnly`` mode, "repeated" means "seen before in the current scope".
+The scope is reset by ``EntityManager::clear()`` and by
+``StrictLoading::reset()`` - call the latter once per request in a long-running
+worker.

docs/en/sidebar.rst

@@ -42,6 +42,8 @@
    reference/php-mapping
    reference/caching
    reference/improving-performance
+   reference/preloading
+   reference/strict-loading
    reference/tools
    reference/metadata-drivers
    reference/best-practices