Merge pull request #11653 from beberlei/GH-8471-RevertPartialObjects2

[GH-8471] Undeprecate PARTIAL for objects in DQL

Author: greg0ire

Diff for: docs/en/index.rst

@@ -74,6 +74,7 @@ Advanced Topics
 * :doc:`Improving Performance <reference/improving-performance>`
 * :doc:`Caching <reference/caching>`
 * :doc:`Partial Hydration <reference/partial-hydration>`
+* :doc:`Partial Objects <reference/partial-objects>`
 * :doc:`Change Tracking Policies <reference/change-tracking-policies>`
 * :doc:`Best Practices <reference/best-practices>`
 * :doc:`Metadata Drivers <reference/metadata-drivers>`

Diff for: docs/en/reference/dql-doctrine-query-language.rst

@@ -533,14 +533,23 @@ back. Instead, you receive only arrays as a flat rectangular result
 set, similar to how you would if you were just using SQL directly
 and joining some data.
 
-If you want to select a partial number of fields for hydration entity in
-the context of array hydration and joins you can use the ``partial`` DQL keyword:
+If you want to select partial objects or fields in array hydration you can use the ``partial``
+DQL keyword:
+
+.. code-block:: php
+
+    <?php
+    $query = $em->createQuery('SELECT partial u.{id, username} FROM CmsUser u');
+    $users = $query->getResult(); // array of partially loaded CmsUser objects
+
+You can use the partial syntax when joining as well:
 
 .. code-block:: php
 
     <?php
     $query = $em->createQuery('SELECT partial u.{id, username}, partial a.{id, name} FROM CmsUser u JOIN u.articles a');
-    $users = $query->getArrayResult(); // array of partially loaded CmsUser and CmsArticle fields
+    $usersArray = $query->getArrayResult(); // array of partially loaded CmsUser and CmsArticle fields
+    $users = $query->getResult(); // array of partially loaded CmsUser objects
 
 "NEW" Operator Syntax
 ^^^^^^^^^^^^^^^^^^^^^
@@ -1427,6 +1436,15 @@ exist mostly internal query hints that are not be consumed in
 userland. However the following few hints are to be used in
 userland:
 
+
+-  ``Query::HINT_FORCE_PARTIAL_LOAD`` - Allows to hydrate objects
+   although not all their columns are fetched. This query hint can be
+   used to handle memory consumption problems with large result-sets
+   that contain char or binary data. Doctrine has no way of implicitly
+   reloading this data. Partially loaded objects have to be passed to
+   ``EntityManager::refresh()`` if they are to be reloaded fully from
+   the database. This query hint is deprecated and will be removed
+   in the future (\ `Details <https://github.com/doctrine/orm/issues/8471>`_)
 -  ``Query::HINT_REFRESH`` - This query is used internally by
    ``EntityManager::refresh()`` and can be used in userland as well.
    If you specify this hint and a query returns the data for an entity

Diff for: docs/en/reference/partial-hydration.rst

@@ -1,11 +1,6 @@
 Partial Hydration
 =================
 
-.. note::
-
-    Creating Partial Objects through DQL was possible in ORM 2,
-    but is only supported for array hydration as of ORM 3.
-
 Partial hydration of entities is allowed in the array hydrator, when
 only a subset of the fields of an entity are loaded from the database
 and the nested results are still created based on the entity relationship structure.

Diff for: docs/en/reference/partial-objects.rst

@@ -0,0 +1,88 @@
+Partial Objects
+===============
+
+A partial object is an object whose state is not fully initialized
+after being reconstituted from the database and that is
+disconnected from the rest of its data. The following section will
+describe why partial objects are problematic and what the approach
+of Doctrine to this problem is.
+
+.. note::
+
+    The partial object problem in general does not apply to
+    methods or queries where you do not retrieve the query result as
+    objects. Examples are: ``Query#getArrayResult()``,
+    ``Query#getScalarResult()``, ``Query#getSingleScalarResult()``,
+    etc.
+
+.. warning::
+
+    Use of partial objects is tricky. Fields that are not retrieved
+    from the database will not be updated by the UnitOfWork even if they
+    get changed in your objects. You can only promote a partial object
+    to a fully-loaded object by calling ``EntityManager#refresh()``
+    or a DQL query with the refresh flag.
+
+
+What is the problem?
+--------------------
+
+In short, partial objects are problematic because they are usually
+objects with broken invariants. As such, code that uses these
+partial objects tends to be very fragile and either needs to "know"
+which fields or methods can be safely accessed or add checks around
+every field access or method invocation. The same holds true for
+the internals, i.e. the method implementations, of such objects.
+You usually simply assume the state you need in the method is
+available, after all you properly constructed this object before
+you pushed it into the database, right? These blind assumptions can
+quickly lead to null reference errors when working with such
+partial objects.
+
+It gets worse with the scenario of an optional association (0..1 to
+1). When the associated field is NULL, you don't know whether this
+object does not have an associated object or whether it was simply
+not loaded when the owning object was loaded from the database.
+
+These are reasons why many ORMs do not allow partial objects at all
+and instead you always have to load an object with all its fields
+(associations being proxied). One secure way to allow partial
+objects is if the programming language/platform allows the ORM tool
+to hook deeply into the object and instrument it in such a way that
+individual fields (not only associations) can be loaded lazily on
+first access. This is possible in Java, for example, through
+bytecode instrumentation. In PHP though this is not possible, so
+there is no way to have "secure" partial objects in an ORM with
+transparent persistence.
+
+Doctrine, by default, does not allow partial objects. That means,
+any query that only selects partial object data and wants to
+retrieve the result as objects (i.e. ``Query#getResult()``) will
+raise an exception telling you that partial objects are dangerous.
+If you want to force a query to return you partial objects,
+possibly as a performance tweak, you can use the ``partial``
+keyword as follows:
+
+.. code-block:: php
+
+    <?php
+    $q = $em->createQuery("select partial u.{id,name} from MyApp\Domain\User u");
+
+You can also get a partial reference instead of a proxy reference by
+calling:
+
+.. code-block:: php
+
+    <?php
+    $reference = $em->getPartialReference('MyApp\Domain\User', 1);
+
+Partial references are objects with only the identifiers set as they
+are passed to the second argument of the ``getPartialReference()`` method.
+All other fields are null.
+
+When should I force partial objects?
+------------------------------------
+
+Mainly for optimization purposes, but be careful of premature
+optimization as partial objects lead to potentially more fragile
+code.

Diff for: docs/en/sidebar.rst

@@ -38,6 +38,7 @@
    reference/native-sql
    reference/change-tracking-policies
    reference/partial-hydration
+   reference/partial-objects
    reference/attributes-reference
    reference/xml-mapping
    reference/php-mapping

Diff for: src/Cache/DefaultQueryCache.php

@@ -16,6 +16,7 @@
 use Doctrine\ORM\PersistentCollection;
 use Doctrine\ORM\Query;
 use Doctrine\ORM\Query\ResultSetMapping;
+use Doctrine\ORM\Query\SqlWalker;
 use Doctrine\ORM\UnitOfWork;
 
 use function array_map;
@@ -210,6 +211,10 @@ public function put(QueryCacheKey $key, ResultSetMapping $rsm, mixed $result, ar
             throw FeatureNotImplemented::nonSelectStatements();
         }
 
+        if (($hints[SqlWalker::HINT_PARTIAL] ?? false) === true || ($hints[Query::HINT_FORCE_PARTIAL_LOAD] ?? false) === true) {
+            throw FeatureNotImplemented::partialEntities();
+        }
+
         if (! ($key->cacheMode & Cache::MODE_PUT)) {
             return false;
         }

Diff for: src/Cache/Exception/FeatureNotImplemented.php

@@ -20,4 +20,9 @@ public static function nonSelectStatements(): self
     {
         return new self('Second-level cache query supports only select statements.');
     }
+
+    public static function partialEntities(): self
+    {
+        return new self('Second level cache does not support partial entities.');
+    }
 }

Diff for: src/Query.php

@@ -74,6 +74,14 @@ class Query extends AbstractQuery
      */
     public const HINT_REFRESH_ENTITY = 'doctrine.refresh.entity';
 
+    /**
+     * The forcePartialLoad query hint forces a particular query to return
+     * partial objects.
+     *
+     * @todo Rename: HINT_OPTIMIZE
+     */
+    public const HINT_FORCE_PARTIAL_LOAD = 'doctrine.forcePartialLoad';
+
     /**
      * The includeMetaColumns query hint causes meta columns like foreign keys and
      * discriminator columns to be selected and returned as part of the query result.

Diff for: src/Query/Parser.php

@@ -9,7 +9,6 @@
 use Doctrine\ORM\EntityManagerInterface;
 use Doctrine\ORM\Exception\DuplicateFieldException;
 use Doctrine\ORM\Exception\NoMatchingPropertyException;
-use Doctrine\ORM\Internal\Hydration\HydrationException;
 use Doctrine\ORM\Mapping\AssociationMapping;
 use Doctrine\ORM\Mapping\ClassMetadata;
 use Doctrine\ORM\Query;
@@ -1702,10 +1701,6 @@ public function JoinAssociationDeclaration(): AST\JoinAssociationDeclaration
      */
     public function PartialObjectExpression(): AST\PartialObjectExpression
     {
-        if ($this->query->getHydrationMode() === Query::HYDRATE_OBJECT) {
-            throw HydrationException::partialObjectHydrationDisallowed();
-        }
-
         $this->match(TokenType::T_PARTIAL);
 
         $partialFieldSet = [];

Diff for: src/Query/QueryException.php

@@ -88,6 +88,15 @@ public static function iterateWithFetchJoinCollectionNotAllowed(AssociationMappi
         );
     }
 
+    public static function partialObjectsAreDangerous(): self
+    {
+        return new self(
+            'Loading partial objects is dangerous. Fetch full objects or consider ' .
+            'using a different fetch mode. If you really want partial objects, ' .
+            'set the doctrine.forcePartialLoad query hint to TRUE.',
+        );
+    }
+
     /**
      * @param string[] $assoc
      * @psalm-param array<string, string> $assoc

Diff for: src/Query/SqlWalker.php

@@ -335,6 +335,11 @@ private function generateClassTableInheritanceJoins(
             $sql .= implode(' AND ', array_filter($sqlParts));
         }
 
+        // Ignore subclassing inclusion if partial objects is disallowed
+        if ($this->query->getHint(Query::HINT_FORCE_PARTIAL_LOAD)) {
+            return $sql;
+        }
+
         // LEFT JOIN child class tables
         foreach ($class->subClasses as $subClassName) {
             $subClass   = $this->em->getClassMetadata($subClassName);
@@ -669,7 +674,8 @@ public function walkSelectClause(AST\SelectClause $selectClause): string
             $this->query->setHint(self::HINT_DISTINCT, true);
         }
 
-        $addMetaColumns = $this->query->getHydrationMode() === Query::HYDRATE_OBJECT
+        $addMetaColumns = ! $this->query->getHint(Query::HINT_FORCE_PARTIAL_LOAD) &&
+            $this->query->getHydrationMode() === Query::HYDRATE_OBJECT
             || $this->query->getHint(Query::HINT_INCLUDE_META_COLUMNS);
 
         foreach ($this->selectedClasses as $selectedClass) {
@@ -1408,28 +1414,30 @@ public function walkSelectExpression(AST\SelectExpression $selectExpression): st
                 // 1) on Single Table Inheritance: always, since its marginal overhead
                 // 2) on Class Table Inheritance only if partial objects are disallowed,
                 //    since it requires outer joining subtables.
-                foreach ($class->subClasses as $subClassName) {
-                    $subClass      = $this->em->getClassMetadata($subClassName);
-                    $sqlTableAlias = $this->getSQLTableAlias($subClass->getTableName(), $dqlAlias);
+                if ($class->isInheritanceTypeSingleTable() || ! $this->query->getHint(Query::HINT_FORCE_PARTIAL_LOAD)) {
+                    foreach ($class->subClasses as $subClassName) {
+                        $subClass      = $this->em->getClassMetadata($subClassName);
+                        $sqlTableAlias = $this->getSQLTableAlias($subClass->getTableName(), $dqlAlias);
 
-                    foreach ($subClass->fieldMappings as $fieldName => $mapping) {
-                        if (isset($mapping->inherited) || ($partialFieldSet && ! in_array($fieldName, $partialFieldSet, true))) {
-                            continue;
-                        }
+                        foreach ($subClass->fieldMappings as $fieldName => $mapping) {
+                            if (isset($mapping->inherited) || ($partialFieldSet && ! in_array($fieldName, $partialFieldSet, true))) {
+                                continue;
+                            }
 
-                        $columnAlias      = $this->getSQLColumnAlias($mapping->columnName);
-                        $quotedColumnName = $this->quoteStrategy->getColumnName($fieldName, $subClass, $this->platform);
+                            $columnAlias      = $this->getSQLColumnAlias($mapping->columnName);
+                            $quotedColumnName = $this->quoteStrategy->getColumnName($fieldName, $subClass, $this->platform);
 
-                        $col = $sqlTableAlias . '.' . $quotedColumnName;
+                            $col = $sqlTableAlias . '.' . $quotedColumnName;
 
-                        $type = Type::getType($mapping->type);
-                        $col  = $type->convertToPHPValueSQL($col, $this->platform);
+                            $type = Type::getType($mapping->type);
+                            $col  = $type->convertToPHPValueSQL($col, $this->platform);
 
-                        $sqlParts[] = $col . ' AS ' . $columnAlias;
+                            $sqlParts[] = $col . ' AS ' . $columnAlias;
 
-                        $this->scalarResultAliasMap[$resultAlias][] = $columnAlias;
+                            $this->scalarResultAliasMap[$resultAlias][] = $columnAlias;
 
-                        $this->rsm->addFieldResult($dqlAlias, $columnAlias, $fieldName, $subClassName);
+                            $this->rsm->addFieldResult($dqlAlias, $columnAlias, $fieldName, $subClassName);
+                        }
                     }
                 }
 

Diff for: src/UnitOfWork.php

@@ -28,7 +28,6 @@
 use Doctrine\ORM\Exception\ORMException;
 use Doctrine\ORM\Exception\UnexpectedAssociationValue;
 use Doctrine\ORM\Id\AssignedGenerator;
-use Doctrine\ORM\Internal\Hydration\HydrationException;
 use Doctrine\ORM\Internal\HydrationCompleteHandler;
 use Doctrine\ORM\Internal\StronglyConnectedComponents;
 use Doctrine\ORM\Internal\TopologicalSort;
@@ -44,7 +43,6 @@
 use Doctrine\ORM\Persisters\Entity\JoinedSubclassPersister;
 use Doctrine\ORM\Persisters\Entity\SingleTablePersister;
 use Doctrine\ORM\Proxy\InternalProxy;
-use Doctrine\ORM\Query\SqlWalker;
 use Doctrine\ORM\Utility\IdentifierFlattener;
 use Doctrine\Persistence\PropertyChangedListener;
 use Exception;
@@ -2356,10 +2354,6 @@ public function isCollectionScheduledForDeletion(PersistentCollection $coll): bo
      */
     public function createEntity(string $className, array $data, array &$hints = []): object
     {
-        if (isset($hints[SqlWalker::HINT_PARTIAL])) {
-            throw HydrationException::partialObjectHydrationDisallowed();
-        }
-
         $class = $this->em->getClassMetadata($className);
 
         $id     = $this->identifierFlattener->flattenIdentifier($class, $data);