Add IQueryable projection support to read-only application services

docs/en/framework/architecture/domain-driven-design/application-services.md

@@ -444,6 +444,7 @@ These methods are low level methods that can control how to query entities from
 * `ApplyPaging` is used to make paging on the query. If your `TGetListInput` already implements `IPagedResultRequest`, you don't need to override this since the ABP automatically understands it and performs the paging.
 * `ApplySorting` is used to sort (order by...) the query. If your `TGetListInput` already implements the `ISortedResultRequest`, ABP automatically sorts the query. If not, it fallbacks to the `ApplyDefaultSorting` which tries to sort by creation time, if your entity implements the standard `IHasCreationTime` interface.
 * `GetEntityByIdAsync` is used to get an entity by id, which calls `Repository.GetAsync(id)` by default.
+* `CreateEntityQueryOrNullAsync` is used to create a query for a single entity by id, which is only needed for the *Query Projection* explained below. It returns `null` if the application service can not create such a query, then `GetEntityByIdAsync` is used.
 * `DeleteByIdAsync` is used to delete an entity by id, which calls `Repository.DeleteAsync(id)` by default.
 
 #### Object to Object Mapping
@@ -456,6 +457,103 @@ These methods are used to convert Entities to DTOs and vice verse. They use the
   * `MapToEntityAsync(TCreateInput)` is used to create an entity from `TCreateInput`.
   * `MapToEntityAsync(TUpdateInput, TEntity)` is used to update an existing entity from `TUpdateInput`.
 
+#### Query Projection
+
+`GetAsync` and `GetListAsync` get the entities from the database, then map them to DTOs in the memory. If your DTO uses only a few properties of a large entity, you can project the query to the DTO instead, so the database returns only the columns you need.
+
+Implement the `IQueryProjector<TEntity, TDto>` interface to define a projection:
+
+````csharp
+using System.Linq;
+using Volo.Abp.ObjectMapping;
+
+namespace MyProject.Books;
+
+public class BookProjector : IQueryProjector<Book, BookDto>
+{
+    public IQueryable<BookDto> ProjectTo(IQueryable<Book> source)
+    {
+        return source.Select(book => new BookDto
+        {
+            Id = book.Id,
+            Name = book.Name
+        });
+    }
+}
+````
+
+You don't have to write the `Select` by hand. Both [Mapperly](https://mapperly.riok.app/) and [AutoMapper](https://docs.automapper.org) can project an `IQueryable`, refer to their own documentation for it and to the [object to object mapping document](../../infrastructure/object-to-object-mapping.md) for their ABP integrations. Your existing maps are not used for the projection, a projector is always a class implementing `IQueryProjector<TSource, TDestination>`.
+
+ABP registers the projectors by convention, you don't need to configure anything else. Implement a projector once for an entity and DTO pair, and use the `ReplaceServices` option of the `DependencyAttribute` to replace an existing one. Filters (like soft delete and multi-tenancy), sorting and paging are still applied to the query before the projection.
+
+> A projection must return one row per entity. The total count and the paging are calculated on the entity query before the projection runs, so a projection that filters out rows (an inner join to an optional relation) or multiplies them (a join to a collection) returns a page that doesn't match the reported total count. Use a left join for optional relations.
+
+The projector is synchronous, so it can not obtain the query of another aggregate root, which is only
+available through the asynchronous `GetQueryableAsync`. Override `CreateGetOutputDtoQueryOrNullAsync` or
+`CreateGetListOutputDtoQueryOrNullAsync` for that. They replace the projector for that application service:
+
+````csharp
+public class BookAppService : ReadOnlyAppService<Book, BookDto, Guid>
+{
+    private readonly IBookDtoQuery _bookDtoQuery;
+
+    //...
+
+    protected override async Task<IQueryable<BookDto>?> CreateGetListOutputDtoQueryOrNullAsync(IQueryable<Book> query)
+    {
+        return await _bookDtoQuery.ProjectAsync(query);
+    }
+}
+
+//The projection is a class of its own, so the other application services returning a BookDto reuse it
+public class BookDtoQuery : IBookDtoQuery, ITransientDependency
+{
+    private readonly IReadOnlyRepository<Author, Guid> _authorRepository;
+
+    //...
+
+    public async Task<IQueryable<BookDto>> ProjectAsync(IQueryable<Book> books)
+    {
+        var authors = await _authorRepository.GetQueryableAsync();
+
+        return from book in books
+               join author in authors on book.AuthorId equals author.Id into bookAuthors
+               from bookAuthor in bookAuthors.DefaultIfEmpty()
+               select new BookDto
+               {
+                   Id = book.Id,
+                   Name = book.Name,
+                   AuthorName = bookAuthor != null ? bookAuthor.Name : null
+               };
+    }
+}
+````
+
+Both queries must come from the same database context, otherwise they can not be executed as a single query,
+and the provider has to be able to translate the join. The one row per entity rule above applies here too,
+that's why the example uses a left join. A joined column can not be used for the sorting, and the paging is
+based on the entity query, since both are applied before this method is called.
+
+A projector is resolved by the `(entity, DTO)` type pair, just like an `IObjectMapper<TSource, TDestination>`, so registering one enables the projection for every application service using that pair. It replaces the way the DTOs are read:
+
+* `GetListAsync` doesn't use `MapToGetListOutputDtosAsync` anymore.
+* `GetAsync` doesn't use `GetEntityByIdAsync` and `MapToGetOutputDtoAsync` anymore, as long as the application service can create a query for a single entity. `ReadOnlyAppService` and `CrudAppService` already do that. A class deriving from `AbstractKeyReadOnlyAppService` has to override `CreateEntityQueryOrNullAsync`, otherwise `GetAsync` keeps loading the entity and mapping it.
+
+The rest of the pipeline is untouched. The authorization policies are still checked, `CreateFilteredQueryAsync`, `ApplySorting` and `ApplyPaging` are still used, the data filters (like soft delete and multi-tenancy) are still applied, and the create, update and delete methods still use the [IObjectMapper](../../infrastructure/object-to-object-mapping.md).
+
+> If an application service needs to keep using the entity based extension points, override the `GetOutputDtoQueryProjector` or `GetListOutputDtoQueryProjector` property and return `null`:
+
+````csharp
+public class BookAppService : CrudAppService<Book, BookDto, Guid>
+{
+    protected override IQueryProjector<Book, BookDto>? GetOutputDtoQueryProjector => null;
+
+    protected override IQueryProjector<Book, BookDto>? GetListOutputDtoQueryProjector => null;
+
+    //...
+}
+````
+
 ## Miscellaneous
 
 ### Working with Streams

framework/src/Volo.Abp.Ddd.Application/Volo/Abp/Application/Services/AbstractKeyReadOnlyAppService.cs

@@ -2,12 +2,14 @@
 using System.Collections.Generic;
 using System.Linq;
 using System.Linq.Dynamic.Core;
+using System.Threading;
 using System.Threading.Tasks;
 using Volo.Abp.Application.Dtos;
 using Volo.Abp.Auditing;
 using Volo.Abp.Domain.Entities;
 using Volo.Abp.Domain.Repositories;
 using Volo.Abp.ObjectMapping;
+using Volo.Abp.Threading;
 
 namespace Volo.Abp.Application.Services;
 
@@ -44,6 +46,20 @@ public abstract class AbstractKeyReadOnlyAppService<TEntity, TGetOutputDto, TGet
 
     protected virtual string? GetListPolicyName { get; set; }
 
+    /// <summary>
+    /// Used by the <see cref="CreateGetOutputDtoQueryOrNullAsync"/> to project the query to the <typeparamref name="TGetOutputDto"/>.
+    /// The <see cref="GetEntityByIdAsync"/> and the <see cref="MapToGetOutputDtoAsync"/> are not used while the query is projected.
+    /// </summary>
+    protected virtual IQueryProjector<TEntity, TGetOutputDto>? GetOutputDtoQueryProjector
+        => LazyServiceProvider.LazyGetService<IQueryProjector<TEntity, TGetOutputDto>>();
+
+    /// <summary>
+    /// Used by the <see cref="CreateGetListOutputDtoQueryOrNullAsync"/> to project the query to the <typeparamref name="TGetListOutputDto"/>.
+    /// The <see cref="MapToGetListOutputDtosAsync"/> is not used while the query is projected.
+    /// </summary>
+    protected virtual IQueryProjector<TEntity, TGetListOutputDto>? GetListOutputDtoQueryProjector
+        => LazyServiceProvider.LazyGetService<IQueryProjector<TEntity, TGetListOutputDto>>();
+
     protected AbstractKeyReadOnlyAppService(IReadOnlyRepository<TEntity> repository)
     {
         ReadOnlyRepository = repository;
@@ -53,6 +69,19 @@ public virtual async Task<TGetOutputDto> GetAsync(TKey id)
     {
         await CheckGetPolicyAsync();
 
+        var dtoQuery = await CreateGetOutputDtoQueryOrNullAsync(id);
+        if (dtoQuery != null)
+        {
+            //TGetOutputDto has no class constraint, so a default value can not be used to detect the missing entity
+            var dtos = await AsyncExecuter.ToListAsync(dtoQuery.Take(1), GetCancellationToken());
+            if (dtos.Count == 0)
+            {
+                throw new EntityNotFoundException<TEntity>(id);
+            }
+
+            return dtos[0];
+        }
+
         var entity = await GetEntityByIdAsync(id);
 
         return await MapToGetOutputDtoAsync(entity);
@@ -65,16 +94,23 @@ public virtual async Task<PagedResultDto<TGetListOutputDto>> GetListAsync(TGetLi
         var query = await CreateFilteredQueryAsync(input);
         var totalCount = await AsyncExecuter.CountAsync(query);
 
-        var entities = new List<TEntity>();
         var entityDtos = new List<TGetListOutputDto>();
 
         if (totalCount > 0)
         {
             query = ApplySorting(query, input);
             query = ApplyPaging(query, input);
 
-            entities = await AsyncExecuter.ToListAsync(query);
-            entityDtos = await MapToGetListOutputDtosAsync(entities);
+            var dtoQuery = await CreateGetListOutputDtoQueryOrNullAsync(query);
+            if (dtoQuery != null)
+            {
+                entityDtos = await AsyncExecuter.ToListAsync(dtoQuery);
+            }
+            else
+            {
+                var entities = await AsyncExecuter.ToListAsync(query);
+                entityDtos = await MapToGetListOutputDtosAsync(entities);
+            }
         }
 
         return new PagedResultDto<TGetListOutputDto>(
@@ -85,6 +121,57 @@ public virtual async Task<PagedResultDto<TGetListOutputDto>> GetListAsync(TGetLi
 
     protected abstract Task<TEntity> GetEntityByIdAsync(TKey id);
 
+    private CancellationToken GetCancellationToken()
+    {
+        return LazyServiceProvider
+            .LazyGetService<ICancellationTokenProvider>(NullCancellationTokenProvider.Instance)
+            .FallbackToProvider();
+    }
+
+    /// <summary>
+    /// Should create a query that selects the entity with the given <paramref name="id"/>.
+    /// It returns null by default, then the entity is not projected.
+    /// </summary>
+    /// <param name="id">The id of the entity.</param>
+    protected virtual Task<IQueryable<TEntity>?> CreateEntityQueryOrNullAsync(TKey id)
+    {
+        return Task.FromResult<IQueryable<TEntity>?>(null);
+    }
+
+    /// <summary>
+    /// Projects the query of the entity with the given <paramref name="id"/> to the <typeparamref name="TGetOutputDto"/>.
+    /// It uses the <see cref="GetOutputDtoQueryProjector"/> and the <see cref="CreateEntityQueryOrNullAsync"/> by default,
+    /// and the <see cref="GetEntityByIdAsync"/> is used when it returns null.
+    /// Override it to await other queries, like the query of another aggregate root to join.
+    /// </summary>
+    /// <param name="id">The id of the entity.</param>
+    protected virtual async Task<IQueryable<TGetOutputDto>?> CreateGetOutputDtoQueryOrNullAsync(TKey id)
+    {
+        var queryProjector = GetOutputDtoQueryProjector;
+        if (queryProjector == null)
+        {
+            return null;
+        }
+
+        var query = await CreateEntityQueryOrNullAsync(id);
+
+        return query == null ? null : queryProjector.ProjectTo(query);
+    }
+
+    /// <summary>
+    /// Projects the given entity query to the <typeparamref name="TGetListOutputDto"/>.
+    /// It uses the <see cref="GetListOutputDtoQueryProjector"/> by default,
+    /// and the <see cref="MapToGetListOutputDtosAsync"/> is used when it returns null.
+    /// Override it to await other queries, like the query of another aggregate root to join.
+    /// The projection must return one row per entity: the total count is already calculated and the paging is
+    /// already applied, so adding or removing rows makes the page inconsistent with the total count.
+    /// </summary>
+    /// <param name="query">The sorted and paged entity query.</param>
+    protected virtual Task<IQueryable<TGetListOutputDto>?> CreateGetListOutputDtoQueryOrNullAsync(IQueryable<TEntity> query)
+    {
+        return Task.FromResult(GetListOutputDtoQueryProjector?.ProjectTo(query));
+    }
+
     protected virtual async Task CheckGetPolicyAsync()
     {
         await CheckPolicyAsync(GetPolicyName);

framework/src/Volo.Abp.Ddd.Application/Volo/Abp/Application/Services/CrudAppService.cs

@@ -84,6 +84,13 @@ protected override async Task<TEntity> GetEntityByIdAsync(TKey id)
         return await Repository.GetAsync(id);
     }
 
+    protected override async Task<IQueryable<TEntity>?> CreateEntityQueryOrNullAsync(TKey id)
+    {
+        var query = await Repository.GetQueryableAsync();
+
+        return query.Where(e => e.Id!.Equals(id));
+    }
+
     protected override void MapToEntity(TUpdateInput updateInput, TEntity entity)
     {
         if (updateInput is IEntityDto<TKey> entityDto)

framework/src/Volo.Abp.Ddd.Application/Volo/Abp/Application/Services/ReadOnlyAppService.cs

@@ -47,6 +47,13 @@ protected override async Task<TEntity> GetEntityByIdAsync(TKey id)
         return await Repository.GetAsync(id);
     }
 
+    protected override async Task<IQueryable<TEntity>?> CreateEntityQueryOrNullAsync(TKey id)
+    {
+        var query = await Repository.GetQueryableAsync();
+
+        return query.Where(e => e.Id!.Equals(id));
+    }
+
     protected override IQueryable<TEntity> ApplyDefaultSorting(IQueryable<TEntity> query)
     {
         if (typeof(TEntity).IsAssignableTo<ICreationAuditedObject>())

framework/src/Volo.Abp.ObjectMapping/Volo/Abp/ObjectMapping/AbpObjectMappingModule.cs

@@ -1,4 +1,5 @@
-using Microsoft.Extensions.DependencyInjection;
+using System.Collections.Generic;
+using Microsoft.Extensions.DependencyInjection;
 using Volo.Abp.DependencyInjection;
 using Volo.Abp.Modularity;
 using Volo.Abp.Reflection;
@@ -18,6 +19,14 @@ public override void PreConfigureServices(ServiceConfigurationContext context)
                     typeof(IObjectMapper<,>)
                 ).ConvertAll(t => new ServiceIdentifier(t))
             );
+
+            //Register types for IQueryProjector<TSource, TDestination> if implements
+            foreach (var serviceType in ReflectionHelper.GetImplementedGenericTypes(
+                         onServiceExposingContext.ImplementationType,
+                         typeof(IQueryProjector<,>)))
+            {
+                onServiceExposingContext.ExposedTypes.AddIfNotContains(new ServiceIdentifier(serviceType));
+            }
         });
     }
 

framework/src/Volo.Abp.ObjectMapping/Volo/Abp/ObjectMapping/IQueryProjector.cs

@@ -0,0 +1,24 @@
+using System.Linq;
+using Volo.Abp.DependencyInjection;
+
+namespace Volo.Abp.ObjectMapping;
+
+/// <summary>
+/// Maps a query to another.
+/// Implement this interface to project a query on the data store side, instead of loading the
+/// source objects into the memory and mapping them one by one.
+/// Implement it once for a source and destination pair. Use the ReplaceServices option of the
+/// DependencyAttribute to replace an existing implementation.
+/// </summary>
+/// <typeparam name="TSource">Type of the source objects</typeparam>
+/// <typeparam name="TDestination">Type of the destination objects</typeparam>
+public interface IQueryProjector<TSource, TDestination> : ITransientDependency
+{
+    /// <summary>
+    /// Projects the given query. The returned query must be built on top of it and must keep its order,
+    /// with a single destination object for each source object, using expressions the query provider can
+    /// translate. The caller may have already sorted, paged or counted the source query.
+    /// </summary>
+    /// <param name="source">The query to project</param>
+    IQueryable<TDestination> ProjectTo(IQueryable<TSource> source);
+}

framework/test/Volo.Abp.Ddd.Application.Tests/Volo/Abp/Application/Services/QueryProjection/Book.cs

@@ -0,0 +1,23 @@
+using System;
+using Volo.Abp.Domain.Entities;
+
+namespace Volo.Abp.Application.Services.QueryProjection;
+
+public class Book : Entity<Guid>
+{
+    public string Name { get; set; } = default!;
+
+    public int Price { get; set; }
+
+    public Book()
+    {
+
+    }
+
+    public Book(Guid id, string name, int price)
+        : base(id)
+    {
+        Name = name;
+        Price = price;
+    }
+}

framework/test/Volo.Abp.Ddd.Application.Tests/Volo/Abp/Application/Services/QueryProjection/BookAbstractKeyAppService.cs

@@ -0,0 +1,27 @@
+using System;
+using System.Linq;
+using System.Threading.Tasks;
+using Volo.Abp.Domain.Repositories;
+
+namespace Volo.Abp.Application.Services.QueryProjection;
+
+public class BookAbstractKeyAppService : AbstractKeyReadOnlyAppService<Book, BookDto, Guid>
+{
+    public BookAbstractKeyAppService(IReadOnlyRepository<Book> repository)
+        : base(repository)
+    {
+
+    }
+
+    protected override async Task<Book> GetEntityByIdAsync(Guid id)
+    {
+        var query = await ReadOnlyRepository.GetQueryableAsync();
+
+        return await AsyncExecuter.FirstAsync(query, book => book.Id == id);
+    }
+
+    protected override IQueryable<Book> ApplyDefaultSorting(IQueryable<Book> query)
+    {
+        return query.OrderBy(book => book.Id);
+    }
+}