Remove the default after adding a required column if HasDefaultValue wasn't used

[marcoatribeiro]

Bug description

I have a User entity which has a UserAccessStatus property configured as an owned entity. This, in turn, as a boolean property named HasEmailAccess, which is configured as follows in the migration configuration class:

builder.Property(accessStatus => accessStatus.HasEmailAccess).IsRequired().HasDefaultValue(false);

Although the above configuration seems redundant (after all, the default bool value is false) it was required to make sure that my migration file (for a PostgreSQL database) was created correctly, setting the column as required and with default value as false.

However, with this configuration when I try to update the owned entity HasEmailAccess property value (effectively deleting and recreating another instance), it works only for one side: when I update its value from false to true. The other way around does not work.

For now, I'm removing the HasDefaultValue clause from the configuration (which, IMO, should not be necessary anyway), but I'd highly appreciate any inputs you guys could provide on this matter. Please let me know if any further info is required.

Your code

internal sealed class UserConfiguration : IEntityTypeConfiguration<User>
{
    public void Configure(EntityTypeBuilder<User> builder)
    {
        builder.ToTable(TableNames.Users);
        builder.Property(user => user.Id)
            .ValueGeneratedNever()
            .HasGuidConversion(UserId.FromGuid);

        builder.OwnsOne(user => user.AccessStatus, ConfigureUserAccessStatus);
        builder.OwnsOne(user => user.Profile, ConfigureUserProfile);
        builder.OwnsOne(user => user.Preferences, ConfigureUserPreferences);

        builder.HasMany(user => user.Permissions)
            .WithOne()
            .HasForeignKey(userPermission => userPermission.UserId);

        builder.HasMany(user => user.Roles)
            .WithOne()
            .HasForeignKey(userRole => userRole.UserId);

        builder.HasIndex(user => user.Email).IsUnique();
        builder.HasIndex(user => user.IdentityProviderId).IsUnique();
    }

    #region Owned entities configuration methods

    private static void ConfigureUserAccessStatus(OwnedNavigationBuilder<User, UserAccessStatus> builder)
    {
        builder.ToTable(TableNames.UsersAccessStatusDetails);

        builder.Property(accessStatus => accessStatus.Type).IsRequired();
        builder.Property(accessStatus => accessStatus.HasEmailAccess).IsRequired(); // .HasDefaultValue(false); --> More details in https://github.com/npgsql/efcore.pg/issues/3470
        builder.Property(accessStatus => accessStatus.AccessGranterUserName).IsRequired(false);
        builder.Property(accessStatus => accessStatus.AccessGrantedOn).IsRequired(false);
        builder.Property(accessStatus => accessStatus.AccessRevokerUserName).IsRequired(false);
        builder.Property(accessStatus => accessStatus.AccessRevokedOn).IsRequired(false);
    }

    private static void ConfigureUserProfile(OwnedNavigationBuilder<User, UserProfile> builder)
    {
        builder.ToTable(TableNames.UsersProfileDetails);
        builder.Property(profile => profile.PersonId)
            .HasNullableGuidConversion(PersonId.FromGuid);

        builder.Property(profile => profile.FullName).IsRequired();
        builder.Property(profile => profile.CpfCnpj).IsRequired();
        builder.Property(profile => profile.Phone).HasMaxLength(MaxLengths.PhoneNumber);
        builder.Property(profile => profile.ProfilePicture).IsRequired(false);

        builder.HasIndex(profile => profile.PersonId).IsUnique();
    }

    private static void ConfigureUserPreferences(OwnedNavigationBuilder<User, UserPreferences> builder)
    {
        builder.ToTable(TableNames.UsersPreferencesDetails);

        builder.Property(preferences => preferences.IsDarkMode).IsRequired(false);
        builder.Property(preferences => preferences.EmailNotificationsEnabled).IsRequired();
        builder.Property(preferences => preferences.SystemNotificationsEnabled).IsRequired();
    }

    #endregion
}


public sealed class User : Entity<UserId>, IAuditable
{
    private readonly HashSet<UserPermission> _permissions = [];
    private readonly HashSet<UserRole> _roles = [];

    private User(UserId id, Guid identityProviderId, bool hasEmailAccess, string fullName, CpfCnpj cpfCnpj, EmailAddress email, 
        string phone, string? profilePicture)
        : base(id)
    {
        IdentityProviderId = identityProviderId;
        Email = email;
        Profile = new UserProfile(fullName, cpfCnpj, phone, null, profilePicture);
        Preferences = new UserPreferences();
        AccessStatus = new UserAccessStatus(UserAccessStatusType.Waiting, hasEmailAccess);
        CreatedOn = TimeProvider.System.GetUtcNow();
    }

    private User() { } // Necessary for EF Core Configuration

    /// <summary>
    /// Id provided by the identity provider module.
    /// </summary>
    public Guid IdentityProviderId { get; }

    /// <summary>
    /// User's e-mail (which is theirs username).
    /// </summary>
    public EmailAddress Email { get; private set; }

    /// <summary>
    /// User's profile data.
    /// </summary>
    public UserProfile Profile { get; private set; } = default!;

    /// <summary>
    /// User's system preferences.
    /// </summary>
    public UserPreferences Preferences { get; private set; } = default!;

    /// <summary>
    /// Last date/time user signed-in to the system.
    /// </summary>
    public DateTimeOffset? LastSignInOn { get; private set; }

    /// <summary>
    /// Contains data related to the status of the user's access to the system.
    /// </summary>
    public UserAccessStatus AccessStatus { get; private set; } = default!;

    public DateTimeOffset CreatedOn { get; private init; }
    public DateTimeOffset? UpdatedOn { get; private set; }

    public IReadOnlyCollection<UserPermission> Permissions => _permissions.ToList().AsReadOnly();
    public IReadOnlyCollection<UserRole> Roles => _roles.ToList().AsReadOnly();

    /// <summary>
    /// Creates a new user with the specified parameters.
    /// </summary>
    /// <param name="identityProviderId">The id provided by the identity provider module.</param>
    /// <param name="hasEmailAccess"><c>True</c> if user has an e-mail/password access to the system or
    /// <c>false</c> if only external provider(s) is(are) used.</param>
    /// <param name="fullName">User's full name.</param>
    /// <param name="cpfCnpj">User's CPF/CNPJ.</param>
    /// <param name="email">User's e-mail (its username).</param>
    /// <param name="phone">User's phone number.</param>
    /// <param name="role">User's <see cref="Role"/>.</param>
    /// <param name="roleCondominiumId">Id of the condominium for which the <see cref="Role"/> applies.</param>
    /// <param name="loginProvider">External login provider name (if any).</param>
    /// <param name="profilePictureUrl">Profile picture URL (if any).</param>
    /// <returns>The new user instance.</returns>
    public static Result<User> Create(
        Guid identityProviderId, 
        bool hasEmailAccess,
        string fullName, 
        CpfCnpj cpfCnpj,
        EmailAddress email, 
        string phone,
        Role role,
        CondominiumId roleCondominiumId,
        string? loginProvider,
        string? profilePictureUrl)
    {
        var user = new User(UserId.New(), identityProviderId, hasEmailAccess, fullName, cpfCnpj, email, phone, profilePictureUrl);

        user._roles.Add(new UserRole(user.Id, role, roleCondominiumId));

        user.RaiseDomainEvent(new UserCreatedDomainEvent(Guid.NewGuid(), GetEventOccurredDateTime(), user.Id,
            identityProviderId, fullName, email, loginProvider, user.Preferences.EmailNotificationsEnabled,
            user.Preferences.SystemNotificationsEnabled));

        return user;
    }

    /// <summary>
    /// Changes the user's e-mail (username).
    /// </summary>
    /// <param name="newEmail">The new e-mail.</param>
    public Result ChangeEmail(EmailAddress newEmail)
    {
        Email = newEmail;
        RaiseDomainEvent(new UserEmailChangedDomainEvent(Guid.NewGuid(), GetEventOccurredDateTime(), 
            IdentityProviderId, Profile.FullName, newEmail));
        return Result.Success();
    }

    /// <summary>
    /// Changes the flag that indicates if user may access the system using e-mail/password.
    /// </summary>
    /// <param name="hasEmailAccess">True if user has added e-mail/password access
    /// or false if this option has been removed.</param>
    public Result ChangeEmailAccess(bool hasEmailAccess)
    {
        AccessStatus = AccessStatus.ChangeEmailAccess(hasEmailAccess);
        return Result.Success();
    }

    /// <summary>
    /// Sets the user's profile person id.
    /// </summary>
    /// <param name="personId">Person id to set.</param>
    public Result SetProfilePersonId(PersonId personId)
    {
        Profile = Profile.WithPersonId(personId);
        return Result.Success();
    }

    /// <summary>
    /// Sets the user's profile picture as a URI.
    /// </summary>
    /// <param name="profilePictureUri">URI of profile picture.</param>
    public Result SetProfilePictureUri(string profilePictureUri)
    {
        Profile = Profile.WithProfilePictureUri(profilePictureUri);
        return Result.Success();
    }

    /// <summary>
    /// Sets the user system access as granted.
    /// </summary>
    /// <param name="granterUserName">Granter user's username.</param>
    /// <param name="grantedOn">Date/time when access has been granted.</param>
    public Result SetAccessStatusAsGranted(EmailAddress? granterUserName, DateTimeOffset? grantedOn)
    {
        AccessStatus = AccessStatus.SetAccessGranted(granterUserName, grantedOn);
        return Result.Success();
    }

    /// <summary>
    /// Sets the user system access as revoked.
    /// </summary>
    /// <param name="revokerUserName">Revoker user's username.</param>
    /// <param name="revokedOn">Date/time when access has been revoked.</param>
    public Result SetAccessStatusAsRevoked(EmailAddress? revokerUserName, DateTimeOffset? revokedOn)
    {
        AccessStatus = AccessStatus.SetAccessRevoked(revokerUserName, revokedOn);
        return Result.Success();
    }

    /// <summary>
    /// Update user's permissions.
    /// </summary>
    /// <param name="permissionsToAdd">Permissions to add to user.</param>
    /// <param name="permissionsToRemove">Permissions to remove from user.</param>
    /// <returns>User with updated permissions list.</returns>
    public Result<User> UpdatePermissions(
        List<(string PermissionName, CondominiumId CondominiumId)> permissionsToAdd,
        List<(string PermissionName, CondominiumId CondominiumId)> permissionsToRemove)
    {
        foreach (var permissionTuple in permissionsToAdd)
        {
            var permission = Permission.FromName(permissionTuple.PermissionName)!;
            _permissions.Add(new UserPermission(Id, permission, permissionTuple.CondominiumId));
        }
        foreach (var permissionTuple in permissionsToRemove)
        {
            var permission = Permission.FromName(permissionTuple.PermissionName)!;
            _permissions.RemoveWhere(x => x.Permission == permission && x.CondominiumId == permissionTuple.CondominiumId);
        }
        return this;
    }

    /// <summary>
    /// Updates user's profile data.
    /// </summary>
    /// <param name="fullName">Person's full name.</param>
    /// <param name="cpfCnpj">Person's CPF or CNPJ.</param>
    /// <param name="phone">Person's main phone number.</param>
    /// <param name="profilePicture">Person's profile picture.</param>
    /// <returns>User instance with updated data.</returns>
    public Result<User> UpdateProfileData(string fullName, CpfCnpj cpfCnpj, string phone, string? profilePicture)
    {
        Profile = Profile.With(fullName, cpfCnpj, phone, profilePicture);
        return this;
    }

    /// <summary>
    /// Update user's preferences.
    /// </summary>
    /// <param name="isDarkMode">Flag which indicates if user uses light or dark mode.</param>
    /// <param name="emailNotificationsEnabled">Flag which indicates if notifications by e-mail are enabled.</param>
    /// <param name="systemNotificationsEnabled">Flag which indicates if system notifications are enabled.</param>
    public Result UpdatePreferences(bool isDarkMode, bool emailNotificationsEnabled, bool systemNotificationsEnabled)
    {
        Preferences = Preferences.With(isDarkMode, emailNotificationsEnabled, systemNotificationsEnabled);

        RaiseDomainEvent(new UserPreferencesUpdatedDomainEvent(Guid.NewGuid(), GetEventOccurredDateTime(), Id,
            Preferences.EmailNotificationsEnabled, Preferences.SystemNotificationsEnabled));

        return Result.Success();
    }
}


public sealed class UserAccessStatus : ValueObject
{
    public UserAccessStatus(UserAccessStatusType type, bool hasEmailAccess,
        EmailAddress? accessGranterUserName = null, DateTimeOffset? accessGrantedOn = null,
        EmailAddress? accessRevokerUserName = null, DateTimeOffset? accessRevokedOn = null)
    {
        Type = type;
        HasEmailAccess = hasEmailAccess;
        AccessGranterUserName = accessGranterUserName;
        AccessGrantedOn = accessGrantedOn;
        AccessRevokerUserName = accessRevokerUserName;
        AccessRevokedOn = accessRevokedOn;
    }

    private UserAccessStatus() { } // Necessary for EF Core Configuration

    public UserAccessStatusType Type { get; } = default!;
    public bool HasEmailAccess { get; }
    public EmailAddress? AccessGranterUserName { get; }
    public DateTimeOffset? AccessGrantedOn { get; }
    public EmailAddress? AccessRevokerUserName { get; }
    public DateTimeOffset? AccessRevokedOn { get; }
    
    protected override IEnumerable<object> GetAtomicValues()
    {
        yield return Type;
        yield return HasEmailAccess;
        yield return AccessGranterUserName ?? default!;
        yield return AccessGrantedOn ?? DateTimeOffset.MinValue;
        yield return AccessRevokerUserName ?? default!;
        yield return AccessRevokedOn ?? DateTimeOffset.MinValue;
    }

    /// <summary>
    /// Changes the flag that indicates if user may access the system using e-mail/password.
    /// </summary>
    /// <param name="hasEmailAccess">True if user has added e-mail/password access
    /// or false if this option has been removed.</param>
    /// <returns>New instance of status with <see cref="HasEmailAccess"/> flag value changed.</returns>
    public UserAccessStatus ChangeEmailAccess(bool hasEmailAccess) => 
        new(Type, hasEmailAccess, AccessGranterUserName, AccessGrantedOn, AccessRevokerUserName, AccessRevokedOn);

    /// <summary>
    /// Sets the user access status as granted.
    /// </summary>
    /// <param name="granterUserName">Granter user's username.</param>
    /// <param name="grantedOn">Date/time when access has been granted.</param>
    /// <returns>New instance of status with status as <see cref="UserAccessStatusType.Active"/> and granter data set.</returns>
    public UserAccessStatus SetAccessGranted(EmailAddress? granterUserName, DateTimeOffset? grantedOn) =>
        new(UserAccessStatusType.Active, HasEmailAccess, granterUserName, grantedOn);

    /// <summary>
    /// Sets the user access status as revoked.
    /// </summary>
    /// <param name="revokerUserName">Revoker user's username.</param>
    /// <param name="revokedOn">Date/time when access has been revoked.</param>
    /// <returns>New instance of status with status as <see cref="UserAccessStatusType.Revoked"/> and revoker data set.</returns>
    public UserAccessStatus SetAccessRevoked(EmailAddress? revokerUserName, DateTimeOffset? revokedOn) =>
        new(UserAccessStatusType.Revoked, HasEmailAccess, AccessGranterUserName, AccessGrantedOn, revokerUserName, revokedOn);
}

Stack traces

Verbose output

EF Core version

8.0.11

Database provider

Npgsql.EntityFrameworkCore.PostgreSQL

Target framework

.NET 8.0

Operating system

Windows 11

IDE

Visual Studio 2022 17.13.2

[AndriySvyryd]

This is by-design. Setting HasDefaultValue means that the specified value will be read from the database when an entity is inserted with the property set to the sentinel value (false for bool by default)

[marcoatribeiro]

I understand the update part. Nevertheless I understand we have an issue in the migration generation.

When I created the migration to add this field to an existing table, I didn't specify the HasDefault clause and it was added in the migration anyway:

Configuration file:

...
builder.Property(accessStatus => accessStatus.HasEmailAccess).IsRequired(true);
...

Migration file:

...
migrationBuilder.AddColumn<bool>(
	name: "has_email_access",
	schema: "users",
	table: "users_access_status_details",
	type: "boolean",
	nullable: false,
	defaultValue: false);
...

However, when I "reset" my migrations and created the table from scratch (my system is still in development), the same column was created without the defaultValue (the configuration file was unchanged):

...
migrationBuilder.CreateTable(
    name: "users_access_status_details",
    schema: "users",
    columns: table => new
    {
        id = table.Column<Guid>(type: "uuid", nullable: false),
        type = table.Column<string>(type: "character varying(100)", maxLength: 100, nullable: false),
        has_email_access = table.Column<bool>(type: "boolean", nullable: false),  // defaultValue not specified
        access_granter_user_name = table.Column<string>(type: "character varying(255)", nullable: true),
        access_granted_on = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
        access_revoker_user_name = table.Column<string>(type: "character varying(255)", nullable: true),
        access_revoked_on = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true)
    }
...

This caused errors upon inserting new rows in this table.

Is there any reason for this behavior?

[AndriySvyryd]

This is simply because when adding a required column to an existing table we have to specify a default value to use in the existing rows. To be consistent with the new table case we should have removed the default value after the column is added.