Site Settings not updated when running on multiple instances

[MarkvDijk]

When running the site on multiple instances, Site Settings are not replicated to the memory of other instances.

When updating a property in the Site Settings and publishing the content, the update is stored in the database. However the SettingsService stores a copy of the settings in memory public ConcurrentDictionary<string, Dictionary<Type, object>> SiteSettings
This is done in the PublishContent event using UpdateSettings(...). This event is only triggered on the instance you are connected to. The other instances don't trigger this event, therefore not updating the SiteSettings dictionary and returning old settings.

Issue can be replicated by using 2 browsers. Check the ARRAffinity cookie for the instance Id you are connect to. They must be different for both browsers. In both browsers open the same page that displays one of the site settings. Update the site setting in browser 1 and refresh the page. the updated setting is visible. Refresh the page in browser 2, it still displays the old setting.

Browser 2 won't display the new setting unless you publish it from that browser as well (causing the dictionary to update)

[Hinneman]

We have exactly the same problem using Site Settings in DXP Production.

Any idea on how to solve this?

[lunchin]

Can you please check the diff of the pull requests #909 if this fixes your issues before I merge.

[Hinneman]

I will implement this change and get back to you when verified.

[Hinneman]

We have now implemented the changes, deployed them to preprod, scaled up environment and verified with two browsers with different ARRAffinity cookies.

A change to the web site settings is still only shown in the browser where the change was published not the other one.

[jonascarlbaum]

My fix was mostly solved by updating GetSiteSettings-method like this. (splitted it up for simplification, can be optimized)

public SettingsBase GetSiteSettings<T>(Guid? siteId = null) where T : SettingsBase
{
    if (!siteId.HasValue)
    {
        siteId = ResolveSiteId();
        if (siteId == Guid.Empty)
        {
            return default;
        }
    }
    try
    {
        if (PageEditing.PageIsInEditMode)
        {
            return GetEditModeSettings<T>(siteId.Value.ToString() + "-common-draft");
        }
        else
        {
            return GetViewModeSettings<T>(siteId.Value.ToString());
        }
    }
    catch (KeyNotFoundException keyNotFoundException)
    {
        _log.Error($"[Settings] {keyNotFoundException.Message}", exception: keyNotFoundException);
    }
    catch (ArgumentNullException argumentNullException)
    {
        _log.Error($"[Settings] {argumentNullException.Message}", exception: argumentNullException);
    }

    return default;
}

private T GetEditModeSettings<T>(string cacheKey) where T : SettingsBase
{
    return _synchronizedObjectInstanceCache.ReadThrough<T>($"{typeof(T).Name}:{cacheKey}",
        () =>
        {
            UpdateSettings();

            if (SiteSettings.TryGetValue(cacheKey, out var siteSettings))
            {
                if (siteSettings.TryGetValue(typeof(T), out var setting))
                {
                    return (T)setting;
                }
            }

            return default;
        },
        (result) => new CacheEvictionPolicy(TimeSpan.FromMinutes(1), CacheTimeoutType.Absolute, null, new[] { Settings.MASTERKEY }),
        ReadStrategy.Wait
    );
}

private T GetViewModeSettings<T>(string cacheKey) where T : SettingsBase
{
    return _synchronizedObjectInstanceCache.ReadThrough($"{typeof(T).Name}:{cacheKey}",
        () =>
        {
            UpdateSettings();

            if (SiteSettings.TryGetValue(cacheKey, out var siteSettings) && siteSettings.TryGetValue(typeof(T), out var setting))
            {
                return (T)setting;
            }

            return default;
        },
        (result) => new CacheEvictionPolicy(TimeSpan.FromMinutes(10), CacheTimeoutType.Absolute, null, new[] { Settings.MASTERKEY }),
        ReadStrategy.Wait
    );
}

The key here is calling UpdateSettings(); when cache invalidates, so the Dictionary is also rebuilt.

and for invalidation purposes we added _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY); on some places, like

        private void SiteCreated(object sender, SiteDefinitionEventArgs e)
        {
            if (_contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot)
                .Any(x => x.Name.Equals(e.Site.Name, StringComparison.InvariantCultureIgnoreCase)))
            {
                return;
            }

            CreateSiteFolder(e.Site);
            _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
        }

        private void SiteDeleted(object sender, SiteDefinitionEventArgs e)
        {
            var folder = _contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot)
                .FirstOrDefault(x => x.Name.Equals(e.Site.Name, StringComparison.InvariantCultureIgnoreCase));

            if (folder == null)
            {
                return;
            }

            _contentRepository.Delete(folder.ContentLink, true, AccessLevel.NoAccess);
            _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
        }

        private void SiteUpdated(object sender, SiteDefinitionEventArgs e)
        {
            var folder = _contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot)
                .FirstOrDefault(x => x.Name.Equals(e.Site.Name, StringComparison.InvariantCultureIgnoreCase));

            if (folder != null)
            {
                return;
            }

            CreateSiteFolder(e.Site);
            _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
        }

        private void PublishedContent(object sender, ContentEventArgs e)
        {
            if (e == null)
            {
                return;
            }

            if (e.Content is SettingsBase)
            {
                var id = ResolveSiteId();
                if (id == Guid.Empty)
                {
                    return;
                }
                UpdateSettings(id, e.Content, false);

                _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
            }
        }

        private void SavedContent(object sender, ContentEventArgs e)
        {
            if (e == null)
            {
                return;
            }

            if (e.Content is SettingsBase)
            {
                var id = ResolveSiteId();
                if (id == Guid.Empty)
                {
                    return;
                }
                UpdateSettings(id, e.Content, true);

                _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
            }
        }

Think we needed to change some stuff, like

T GetSiteSettings<T>(Guid? siteId = null);

to

SettingsBase GetSiteSettings<T>(Guid? siteId = null) where T : SettingsBase;

and do some type casting in code, otherwise the code changes above (if I remember correctly) works well for us.
Was your solution similar @Hinneman?

We have some more changes in our solution, so I can't just copy&paste the whole stuff, and won't do a PR now.
But if someone can test these changes out on the vanilla version and commit a PR it would be great... ;)

Addition, I forgot to mention I added a simple static class in ServiceSettings.cs

public static class Settings
{
    public static string MASTERKEY = "MASTERKEY";
}

[optidada]

Since some time (1-2 years?) this issue is generating a lot of high priority support cases and headache with Optimizely Support. It seems like there is a proper fix/workaround since some time back, but can we have it merged as well? Thank you. @daniel-isaacs @lunchin

[jonascarlbaum]

@optidada my code work in DXP.

I didn’t do a PR, since we had customized changes and I didn’t have the time to clone and fix. I guess anyone can do it, from my comment, and do a PR!?

[optidada]

Thanks @jonascarlbaum. You're absolutely right. Just to be clear - I wasn't necessarily asking you for a PR but anyone really that feel that it's beneficial for the project. Every solution making use of this SettingsService on Azure (with multiple instances) seems to hit this bug at some point :)

[jonascarlbaum]

Thanks @jonascarlbaum. You're absolutely right. Just to be clear - I wasn't necessarily asking you for a PR but anyone really that feel that it's beneficial for the project. Every solution making use of this SettingsService on Azure (with multiple instances) seems to hit this bug at some point :)

Yep, on every update of a setting only the instance the editor was on will be ok, all other instances would return old settings.

As a workaround, before we implemented the fix, was restarting the site in PaaS-portal, after someone changed a setting, which wasn’t a very nice…

So, it would very much need an update, if this is a project still being cloned and referenced etc., which I guess is the case.

But maybe this project isn’t a high priority!? 🤔🤷‍♂️

[avinashmpawar]

@jonascarlbaum we have done the fix suggested however we are stuck with how to handle or pass this Settings.MASTERKEY _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY);
Appreciate your valuable suggestion on this, I am getting compile time error if used code as is
The type or namespace name 'MASTERKEY' does not exist in the namespace 'Foundation.Infrastructure.Cms.Settings' (are you missing an assembly reference?)
Thank you so much in advance

[lunchin]

I am going to raise a pr to fix this for cms 12. Here is the code I am using @avinashmpawar if you want to try iy.

using System.Globalization;
using EPiServer.DataAccess;
using EPiServer.Events;
using EPiServer.Events.Clients;
using EPiServer.Framework.TypeScanner;
using EPiServer.Logging;
using EPiServer.Security;

namespace lunchin.Optimizely.Cloud.Extensions.Settings;

public class SettingsService : ISettingsService, IDisposable
{
    //Generate unique id for your event and the raiser
    private readonly Guid _raiserId;
    private static Guid EventId => new("b29b8aef-2a17-4432-ad16-8cd6cc6953e3");

    private readonly IContentRepository _contentRepository;
    private readonly ContentRootService _contentRootService;
    private readonly IContentTypeRepository _contentTypeRepository;
    private readonly ILogger _log = LogManager.GetLogger();
    private readonly ITypeScannerLookup _typeScannerLookup;
    private readonly IContentEvents _contentEvents;
    private readonly ISiteDefinitionEvents _siteDefinitionEvents;
    private readonly ISiteDefinitionRepository _siteDefinitionRepository;
    private readonly ISiteDefinitionResolver _siteDefinitionResolver;
    private readonly IHttpContextAccessor _httpContext;
    private readonly IEventRegistry _eventRegistry;
    private readonly IContentLanguageAccessor _contentLanguageAccessor;

    public SettingsService(
        IContentRepository contentRepository,
        ContentRootService contentRootService,
        ITypeScannerLookup typeScannerLookup,
        IContentTypeRepository contentTypeRepository,
        IContentEvents contentEvents,
        ISiteDefinitionEvents siteDefinitionEvents,
        ISiteDefinitionRepository siteDefinitionRepository,
        ISiteDefinitionResolver siteDefinitionResolver,
        IHttpContextAccessor httpContext,
        IEventRegistry eventRegistry,
        IContentLanguageAccessor contentLanguageAccessor)
    {
        _contentRepository = contentRepository;
        _contentRootService = contentRootService;
        _typeScannerLookup = typeScannerLookup;
        _contentTypeRepository = contentTypeRepository;
        _contentEvents = contentEvents;
        _siteDefinitionEvents = siteDefinitionEvents;
        _siteDefinitionRepository = siteDefinitionRepository;
        _siteDefinitionResolver = siteDefinitionResolver;
        _httpContext = httpContext;
        _eventRegistry = eventRegistry;
        _raiserId = Guid.NewGuid();
        _contentLanguageAccessor = contentLanguageAccessor;
    }

    public ConcurrentDictionary<Guid, Dictionary<Type, Guid>> SiteSettings { get; } = new ConcurrentDictionary<Guid, Dictionary<Type, Guid>>();

    public ContentReference? GlobalSettingsRoot { get; set; }

    public List<T> GetAllSiteSettings<T>() where T : SettingsBase
    {
        var sites = _siteDefinitionRepository.List();
        var siteSettings = new List<T>();

        foreach (var site in sites)
        {
            var settings = GetSiteSettings<T>(site.Id);
            if (settings != null)
            {
                siteSettings.Add(settings);
            }
        }

        return siteSettings;
    }

    public T? GetSiteSettings<T>(Guid? siteId = null, string? language = null) where T : SettingsBase
    {
        if (!siteId.HasValue)
        {
            siteId = ResolveSiteId();
            if (siteId == Guid.Empty)
            {
                return default;
            }
        }

        try
        {
            if (SiteSettings.TryGetValue(siteId.Value, out var siteSettings) &&
                siteSettings.TryGetValue(typeof(T), out var settingId))
            {
                return _contentRepository.Get<T>(settingId, language == null ? _contentLanguageAccessor.Language : CultureInfo.GetCultureInfo(language));
            }
        }
        catch (KeyNotFoundException keyNotFoundException)
        {
            _log.Error($"[Settings] {keyNotFoundException.Message}", exception: keyNotFoundException);
        }
        catch (ArgumentNullException argumentNullException)
        {
            _log.Error($"[Settings] {argumentNullException.Message}", exception: argumentNullException);
        }

        return default;
    }

    public void InitializeSettings()
    {
        try
        {
            RegisterContentRoots();
        }
        catch (NotSupportedException notSupportedException)
        {
            _log.Error($"[Settings] {notSupportedException.Message}", exception: notSupportedException);
            throw;
        }
        catch (InvalidOperationException ex)
        {
            _log.Error(ex.Message, ex);
        }
        _contentEvents.PublishedContent += PublishedContent;
        _siteDefinitionEvents.SiteCreated += SiteCreated;
        _siteDefinitionEvents.SiteUpdated += SiteUpdated;
        _siteDefinitionEvents.SiteDeleted += SiteDeleted;

        var settingsEvent = _eventRegistry.Get(EventId);
        settingsEvent.Raised += SettingsEvent_Raised;
    }

    public void UpdateSettings(Guid siteId, IContent content)
    {
        var contentType = content.GetOriginalType();
        try
        {
            if (!SiteSettings.ContainsKey(siteId))
            {
                SiteSettings[siteId] = [];
            }

            SiteSettings[siteId][contentType] = content.ContentGuid;
        }
        catch (KeyNotFoundException keyNotFoundException)
        {
            _log.Error($"[Settings] {keyNotFoundException.Message}", exception: keyNotFoundException);
        }
        catch (ArgumentNullException argumentNullException)
        {
            _log.Error($"[Settings] {argumentNullException.Message}", exception: argumentNullException);
        }
    }

    public void UpdateSettings()
    {
        var root = _contentRepository.GetItems(_contentRootService.List(), [])
             .FirstOrDefault(x => x.ContentGuid == SettingsFolder.SettingsRootGuid);

        if (root == null)
        {
            return;
        }

        GlobalSettingsRoot = root.ContentLink;
        var children = _contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot).ToList();
        foreach (var site in _siteDefinitionRepository.List())
        {
            var folder = children.Find(x => x.Name.Equals(site.Name, StringComparison.InvariantCultureIgnoreCase));
            if (folder == null)
            {
                CreateSiteFolder(site);
                return;
            }

            var settingsModelTypes = _typeScannerLookup.AllTypes
                    .Where(t => t.GetCustomAttributes(typeof(SettingsContentTypeAttribute), false).Length > 0);

            foreach (var settingsType in settingsModelTypes)
            {
                if (settingsType.GetCustomAttributes(typeof(SettingsContentTypeAttribute), false)
                    .FirstOrDefault() is not SettingsContentTypeAttribute attribute)
                {
                    continue;
                }

                var siteSetting = _contentRepository.GetChildren<SettingsBase>(folder.ContentLink, _contentLanguageAccessor.Language)
                    .FirstOrDefault(x => x.Name.Equals(attribute.SettingsName));

                if (siteSetting != null)
                {
                    UpdateSettings(site.Id, siteSetting);
                }
                else
                {
                    var contentType = _contentTypeRepository.Load(settingsType);
                    var newSettings = _contentRepository.GetDefault<IContent>(folder.ContentLink, contentType.ID);
                    newSettings.Name = attribute.SettingsName;

                    try
                    {
                        _ = _contentRepository.Save(newSettings, SaveAction.Publish, AccessLevel.NoAccess);
                        UpdateSettings(site.Id, newSettings);
                    }
                    catch (Exception e)
                    {
                        _log.Error(e.Message);
                    }
                }

            }
        }
    }

    public void RegisterContentRoots()
    {
        var registeredRoots = _contentRepository.GetItems(_contentRootService.List(), []);
        var settingsRootRegistered = registeredRoots.Any(x => x.ContentGuid == SettingsFolder.SettingsRootGuid && x.Name.Equals(SettingsFolder.SettingsRootName));

        if (!settingsRootRegistered)
        {
            _contentRootService.Register<SettingsFolder>(SettingsFolder.SettingsRootName, SettingsFolder.SettingsRootGuid, ContentReference.RootPage);
        }

        UpdateSettings();
    }

    public void Dispose()
    {
        if (_contentEvents != null)
        {
            _contentEvents.PublishedContent -= PublishedContent;
        }

        if (_siteDefinitionEvents != null)
        {
            _siteDefinitionEvents.SiteCreated -= SiteCreated;
            _siteDefinitionEvents.SiteUpdated -= SiteUpdated;
            _siteDefinitionEvents.SiteDeleted -= SiteDeleted;
        }

        if (_eventRegistry != null)
        {
            var settingsEvent = _eventRegistry.Get(EventId);
            settingsEvent.Raised -= SettingsEvent_Raised;
        }
        
        GC.SuppressFinalize(this);
    }

    private void CreateSiteFolder(SiteDefinition siteDefinition)
    {
        var folder = _contentRepository.GetDefault<SettingsFolder>(GlobalSettingsRoot);
        folder.Name = siteDefinition.Name;
        var reference = _contentRepository.Save(folder, SaveAction.Publish, AccessLevel.NoAccess);

        var settingsModelTypes = _typeScannerLookup.AllTypes
            .Where(t => t.GetCustomAttributes(typeof(SettingsContentTypeAttribute), false).Length > 0);

        foreach (var settingsType in settingsModelTypes)
        {
            if (settingsType.GetCustomAttributes(typeof(SettingsContentTypeAttribute), false)
                .FirstOrDefault() is not SettingsContentTypeAttribute attribute)
            {
                continue;
            }

            var contentType = _contentTypeRepository.Load(settingsType);
            var newSettings = _contentRepository.GetDefault<IContent>(reference, contentType.ID);
            newSettings.Name = attribute.SettingsName;

            try
            {
                _contentRepository.Save(newSettings, SaveAction.Publish, AccessLevel.NoAccess);
                UpdateSettings(siteDefinition.Id, newSettings);
            }
            catch (Exception e)
            {
                _log.Error(e.Message);
            }
        }
    }

    private void SiteCreated(object? sender, SiteDefinitionEventArgs e)
    {
        if (_contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot)
            .Any(x => x.Name.Equals(e.Site.Name, StringComparison.InvariantCultureIgnoreCase)))
        {
            return;
        }

        CreateSiteFolder(e.Site);
    }

    private void SiteDeleted(object? sender, SiteDefinitionEventArgs e)
    {
        var folder = _contentRepository.GetChildren<SettingsFolder>(GlobalSettingsRoot)
            .FirstOrDefault(x => x.Name.Equals(e.Site.Name, StringComparison.InvariantCultureIgnoreCase));

        if (folder == null)
        {
            return;
        }

        _contentRepository.Delete(folder.ContentLink, true, AccessLevel.NoAccess);
    }

    private void SiteUpdated(object? sender, SiteDefinitionEventArgs e)
    {
        if (e is SiteDefinitionUpdatedEventArgs updatedArgs)
        {
            var prevSite = updatedArgs.PreviousSite;
            var updatedSite = updatedArgs.Site;
            var settingsRoot = GlobalSettingsRoot;
            if (_contentRepository.GetChildren<IContent>(settingsRoot)
                .FirstOrDefault(x => x.Name.Equals(prevSite.Name, StringComparison.InvariantCultureIgnoreCase)) is ContentFolder currentSettingsFolder)
            {
                var cloneFolder = currentSettingsFolder.CreateWritableClone();
                cloneFolder.Name = updatedSite.Name;
                _contentRepository.Save(cloneFolder);
                return;
            }
        }

        CreateSiteFolder(e.Site);
    }

    private void PublishedContent(object? sender, ContentEventArgs e)
    {
        if (e?.Content is not SettingsBase)
        {
            return;
        }

        var parent = _contentRepository.Get<IContent>(e.Content.ParentLink);
        var site = _siteDefinitionRepository.Get(parent.Name);

        var id = site?.Id;
        if (id == null || id == Guid.Empty)
        {
            return;
        }
        UpdateSettings(id.Value, e.Content);
        RaiseEvent(new SettingEventData
        {
            SiteId = id.ToString(),
            ContentId = e.Content.ContentGuid.ToString()
        });
    }

    private Guid ResolveSiteId()
    {
        var request = _httpContext.HttpContext?.Request;
        if (request == null)
        {
            return Guid.Empty;
        }

        var site = _siteDefinitionResolver.GetByHostname(request.Host.Value, false, out _);
        if (site != null)
        {
            return site.Id;
        }

        site = _siteDefinitionRepository.List()
            .FirstOrDefault(x => x.Hosts.Any(x => x.Url.Host.Equals(request.Host.Value, StringComparison.OrdinalIgnoreCase)));

        if (site != null)
        {
            return site.Id;
        }

        return Guid.Empty;
    }

    private void SettingsEvent_Raised(object sender, EventNotificationEventArgs e)
    {
        // don't process events locally raised
        if (e.RaiserId != _raiserId && e.Param is SettingEventData settingUpdate && settingUpdate != null)
        {
            if (Guid.TryParse(settingUpdate.ContentId, out var contentId))
            {
                var content = _contentRepository.Get<IContent>(contentId);
                if (content != null && settingUpdate.SiteId != null)
                {
                    UpdateSettings(Guid.Parse(settingUpdate.SiteId), content);
                }
            }
        }
    }

    private void RaiseEvent(SettingEventData message) => _eventRegistry.Get(EventId).Raise(_raiserId, message);

}

[jonascarlbaum]

@jonascarlbaum we have done the fix suggested however we are stuck with how to handle or pass this Settings.MASTERKEY _synchronizedObjectInstanceCache.Remove(Settings.MASTERKEY); Appreciate your valuable suggestion on this, I am getting compile time error if used code as is The type or namespace name 'MASTERKEY' does not exist in the namespace 'Foundation.Infrastructure.Cms.Settings' (are you missing an assembly reference?) Thank you so much in advance

@avinashmpawar I only added this in the bottom of SettingsService.cs...

public static class Settings
{
    public static string MASTERKEY = "MASTERKEY";
}

Also updated my answer above #895 (comment)

[lunchin]

If someone check PR to see it works for them, @daniel-isaacs can merge this.