umbraco 9: imagesharp and custom virtual directory not working together

[shearer3000]

Which exact Umbraco version are you using? For example: 9.0.1 - don't just write v9

9.0.1

Bug summary

Hello

Umbraco out-of-the-box seems to expect that media will reside in wwwroot, but our requirement (and I suspect many others too) is the need to store umbraco media in a custom directory (say a UNC network path) or even an azure blob for production.

I have registered a custom media directory as per https://our.umbraco.com/Documentation/Extending/FileSystemProviders/ but then the issue seems to become how to tell ImageSharp about that location too (for crops on the website frontend). We found a ImageSharp thread about doing this SixLabors/ImageSharp.Web#135

Following both of the above ultimately seems to clash with each other as Umbraco middleware already registers Imagesharp in the start-up routine.

Is there a documented working solution or this? (anyone with a CD/automated deployment system will most likely need media to sit “outside” of the deployment package so that it “survives” deployments, so might be missing why this isn’t capability that more people are requesting?)

Thanks
Andrew

Specifics

No response

Steps to reproduce

no additional info to ticket summary

Expected result / actual result

No response

---

This item has been added to our backlog AB#14906

[creativesuspects]

I'd also like to use a static content folder outside of wwwroot on my Umbraco 9 projects. I've been able to get this working using a custom FileSystemProvider, but that breaks ImageSharp as @shearer3000 has already pointed out. I've even tried implementing a custom IImageProvider, but I'm a bit confused as to how (or where) I should tell ImageSharp to use the custom IImageProvider since ImageSharp is already being registered as middleware by Umbraco itself. It would be great if Umbraco 9 offered this feature out of the box with a simple appsettings.

[shearer3000]

are we missing something here @creativesuspects ? i wonder why we're the only ones requiring this? are we doing something wrong or is everyone else using an azure blog for all environments including dev and test? would be great to hear some insight / thoughts from someone at HQ @bergmania . thanks

[creativesuspects]

I’m also curious to know why there aren’t more devs running into this issue. I’m developing solo and I need this to easily share media between local and staging environments. But I would imagine that devs working in teams especially would use a “shared folder” setup all the time.

I haven’t had time to really dive into the Azure Blob Storage solution yet. But by the looks of it you’d need quite a bit of code to get it working. For something that could be fixed with a simple virtual directory in IIS prior to v9...

It’s great that Umbraco 9 offers the IFileSystem interface, but it’s kind of useless when it breaks ImageSharp in the process.

@shearer3000: Did you ask around on the Discord channel by any chance?

[shearer3000]

Discord channel? no not sure what that is

[shearer3000]

hi @bergmania any insight from HQ on this? thanks

[bergmania]

Hi @shearer3000 ...

We have assigned time to look at this in our upcoming sprint.

You should be able to configure ImageSharp even that it is also added by Umbraco. Maybe by calling services.Configure<T>() or services.AddImageSharp().

But hopefully we can make it easier :)

[Wiggee11]

For all of our V8 projects we stored the media files outside of the application directory for deployments to various environments, and now porting everything across to V9 we will need to find a solution for this.

@bergmania Can you shed a bit more light/instruction on the issue and how we can resolve/workaround before a permanent fix is available?

[creativesuspects]

Hi @shearer3000 ...

We have assigned time to look at this in our upcoming sprint.

You should be able to configure ImageSharp even that it is also added by Umbraco. Maybe by calling services.Configure<T>() or services.AddImageSharp().

But hopefully we can make it easier :)

I will test if (re)adding ImageSharp after initializing the FileSystemProvider makes a difference.

[bergmania]

I have something like this working.. Based on the two links you provided..

The important thing is that the ImageSharpMiddleware are running before the StaticFileMiddleware. Otherwise the request are handled by the StaticFileMiddleware (and ignoring the query)

using System.IO;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using SixLabors.ImageSharp.Web;
using SixLabors.ImageSharp.Web.DependencyInjection;
using SixLabors.ImageSharp.Web.Providers;
using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Core.Configuration.Models;
using Umbraco.Cms.Core.DependencyInjection;
using Umbraco.Cms.Core.IO;
using Umbraco.Cms.Infrastructure.DependencyInjection;
using Umbraco.Cms.Web.Common.ApplicationBuilder;
using IHostingEnvironment = Umbraco.Cms.Core.Hosting.IHostingEnvironment;

namespace UmbracoExamples.Composition
{
    public class SetMediaFileSystemComposer : IComposer
    {
        public void Compose(IUmbracoBuilder builder)
        {
            var rootPath = Path.Combine("C:", "storage", "umbracoMedia","media");
            
            builder.SetMediaFileSystem((factory) =>
            {
                IHostingEnvironment hostingEnvironment = factory.GetRequiredService<IHostingEnvironment>();
                IOptions<GlobalSettings> globalSettings = factory.GetRequiredService<IOptions<GlobalSettings>>();
              
                var rootUrl = hostingEnvironment.ToAbsolute("/media");

                return new PhysicalFileSystem(
                    factory.GetRequiredService<IIOHelper>(),
                    hostingEnvironment,
                    factory.GetRequiredService<ILogger<PhysicalFileSystem>>(),
                    rootPath,
                    rootUrl);
            });

            var fileProvider = new PhysicalFileProvider(rootPath);
            builder.Services.Configure<UmbracoPipelineOptions>(options =>
                options.AddFilter(new UmbracoPipelineFilter(nameof(SetMediaFileSystemComposer))
                {
                    PrePipeline = app =>
                    {
                        IOptions<GlobalSettings> globalSettings = app.ApplicationServices.GetRequiredService<IOptions<GlobalSettings>>();

                        app.UseImageSharp();
                        app.UseStaticFiles(new StaticFileOptions
                        {
                            FileProvider = fileProvider,
                            RequestPath = "/media"
                        });
                    }
                }));

            builder.Services.AddImageSharp().ClearProviders().AddProvider<CustomMediaPhysicalFileSystemProvider>();
         }
    }

    
    
    public class CustomMediaPhysicalFileSystemProvider : PhysicalFileSystemProvider
    {
        public CustomMediaPhysicalFileSystemProvider(
            IWebHostEnvironment environment,
            FormatUtilities formatUtilities) : base(UseCustomWebRootFileProvider(environment), formatUtilities)
        {
        }

        private static IWebHostEnvironment UseCustomWebRootFileProvider(IWebHostEnvironment environment)
        {
            return new CustomMediaWebHostEnvironment(environment);
        }
    }

    internal class CustomMediaWebHostEnvironment : IWebHostEnvironment
    {
        private readonly IWebHostEnvironment _environment;

        public CustomMediaWebHostEnvironment(IWebHostEnvironment environment)
        {
            _environment = environment;
            WebRootFileProvider = new PhysicalFileProvider(Path.Combine("C:", "storage", "umbracoMedia"));
        }

        public string ApplicationName
        {
            get => _environment.ApplicationName;
            set => _environment.ApplicationName = value;
        }

        public IFileProvider ContentRootFileProvider
        {
            get => _environment.ContentRootFileProvider;
            set => _environment.ContentRootFileProvider = value;
        }
        
        public string ContentRootPath
        {
            get => _environment.ContentRootPath;
            set => _environment.ContentRootPath = value;
        }
        public string EnvironmentName
        {
            get => _environment.EnvironmentName;
            set => _environment.EnvironmentName = value;
        }
        
        public string WebRootPath        
        {
            get => _environment.WebRootPath;
            set => _environment.WebRootPath = value;
        }
        public IFileProvider WebRootFileProvider { get; set; }
    }
}

[creativesuspects]

@bergmania Looking forward to test your code tomorrow!

[creativesuspects]

@bergmania I used your code to add ImageSharp followed by my custom FileSystemProvider, but I implemented the FileSystemProvider slightly different based on these comments:

SixLabors/ImageSharp.Web#135 (comment)
https://github.com/SixLabors/ImageSharp.Web/blob/master/src/ImageSharp.Web/Providers/PhysicalFileSystemProvider.cs

So on Development and Staging I'm loading media files from a shared Dropbox folder (outside of wwwroot) and on Production I'm using the default wwwroot/media location. The path for the Dropbox folder is configured in appsettings.Development.json and appsettings.Staging.json. The only issue I've noticed so far with this setup is that it seems physical media files that were added on Development are not being deleted when the media item is deleted on Staging and the recycle bin is emptied. When I upload and delete media items on the same environment the physical file does get deleted. So that's a weird one.

public class CustomMediaFileSystemComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        var mediaPath = "";

        builder.SetMediaFileSystem((factory) =>
        {
            var webHostEnvironment = factory.GetRequiredService<IWebHostEnvironment>();
            var hostingEnvironment = factory.GetRequiredService<IHostingEnvironment>();
            var rootUrl = hostingEnvironment.ToAbsolute("~/media");

            mediaPath = hostingEnvironment.MapPathWebRoot("~/media");

            if (webHostEnvironment.IsProduction() == false)
            {
                var mediaSettings = new MediaSettings();
                builder.Config.GetSection(MediaSettings.ConfigurationKey).Bind(mediaSettings);

                if (string.IsNullOrEmpty(mediaSettings.RootPath) == false)
                {
                    mediaPath = mediaSettings.MediaPath;
                }
            }

            return new PhysicalFileSystem(
                factory.GetRequiredService<IIOHelper>(),
                hostingEnvironment,
                factory.GetRequiredService<ILogger<PhysicalFileSystem>>(),
                mediaPath,
                rootUrl);
        });

        builder.Services.Configure<UmbracoPipelineOptions>(options =>
            options.AddFilter(new UmbracoPipelineFilter(nameof(CustomMediaFileSystemComposer))
            {
                PrePipeline = app =>
                {
                    var fileProvider = new PhysicalFileProvider(mediaPath);

                    app.UseImageSharp();
                    app.UseStaticFiles(new StaticFileOptions
                    {
                        FileProvider = fileProvider,
                        RequestPath = "/media"
                    });
                }
            }));

        builder.Services.AddImageSharp().ClearProviders().AddProvider<CustomMediaFileSystemProvider>();
    }
}

public class CustomMediaFileSystemProvider : IImageProvider
{
    private readonly IFileProvider fileProvider;
    private readonly FormatUtilities formatUtilities;
    private readonly IConfiguration config;

    public CustomMediaFileSystemProvider(
        IWebHostEnvironment environment,
        FormatUtilities formatUtilities,
        IConfiguration config)
    {
        this.formatUtilities = formatUtilities ?? throw new ArgumentNullException(nameof(formatUtilities));
        this.config = config ?? throw new ArgumentNullException(nameof(config));

        var mediaSettings = new MediaSettings();
        this.config.GetSection(MediaSettings.ConfigurationKey).Bind(mediaSettings);

        if (environment.IsProduction() == false && string.IsNullOrEmpty(mediaSettings.RootPath) == false)
        {
            this.fileProvider = new PhysicalFileProvider(mediaSettings.RootPath);
        }
        else
        {
            this.fileProvider = environment?.WebRootFileProvider ?? throw new ArgumentNullException(nameof(environment));
        }
    }

    public ProcessingBehavior ProcessingBehavior { get; } = ProcessingBehavior.CommandOnly;

    public Func<HttpContext, bool> Match { get; set; } = _ => true;

    public bool IsValidRequest(HttpContext context) => this.formatUtilities.GetExtensionFromUri(context.Request.GetDisplayUrl()) != null;

    public Task<IImageResolver> GetAsync(HttpContext context)
    {
        // Path has already been correctly parsed before here.
        IFileInfo fileInfo = this.fileProvider.GetFileInfo(context.Request.Path.Value);

        // Check to see if the file exists.
        if (fileInfo.Exists == false)
        {
            return Task.FromResult<IImageResolver>(null);
        }

        var metadata = new ImageMetadata(fileInfo.LastModified.UtcDateTime, fileInfo.Length);
        return Task.FromResult<IImageResolver>(new PhysicalFileSystemResolver(fileInfo, metadata));
    }
}

And the ApplicationSettings model and json:

public class ApplicationSettings
{
    public const string ConfigurationKey = "Application";

    public MediaSettings Media { get; set; }
}

public class MediaSettings
{
    public const string ConfigurationKey = "Application:Media";

    public string RootPath { get; set; }

    public string MediaPath { get; set; }
}

{
  "$schema": "./umbraco/config/appsettings-schema.json",
  "Application": {
    "Media": {
      "RootPath": "D:\\Dropbox\\UmbracoMedia\\project",
      "MediaPath": "D:\\Dropbox\\UmbracoMedia\\project\\media"
    }
  }
}

Maybe the physical file deletion problem is a Dropbox related issue, I'm not sure. This shouldn't be necessary, but I've added an INotificationHandler to solve this issue:

public class DeletePhysicalMediaFileHandler : INotificationHandler<MediaEmptiedRecycleBinNotification>
{
    private readonly ILogger<DeletePhysicalMediaFileHandler> logger;
    private readonly ApplicationSettings applicationSettings;
    private readonly IWebHostEnvironment webHostEnvironment;

    public DeletePhysicalMediaFileHandler(
        ILogger<DeletePhysicalMediaFileHandler> logger,
        IOptions<ApplicationSettings> applicationSettings,
        IWebHostEnvironment webHostEnvironment)
    {
        this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
        this.applicationSettings = applicationSettings?.Value ?? throw new ArgumentNullException(nameof(applicationSettings));
        this.webHostEnvironment = webHostEnvironment ?? throw new ArgumentNullException(nameof(webHostEnvironment));
    }

    public void Handle(MediaEmptiedRecycleBinNotification notification)
    {
        foreach (var media in notification.DeletedEntities)
        {
            try
            {
                if (media.HasProperty("umbracoFile") == false)
                    continue;

                var cropInfo = media.GetValue<string>("umbracoFile");
                var mediaSrc = JsonConvert.DeserializeObject<ImageCropperValue>(cropInfo).Src;

                if (mediaSrc.StartsWith("/media/") == false)
                    continue;

                var path = Path.Combine(GetMediaRootPath(), mediaSrc.Substring(1).Replace("/", "\\"));

                File.Delete(path);
            }
            catch (Exception ex)
            {
                this.logger.LogError($"Error deleting media file: { ex.Message }", ex);
            }
        }
    }

    private string GetMediaRootPath()
    {
        if (this.webHostEnvironment.IsProduction())
            return webHostEnvironment.WebRootPath;

        return this.applicationSettings.Media.RootPath;
    }
}

[shearer3000]

Hi @bergmania thanks for the code sample above. Just checking this definitely worked for you? When I updated our website to use it, the media section all works fine but the frontend continues to show the original image rather than the crop the url specifies (querystring). Its like Imagesharp still isn't handling the request and is still falling back to the static file provider.

You mentioned:

important thing is that the ImageSharpMiddleware are running before the StaticFileMiddleware

Is that regarding these lines of code?

                    app.UseImageSharp();
                    app.UseStaticFiles(new StaticFileOptions
                    {
                        FileProvider = fileProvider,
                        RequestPath = "/media"
                    });

any other thoughts or ideas why Imagesharp continues to ignore handling the image request would be so appreciated.

@creativesuspects i take it your code works for you now?

thanks
Andrew

[creativesuspects]

@creativesuspects i take it your code works for you now?

@shearer3000 Yes, it works. I’m actively using it on dev, staging and production for a project. Although I still need the NotificationHandler to make sure the physical files are being deleted on dev and staging. But again, this may be a Dropbox related issue. I’m not sure. Maybe you could use my code (without the NotificationHandler) on a UNC network path to see what happens?

Even with this code working and even if Umbraco would build something similar into the core to make it easier to change the media file location on a specific environment it still sucks that you can’t use a simple virtual directory on IIS. I often use additional virtual directories for other files that are being stored, like PDF invoices or reports. These files also need to be shared across dev and staging. Of course this is not an Umbraco issue, but a .NET Core issue. Luckily with these files I don’t have to worry about ImageSharp, so it can probably be fixed with just a few lines of code:

https://www.jauernig-it.de/asp-net-coreiis-serving-content-from-a-file-share/