Julia is unable to launch in the Universal Windows Platform (UWP)

[JanisErdmanis]

One of the prerequisites for applications to be accepted in the Microsoft marketplace is that they must run within a UWP containerised environment. This means that fullTrust capability should only be used if there is a good reason for doing so. Unfortunately now when Julia 1.9.3 on Windows 11 is started from the UWP environment Julia crashes at launch with the following error:

fatal: error thrown and no exception handler available.
InitError(mod=:Sys, error=ErrorException("could not load library "libpcre2-8"
The parameter is incorrect. "))
ijl_errorf at C:/workdir/src\rtutils.c:77
ijl_load_dynamic_library at C:/workdir/src\dlload.c:369
jl_get_library_ at C:/workdir/src\runtime_ccall.cpp:48
jl_get_library_ at C:/workdir/src\runtime_ccall.cpp:39 [inlined]
ijl_load_and_lookup at C:/workdir/src\runtime_ccall.cpp:61
jlplt_pcre2_compile_8_51593 at C:\Program Files\WindowsApps\JuliaUWP_0.1.0.0_neutral__s0by06ay2112j\lib\julia\sys.dll (unknown line)
compile at .\pcre.jl:161
compile at .\regex.jl:75
match at .\regex.jl:376
match at .\regex.jl:376 [inlined]
match at .\regex.jl:395 [inlined]
splitdrive at .\path.jl:38
joinpath at .\path.jl:264
joinpath at .\path.jl:327 [inlined]
abspath at .\path.jl:449 [inlined]
__init_build at .\sysinfo.jl:128
__init__ at .\sysinfo.jl:120
jfptr___init___34208 at C:\Program Files\WindowsApps\JuliaUWP_0.1.0.0_neutral__s0by06ay2112j\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1880 [inlined]
jl_module_run_initializer at C:/workdir/src\toplevel.c:75
_finish_julia_init at C:/workdir/src\init.c:855
jl_repl_entrypoint at C:/workdir/src\jlapi.c:711
mainCRTStartup at C:/workdir/cli\loader_exe.c:59
uaw_wcsrchr at C:\Windows\System32\KERNEL32.DLL (unknown line)
uaw_wcsrchr at C:\Windows\System32\KERNEL32.DLL (unknown line)
ZwWaitLowEventPair at C:\Windows\SYSTEM32\ntdll.dll (unknown line)

Introduction

To reproduce the error, let’s start with something that works, using a fullTrust capability. To do so, we can unzip Julia for Windows and, within it, place the following AppxManifest.xml file:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Package xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10" 
         xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
         xmlns:uap10="http://schemas.microsoft.com/appx/manifest/uap/windows10/10"
         xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities">

  <Identity Name="JuliaUWP" 
            Publisher="CN=JuliaUser" 
            Version="0.1.0.0" />

  <Properties>
    <DisplayName>JuliaUWP</DisplayName>
    <PublisherDisplayName>AppBundler</PublisherDisplayName>
    <Logo>assets\icon.png</Logo>
  </Properties>

  <Dependencies> 
    <TargetDeviceFamily Name="Windows.Desktop" MinVersion="10.0.10240.0" MaxVersionTested="10.0.19041.0" />
  </Dependencies>

  <Resources>
    <Resource Language="en-us" />
  </Resources>

  <Applications>
    <Application EntryPoint="Windows.FullTrustApplication" 
		 Executable="bin\julia.exe" 
		 Id="JuliaUWP" 
		 uap10:RuntimeBehavior="packagedClassicApp" 
		 uap10:Subsystem="console" 
		 uap10:SupportsMultipleInstances="true" 
		 uap10:TrustLevel="mediumIL">
		     
    <uap:VisualElements DisplayName="Julia UWP" 
                         Square150x150Logo="icon.png"
                         Square44x44Logo="icon.png"
                         BackgroundColor="transparent"
                         Description="Julia UWP">
       <uap:DefaultTile
           Wide310x150Logo="icon.png"
           ShortName="Julia UWP"
           Square71x71Logo="icon.png"
           Square310x310Logo="icon.png">
       </uap:DefaultTile>
     </uap:VisualElements>
    </Application>
  </Applications>

  <Capabilities>
    <rescap:Capability Name="runFullTrust"/>
  </Capabilities>

</Package>

This file shall be placed within a Julia directory from which a PowerShell can register it with

Add-AppPackage -register AppxManifest.xml

That adds a start menu item for Julia UWP which should work and open Julia console when launched.

Reproducing the Error

The resulting application with the full trust capability works so let’s take the next step. We can remove full trust capability from the AppxManifest.xml by setting EntryPoint="app", RuntimeBehavior="windowsApp" and TrustLevel="appContainer", which results in:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Package xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10" 
         xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
         xmlns:uap10="http://schemas.microsoft.com/appx/manifest/uap/windows10/10"
         xmlns:uap11="http://schemas.microsoft.com/appx/manifest/uap/windows10/10" 
         xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities">

  <Identity Name="JuliaUWP" 
            Publisher="CN=JuliaUser" 
            Version="0.1.0.0" />

  <Properties>
    <DisplayName>JuliaUWP</DisplayName>
    <PublisherDisplayName>AppBundler</PublisherDisplayName>
    <Logo>assets\icon.png</Logo>
  </Properties>

  <Dependencies> 
    <TargetDeviceFamily Name="Windows.Desktop" MinVersion="10.0.10240.0" MaxVersionTested="10.0.19041.0" />
  </Dependencies>

  <Resources>
    <Resource Language="en-us" />
  </Resources>

<Applications>
    <Application EntryPoint="app" 
		 Executable="bin\julia.exe" 
		 Id="JuliaUWP" 
		 uap10:RuntimeBehavior="windowsApp" 
		 uap10:Subsystem="console" 
		 uap10:SupportsMultipleInstances="true" 
		 uap10:TrustLevel="appContainer">
		     
    <uap:VisualElements DisplayName="Julia UWP" 
                         Square150x150Logo="icon.png"
                         Square44x44Logo="icon.png"
                         BackgroundColor="transparent"
                         Description="Julia UWP">
       <uap:DefaultTile
           Wide310x150Logo="icon.png"
           ShortName="Julia UWP"
           Square71x71Logo="icon.png"
           Square310x310Logo="icon.png">
       </uap:DefaultTile>
     </uap:VisualElements>
    </Application>
  </Applications>
</Package>

Adding it with Add-AppPackage -register AppxManifest.xml and launching it from the menu reproduces the error.

[PallHaraldsson]

TL;DR Maybe close this issue and/or rename its title to support rather WinUI 3/Windows App SDK. Though likely such support belongs elsewhere, and then please add a link here to that issue you may open. Tip, maybe just try MSIX tool I point to below, or WinUI 3?

One of the prerequisites for applications to be accepted in the Microsoft marketplace is that they must run within a UWP containerised environment.

That may not be correct. I was thinking if UWP is needed, then this issue should be given "priority" label, but I see UWP is "deprecated" and then it should not...

Traditional desktop apps in the Microsoft Store on Windows
https://developer.microsoft.com/en-us/microsoft-store/desktop-apps

But the utility and desirability of the Windows platform comes largely from the huge selection of powerful and productive traditional desktop applications using traditional installers [..] In recognition of that, last year, we introduced a preview program to enable those traditional desktop apps, commonly called “Win32” apps, packaged in .EXE or .MSI installers, and built using anything [..]

APPX, the UWP format, is not needed:
Microsoft Store submission API for MSI or EXE app
https://learn.microsoft.com/en-us/windows/apps/publish/store-submission-api

UWP is containerized, and provides limited API (maybe more limited than Julia needs), e.g. restricts access to files (but seemingly that and any and all that restriction can be turned off), i.e. is a sandbox, so NOT appropriate for Julia itself? As opposed to Julia apps, where you may want to limit access to e.g. a microphone. Did you mean the issue here for Julia itself, or for apps made with (then this might belong at PackageComplier.jl). A first step might be to make only Julia work with UWP or whatever alternative, e.g. WinUI [3], (as an experiment, to) then make Julia apps work, but possibly e.g. UWP will not work for Julia since its .exe is a console type (not GUI type), only for compiled (GUI) Julia apps.

I believe this source usually reliable, but I don't see UWP deprecated at e.g. its Wikipedia page, maybe WinUI 3 is similar enough, or UWP was undeprecated:
https://www.thurrott.com/dev/258377/microsoft-officially-deprecates-uwp

Microsoft continues to baby-step around the obvious, but it has officially deprecated the Universal Windows Platform (UWP) as it pushes the desktop-focused Windows App SDK (formerly called Project Reunion) and WinUI 3 as the future of Windows application development.
[..]
For those unclear on the matter, the Windows App SDK basically takes key UWP technologies and new technologies like WinUI 3 that will not be backported to UWP and makes them available to developers

I dug in, to find out the truth and I find:
Create app store listings
https://learn.microsoft.com/en-us/windows/apps/publish/publish-your-app/create-app-store-listing?source=recommendations&pivots=store-installer-msi-exe

Yes, MSIX is the default, but next option to click is "MSI or EXE" (I see elsewhere MSIX allows "in-app ads", and UWP seemingly, but not mentioned for "MSI or EXE", then likely not allowed?). I.e. I think traditional EXE files are allowed is for store apps. At least traditional desktop/GUI .exe, but note .exe comes in many forms, also console apps, and Julia is a console app, not a GUI app. And an .exe can't be both, its header encodes the type.

So that type of console app, .exe may be forever blocked from the store? And actually possibly then no alternative?

Also seemingly MSIX can be used:

The MSIX package format preserves the functionality of existing app packages and/or install files in addition to enabling new, modern packaging and deployment features to Win32 [..] apps.

See here to try it out:
https://learn.microsoft.com/en-us/windows/msix/packaging-tool/tool-overview

The MSIX Packaging Tool enables you to repackage your existing desktop applications to the MSIX format. It offers both an interactive UI and a command line for conversions [..]

Prerequisites

• Windows 10, version 1809 (or later)

[MSIX is latest, for Win10+, MSI older, and APPX file format, maybe better if you want to support down to Windows 8? "Unlike legacy desktop apps, APPX is the only installation system allowed for UWP apps."]

Compiled programs can be made, i.e. compiled console apps, and I think GUI apps (then the "non-console apps" too, not sure, if yet supported, maybe GUI programs are a hack running from an open console windows, if that's possible?). If the latter can be done, great, then UWP or WinUI 3 would possibly be a better restricted/sandboxed variant, to support making.

https://learn.microsoft.com/en-us/windows/apps/winui/winui3/release-notes/winui3-project-reunion-0.5

Windows UI Library (WinUI) 3 is a native user experience (UX) framework for building modern Windows apps. It ships independently from the Windows operating system as a part of Project Reunion (now called the Windows App SDK).

See here video on Project Reunion, which is about merging desktop APIs with UWP, not being restricted to either, but also interestingly having those Win11 APIs, available in older versions of Windows, though seemingly only down to some Windows 10 version, see below (I suppose then bundling proprietary Windows components with the app, if actually non-free then the GPL system exception would not apply for such distribution/conveying, though Julia is recently MIT-only, while bundling GPL component that could then not be used):
https://learn.microsoft.com/en-us/shows/visual-studio-toolbox/project-reunion

https://learn.microsoft.com/en-us/windows/apps/windows-app-sdk/

The Windows App SDK provides a unified set of APIs and tools that can be used in a consistent way by any desktop app on Windows 11 and downlevel to Windows 10, version 1809.

Latest version is currently 1.4 with "End of servicing" "08/29/2024"

Customers can choose Current releases or Maintenance releases.

"Customers" seems to imply non-free/proprietary components.

I could download the Windows App SDK, and while it has a LICENCE folder it only has two such cryptic XML files that didn't have the info I wanted to see:

<License xmlns:xsd="http://www.w3.org/2001/XMLSchema" ...

https://learn.microsoft.com/en-us/windows/apps/publish/publish-your-app/create-app-store-listing?source=recommendations&pivots=store-installer-msi-exe

https://learn.microsoft.com/en-us/windows/apps/winui/

At this time, there are two generations of the Windows UI Library (WinUI): WinUI 2 for UWP and WinUI 3 in the Windows App SDK. While both can be used in production-ready apps on Windows 10 and later, each have different development targets.

See Comparison of WinUI 3 and WinUI 2.

WinUI 3 can be used to build production-ready desktop/Win32 Windows apps.

WinUI 2 can be used in UWP applications and [..]

WinUI 3 is supported only in desktop-based projects. To use WinUI 3, UWP projects can migrate their project type to desktop (see how to migrate your UWP app to the Windows App SDK).

https://blogs.embarcadero.com/the-future-of-windows-gui-for-python-java-c-javascript/

Why are we not all writing programs that use UWP?

After some time, Microsoft claimed to have a single OS that could power the Surface devices, Xbox, desktop apps, touch enabled devices such as tablets, and Windows Phones. So, Windows 10 brought the UWP (Universal Windows Platform), where you build a single app that can run on all these devices, at least theory. One of the most remarkable things about UWP was its security, which isolates individual apps from each other and the underlying OS into a kind of sandbox.

However, these technologies struggled to adopt widespread development for all developers. For instance apps created with UWP are restricted to running in a closed sandbox with a user interface which was very much prescribed by Microsoft.

Issue 513976: Chrome is not a UWP App on Win 10
https://bugs.chromium.org/p/chromium/issues/detail?id=513976

WontFix (Closed)

Intriguingly Python is available as UWP (for IoT), but it's very restrictive, I'm not sure if "background UWP application" is a separate type, or all UWP have those restrictions (shared by all GUI apps?):

https://github.com/ms-iot/python/blob/develop/README_uwp.md

PythonUWP is based on a modified version of CPython 3.5 which supports running as a Windows 10 UWP (Universal Windows Platform) application. Specifically, this supports running in a background application on Windows 10 IoT core.

Since there are a lot of differences between a win32 application and an background UWP application, naturally PythonUWP also inherited some of these differences. This document tries to highlight some of them. This is not an exhaustive list by any means.

Input/Output
As the name suggests, a background UWP application runs in the background and doesn't contain any user interface elements, specifically a console. Thus, the standard outputs -- sys.stdout, sys.stdout, sys.stderr, and sys.stderr -- have been redirect to OutputDebugString() or the "debug output window" when running under PTVS in Visual Studio (https://github.com/Microsoft/PTVS).

The standard inputs -- sys.stdin and sys.stdin -- are set to None. Any attempts to read from it by calling Input() without redirecting the sys.stdin will result in a RuntimeError.

As usual, the user can the sys.stdin, sys.stdout, and/or sys.stderr to file and/or sockets as needed.

Environment variables
UWP applications do not have the concept of environment variables.

https://learn.microsoft.com/en-us/windows/uwp/get-started/universal-application-platform-guide

UWP is one of many ways to create client applications for Windows. UWP apps use WinRT APIs [..]

UWP is one choice for creating apps that run on Windows 10 and Windows 11 devices, and can be combined with other platforms. UWP apps can make use of Win32 APIs and .NET classes (see API Sets for UWP apps, Dlls for UWP apps, and .NET for UWP apps).

The Microsoft development story continues to evolve, and along with initiatives such as WinUI, MSIX, and the Windows App SDK, UWP is a powerful tool for creating client apps.

Features of a UWP app

A UWP app is:

• Secure: UWP apps declare which device resources and data they access. The user must authorize that access.
• Able to use a common API on all devices that run Windows. [I.e. not all APIs]
  [..]
• Programmable in C#, C++, Visual Basic, and JavaScript. For UI, use WinUI, XAML, HTML, or DirectX.
  [..]

Let's look at these in more detail.

Secure

UWP apps declare in their manifest the device capabilities they need such as access to the microphone, location, Webcam, USB devices, files, and so on. The user must acknowledge and authorize that access before the app is granted the capability.

A common API surface across all devices

Windows 10 introduced the Universal Windows Platform (UWP), which provides a common app platform on every device that runs Windows. The UWP core APIs are the same on all Windows devices. If your app only uses the core APIs, it will run on any Windows device no matter whether you are targeting a desktop PC, Xbox, Mixed-reality headset, and so on.

A UWP app written in C++/WinRT has access to the Win32 APIs that are part of the UWP. These Win32 APIs are implemented by all Windows devices.
Extension SDKs expose the unique capabilities of specific device types

If you target the universal APIs, then your app can run on all devices that run Windows 10 or later. But if you want your UWP app to take advantage of device-specific APIs, then you can do that, too.

Monetize your app
[..]

• In-app purchases.

[..]

• Integrate Cortana to add voice command capability to your app. [Cortana app has been discontinued...]

Use a language you already know
[..]
The Windows Runtime, and WinRT APIs, are an evolution of Windows APIs. Originally, Windows was programmed via flat, C-style Win32 APIs. To those were added COM APIs [..]
And, of course, you can call WinRT APIs from your UWP app. UWP is an application model built on top of the Windows Runtime. Technically, the UWP application model is based on CoreApplication, although that detail may be hidden from you, depending on your choice of programming language. As this topic has explained, from a value proposition point of view, the UWP lends itself to writing a single binary that can, should you choose, be published to the Microsoft Store and run on any one of a great range of device form factors. The device reach of your UWP app depends on the subset of Windows Runtime APIs that you limit your app to calling, or that you call conditionally.

How the Universal Windows Platform relates to Windows Runtime APIs
[..] Hopefully, this section has been successful in describing the difference between the technology underlying Windows Runtime APIs, and the mechanism and business value of the Universal Windows Platform.

[JanisErdmanis]

I have not been clear. This is not an issue for the distribution of Julia itself on Windows. Even if Microsoft pushes hard for Windows S, I have no doubts that Julia would get accepted with fullTrust capability in the Windows app store. It just does not make sense to run Julia in a containerised development environment.

The issue is with GUI applications, which will be built with sophisticated frameworks like QML.jl and Gtk.jl, Mousetrap.jl, and others which would face this issue. Nevertheless, I don’t think this should be considered a priority, as currently, a workaround is available for using fullTrust capability. The only drawback is that it may depend on the reviewer, who may insist that the application’s functionality does not require full system access. In that scenario, applications can be distributed outside the MS store without a review procedure.

One of the likeable things about containerisation is the promise to the user that the application won’t mess up your system. Windows users may become more conscious and eventually avoid installing applications requiring fullTrust capability. They may even use Windows S, which only allows them to install apps from the Windows app store. And then this issue could become pressing.

The idea for this issue is that perhaps someone could have seen a case like this before and have suggestions on how to proceed and what to check. And in this chaotic way, the core issue and solution may become clear.

I will get back to this. It is hard to digest your reply in one attempt.

[PallHaraldsson]

About actually fixing your issue, i.e. I recalled
"could not load library "libpcre2-8"
from an otherwise unrelated issue #52205

and I think PCRE library maybe one of the first Julia dependency loaded, i.e. DLL. And it's supposed to open always, so no exception handler for it isn't bad, but I'm not sure where "The parameter is incorrect." comes from since I don't find it in the Julia source code. I think PCRE DLL may actually be starting and failing for some reason. At first I though this might be a path problem, I'm not sure what instructed Windows where to find/look up DLLs, in Linux/Unix it may be an ENV var and at least UWP doesn't support such.

[PallHaraldsson]

I looked into and seemingly the "reason" comes from Windows complaining about the DLL:

julia/src/dlload.c

Line 388 in e75dd47

jl_errorf("could not load library \"%s\"\n%s", modname, reason);

Since DLLs aren't supported in UWP, in the same way, or need some fixing, as explained here:
https://learn.microsoft.com/en-us/cpp/porting/how-to-use-existing-cpp-code-in-a-universal-windows-platform-app?view=msvc-170

I also found here something that might be relevant:
https://github.com/Microsoft/vcpkg/blob/master/ports/pcre/portfile.cmake

or not, but PCRE would likely just be the first of may DLLs Julia dependencies with potential problems... also I guess more from e.g. Mousetrap.jl. I believe there's a global workaround as you stated, might not work in Windows S.

I'm not on Windows so I can't test anything. It might be helpful to ask about this on Julia discourse, how to support UWP, or rather the similar WinUI 3. It IS desirable to support Julia apps, in a restrictive containerized scenario, since it seemd to be desirable and the future; maybe even have it a supported option for PackageCompiler.jl, and the first step would be to get Julia alone to work. @KristofferC is its main developer, also a core dev here, and he might have an opinion if an issue should rather be opened there, or kept open here, at least for now.

[davidanthoff]

I have no doubts that Julia would get accepted with fullTrust capability in the Windows app store

Just FYI, Julia is in the Windows Store, and it did get fullTrust permissions (and actually a bit more than that).

One option might be that a GUI application could take a dependency on Julia in the Windows Store, so that is could get the already approved version of Julia installed. I forgot the details, but I think there is something like a framework dependency with MSIX packages, or something like that.

[JanisErdmanis]

Taking Julia as a dependency from the Windows store seems like an interesting option to explore. I have seen some examples of Python app bundling where it is done that way. However, I have some doubts that it would escape this error when the application runs in UWP confinement. The reason is that otherwise, it would provide an obvious security loophole where containerised apps would be able to access and modify the system.

[MilesCranmer]

Seems like some PySR users have run into this bug on what I presume are normal Windows installations:

• [BUG]: libpcre2-8 bug on Windows MilesCranmer/PySR#636
• Windows Julia Install - could not load library "libpcre2-8" The specified module could not be found.  MilesCranmer/PySR#566

Is there any workaround?

[Angelld23]

After spending all day trying to work around this issue, I was able to sort of solve it. I was having issues accessing the path for other reasons but adding the julia's bin path (where all the dll files are) to the PATH environment variable should solve it. However, due to my specific problems I had to write a script to load each DLL file in the bin directory before importing anything else. I was trying to use PySR, but I guess this might work for everybody.

import os
import ctypes
import glob

Path to the bin directory of your Julia installation

julia_bin_path = C:\path\to\your\dll_directory (which is the same as the julia.exe)

Add the bin directory to PATH

os.environ["PATH"] += ";" + julia_bin_path

Load each DLL file in the bin directory

for dll_path in glob.glob(os.path.join(julia_bin_path, "*.dll")):
try:
ctypes.CDLL(dll_path)
print(f"Loaded {dll_path} successfully.")
except OSError as e:
print(f"Could not load {dll_path}: {e}")

from pysr import PySRRegressor

[JanisErdmanis]

This looks promising. I will see if I can get Julia’s REPL to launch in a sandboxed environment over the next week with this approach. @Angelld23 Have you perhaps already managed to do that?

[Angelld23]

@JanisErdmanis My issue was based on the PySR library installation in python, but shared the solution here to see if it would be helpful for you guys. I believe it should work anyways. Basically, due to the UWP issues (not sure if it is the source), you should find a way to load all the dll files manually into the path. After I added "libpcre2-8", I started getting the same error but with the others dll files. that is why in the script I had them all to be more specific. So the issues is not just libpcre2-8, but almost if not all .dll files. There should be a better way to solve it, but that is what I managed to do for today. Please let me know if it helped you solve it too.

[JanisErdmanis]

I played around with the issue yesterday by making a C# launcher for loading the libraries manually and then starting Julia (see gist). It went nowhere, but I finally understand the issue's root cause, which is the dependency on msvcrt.dll which is not available in the UWP containerized environment.

To strengthen my hypothesis I made a test with Python to see that it also suffers from the same issue. The last version of Python that depended on msvcrt.dll was Python 2.5 which was released in early 2010. I verified that by executing:

dumpbin /dependents python.exe

I tested that it indeed fails to load in a UWP containerised environment. Then to check that my configuration was correct I picked Python 2.7 which depends on MSVCR90.dll and it loaded just fine.

I also found that Python and C# launcher itself is capable of loading libraries that depend on msvcrt.dll. As I understand that is because it uses the Windows API redirection layer to handle the msvcrt.dll calls.

This led me to the idea that perhaps full recompilation of Julia's ecosystem with UCRT is not needed. Instead, it is perhaps possible to compile Julia launcher julia.exe with UCRT which would do the redirection calls made to msvcrt.dll automatically.