Proposal: Unified Mado API

[kkoreilly]

The README states that the purpose of mado is to be a base layer for Go GUI frameworks. Currently, various Go GUI frameworks depend on go-gl/glfw, which is why one of the goals of mado is to provide a glfw compatible API. However, glfw is not the best basis for an elegant, cross-platform Go library, given that it is a C library written for desktop only. Trying to satisfy the glfw API limits the scope of the mado project and forces GUI frameworks to figure out mobile and web with their own abstractions.

Because of the lack of a unified, cross-platform system API, many Go GUI frameworks like Cogent Core, Fyne, Gio, and Ebiten have written their own versions of the same fundamental API: an interface between messy platform-specific code and a simple, cross-platform Go API. Therefore, significant amounts of time are wasted implementing the same things across the different frameworks, with each one having to figure out each new feature manually. Additionally, this leads to various features being missing in some or all frameworks, including accessibility, push notifications, and IME support, the original motivation for mado.

Instead of each framework individually implementing IME support or waiting for a PR almost nine years old to be merged (glfw/glfw#658), I propose that we collaborate on turning mado into a fully functional, cross-platform, completely glfw-independent API designed specifically for Go GUI frameworks. The core structure behind all of the system app driver implementations is very similar across the different frameworks, and I believe that we can reach a consensus on an API that will work for all of them and reduce the amount of time all of us have to spend writing code in C, Objective-C, and Java.

This is a concrete outline of my proposal for the new structure of mado:

• Remove all existing directories, including glfw
• Form the following package structure:
  • mado (the root of the repository) contains cross-platform interfaces like App and Window that importers of mado interact with and platform-specific packages implement
    • windows, mac, x11, wayland, android, ios, web, and offscreen provide platform-specific implementations of the interfaces in the mado package using platform-specific code that extends the base implementations
    • base provides base implementations of the mado interfaces that the platform-specific implementations can extend
    • events provides a comprehensive event system that allows the transmission of various types of events from the system to the consumer of the Go API using an event deque

This structure is based on the structure of goosi, the cross-platform system app interface API that I developed for Cogent Core. It already supports mobile, web, and offscreen without any glfw dependency, and it uses glfw for desktop, but that will be replaced with direct code in this new mado structure. In addition to being based on goosi, this new mado structure would also take code from Gio, Fyne, and glfw, which would allow us to get a fully functional, cross-platform, glfw-independent, IME-supporting Go system app interface API within a couple of weeks.

[kanryu]

@kkoreilly Thank you for your concrete suggestions! :)

This is after mado achieves implementation of glfw layer.
A suggestion regarding the roadmap for the mado framework itself.

Before I answer each of the many items you suggested, I will first write about my planned mado configuration. This is not to deny your idea, but to clarify the points that should be discussed by first showing the overall picture of my concept.

• mado
  • Provides interfaces and constants for basic concepts such as applications, windows, ime context, and renderers defined by the mado framework.
• mado/io(event)
  • This is a basic library that does not depend on the OS, such as events and coordinate calculations used by the mado framework as infrastructure.
• mado/glfw
  • Provides a compatibility layer for glfw. Just rewrite go-gl/glfw and the import statement to make it work. It may be possible to separate it from the mado project in the future.
• mado/window
  • Window implementation for each desktop OS is here. In principle, mado users do not need to be aware of this, but users can extend this package to react to new events when they want to intervene in the implementation of a specific OS beyond mado.
  • The distinction between events for application processes and events for windows is ambiguous, and there is little point in separating them, so it is also effective to package them as mado/os.
• mado/renderer
  • Implement various hardware renderers that mado expects here. Contains special initialization routines required by each renderer. External libraries and mado users can add new renderers without modifying mado/renderer. (cairo, skia etc.)
• Unique features like joystick, push notifications, camera, security unlock, etc.
  • Although it has not been decided whether to incorporate it into the mado framework, it would be ideal to combine it with mado as a library other than mado so that users can import it and use it.

[kanryu]

As you point out, I agree that the environment covered by GLFW is biased towards desktop OS.

Several Go toolkits can display their GUIs on Android, iOS, and interestingly on web browsers.

These environments are fundamentally different from those assumed by traditional desktop operating systems.

In fact, since gioui also supports these historically new execution environments, it would be possible for mado to consider them as operational targets as well. However, I have never run gioui or mado in these environments, and additional development is still needed to be confident that they will work.

I hope that such enhancements that expand the possibilities of mado meet your wishes.

• Remove all existing directories, including glfw
  • I don't agree with removing the glfw package at this time, but I think it's possible to move it to other projects since it's originally highly independent.
• mado
  • The specific package contents may differ from what you expected, but I think we can generally agree with you on what the mado package provides.
• base
  • As I read your proposal, this is what the current mado package contains. I've read quite a bit of her Fyne source code, and I think the API is beautiful, with many interfaces and structs accessible in the form of fyne.Canvas, fyne.Point, etc. I think the relationship between mado and base packages is debatable.
• event
  • The current io/event has already implemented almost the same content as your suggestion. What I am concerned about is that there are roughly two types of events: those that are only for receiving information, and those that require an immediate response. I think you should think of a better way to implement events that require responses.

[kkoreilly]

My GUI framework, Cogent Core, already works on Android, iOS, and Web, and I have tested it extensively there. Across Cogent Core, Fyne, and Gio, there is already plenty of code to implement everything needed for those platforms, without any dependency on GLFW required. I think that mobile and web are possibly the most important platforms to support, since the majority of all online activity happens there now.

If you look at the GLFW API (https://pkg.go.dev/github.com/go-gl/glfw/v3.3/glfw), you can see that there is a significant amount of event callback and window hint code that is fundamentally inconsistent with standard Go principles, since the API was designed for C code on desktop without IME support, which is an environment completely different from the one that mado is trying to support.

I believe that it is in the best interest of downstream Go GUI frameworks to have a single, unified solution that supports all platforms with an idiomatic Go API. However, I understand that it may be easier for frameworks to simply switch to just using the new GLFW API while also keeping their old platform-specific wrappers. Regardless, any GLFW API that we implement will be a wrapper around the unified mado API, and thus I think that it makes sense to focus on developing a high-quality mado API first, from which we can consider a GLFW API. In other words, I do not think that we want to structure the mado API around satisfying the GLFW API.

I agree that our foundational designs of the core mado API are similar, and specific issues regarding that can be resolved as we go. My proposal for the base package was not particularly clear, and it may not be necessary for how we structure things, so we do not have to worry about that for now.

For the event API, I do not understand your distinction between two types of events, so it would be helpful if you could elaborate on that further. Additionally, I do not like the io directory name, since there is already the io standard library package, and there is not much "o" (output) contained in that directory. In general, I would prefer to have one or two centralized event packages instead of individual packages for each type of event.

Another issue is that Gio has all of the platform-specific code in one directory, while I prefer to have it separated into different directories for each platform, which makes the code less jumbled in my opinion. We can discuss further what we think is the best approach for that. Regardless, I think that the broader container should be called driver or app, not window, since it contains platform-specific code that goes beyond windows, and it should not be limited to just desktop platforms.

[kkoreilly]

@kanryu, what do you think about my updated ideas above? Would you be willing to consider a PR that moves mado more in the direction of my proposal?

[kanryu]

1. First, the vast majority of GUI frameworks are based on GLFW or SDL, and I don't think building new frameworks without compatible support for these will gain users.
2. Next, I am currently facing unexpected difficulties with Linux support and need to resolve a situation where gioui behaves differently on x11 and Wayland with no satisfactory results. At the very least, I cannot appeal to users to use this library unless it shows the behavior I expect on Windows, Mac, and Linux.
3. In addition, the gioui included in mado faces many challenges in order to grow as mado. Much of this depends on how the Context object is handled. It would be desirable to have a design that makes the behavior of Window and Context (Renderer) simpler and allows Mado users, including us, to freely combine them, but the current implementation needs to be carefully considered.
4. Our goal lies ahead. I think it would be very attractive if Go could be used to freely perform direct rendering in OpenGL ES, Vulkan, Metal, etc. in fields such as Android, iOS, and web browsers, which are far from the Go language. If it can be selectively combined with powerful graphics engines such as cairo and skia, it will grow into a product comparable to Flutter. I think this part is also the goal your library will achieve.

[kkoreilly]

@kanryu, I agree that providing compatibility with GLFW will increase the chance that others use mado, so I agree that it makes sense to support it. I plan to file several PRs in the next few days to improve mado's interfaces and separate rendering from other app logic. Also, I will start working on adding in mobile and web support.

[kanryu]

@kkoreilly Thank you! :)

I appreciate your efforts and are ready to accept your contributions.
Please let me know if there is anything additional you need to work on.
I have split the implementation of each renderer by OS (mswindows/darwin/unix). These may be grouped together in the OS directory.

[kanryu]

I made a spreadsheet for supporting hardware renderers

• OS and Renderers for Mado
  • https://docs.google.com/spreadsheets/d/1_vjzgv1Y6C_HzBj-ezJ7bXJ2_oJ0Dry-lbHNB4-CvR0/edit#gid=0

[kkoreilly]

@kanryu, I looked at the mado API and the Cogent Core system app interface API and realized that there are a lot of major differences. Also, I realized that the Cogent Core system app interface API is heavily interdependent on the rest of Cogent Core. For those reasons, creating a mado API that works on all platforms and preserves compatibility with Cogent Core will take a long time. My highest priority right now is to release v1 of Cogent Core in early June, and I do not think that I have enough time before then to dedicate to working on mado.

However, I am still very interested in contributing to mado, and I think that it is the best way to structure things in the long-term. Therefore, once I release v1 of Cogent Core in early June, I should be able to work with you on improving mado and using it in Cogent Core.

[kanryu]

It makes sense that you would prioritize development of Cogent Core. I don't think I'll complain about that decision.
I looked at go.mod in your project (cogentcore/core), and it looks like you are importing the framebuffer library directly without using glfw.
In fact, I think GUI frameworks usually use a specific version to initialize GL. mado (gioui) also has such a setting, and each framebuffer is operated with fixed parameters and modes. If the settings don't match your library, it probably won't work.
When combining your library with mado, a good solution is to disable mado (gioui)'s rendering engine. In other words, the design uses mado only as a Window client and gl initialization library, and all drawing is implemented on the Cogent Core side.
If you want this behavior, create mado.Window with app.CustomRenderer(true). In this case, mado(gioui) will not draw anything. Instead, you can draw freely.

I look forward to the day when we can develop together with you.

[kkoreilly]

Yes, Cogent Core manages the framebuffers and draws images directly to them using Vulkan. It mainly relies on glfw for events and getting the window surface pointer. I also look forward to working with you on mado once I release v1 of Cogent Core.