NSWindows95InterfaceStyle menubars not working in Debian 13

[dboyd-invitoep]

Environment

• Debian 13 (testing/trixie), GNOME 46, Wayland session (XDG_SESSION_TYPE=wayland)
• GNUstep base 1.31.1, gui 0.32.0, back 0.32.0 (built from upstream git)
• NSMenuInterfaceStyle = NSWindows95InterfaceStyle set in NSGlobalDomain
• Observed with Gorm, Ink, and other apps that embed the Win95-style menu inside the main window

Symptoms
Launching any app shows no menu bar; only a thin sliver appears beneath GNOME’s rounded title bar. Console shows no errors once the Cairo null-context fix
is applied, but the menu never becomes visible.

Cause
In GNOME 46 on Wayland, Mutter now draws a 36 px client-side header bar on top of every window. GNUstep’s Win95-style menu is part of the window frame,
so it’s still positioned immediately beneath the traditional 22 px title area. The compositor’s header overlaps that region, hiding the entire menu. This
regressed between Debian 12 (GNOME 43) and Debian 13 (GNOME 46) even though GNUstep’s code didn’t change.

Proposed Fix
Teach GSWindowDecorationView to detect this configuration and reserve extra space above the menu view:

• Detect NSWindows95InterfaceStyle, XDG_SESSION_TYPE=wayland, and XDG_CURRENT_DESKTOP/DESKTOP_SESSION containing “gnome”.
• When those conditions hold, add a 36 px “header bar” compensation to every place we account for menu height:
  • contentRectForFrameRect: and frameRectForContentRect:
  • addMenuView, removeMenuView, layout, and toolbar placement
• The menu view itself stays 22 px tall; we just lower it by 36 px so it sits below Mutter’s header.
• This keeps behaviour unchanged on Xorg or other desktops.

I’ve attached a patch that adds _GSHeaderBarCompensation() in GSWindowDecorationView.m and applies the offset (requires <strings.h> for strcasecmp).

gnome-wayland-menu-compensation.patch

[dboyd-invitoep]

whoops -- just noticed this is libs-gui, not libs-back

[fredkiefer]

There are two subclasses of the GSWindowDecorationView class where you added your patch. GSBackendWindowDecorationView lives in the same file and handles the case where the window decoration is drawn by the native backend implementation. And GSStandardWindowDecorationView.m where GNUstep guy itself draws this. On Wayland we should be in the second case, so your patch would rather belong into that class. Also this could mean that your change is not Wayland specific, but rather always required when we draw the window decoration ourselves and use the NSWindows95InterfaceStyle. Most likely nobody ever used this combination before. Could you please check that?
In that case we should rather check for this style in the -offsets::::: method and handle the case there correctly. That should give a smaller and more clear patch. Please give this a try.

[danjboyd]

I was running wayland with NSWindows95InterfaceStyle in Debian 12 and everything was working there. It was something in the Debian 12 -> Debian 13 upgrade that messed it up.

Happy to help with testing, but I'm not clear exactly what you're asking me to test.

What is everyone else running? NeXT-style menus?

[fredkiefer]

Thank you, I am now hopefully understanding better what you are facing. If I understand it correctly in Debian 12 the window decoration was drawn by GNUstep, via GSStandardWindowDecorationView and in Debian 13 this now gets drawn by Mutter.
This means that you should switch to using GSBackendWindowDecorationView. This switch gets done by returning YES from handlesWindowDecorations in WaylandServer (of course only when Mutter is detected). Next you have to adjust the values returned by the -styleoffsets:::: method also in this class to handle this 36 pixels. Again with some checks for Mutter. The rest should then fall into place by itself. If not we need to analyze deeper what is still missing.
The good thing is, this now seems to be a back issue, so your bug report was placed in the correct library :-)

[tedge]

I can reproduce the hidden menubars on Ubuntu 25.10. It's caused by gnustep-back getting incorrect window style-offset values from GNOME's window manager, Mutter. (Most WMs have at least minor style-offset issues on GNUstep, but recent releases of GNOME are particularly egregious).

PikoPixel has had workarounds for style-offset issues for awhile, but I'm now rewriting them (again) because the current workarounds stopped working on recent GNOME versions.

Please learn from my mistakes and avoid using 'magic' numbers like I did - they can be a quick fix, but they just mask the underlying issue, which can reappear after only minor changes in the environment.

For instance, the magic value in this case (36) represents the height of UI window title-bars on a specific version of a window theme, displayed at a specific scale (100%).

There's various ways that title-bar height could change, just on that same machine/distro/GNOME-version:

• User installs a different window theme
• Future version of the window theme tweaks its geometry
• User changes the display's scale

Ubuntu 25.10's default GNOME window theme has a different title-bar height (37), and other distros' default themes probably differ as well.

Rather than continuing to find offsets for gnustep-back's incorrect style-offset values (which is how PikoPixel's style-offsets work now, & can easily break), I've been trying to figure out why gnustep-back gets bad values in the first place, & seeing if it can be redirected into reading the correct values.

There's more to the workarounds that I won't include here - this'll just be about fixing in-window menus on GNOME.

A bit of background about what style-offset values are:

Cocoa windows & X11 windows are quite different:

• X11 & Cocoa's coordinate systems are flipped (X11's origin is top-left, Cocoa's is bottom-left).
• In Cocoa, a window's frame rect represents the entire area covered by a window (content, title-bar, and side & bottom edge decorations); In X11, a window's frame rect only represents the content area (title-bars & edge decorations are not part of the window - they're separate window(s) owned & managed by the window manager).

In order for GNUstep to convert points representing window locations between X11 coordinates & Cocoa coordinates, it needs to know not just the X11 window's content area, but also how far the X11 window's title-bar & decorations extend beyond it.

These are what GNUstep gui & back call "style-offset values": Four values (top, bottom, left, right) representing the distance that decorations extend off the edges of the X11 content/window frame, so that when added to the X11 window's frame geometry, it will represent the same area as represented by a Cocoa window frame (which already includes the title-bar/decorations).

X11 provides 2 ways to read a window's style-offsets (known in X11 as frame-extents):

• Querying a window's property: _NET_FRAME_EXTENTS
• Sending a window message: _REQUEST_NET_FRAME_EXTENTS

The window's NET_FRAME_EXTENTS property is more accurate, but it's only valid after the window has been mapped to the screen (and the window manager has relocated it on the screen (if needed) and created the separate title-bar/decoration window(s)). If an X11 window is unmapped, you can get its extents by sending it a _REQUEST_NET_FRAME_EXTENTS message, though this can be less accurate - the Window Manager is just making an educated guess, since the decoration window(s) haven't actually been created yet.

GNUstep-back also sets up a table of generic style-offset values, indexed by the 16 permutations of the following NSWindowStyleMask bit values: Titled, Closable, Resizable, Miniaturizable. The generics table is used to get the style-offsets for a window style when you don't have a specific window instance. In fact, the gnustep-back API that's called by gnustep-gui to get style-offset values doesn't include a window-reference parameter, so when gnustep-gui requests style-offsets values from gnustep-back, the values can only come from this generics table, since there's no window specified.

GNUstep-back populates the generics table by creating a set of temporary windows - one for each permutation of the NSWindowStyleMask values - and first trying a _REQUEST_NET_FRAME_EXTENTS message. If the message doesn't succeed, then it maps the windows to an offscreen location and queries the window's _NET_FRAME_EXTENTS property.

The generics table is saved to disk & reloaded when an app starts. This means those values can persist after installing a new version of gnustep; Upgrading from a version that read bad style-offset values to one that can read them correctly also requires forcing the generics table to be recalculated (this is done by setting a default, GSIgnoreRootOffsets).

Now, the 2 problems causing hidden in-window menus on GNOME:

1. Recent versions of GNOME always return garbage values when responding to the _REQUEST_NET_FRAME_EXTENTS message; GNUstep-back receives a valid reply event, but the frame-extent values are all zero. This causes the generics table to be populated with incorrect (zeroed) style-offset values. (This is why 36 works as the magic value - it offsets the -36 error caused by setting the top style-offset value to 0 instead of the correct title-bar height).

2. On GNOME & other desktops with Mutter-based window-managers (Cinnamon, Budgie, Pantheon), a window's _NET_FRAME_EXTENTS property will sometimes be set incorrectly if the window was mapped to offscreen coordinates. (I see incorrect values roughly 5%-10% of the time).

To get in-window menus working:

1. Install the attached patch for gnustep-back (&recompile,reinstall) - it's a one-line fix that allows you to disable sending _REQUEST_NET_FRAME_EXTENTS messages (via default-value), which will cause gnustep-back to fall through to querying the _NET_FRAME_EXTENTS property instead.

2. Set up the following default values:

defaults write NSGlobalDomain GSIgnoreRootOffsets TRUE

defaults write NSGlobalDomain GSBackChecksOffsetsWithoutNetRequests TRUE

defaults write NSGlobalDomain GSBackChecksOffsetsOnScreen TRUE

(GSIgnoreRootOffsets forces gnustep-back to recalculate the generics table - it can be turned off once you have it working with valid values; GSBackChecksOffsetsWithoutNetRequests disables _REQUEST_NET_FRAME_EXTENTS messages (requires the attached patch to be installed); GSBackChecksOffsetsOnScreen forces gnustep-back to use on-screen windows instead of off-screen windows when populating the generics table - this default feature is already built-in).

libs-back_OptionToDisableNetRequestFrameExtents.patch