Open Source Program Revamp

[Geenz]

Hey everyone! So following up with our changes to develop, I thought I would share a proposal for an improved open source program that we’re currently working on and are seeking feedback around. To be clear, we don’t expect this to go off without a hitch, and we are expecting to have to make changes as we go here. We’re providing this now so we can start filling in the gaps from the community’s feedback. We are allowing the community to comment and provide feedback that we can incorporate into the final plan prior to its implementation.

Current Challenges

• We only have so many Lindens, and we haven’t yet perfected cloning
  • Those Geenz clones were getting kinda weird… A few of them were missing feathers!
• We have some pretty big piles of work that needs doing “eventually”
  • Various long standing bugs
  • Library maintenance
  • Performance tweaks
• The bar is not well defined for merge readiness to develop
  • This has resulted in many seemingly random bugs finding their way into shipping code
    • Good one: create a torus, hollow it out, slice it up, then observe the lighting on the interior hollow bit
  • Various unfinished bits making their way into develop
    • We were bad about this especially - see: adding feature flags everywhere for unfinished features
• Documentation tends to get scattered
  • New contributors must go through the ritual of finding the right documentation in the right place to get started
  • Some internal docs that aren’t available on how to setup a build environment
• Tooling to get started is next to non-existent
  • Getting setup is currently a manual process that can be error prone for new contributors

Proposed Solution

• We’ll be using the “Help Wanted” tag in both Canny and GitHub more, and we invite others to use it as well: https://github.com/secondlife/viewer/labels/help-wanted
  • These are things we are actively seeking help on! For example: Linux
• Consider starting a bounty program
  • Give us ideas for rewards!
• Provide well defined contribution guidelines
  • Smoke tests available for contributors to better do basic regression testing against their code with in order to get their contributions merged in
  • Unit test documentation
  • Contributor provided test plans moving forward
  • Emphasize shippable, but iterative development into develop
  • Encourage side branches and forks that track develop until they have something that meets whatever ends up as our merge criteria
• Better communicate with contributors
  • Discussion with maintainers before beginning work
  • Discussion in a real-time medium
    • GitHub is fine for asynchronous discussions - but sometimes you just need to talk in realtime! Public Discord coming soon!
• Introduce more automation
  • Visual result automation
    • Define a % change that is acceptable for some changes, e.g., shader changes
  • Detect what modules were changed to determine more targeted testing
    • We already do this to a degree - but it’s not currently wired up to any test plan generation
  • Automatic performance smoke test
    • An idea that’s been floating around the lab for a bit. We’re getting there!
• Move more of the QA process into public
  • More information coming soon!
• Migrate build and development environment documentation to GitHub wiki
  • Some work has been done on this - macOS is mostly complete, Windows is lacking.
  • Linux too!
• Ease development environment setup
  • Some work has been done here - mostly environment setup scripts that can largely get a terminal session setup with the environment variables you need

We know that this will be quite the change for many of you - and we’re sorry for the inconvenience that this creates. However, as we work towards making contributing to Second Life a more structured experience and easier to get started, our hope is that overall quality of the viewer will continue to improve for everyone on the platform. Residents, content creators, and TPVs all included.

On the subject of TPVs - we will be opening up more communication channels for TPVs that will enable closer collaboration with LL moving forward. Additional information will be coming soon.

[Ansariel]

While these are plans focused on the future, I would like to take the chance and talk about the current state of the archived develop branch: I warned right in the beginning that throwing everything into a giant bin is causing giant problems - my concerns were dismissed by pointing at the feature flags - good luck adding feature flags to core components of the viewer btw - and that in the case bugs would be found everyone would jump at them to fix them immediately. The result of this disastrous development process can now be seen with the develop branch worth 1600+ changes being archived with the plan to dissect it and cut worthy pieces out of it.

However, in my opinion that plan is deemed to fail. For instance, the game controller and viewer-side LUAU scripting support received a lot of fixes after they went into develop which likely can't be simply taken from there - unless you plan to dig out an old revision before those branches were merged into develop. Even worse for former main-b and maint-c branches: Part of their merge was applying the whitespace and BOOL/bool conversion changes, resulting in catastrophic merges. Digging out and taking revisions before that merge to continue working on? Well, as far as I can tell the last revisions before they got merged date from June and August 2024 - good luck with that!

In my opinion, you have to bite the bullet at this point and sort out the archived develop branch before moving forward instead of picking pieces here and there and relying on 6+ months old branches that received follow-up fixes you will never going to find in the pile of 1600+ commits.

[Geenz]

Since I touched on it as a challenge up above - I'll talk about it here, what our current plan is, and where we're at.

So we're already biting the bullet so to speak. As I speak, we're in the process of getting maint-b and other things merged into 2025.03, which in turn trickles down to develop once we've decided that it's more or less ready to go as in has no major crashes, bugs, or performance regressions, and if we wanted to ship it we could. Maint-b is one of the less problematic changesets to try and ship, and will hopefully set the stage for bringing in additional work (which we've already identified). Chunks of that additional work needed more time to bake, and should have not been approved for develop in the states they were in.

We have no choice at this point but to take a piecemeal approach to this. The alternative is we completely halt our releases for many months in a stabilization effort that ultimately deprives everyone of work going into bug fixes, performance improvements, crash fixes, etc. With where archive/develop is right now - that doesn't seem worth an outright shipping stop. We have a lot of things we need to improve today. Not push to the side for potentially several months as we feverishly work to fix every small bug we can that was introduced into develop with the limited time we actually have available. We don't have a large team of people to do this work.

However, we are making headway on getting those changes into shipping releases. We will need some help from original contributors along the way here and there if we want to accelerate the process of getting develop into a good state. Otherwise this is going to be an arduous effort that we can only take in smaller bits over the course of many months.

Let's also talk about feature flags, and how they were previously being used as that has a massive impact on this kind of work:
You used to use a feature flag for:

• A feature that is shippable but the server may not have it enabled for whatever reason, for example, WebRTC voice

We started to use feature flags for:

• A feature that is shippable but the server may not have it enabled for whatever reason, for example, WebRTC voice
• A feature that's still in development and may have many deficiencies but you think you can disable it at runtime safely for a shippable release so you can get it into develop and so hopefully a TPV can merge it to increase testing surface area, for example, PMFP

We are going to start only using feature flags for:

• A feature that is shippable but the server may not have it enabled for whatever reason, for example, WebRTC voice

Note the subtle but important difference here. I think you can appreciate it.

[Nex-Pro]

We should rename the project "Geenz" then everyone can git clone the project and we'll have all the Geenz clones we'll ever need. :)

[vldevel]

I'm glad changes are being made. :-)

  * New contributors must go through the ritual of finding the right documentation in the right place to get started
  * Some internal docs that aren’t available on how to setup a build environment

* Tooling to get started is next to non-existent
  * Getting setup is currently a manual process that can be error prone for new contributors

I spent a full Sunday trying to figure out why LL's viewer refused to build manually (from a Windows VM). I followed the provided build instructions (1), just to find out they were totally wrong and won't allow to build the viewer. I finally had to delve into LL's autobuild sources to find out what was missing (2) and what was wrong with the pre-built package (libsqlite) I needed to add for my PR (3).

* Ease development environment setup
  * Some work has been done here - mostly environment setup scripts that can largely get a terminal session setup with the environment variables you need

To build the Cool VL Viewer, all you need to do is open a terminal in the source tree and launch a single script (or batch file for Windows). Refreshing, isn't it ?

  * Smoke tests available for contributors to better do basic regression testing against their code with in order to get their contributions merged in

Not sure what you mean by this, but I have asked several times (e.g. during Open Development User Group meetings) to be able, as a contributor, to trigger a viewer build (and to be able to recover the build for testing) in the git on a forked branch with the PR(s) I want to test and validate before submitting it(them). That would be much easier than having to build the viewer locally.

* Consider starting a bounty program
  * Give us ideas for rewards!

I have not been programing my own fork of a SL viewer, on my free time, for the past 18 years, to make money or gain anything. I did it to contribute to the SL community in the best way I could, and yes, to try and make SL better for myself as well.

My only "reward" is to see SL getting better, the viewer getting faster and stabler, and to see my benevolent work respected.

Rejecting a perfectly valid PR (4) that solves a current, abyssal, security risk with LL's official viewer (namely, an outdated CEF version, riddled with security holes and known exploits), and doing it stealthily after having let the PR rotting for 2 months, without a single explanation, is not a proper show of respect for my work.

If you guys want Open Source developers to contribute cool stuff, you should be more considerate for their work and advice. And the advice part brings us to:

 * Better communicate with contributors
   * Discussion with maintainers before beginning work

.../...
* GitHub is fine for asynchronous discussions
.../...
On the subject of TPVs - we will be opening up more communication channels for TPVs that will enable closer collaboration with LL moving forward.

Well, I lost the count of the times (but (5) is certainly the most prominent) I have warned or advised, or commented in git commits, about an issue, only to see my warning or advice ignored, and my predictions sadly proving to be exact in the end.

So I am glad that some change is coming about this...

  * Discussion in a real-time medium
    * GitHub is fine for asynchronous discussions - but sometimes you just need to talk in realtime! Public Discord coming soon!

I'm not using Discord, due to their unacceptable "privacy" policy. So you'll not see me there...
And "real time" is a bit of an utopia, given the time zone discrepancies.

  * Automatic performance smoke test
    * An idea that’s been floating around the lab for a bit.  We’re getting there!

Even if not automatic, I have advocated in the past for getting a performance test ground in Aditi with a sim containing a lot of render-heavy stuff, and with bot avatars that would always be the same and always be logged in, so that we can test code changes in a perfectly reproducible environment.

At some point (for the HTTP Pipelining introduction in the texture fetcher), there has been one such test ground, but restricted only to texture fetching tests, and this was way cool to fine-tune the texture fetcher code.

* Move more of the QA process into public
  * More information coming soon!

It would be nice to be able to raise a "showstopper" flag when we see a change that breaks some aspect of SL, and before that change makes its way to release viewers (or servers). That flag could then trigger an action of the QA team for verification/investigation.

---

(1) https://wiki.secondlife.com/wiki/Build_the_Viewer_on_Windows
(2) LL_BUILD env variable was not set by autobuild...
(3) Did you know that:
tar cjf path/pre_built_package.tar.bz2 .
does not work with LL's autobuild. You need to package with:
tar cjf path/pre_built_package.tar.bz2 *
which we normally don't use under Linux, because the '*' won't expand to .hidden_files_and_folders
This one took me a lot of time to find out... And that's just because the tared files are rooted to ./ (current directory) in the first case, and autobuild does not comprehend it...
(4) #3149
(5) I told LL about the upcoming PBR fiasco:
https://bitbucket.org/lindenlab/viewer/commits/aaf7b17db047f0cb2630b479d5468062e6ca815e

[Geenz]

Thanks for the productive feedback!

Just addressing a few points:

• Smoke tests are more or less simplified test plans to determine if something meets some sort of minimum "bar".
  • Examples: does it run? Does it render? Does it get the packets it needs?
• Regarding PR acceptance - I'm going to point at the the "The bar is not well defined for merge readiness to develop" bullet point above.
  • I know this is frustrating to consider - however we didn't do a great job at making this public in terms of what should and should not go into the codebase. Regarding your CEF changes, I'd like to understand more in detail before I really make any kind of judgement there. I'll review whether we need a policy change, need to define a new vision for that specific part of the codebase, or just generally determine what we need to do here. Thank you for raising this to my attention.
• Agreed on the showstopper flag. This is a big reason why we're looking to move more of the QA process into the public eye. We often will get showstoppers from places we never expected, and at the worst possible times.

[cosmic-linden]

At some point (for the HTTP Pipelining introduction in the texture fetcher), there has been one such test ground, but restricted only to texture fetching tests, and this was way cool to fine-tune the texture fetcher code.

Not sure what you are referring to, but there is a public region on Agni still around called LagLand along those lines.

We do have a few other regions used by QA for testing and some of them are public. Maybe that's something we could make more widely known, assuming there isn't already a wiki page for that.

[Dzonatas]

I wonder if there has been a HTTP texture fetcher code that handles "spatial" images. That occurs like when one guid/uuid is shared with different extensions (jpg, j2k, etc). What I'm aiming at is it being smart enough to send the right image for each component, which may be formatted for jpg, j2k, etc. Without the extension specified (on the filename), The response could include so many components from jpg and so many components from j2k as an optimization (instead of "all" components).

Does the revamp let me express such replies as above? I'm mixing open and non-open code together. I'm fed up with it. I digress.

[Dzonatas]

Hello,

My main interest here is WSL (Windows Subsystem for Linux). I want to do something small (and automated) to the game server. Instead of installing Linux and running the entire viewer on Linux, I'm just pointing out basic lambda code running on a Windows machine that automates that interaction. This, ideally, would not support Linux outside of a game server on that subsystem. It opens doors.

Once installed, WSL could hang onto assets in a more secure manner and use it as a cache... a secure cache.

The reason for above is to run either kakadu or openjpeg in a way that doesn't bomb the data flow. There are related features not being used by jpeg or j2k formats. I've been looking the ability to store scripts in those image formats.

I have more ideas for WSL (and "permastructure"). I think 32GB is now recommended for these newer systems... for the cache. I just wanted to nail this one down.