Enhanced Layering facilities

[RadianM]

Combining multiple effects into one pattern is relatively easily done by overlaying certain effects in overlapping Segments but it's a bit hit and miss. Only a handfull of effects actually cater for this. I think the potential for combining effects is underestimated.

Embracing "Photoshop like layering" would literally add many new dimensions to the ability of WLED to allow the creation of interesting composite patterns. For example, I hacked some existing patterns with code such as...

aRGB=effectPixel(i);   //pixel created by effect
if(overlay){
 aRGB.nscale8_video(SEGMENT.intensity);  //"Opacity"
 bRGB=SEGMENT.getPixelColor(i);          //underlying pixel
 bRGB|=aRGB;                             //"or" operator brings each channel up to the higher of the two values
 SEGMENT.setPixelColor(i, bRGB);
}

..to make them mix the RGB from the effect with the underlying RGB from lower numbered segments.

Using code tweaks such as the above I created a "Bonfire night party" with a "smoky sky" background in Segment 0 (Courtesy of Aurora) overlayed with licking bonfire flames (Coutesy of Fire 2012) and the fireworks effect. All pretty simple to arrange - but the results are so much more engaging than the effects in isolation,

Unfortunately the Segment controls don't make life quite so easy (although recent fixes to issues #3403 #3405 have helped). The Segment brightness slider is the main issue: as it's applied to the pixels written by the effect, it also affects the brightness of all layers beneath. For layering applications, as a crude solution, it could be exposed to the effect code which would apply it before merging with underlying pixel data. In this way it could operate more like "Layer Opacity". But ideally, it would be applied in interactions with the bus_manager.

One other unfortunate consequence of the layering as-is is that Mirror effect also mirrors all underlying segments as the mirror is applied to the pixels as they arrive via the setPixelColor (the same problem as with brightness I mentioned above) so it does seem to make it vital for layering to be handled deeper down in the architecture (as opposed to being up to each individual effect)

Then, along with a new drop-down menu for different "Layer mode" options (simulated in image below), it could be used to apply the overlay using some of the popular image blending techniques and make layering a possibility for all effects. How much of this could practically be rolled into the existing code I'm not sure. I'm still finding my way around it.

Proposal for enhanced Segment controls:

[blazoncek]

Easier said than done.
Would require re-writing majority of effects, which is impractical and for some of them impossible.

[RadianM]

Sure, I agree that hacking effects (as I've done) is impractical - and doesn't get around the Segment brightness and mirror issues.

That's why I queried the possibility of putting the layering enhancements "on the other side" of the calls to setPixelColor i.e. the effects just write out pixel data as usual and the layering is handled further down - thus making it available to all effects.

[RadianM]

With this tiny change to FX_fcn.cpp to re-purpose the segment opacity sliders:

////  uint8_t _bri_t = currentBri(on ? opacity : 0);  //Opacity slider mod...
  uint8_t _bri_t = currentBri(on ? 255 : 0);          //...force segment opacity sliders to 255 and use value in effect code instead.

Then in the effect code, at the time of writing out pixel data:

aRGB=effectPixel(i);   //pixel created by effect
if(overlay){
 aRGB.nscale8_video(SEGMENT.opacity);  //scale RGB by segement opacity slider
 bRGB=SEGMENT.getPixelColor(i);          //underlying pixel
 bRGB|=aRGB;                             //"or" operator brings each channel up to the higher of the two values
 SEGMENT.setPixelColor(i, bRGB);
}

This achieves my goal of getting individual layer opacity from each segment. The control is even labelled opacity in the code already.

(The above example for the effect code is exactly the same as I posted earlier except SEGMENT.opacity is used instead of SEGMENT.intensity - which is reserved for its usual role of modulating some other aspect of the effect.)

I even appreciate the transition that kicks-in on the slider change as it draws attention to the layers presence in the stack!

Of course, for unmodified effect code, effects will always be rendered at full brightness. No big deal for me as I'm only interested in a using few as candidates for layering.

[blazoncek]

This achieves my goal of getting individual layer opacity from each segment. The control is even labelled opacity in the code already.

Thanks, but WLED is used by thousands of people. Considering only "my goal" is a wrong approach as "your goal" may not align with everyone else's "goals".

(The above example for the effect code is exactly the same as I posted earlier except SEGMENT.opacity is used instead of SEGMENT.intensity - which is reserved for its usual role of modulating some other aspect of the effect.)

The above approach would require re-writing of every effect and some effects do not like to be rewritten. 😄 If you want to make such fundamental change it has to be on a different layer.

Of course, for unmodified effect code, effects will always be rendered at full brightness. No big deal for me as I'm only interested in a using few as candidates for layering.

Again "not a big deal for me" is a wrong approach. Consider rather thinking "how can I achieve my goal without disturbing everyone else".

I've managed to create effect blending without any change in the looks of effects and without too much code changes. I consider the same for segment "layering" or blending. If it is to be done it must not require any change in effect logic and has to be done in appropriate layer.

Currently I can see the possibility only by adding triple buffering to the code, with the third buffer used in Segment class (it used to have it by calling SEGMENT.setUpLeds() in the past but we removed it in favour of global buffering).

I'll tag @softhack007 and @ewoudwijma to join the discussion as they have plenty of experience by writing effect code and know how buffering and underlying pixel rendering works.

[RadianM]

I appreciate the fact that you've opened this up for discussion. I think the emphasis I placed on this "working for me" could have been explained better. What I intended to convey is that if a full embrace of layering is too big a step for the general release of WLED then I'm content with compiling my own private version given how one small change makes it so much more workable for my own application. I feel additional buffering should be avoided at all costs.

But I'm still blown away by the artistic possibilities pattern layering brings. Behind an acrylic diffuser, 1D and 2D scenes can be constructed in Bob Ross style. It's a massive step up from staring at a single twinkly effect. However I also acknowledge that the latter is far more appropriate for festive light strings.

[blazoncek]

But I'm still blown away by the artistic possibilities pattern layering brings.

As a photographer I know the possibilities of layers. 😄 Please prepare POC with the above considerations I mentioned and we'll be able to talk in more detail.

[softhack007]

Hi, i appreciate the idea, too.

Just a few additional points to consider for a generic implementation:

• optimization - a few extra rounds of getPixelColor/setPixelColor per effect are expensive in terms of CPU time. Its means that FPS will drop (maybe by 50%), especially on larger fixtures.
• I expect it will be too much computation/memory load for 8266.
• Best to make the blending feature an option that users can switch on/off (similar to global buffer option)
• currently there is no explicit concept for Segments that are "above" and "below". The result of blending several layers will depend on order, so its important that users get a possibility to specify which segment is "in from of" another segment.
• Probably @blazoncek is right, we might need some tripple buffering, as many effects render based on previous colors (for example: scrolling left/light/up/down, fading out (->trails), or "growing" a pattern). If the global buffer would return merged (blended) pixels that will be blended again, some effects will look very unexpected.

[blazoncek]

Currently I only see the option to make setPixelColor/getPixelColor paint segment's local buffer and then "blend" segment buffers in show call.

[softhack007]

Currently I only see the option to make setPixelColor/getPixelColor paint segment's local buffer and then "blend" segment buffers in show call.

agreed. and we need an idea how to control the order of painting segments that overlap.

[blazoncek]

The "rendering" order is the same as seen in UI (from top-down). There is currently no way to reorder segments as they are stored in vector. Rearranging vector items should be avoided for performance reasons.

[RadianM]

The "rendering" order is the same as seen in UI (from top-down). There is currently no way to reorder segments as they are stored in vector. Rearranging vector items should be avoided for performance reasons.

Can't just the properties of two segments be swapped with each other? Rendering order can then remain top-down if the UI lets the user 'move' a segment within the list.