Spatial Format

[revisionfx]

Open Effects Proposal for Standard Change

Please read the contribution guidelines first.

Standard Change Workflow

• [X ] Create proposal as issue (you're doing this now!)
• Tag this issue with standard change tag
• Identify subcommittee: at least one plug-in vendor, and at least one host
• Discuss the idea in this issue
• Write new or updated code and doc
• Publish updates as a pull request (ideally on a feature/PROPOSAL-NAME branch)
  • Make sure that PR references this issue number to keep them in sync
  • Discuss and review code in the PR
  • Meet all requirements below for accepting PR
• When subcommittee signs off and other members don't have any further review comments,
  maintainer merges PR to master which closes PR and issue

Requirements for accepting a standard change:

• Header files updated
• Documentation updated
• Release notes added
• Compatibility review completed
• Working code demonstrated with at least one host and one plugin
• At least two members sign off
• No further changes requested from membership

Summary

We need a spatial format model separate from project size. Already a PR #61 done before issue filed.

Motivation

We have an issue of spatial representation across different hosts that this would address.

Problem

Right now we have no way to set output size of a generator other than default project size and the render region with project size is not complete or that useful for many hosts.

Initial Spatial format from http://openeffects.org/standard_changes/spatial-formats-are-broken-replace-with-new-properties-and-a-new-action without discussions

Regions of definition are well defined concepts within OFX, the ‘format’ of a clip is not. For example a clip may be HD, 4K, SD etc… while the RoD need not be congruent to the ‘format’ at all. Furthermore, effects have no way of changing the spatial format. For example an effect that up scales images from SD to 4K would want to flag the change in format as well as the RoD, otherwise it will be interpreted as a zoom.

The 'format' nomenclature is taken from Nuke.

(Pierre: Adding this thread from Nuke dev forum as reference

http://community.thefoundry.co.uk/discussion/topic.aspx?f=191&t=104638&show=openfx%2cgenerator

)

However as is, it's incomplete so I relabel the PR WIP - we have 3 basic host "contexts" maybe to address at once. Some hosts might not have the same power in different part of their app with regards to that. Some "editing" app might have some strong conforming per timeline/comp/sequence and be limited perhaps to provide a distinction between timeline size and clip size. Some host might not be able to grow the image definition size post say a blur but should maintain spatial reference otherwise say a point positioner could relatively move... Some host might be a layer stack and have a distinction between layer size and comp size (the final crop). Anyhow the render region optimization is something that can change every frame but for cross-compatibility we need a stable spatial format. Even some exotic compositing system might have no concept of project space, spatial format is sort of separately carried across a graph (with some rule when 2 different ones are merged) and what we have as project size should really just be a default.

Impact

Initial Solution to refine

Currently we use the clip properties kOfxImageEffectPropProjectExtent, kOfxImageEffectPropProjectSize and kOfxImageEffectPropProjectOffset to represent the format. This should be simplified to...

kOfxImageEffectImageFormatResolution a 2D integer indicating the resolution in pixels of the output image, this is always full res with no render scale applied,
kOfxImageEffectImageFormatAspectRatio as per normal.

These are present on all input clips.

We have a new action to compute the format of the output clip, kOfxImageEffectActionGetImageFormat, which would ask the effect to set the appropriate properties on the outArgs.

Documentation Impact

What changes to the docs are needed for this change? TBA

Stakeholders

All

Discussion

[peterl-bmd]

Note that Fusion (and I suspect most node based systems) will not always have the same image format on a Clip for the duration, as opposed to a nominal "project resolution".

I'm not clear what the clip properties are meant to reflect here vs kOfxImagePropBounds.

[revisionfx]

I reformatting all this so it works for different kinds of apps, likely would need a few additional description strings. Image Bounds are read-only, there are issues across different types of apps. It's quite possible in Fusion a generator can create resolution as parameter and be able to set Region of Definition to a specific constant size (e.g. color bars 1080 versus 2160 without reference to Project Resolution) and it's possible Fusion (never tried it) supports generators with mandatory input which then would be another way to deal with this in Fusion. Not something that can be assumed across all hosts. In the pure compositing scenario for example in Interact (to draw Overlays) the Regions are not defined yet (it's a render concept). For any "UV to ST" mapping (any projection, e.g to equirectangular ) a static spatial size reference is necessary. For NLE type application or layer-stack compositing (with a transform per layer), assuming Project Resolution is Timeline resolution (timeline might be sequence or comp in another system, same thing), you need a second reference that is the layer dimension. For NLE this could be for images larger or smaller than Project resolution a way to get the Input Clip format versus the padded with black image bounds. Note in a layer stack (chain of effects on a layer), local world for effect starts at root layer, and region of definition can grow (yes frame by frame different) but then you typically loose reference to where a point parameter for example actually is... Also some hosts only natively support a render window within what I call Spatial Format here (cannot be larger) - where the layer/video track is essentially center-aligned to Project. Also although such system can support accessing another layer from a plugin on a layer, it typically has issues with the transform (e.g. a scale of 100X) it must clip to something. Another issue is say you have an animated horizontal blur, the application should tell you if pixels are used outside of final render region or not - so you know how to handle edges so if not you don't grow transparency at the edges.

[revisionfx]

Wait! I just read Resolve latest chapter 9 about Spatial Formats (13 pages) - this is documentary evidence - cheers :)

[revisionfx]

If you put a small clip in a larger timeline in Resolve for example (assuming you set properly the mismatch resolution menus), you get a small image in a sea of black pixels. As far as I know in such Filter Context (or Generator context) there is no GetRoi, GetRod. Wouldn't it be useful to get 1) the source clip window 2) also maybe the output render window when this is pull from render