Proposal: Viewer experience v2 update

[slimbuck]

NOTE: this proposal is a work in progress and will be updated based on feedback.

Intro

This document proposes an update to the viewer experience to introduce the following features:

• annotations
• constrained cameras
• high precision rendering
• post effects
• tonemapping
• include fov in camera animations
• skybox (optional)
• audio (optional)

Out-of-scope features

This update will not address the following features, which will be added in a subsequent round of updates:

• 1st person walk mode
• mesh/3d collisions

Schema

type AnimTrack = {
    name: string,
    duration: number,
    frameRate: number,
    loopMode: 'none' | 'repeat' | 'pingpong',
    interpolation: 'step' | 'spline',
    smoothness: number,
    keyframes: {
        times: number[],
        values: {
            position: number[],
            target: number[],
            fov: number[],
        }
    }
};

type Constraint = {
    min: number,
    max: number
};

type PositionConstraints = {
    x: Constraint,
    y: Constraint,
    z: Constraint
};

type OrbitConstraints = {
    distance: Constraint,
    azimuth: Constraint,
    elevation: Constraint
};

type CameraConstraints = {
    position?: PositionConstraints,
    orbit?: OrbitConstraints
};

type CameraPose = {
    position: [number, number, number],
    target: [number, number, number],
    fov: number
};

type Camera = {
    initial: CameraPose,
    constraints: CameraConstraints
};

type Annotation = {
    position: [number, number, number],
    title: string,
    text: string,
    extras: any,
    camera: Camera;
};

type PostEffectSettings = {
    sharpness: {
        enabled: boolean,
        amount: number,
    },
    bloom: {
        enabled: boolean,
        intensity: number,
        blurLevel: number,
    },
    grading: {
        enabled: boolean,
        brightness: number,
        contrast: number,
        saturation: number,
        tint: [number, number, number],
    },
    vignette: {
        enabled: boolean,
        intensity: number,
        inner: number,
        outer: number,
        curvature: number,
    },
    fringing: {
        enabled: boolean,
        intensity: number
    }
};

type ExperienceSettings = {
    version: 2,
    tonemapping: 'none' | 'linear' | 'filmic' | 'hejl' | 'aces' | 'aces2' | 'neutral',
    highPrecisionRendering: boolean,
    soundUrl?: string,
    background: {
        color: [number, number, number],
        skyboxUrl?: string
    },
    postEffectSettings: PostEffectSettings,

    animTracks: AnimTrack[],
    cameras: Camera[],
    annotations: Annotation[],

    startMode: 'default' | 'animTrack' | 'annotation'
};

export type { AnimTrack, Camera, Annotation, PostEffectSettings, ExperienceSettings };

Notes

• camera 0 is the default camera
• any camera with an orbit constraint will not allow switching to fly mode
• the camera position constraint applies to fly and orbit camera position and to the orbit camera focus target
• user can click on an annotation in the 3d viewport to activate it or step through the set of annotations in the UI

[nancywan1004]

Another thing to consider:

• support default/custom reveal shader effects

[PLAN8VR]

+1 for skybox please!

[slimbuck]

The first phase of these updates were done in #94

[PLAN8VR]

I thought of a nice option the other day, to be able to display time-lapse splats. (So, same object/view taken at different time intervals.) if the 4dgs frames had the option to fade from one frame to the next, this could be quite beautiful. So just use the transparency to drop one frame down whilst bringing the new one up.

[simonbethke]

I would like to add settings for the XR experience like this:

XRExperience:
  scale: float
  virtualGroundPlane: position + normal
  controlOptions: walk|grab|fly
  audio: [
    {
      source: libID|url
      channel: int
      origin: position
    }
  ]

This is an extra object, because I have the idea to enable people to edit or create such settings for splats that have been published by others.

scale - This is simply a uniform scale of the splat and should be set in a way that measurements match realworld measurements
virtualGroundPlane - This defines the startposition and the plane that is used for walking and teleporting. Maybe the magnitude of the normal can be used to limit the distance a user can move away from the origin.
controlOptions - This is a set of 1-3 of the following options
controlOptions.walk - This already exists mostly. The user can walk and teleport on the virtual ground plane
controlOptions.grab - This mode allows to grab an object and move it with the controller. When using two controllers a user can grab and 3D-Pinch-Zoom by moving the controllers apart (like in Gracia VR)
controlOptions.fly - With this option there can be a button on a controller, that allows to fly in direction of that controller (like in Splatshop)
audio - This allows to place n audio-channels in space. The source "libID" should allow to choose an ambient track from a factory-library.