obs-nvenc: Only update Target Quality in CQVBR mode

[Lordmau5]

Description

Previously while CQVBR (Variable Bitrate with Target Quality) is enabled and the encoder is updated (e.g. changing target quality, b-frames, etc.) it was setting the averageBitRate value to what is being set in Variable Bitrate rate control mode.

However, CQVBR is initialized with a value of 0 and instead Target Quality is being used. This adds a check for the CQVBR mode and if it's updated in the Output settings will update the targetQuality value for the encoder, otherwise (if VBR is active) it will update averageBitRate.

(It was also updating it internally when the Replay Buffer was started while a recording was on-going (or the other way around) since OBS is calling the UpdateRecordingSettings method)

Big thanks to Bleuzen on the OBS Discord for spotting where in the code the issue is occuring.

Motivation and Context

Fixing the newly (32.0) added CQVBR / Variable Bitrate with Target Quality mode when using both Replay Buffer and Recording.

How Has This Been Tested?

Latest 32.1 portable instance - Setting the recording to CQVBR with a Target Quality of 20, then playing some high motion content and starting the recording, then observing the Stats window / dock. Afterwards either starting Replay Buffer or updating the Target Quality setting in the Output tab.
In the latest public build it will drop the bitrate massively (as in, to the average that is set in Variable Bitrate rate control mode).

Additionally the same in this build. However, updating the settings while this mode is active works properly.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[DeeDeeG]

Cool that this troubleshooting effort was able to proceed to a PR!

BTW, I think if cqvbr is true, then vbr is true (as enc->config.rcParams.rateControlMode == NV_ENC_PARAMS_RC_VBR should be true; no separate CQVBR rc mode according to the driver's logic, just RC_VBR). So, this line might not need updating from how it was, and you can have a smaller diff for this PR?

(I'm not able to test right now, but I'll drop another comment if I do get a chance later. Not that anyone should wait for me as I might not have time to.)