Ultimate recommended translation setting with Quality in mind

[aikotanaka6699-cmyk]

Greatly appreciate the work.

I was wondering if there's a ultimate setting for high end machine assuming having Faster-Whisper-XXL Pro.
The desire is for accurate translation with irrelevant noise filtered out at the highest quality of translation.

I arrived at this setup just trying to read around but couldn't find a precise place where it says definitively a "best" setting, which I know is hard to specify since everything "depends" on the scenario. But I was wondering if there was anything obvious with current setup that I should look into or swap out.

Transcription

"--compute-type", "float16",
"--ensemble", "--ensemble-serial",                
"--condition-on-previous-text", "true",

"--pass1-pipeline", "fidelity",
"--pass1-model", "large-v2",
"--pass1-sensitivity", "balanced",
"--pass1-scene-detector", "semantic",       
"--pass1-speech-segmenter", "ten",

"--pass2-pipeline", "xxl",
"--xxl-exe", $XXLExe,  # this is faster-whisper-xxl pro
"--pass2-model", "large-v2",                 
"--pass2-sensitivity", "aggressive",          
"--pass2-vad-threshold", "0.30",
"--pass2-speech-enhancer", "clearvoice:MossFormer2_SE_48K", 
"--pass2-speech-segmenter", "silero-v6.2", 
"--pass2-scene-detector", "semantic",
"--merge-strategy", "smart_merge"

Translation
ollama with mlabonne_gemma-3-27b-it-abliterated-GGUF

Any guidance would be appreciated.

[meizhong986]

You're way ahead of me :)
Does xxl work with pass2-speech-enhancer? I haven't tested it at all :)

My to go is: pass1 aggressive fidelity. It does take longer and it does have more risk of halucination though.

[aikotanaka6699-cmyk]

Time it takes for me does not really matter as long as it gives quality/accuracy.

I'm still new to a lot of this. I used AI to guide me to my current settings. Hence the point of the post haha.

The pass2 settings definitely runs and generates pass2.srt that has the transcription but I can't really determine whether or not it's effective yet. During normal dialog, it seems to do a decent job but it struggles outside that.

I'm still trying to play around to figure out optimal settings. I haven't been satisfied with any of combination I've tried so far but maybe I'm just being too picky. I think I need to test these passes in isolation to get a better idea how well they are doing.

[justantopair-ai]

is sematic + ten_vad is the goto for pass 1 now instead of audiotok + silero 6.2?
@meizhong986 I saw some qwen3.5 release recently, is there any model that's better than current gemma3:12b/qwen2.5:14b for 16GB VRAM?
also I think the pornify instruction is kinda overtuned when using with the 2 models above. They kinda make even normal convo escalated/erotic, which is funny but affecting the plot lines/the way they address others in drama/dialogue heavy jav.

[meizhong986]

The "customize parameters" in the ensemble TAB gives you all the levers you need to mix and match. I try to keep the defaults to be optimal for each pipeline but sometimes bugs creep in.

Yes, I've noticed as well that the pornify goes way way into fantasy-novel too fast in many of LLMs.

[justantopair-ai]

Google releases Gemma4 today with improvement on various aspects, including language processing.
https://ollama.com/library/gemma4
the 26b-a4b has better translation than gemma3 27b while requiring less vram afaik
there might be 9b and 12b models later for 8-16 GB VRAM

So far, I'm quite satisfied with translategemma:12b comparing to qwen. There was some benchmark and translategemma:27b is one of the best translation model right now iirc. Sadly, we need more than 16 GB VRAM for 27b model.
My usual default ensemble setup for whisperjav:

For AI translate, I add translated title + plot from japan sites (database, news, shopping, etc.) to aid. I have translategemma:27b-it running on a mac with 36 GB RAM for this task. I'm thinking of using it for subtitle translation too, but the speed tradeoff would be huge to run without cuda.

[TinyRick1489]

You can use translategemma:27b model on Kaggle. I ran all my models on Kaggle.

[justantopair-ai]

@TinyRick1489 current notebook version is 1.8.5, did I miss anything? I couldn't find 1.8.10 to run Ollama on Kaggle

[TinyRick1489]

The Kaggle notebook is not updated yet. You can use step 1 and 2 to install WhisperJAV but after that you have to write you own code if you are familiar with CLI.

[TinyRick1489]

I tried gemma4:26b and qwen3.5:9b. By default these models use thinking mode and I think that is the cause of error like API limit, No match found or Untranslated lines.
In the debug log if you see DEBUG:root:Response:, it contains both translated dialogues as well as chain of thoughts.
For models like translategemma and qwen2.5-instruct the DEBUG:root:Response: only contains translated dialogues.

Gemma4 debug log - gemma4_debug_log.txt
Qwen3.5 debug log - qwen3.5_debug_log.txt
Translategemma debug log - translategemma_debug_log.txt

[aikotanaka6699-cmyk]

Was playing around with it some more and thought I'd share some ideas. Again, I'm by no means an expert. Most of this is just through a lot of trial/error. I do run all this locally with a powerful machine but I think the basic concepts are transferable no matter what you are doing.

General

I opted to utilize the python installation instead of the GUI to give me flexibility in automating most of the process as much as possible with whisperjav's cli. It has several advantages:

• Automate the process to integrate other pieces not part of whisperjav
• Allows running the process in parts instead of all at once. This let's me continue at halfway point and let's me try out different translation settings without having to transcribe again.
• Allows to modify whisperjav's internals a bit.

I have divided the process in 4 main processes:
Audio cleanup => Transcription => Transcription cleanup => Translation

To accomplish this, I've created 3 powershell scripts

Script 1. Preprocessing the Audio

Generating a clean transcribed file may be the most important part of the process as less noise that can confuse and interfere with AI, the better. I wanted to add another layer of preprocessing to what already exists in whisperjav. My Script 1 does:

1. Generate a temp wav audio file of the video to ssd C: drive so that there's no issue with speed in processing the file since my video resides in a slow storage location.
2. Run the generated wav audio file through "audio-separator" through installing pip install audio-separator. It's actually amazing and does a great job filtering out a lot of noise. Beginning of video often times have music with introduction narration and you can hear the clear difference.
3. Using the newly cleaned audio, I will run it through the normal whisperjav transcription process where I use the 2 pass approach. I don't really know which settings is best, but I use Fidelity (aggressive) with Faster-Whisper-XXL (balanced) and smart_merge.

Script 2. Cleaning up the Transcription even more

I run it through a ollama model whose sole job is to figure out if it's actual dialogue that needs to be stay there to be translated; otherwise it's removed. The idea is that it would allow the AI to have better context to filter out hallucinations before feeding it into the translator.

Script 3. Translation

I feed it through another ollama model who's sole job is to translate very strictly.

I think technically, you can combine the two ollamas instructions, but my observation/guess is the results are better when ollama only has to worry about 1 job at a time.

Failed approach

One thing I tried was to attach role to each transcription line with ollama to give it more clear context on what it's translating. For example:
[Student]: Blah Blah Blah

This was very difficult to implement and gave very bad results as the AI needs more clues than just text transcripts to decipher who's speaking. I did try to instruct it to put [Unknown] if it's not 100% sure but ultimately, I felt this would increase the pipeline's complexity too much for so little gain.

[TinyRick1489]

Did you try using --diarize flag in Faster-Whisper-XXL Pro?

[aikotanaka6699-cmyk]

For the failed approach I mentioned, I didn't use --diarize flag. I didn't even know it existed. I'll have to look into that later.

I attempted to just feed it through a different ollama model to determine character but it didn't a pretty horrible job.

[justantopair-ai]

yea I think it's pretty hard to put roles on lines and dismissing return.
even for us human without audio + video, it's challenging to figure out whose lines. Some lines are just neutral even with given context.

A proper/correct transcription (text and timestamp) is still the core priority imo. Translator might not be perfect guessing the roles to put matching pronoun, but at least we can understand the content when watching.

[TinyRick1489]

@aikotanaka6699-cmyk
Which model do you use in audio-separator for preprocessing the audio?

[aikotanaka6699-cmyk]

I used audio-separator model model_bs_roformer_ep_317_sdr_12.9755.ckpt