MiniMaxAI

/

MiniMax-H3

like 3.7k

Follow

MiniMax 8.96k

I'm just responding to say that I don't think it works. I have been over the guide in every way possible, and the only reference audio that works for me is direct pass-through. I would love to be wrong and have someone explain how they did it (and not just say "read the guide").

I haven't been able to get it to work either and I've followed the prompt guide basically perfectly. It always just starts with gibberish or just plays the audio of the file itself until it may or may not interrupt it with my prompted dialogue.
Lipsyncing the referenced audio works perfectly but actually transforming the voice to make it say what you want without the original file being imposed into the scene seems impossible.

It works, but the comfy node passes the audio directly as well as copy the voice.

I can clone any voice but the audio goes all messed up trying to speak a garbled version of the input audio as well for any non speaking moments. You can force speaking throughout the video and it will work. But any silence becomes a garbled version of the input audio.

I think the comfy node is to blame tbh. The node messes up A LOT when trying to disconnect audio. It wont allow me to disconnect it until i refresh the page.

I actually got it to work now. Just gotta connect it to the ref_video_audio connections instead of ref_audio and prompt accordingly...

For example:
subject_definitions:
...
<Subject 1> ...
<Audio 1> is the voice-timbre reference for <Subject 1>, containing a spoken english vocal layer.

summary:
...
<Audio 1> is used as the voice-timbre reference for <Subject 1>.

retention_analysis:
...
<Audio 1>: reference - it's vocal timbre guides the spoken voice of <Subject 1> without copying the original signal.

and then prompt whatever the character is supposed to say in the full prompt:
<Subject 1> says : <d>[English] This finally seems to work. </d>

Edit: Seems to have been a lucky gen with the pitch and voice adjusting to whatever I needed but it doesn't seem to work that way. Still getting the gibberish/imposing when doing it how it's meant to be used. I guess it's a conditioning issue with the node as rocky533 said.

What about using a singing clip for lipsync

Lip sync works fine, you can use custom audio fine with music.

I think the issue is the turbo loras. They are trained on the first/last frame model. We are using ref model. It works but its does not do audio properly. I think a proper ref distilled lora is needed to get the audio proper with this. The audio issues are not with the node or the inputs. Because the audio problems happen even with no audio ref input. It makes jibberish for any silent moments in the video.

The issues with the vocal references imposing the video doesn't seem to have anything to do with the turbo LoRAs (I don't recommend using them anyway because they reduce quality massively). I have not used them once and I still get the gibberish relatively often. It's basically a 50/50 chance from what I've seen so far unless I make my character talk from start to finish...

I agree most of the loras degrade things.

This one works best in comfy
https://huggingface.co/drbaph/MiniMax-H3-Turbo-Lora-ComfyUI/tree/main
minimax_h3_turbo_v4_step600_ema_pruned_comfyui.safetensors

The quality of the video is not an issue with it. The audio however is... something else...
Video quality and prompt understanding is there... with the lora above
https://civitai.red/user/sy0ww4bb1984/videos
But the audio....
https://civitai.red/images/139446365
https://civitai.red/images/139445743

My workflow for cloning voices is 20 steps with no speedup loras at 0.2MP output. From there, I use that audio track into a new ref flow and use audio "passthrough" for the "real video. Honestly, at this point I do that with every generation because the low-resolution output seems to follow the prompts better, so I can fire off a few low-res ones, pick the best, and use that as the reference for the high resolution (with speedup lora) one with no big surprises.