NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

The awkward dance of conversational voice AI is finally over. Every voice AI you have used feels broken in the same way. Ask Siri a question and wait through that painful silence before she responds. Try to interrupt any other voice AI agent mid-sentence and it will keep talking like you don't even exist. These systems chain together separate models, one to hear, one to think, and one to speak, creating robotic conversations full of unnatural pauses and zero awareness of when you are trying to jump in.

This is where this new model from Nvidia comes into play. PersonaPlex resolves this issue.

NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

Moshi made conversations feel human with real-time back and forth but locked you into a single personality and voice. You couldn't make it a customer service agent, a casual friend, or anything specific to your needs. Nvidia's Persona kills both problems at once. It's a 7 billion parameter model that listens and speaks at the same time while letting you define any role through text and pick from diverse voices.

How it works

PersonaPlex takes its innovation from Moshi. Under the hood, it's built on the Moshi architecture, but fine-tuned with a mix of real human conversation and synthetic dialogues, which gives it both natural speech patterns and the ability to follow complex instructions for specific roles.

It handles interruptions naturally, throws in conversational back channels like pauses at the right moments, and maintains whatever persona you have set throughout the entire conversation.

Installing NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

I used an Ubuntu system with one GPU card, Nvidia RTX 6000 with 48 GB of VRAM.

Step-by-step setup

• Create a virtual environment.
• Clone the PersonaPlex repository.
• Install all the prerequisites. This takes a minute.
• Log into Hugging Face. It's a gated model, so you need to accept the terms and conditions.
• Grab a free read token from Hugging Face.
• Install the Hugging Face Hub package.
• Log in using your token or set it in your environment.
• Launch the server.
  • On first run it downloads the model. The model is just over 16.7 GB in size.

Running NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

The model loads and the server runs on localhost. It also exposes a URL. VRAM consumption sits at over 20 GB, which is not bad for this sort of model.

Access it in the browser, connect your local mic, and you can start talking.

Testing NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

Speed is quite good. There is a lot of room for improvement, but the actual problem of robotic overlap and missed interruptions feels resolved.

Bank service persona trial

I switched to a bank service prompt and connected:

• PersonaPlex: Calling First Neuron Bank. How can I assist you today?
• Me: What is your name?
• PersonaPlex: I'm Alexis. How can I help?
• Me: My salary hasn't arrived in my bank account and I need to pay my rent for this week. Can you...
• PersonaPlex: I understand that can be frustrating. Let me check that quickly for you. Can I get the last four digits of your card?
• Me: 1 2 3 4.
• PersonaPlex: Thanks. I found your account. There was a recent transaction of $1,200 at Home Depot that was declined. Did you attempt that transaction?
• Me: I'm not talking about that transaction. I'm talking about a new transaction. Just transfer it.
• PersonaPlex: Unfortunately, the system has flagged your account for unusual activity. I can't process a large transaction without confirming your identity. Would you like to confirm your location first?
• Me: Yes, I am in Sydney, Australia.
• PersonaPlex: Yes, I see that. Your typical location is Seattle, Washington. There was...
• Me: My location is Sydney. Are you there?
• PersonaPlex: Yes, I'm here. Still showing Miami as the last place for your card. Would you like to try another transaction to confirm your location?

What the logs show

Behind the scenes, it does the handshake and then opens and closes the connection.

Quality observations

• The responses look more natural than Moshi.
• The model should not be hallucinating, as seen in both conversations where I was asking something else and the model responded with something else.
• At times it felt lost.

Resource usage snapshot

• Model size on disk - just over 16.7 GB
• VRAM consumption during serving - a bit over 20 GB
• VRAM remained stable across tests

Final thoughts

PersonaPlex makes real-time voice back-and-forth feel human and fixes the big problem of interruptions while letting you define any role and voice. The speed is solid and the persona control is flexible. Nvidia needs to improve response quality and reduce hallucinations, but this is a good evolution. I expect a lot of improvement in voice AI this year.