Guides

AI Voice Clone Singing, Open Source: No Training, 6 GB

AI voice clone singing is a technique that makes open-source models sing a melody or a guide song in your own cloned voice, using about 10 seconds of speech. A written melody goes straight to SoulX-Singer; a rap goes through an ACE-Step 1.5 guide song first, and Whisper checks every line.

David Marsa

David Marsa

Founder & CEO

Intermediate11 min readPublished Nov 18, 2026

Last verified Nov 18, 2026

Ten seconds of speech, plus written notes or a generated beat, merge into one singing voice.

What you will be able to do

  • Make a 10-second voice prompt that both SoulX-Singer modes can use.
  • Write a short melody as notes and get every line sung right in four takes.
  • Generate a guide rap with ACE-Step 1.5 on a 6 GB GPU and re-sing it in your voice with SoulX SVC.
  • Score takes and seeds line by line with Whisper and catch unsafe mishearings.
  • Mix the result to -14 LUFS and publish it with an AI-content disclosure.

Key takeaways

  • AI voice clone singing makes open-source models sing in your own cloned voice from about 10 seconds of speech, with no per-voice training.
  • A written melody goes straight to SoulX-Singer score mode; a rap goes through an ACE-Step 1.5 guide song that SoulX SVC re-sings in your voice with the same timing.
  • Whisper large-v3 reads every line of every take, so the best take per line wins and unsafe mishearings are flagged before you publish.
  • Six gigabytes is enough: ACE-Step with no language model, INT8 weights and CPU offload, and one GPU job at a time.
  • Clone only your own voice or one with written consent, and disclose the song as AI-generated music when you upload it.

AI voice clone singing is a technique that makes open-source models sing a melody or a guide song in your own cloned voice, using about 10 seconds of speech. By the end, you have a short song in your own voice, made on a 6 GB GPU, with every line checked by Whisper.

Installing takes an afternoon. A 4-line written song then takes about 12 minutes of GPU time: 4.5 minutes for the voice prompt, four takes at about 1.5 minutes each, 1 minute of judging.

At Neomanex I run this pipeline for the songs in our videos, and I listen to every flagged line before it ships. The examples use Pebblewick Coffee, a fictional neighbourhood roaster.

What AI voice clone singing is, and the two routes

Pick the route by one question: can you write the melody down? If yes, the job is easy. If not, you need a guide singer first.

Route A: written melodyRoute B: rap, or a style you cannot notate
InputLyrics plus notesLyrics plus a style caption
SingerSoulX-Singer score mode sings your notes in your voiceACE-Step 1.5 makes a guide song; SoulX-Singer SVC re-sings it in your voice
JudgeWhisper large-v3 picks the best take per lineWhisper large-v3 scores every seed per line

SVS (singing voice synthesis) sings from notes and words. SVC (singing voice conversion) takes an existing vocal and swaps the voice, keeping its timing and melody. SoulX-Singer does both.

If you can write the melody, SoulX-Singer sings it; if not, ACE-Step makes a guide that SoulX re-sings in your voice, and both routes end in a mix at -14 LUFS.

No training: a 10-second voice prompt

Training an RVC model is the other road. The RVC project(opens in new tab) recommends at least 10 minutes of low-noise speech and a training run per voice. SoulX-Singer is zero-shot: it reads a 10-second prompt and trains nothing, so the first take comes the same afternoon. I have not measured it against a trained RVC model.

Here is route B on a real song: a rap sung in my own cloned voice.

What you need

Six gigabytes is enough if you treat the GPU as one queue. I promise nothing for other cards.

NeedVersion I run
NVIDIA GPU with CUDARTX 3060 Laptop GPU, 6 GB
Python and ffmpegPython 3.10
SoulX-Singercommit 81aeb3a, transformers 4.41.2
ACE-Step 1.5commit ca1e85f
faster-whisper1.2.1
A voice you may useYour own, or one with its owner's written consent

The Neomanex model directory lists the open-weight models we track.

Six gigabytes is enough when the language model stays off and only one model sits on the card.

One GPU job runs at a time, behind a lock. Before a model loads, the card stays under 600 MiB for 45 seconds. Whisper runs on CUDA in float16, never int8, never on the CPU.

Step 1: Record a 10-second voice prompt

The prompt decides how much of you comes through, so keep it clean and short. Use about 10 seconds of speech in your voice: a clean recording, or a clean take from a text-to-speech (TTS) clone of your own voice. A TTS take works as a prompt; Step 5 covers where synthetic audio hurts. Pebblewick's says "This is my voice, recorded for a song I write myself."

The pipeline keeps the first 10 seconds and builds two prompts. The SVC prompt is 24 kHz mono plus an RMVPE pitch file. The score-mode prompt is 44.1 kHz mono, run through SoulX's preprocessing for word times, notes and pitch. Denoise it for score mode, where a raw room recording loses words. Either works for SVC.

Step 2: Write the melody as notes

Writing notes feels slow, and it is what makes every later line come out right. A song file holds one token per note: a MIDI pitch and a length in beats. ~ joins a syllable to the next note. _ holds a melisma and ends the word; without it, the next word glues onto the held vowel.

"Wake Up Warm", the Pebblewick jingle, runs at 100 BPM in C major:

LineLyric
1Wake up warm, the kettle's on
2Pebblewick pours it slow
3Beans we roast before the dawn
4One more cup before you go

I keep a male voice between D3 and D4. Each word's last note holds for 92% of its length, then rests, so consonants stay apart. Use score mode for a written song: melody mode on a borrowed vocal smears new consonants into gibberish.

Line 2 as tokens (syllable, MIDI pitch, beats):

Peb~ 52 0.5 | ble~ 55 0.5 | wick 57 1 | pours 60 1 | it 57 0.5 | slow~ 55 1 | _ 52 2

Score mode does not read tokens. A short converter turns each line into one target segment: every note carries its word, the word's ARPAbet from g2p_en, a pitch (0 for a rest) and a note type (2 starts a word, 3 carries it on, 1 is a rest). At 100 BPM a beat is 0.6 seconds:

{"text": "<SP> Pebblewick Pebblewick Pebblewick <SP> pours <SP> ...",
 "note_pitch": "0 52 55 57 0 60 0 ...",
 "note_type": "1 2 3 3 1 2 1 ...",
 "duration": "0.200 0.300 0.300 0.552 0.048 0.552 0.048 ..."}

The list of segments is song/lead.json, the target in the build commands.

Step 3: Sing four takes and let Whisper pick each line

Never trust your ear alone on a take. Let Whisper read every line, then listen to the pick.

SoulX-Singer samples, so every take differs. I sing four. Whisper large-v3 (beam 5, no conditioning on previous text) transcribes each line of each take alone. Each cell scores the text similarity between what Whisper heard and the lyric, from 0 to 1. The highest score wins the line; a tie goes to the lower take.

LineTake 1Take 2Take 3Take 4Pick
1"Wake up one, the kettle's on" 0.831.001.001.00take 2
2"Pebble wick pours it slow" 0.67"Pebblewick pause it slow" 0.751.001.00take 3
31.00"Beings we roast before the dawn" 0.831.001.00take 1
4"One more cut before you go" 0.831.001.001.00take 2

Five misses in 16 cells, picks 2, 3, 1, 2, and every picked line is right. The second-best take can sit under the pick as a vocal double.

Whisper reads every line of every take, and the take closest to the lyric wins each line.

Step 4: Generate a guide rap with ACE-Step 1.5

The guide singer is a stand-in: you want its timing, not its voice, so ask for a dry, plain vocal. Pebblewick's caption is "boom bap hip hop, solo male rapper, clear diction, dry upfront vocal", 90 BPM, 30 seconds, four lines of 10 words:

LineLyric
1Up at five with the drum turning beans to gold
2Pebblewick roast got a caramel kick when it gets cold
3We ship the beans the day we roast, never stale
4Pour it slow, raise a cup, every sip tells the tale

On 6 GB, ACE-Step 1.5 runs the 2B turbo DiT with no language model (thinking and chain of thought off), INT8 weight-only quantization and CPU offload. The 1.7B language model does not fit beside the DiT, and it rewrites your caption and lyrics anyway.

The same seed and words give a byte-identical guide. ACE-Step adds ad-libs and repeats or drops a hook line, so part of every score is structure.

Step 5: Re-sing the guide in your voice with SoulX SVC

SVC copies the performance exactly, mistakes included, so a clean guide matters more than the prompt. A mel-band roformer karaoke model splits the lead vocal from the beat, and RMVPE extracts pitch. Nothing removes reverb, so the dry vocal from Step 4 matters: SVC loses consonants under reverb. SoulX-Singer SVC with --auto_shift sings it again in your voice and range, with the same timing, back over ACE-Step's beat.

SVC keeps the guide's timing word for word and swaps the voice; hard consonants are where it loses words.

SVC keeps the flow and blurs hard consonants. It needs a real performance underneath: on a TTS clone of your voice reading the rap, it makes the words worse. A synthetic prompt is fine; a synthetic vocal to convert is not.

I build pipelines like this with AI agents on written workflows and show the work every week. See how we run it, from the inside, at neomanex.com/join.

Step 6: Score every seed, line by line

I score every seed against the lyric sheet. Whisper reads the converted vocal in 5-second windows, each edge snapped to the quietest 20 ms within 0.3 seconds so no cut lands inside a word. Heard words align to the sheet by edit distance, and each error is charged to its line. WER (word error rate) is errors divided by the 40 lyric words. A line is flagged when Whisper hears a word from an unsafe list (profanity and sexual words).

LineSeed 1Seed 2Seed 3
1634
2422
353, unsafe: "We sh*t the beans"3
4523
WER0.500.25 (lowest)0.30
Verdictruled out by scoreruled out by line 3publish after listening

Seed 2 scores best and still loses. I pick seed 3 by score and by ear.

The lowest score is not always the take to publish: an unsafe mishearing on one line rules a seed out.

Step 7: Mix to -14 LUFS and label it as AI-made

A mix that buries the words undoes the judging. Whisper judges the dry take; the mix adds only light effects. Both routes high-pass the vocal at 90 Hz and land at -14 LUFS, true peak -1.5 dBTP, loudness range 9. Route B adds a 2.5:1 compressor and +2 dB at 3.5 kHz, matching the guide's vocal level. Route A dips the band's 1 to 5 kHz range under the voice.

Keep the wav of every picked take: SoulX never sings the same input twice the same way. Listen to every flagged line. Then label the upload: YouTube asks creators to disclose AI-generated music(opens in new tab) with the "AI use" setting in YouTube Studio.

What goes wrong, and the rule each failure produced

Every failure here was quiet: the take sounded fine and the words were wrong.

Failure (retold on Pebblewick)Rule now
Seeds on the same words and settings ranged from clean to unusableRun several seeds, score each line by line, pick by score, then by ear
SVC kept the timing and blurred "kick"Ask for a dry, solo, clear-diction guide; judge after conversion
Whisper heard "ship" as profanityFlag unsafe words per line; never publish a flagged take unheard
A fast rap transcribed whole lost its opening secondsJudge in 5-second windows snapped to silence
A new lyric on a borrowed pitch curve came out unintelligibleWrite the melody and use score mode
Another job took the card between the free check and the load: out of memoryOne GPU job behind a lock, wait for an idle card; singing retries after an out-of-memory

Turning failures into rules is its own loop: build a self-improving AI system.

Build your own: the minimal command set

Start with route A on four lines. Once one line comes back clean, the rest is repetition. These commands use the upstream projects at the versions above.

SoulX-Singer, one take in score mode (prompt metadata from SoulX's preprocessing, song/lead.json from your converter):

git clone https://github.com/Soul-AILab/SoulX-Singer.git && cd SoulX-Singer && git checkout 81aeb3a
python3.10 -m venv .venv && .venv/bin/pip install -r requirements.txt huggingface_hub
.venv/bin/hf download Soul-AILab/SoulX-Singer --local-dir pretrained_models/SoulX-Singer
.venv/bin/hf download Soul-AILab/SoulX-Singer-Preprocess --local-dir pretrained_models/SoulX-Singer-Preprocess

.venv/bin/python -m cli.inference --device cuda --control score \
  --model_path pretrained_models/SoulX-Singer/model.pt --config soulxsinger/config/soulxsinger.yaml \
  --prompt_wav_path prompt/vocal.wav --prompt_metadata_path prompt/metadata.json \
  --target_metadata_path song/lead.json --phoneset_path soulxsinger/utils/phoneme/phone_set.json \
  --save_dir out/take1

ACE-Step 1.5, one guide rap in the 6 GB tier:

cd .. && git clone https://github.com/ACE-Step/ACE-Step-1.5.git && cd ACE-Step-1.5 && git checkout ca1e85f && uv sync

Save this as guide.py inside ACE-Step-1.5, with your lyrics in rap.txt beside it, and run it with uv run python guide.py:

from acestep.handler import AceStepHandler
from acestep.llm_inference import LLMHandler
from acestep.inference import GenerationParams, GenerationConfig, generate_music

dit = AceStepHandler()
dit.initialize_service(project_root=".", config_path="acestep-v15-turbo", device="cuda",
                       offload_to_cpu=True, offload_dit_to_cpu=True, quantization="int8_weight_only")
p = GenerationParams(caption="boom bap hip hop, solo male rapper, clear diction, dry upfront vocal",
                     lyrics=open("rap.txt").read(), vocal_language="en", bpm=90, duration=30,
                     thinking=False, use_cot_metas=False, use_cot_caption=False, use_cot_language=False, seed=3)
generate_music(dit, LLMHandler(), p, GenerationConfig(batch_size=1, audio_format="wav",
               use_random_seed=False, seeds=[3]), save_dir="guide")

Separation and pitch, with SoulX's own preprocessing tools: back in SoulX-Singer with its venv, copy the wav ACE-Step wrote to guide/seed3.wav (cd ../SoulX-Singer && mkdir -p guide, then copy the file from ../ACE-Step-1.5/guide/):

import soundfile as sf
from preprocess.tools import VocalSeparator, F0Extractor

P = "pretrained_models/SoulX-Singer-Preprocess/"
sep = VocalSeparator(
    sep_model_path=P + "mel-band-roformer-karaoke/mel_band_roformer_karaoke_becruily.ckpt",
    sep_config_path=P + "mel-band-roformer-karaoke/config_karaoke_becruily.yaml",
    der_model_path=P + "dereverb_mel_band_roformer/dereverb_mel_band_roformer_anvuew_sdr_19.1729.ckpt",
    der_config_path=P + "dereverb_mel_band_roformer/dereverb_mel_band_roformer_anvuew.yaml", device="cuda")
r = sep.process("guide/seed3.wav")
sf.write("guide/vocal.wav", r.vocals.T, r.sample_rate)
sf.write("guide/beat.wav", r.accompaniment.T, r.sample_rate)

f0 = F0Extractor(model_path=P + "rmvpe/rmvpe.pt", device="cuda")
for w in ("guide/vocal.wav", "prompt/vocal.wav"):
    f0.process(w, f0_path=w.replace(".wav", "_f0.npy"))

SoulX SVC, the separated guide vocal in your voice:

.venv/bin/python -m cli.inference_svc --device cuda --auto_shift \
  --model_path pretrained_models/SoulX-Singer/model-svc.pt --config soulxsinger/config/soulxsinger.yaml \
  --prompt_wav_path prompt/vocal.wav --prompt_f0_path prompt/vocal_f0.npy \
  --target_wav_path guide/vocal.wav --target_f0_path guide/vocal_f0.npy --save_dir out/svc

faster-whisper, one line or one window:

from faster_whisper import WhisperModel

m = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, _ = m.transcribe("line2.wav", language="en", beam_size=5, condition_on_previous_text=False)
heard = " ".join(s.text.strip() for s in segments)

Want to watch the next song get made, take by take? See how we run it, from the inside, at neomanex.com/join.

Steps

  1. Record a 10-second voice prompt

    Use about 10 seconds of clean speech in your voice, cut to the first 10 seconds. Build a 24 kHz mono prompt with a pitch file for SVC and a SoulX-preprocessed prompt for score mode.

  2. Write the melody as notes

    Write one token per note with a MIDI pitch and a length in beats. Join syllables across notes with ~ and end a held melisma with _, and keep a male voice between D3 and D4.

  3. Sing four takes and let Whisper pick each line

    Sing four takes with SoulX-Singer in score mode. Whisper large-v3 transcribes every line of every take, and the take closest to the lyric wins each line.

  4. Generate a guide rap with ACE-Step 1.5

    Write a caption that asks for a solo rapper, clear diction and a dry upfront vocal. Run the 2B turbo model with no language model, INT8 weights and CPU offload, one seed per guide.

  5. Re-sing the guide in your voice with SoulX SVC

    Split the guide vocal from the beat and extract pitch. Run SoulX-Singer SVC with auto shift and lay your new vocal back over the beat.

  6. Score every seed, line by line

    Transcribe the converted vocal in 5-second windows snapped to silence, align the words to the lyric sheet and count errors per line. Rule out any seed with an unsafe mishearing, then pick by score and by ear.

  7. Mix to -14 LUFS and label it as AI-made

    High-pass the vocal at 90 Hz, keep effects light and normalize to -14 LUFS with a true peak of -1.5 dBTP. Listen to every flagged line and disclose the upload as AI-generated music.

Frequently asked questions

What is AI voice clone singing?

AI voice clone singing is a technique that makes open-source models sing a melody or a guide song in your own cloned voice, using about 10 seconds of speech. A written melody goes to SoulX-Singer score mode, which sings your notes. A rap goes through an ACE-Step 1.5 guide song that SoulX-Singer SVC re-sings in your voice with the same timing. No per-voice training is needed.

Who is open-source AI voice clone singing for?

Open-source AI voice clone singing is for makers, creators and small studios with a consumer NVIDIA GPU of 6 GB or more who want jingles, intros or short songs in their own voice. SoulX-Singer, ACE-Step 1.5 and Whisper all run locally on that card, and Whisper checks every sung line against the lyric sheet before anything is published.

How do I make AI sing a song in my own voice for free?

Record about 10 seconds of clean speech in your voice. Then write the melody as notes, one token per note, and sing four takes with SoulX-Singer in score mode. Whisper large-v3 transcribes every line of every take, and the take closest to the lyric wins each line. Every model is open source and runs on your own GPU. Use only your own voice or one with written consent.

Do I need to train a model on my voice to make it sing?

No. SoulX-Singer is zero-shot: it reads a 10-second voice prompt at inference time and trains nothing per voice. Training an RVC model is the other route, and the RVC project recommends at least 10 minutes of low-noise speech plus a training run for each voice. Zero-shot is faster to set up, and a trained RVC model stays the alternative when you have the recordings.

Can AI rap in my own voice with open-source models?

Yes. ACE-Step 1.5 generates a guide rap from your lyrics and a style caption, and SoulX-Singer SVC sings that guide again in your voice with the same timing. SVC keeps the flow and blurs hard consonants, so every seed comes out differently. Generate several seeds, score each one line by line with Whisper large-v3, and pick by score and then by ear.

Can ACE-Step 1.5 and SoulX-Singer run on a 6 GB GPU?

Yes, on an RTX 3060 Laptop GPU with 6 GB. ACE-Step 1.5 runs the 2B turbo DiT with no language model, INT8 weight-only quantization and CPU offload. SoulX-Singer and Whisper large-v3 (float16) each load alone, because only one GPU job runs at a time. Before each load, the card stays under 600 MiB for 45 seconds. Other cards are untested.

Can I clone someone else's voice to make it sing?

Only with written consent from the voice's owner. AI voice clone singing works on any 10-second prompt, so consent is the rule that limits it: clone your own voice, or one whose owner agreed in writing, and never a public figure's voice without that agreement. When you publish the song, YouTube asks creators to disclose AI-generated music with the AI use setting at upload.

How do I know the AI sang the right words?

Transcribe every take with Whisper large-v3 and compare it to the lyric sheet. For a written melody, Whisper reads each line alone and the closest take wins the line. For a rap, Whisper reads 5-second windows snapped to silence, and every error is charged to its lyric line. Lines where Whisper hears profanity are flagged, so listen to each one before publishing.