What you will be able to do
- Make a 10-second voice prompt that both SoulX-Singer modes can use.
- Write a short melody as notes and get every line sung right in four takes.
- Generate a guide rap with ACE-Step 1.5 on a 6 GB GPU and re-sing it in your voice with SoulX SVC.
- Score takes and seeds line by line with Whisper and catch unsafe mishearings.
- Mix the result to -14 LUFS and publish it with an AI-content disclosure.
Key takeaways
- AI voice clone singing makes open-source models sing in your own cloned voice from about 10 seconds of speech, with no per-voice training.
- A written melody goes straight to SoulX-Singer score mode; a rap goes through an ACE-Step 1.5 guide song that SoulX SVC re-sings in your voice with the same timing.
- Whisper large-v3 reads every line of every take, so the best take per line wins and unsafe mishearings are flagged before you publish.
- Six gigabytes is enough: ACE-Step with no language model, INT8 weights and CPU offload, and one GPU job at a time.
- Clone only your own voice or one with written consent, and disclose the song as AI-generated music when you upload it.
AI voice clone singing is a technique that makes open-source models sing a melody or a guide song in your own cloned voice, using about 10 seconds of speech. By the end, you have a short song in your own voice, made on a 6 GB GPU, with every line checked by Whisper.
Installing takes an afternoon. A 4-line written song then takes about 12 minutes of GPU time: 4.5 minutes for the voice prompt, four takes at about 1.5 minutes each, 1 minute of judging.
At Neomanex I run this pipeline for the songs in our videos, and I listen to every flagged line before it ships. The examples use Pebblewick Coffee, a fictional neighbourhood roaster.
What AI voice clone singing is, and the two routes
Pick the route by one question: can you write the melody down? If yes, the job is easy. If not, you need a guide singer first.
| Route A: written melody | Route B: rap, or a style you cannot notate | |
|---|---|---|
| Input | Lyrics plus notes | Lyrics plus a style caption |
| Singer | SoulX-Singer score mode sings your notes in your voice | ACE-Step 1.5 makes a guide song; SoulX-Singer SVC re-sings it in your voice |
| Judge | Whisper large-v3 picks the best take per line | Whisper large-v3 scores every seed per line |
SVS (singing voice synthesis) sings from notes and words. SVC (singing voice conversion) takes an existing vocal and swaps the voice, keeping its timing and melody. SoulX-Singer does both.

No training: a 10-second voice prompt
Training an RVC model is the other road. The RVC project(opens in new tab) recommends at least 10 minutes of low-noise speech and a training run per voice. SoulX-Singer is zero-shot: it reads a 10-second prompt and trains nothing, so the first take comes the same afternoon. I have not measured it against a trained RVC model.
Here is route B on a real song: a rap sung in my own cloned voice.
What you need
Six gigabytes is enough if you treat the GPU as one queue. I promise nothing for other cards.
| Need | Version I run |
|---|---|
| NVIDIA GPU with CUDA | RTX 3060 Laptop GPU, 6 GB |
| Python and ffmpeg | Python 3.10 |
| SoulX-Singer | commit 81aeb3a, transformers 4.41.2 |
| ACE-Step 1.5 | commit ca1e85f |
| faster-whisper | 1.2.1 |
| A voice you may use | Your own, or one with its owner's written consent |
The Neomanex model directory lists the open-weight models we track.

One GPU job runs at a time, behind a lock. Before a model loads, the card stays under 600 MiB for 45 seconds. Whisper runs on CUDA in float16, never int8, never on the CPU.
Step 1: Record a 10-second voice prompt
The prompt decides how much of you comes through, so keep it clean and short. Use about 10 seconds of speech in your voice: a clean recording, or a clean take from a text-to-speech (TTS) clone of your own voice. A TTS take works as a prompt; Step 5 covers where synthetic audio hurts. Pebblewick's says "This is my voice, recorded for a song I write myself."
The pipeline keeps the first 10 seconds and builds two prompts. The SVC prompt is 24 kHz mono plus an RMVPE pitch file. The score-mode prompt is 44.1 kHz mono, run through SoulX's preprocessing for word times, notes and pitch. Denoise it for score mode, where a raw room recording loses words. Either works for SVC.
Step 2: Write the melody as notes
Writing notes feels slow, and it is what makes every later line come out right. A song file holds one token per note: a MIDI pitch and a length in beats. ~ joins a syllable to the next note. _ holds a melisma and ends the word; without it, the next word glues onto the held vowel.
"Wake Up Warm", the Pebblewick jingle, runs at 100 BPM in C major:
| Line | Lyric |
|---|---|
| 1 | Wake up warm, the kettle's on |
| 2 | Pebblewick pours it slow |
| 3 | Beans we roast before the dawn |
| 4 | One more cup before you go |
I keep a male voice between D3 and D4. Each word's last note holds for 92% of its length, then rests, so consonants stay apart. Use score mode for a written song: melody mode on a borrowed vocal smears new consonants into gibberish.
Line 2 as tokens (syllable, MIDI pitch, beats):
Peb~ 52 0.5 | ble~ 55 0.5 | wick 57 1 | pours 60 1 | it 57 0.5 | slow~ 55 1 | _ 52 2
Score mode does not read tokens. A short converter turns each line into one target segment: every note carries its word, the word's ARPAbet from g2p_en, a pitch (0 for a rest) and a note type (2 starts a word, 3 carries it on, 1 is a rest). At 100 BPM a beat is 0.6 seconds:
{"text": "<SP> Pebblewick Pebblewick Pebblewick <SP> pours <SP> ...",
"note_pitch": "0 52 55 57 0 60 0 ...",
"note_type": "1 2 3 3 1 2 1 ...",
"duration": "0.200 0.300 0.300 0.552 0.048 0.552 0.048 ..."}
The list of segments is song/lead.json, the target in the build commands.
Step 3: Sing four takes and let Whisper pick each line
Never trust your ear alone on a take. Let Whisper read every line, then listen to the pick.
SoulX-Singer samples, so every take differs. I sing four. Whisper large-v3 (beam 5, no conditioning on previous text) transcribes each line of each take alone. Each cell scores the text similarity between what Whisper heard and the lyric, from 0 to 1. The highest score wins the line; a tie goes to the lower take.
| Line | Take 1 | Take 2 | Take 3 | Take 4 | Pick |
|---|---|---|---|---|---|
| 1 | "Wake up one, the kettle's on" 0.83 | 1.00 | 1.00 | 1.00 | take 2 |
| 2 | "Pebble wick pours it slow" 0.67 | "Pebblewick pause it slow" 0.75 | 1.00 | 1.00 | take 3 |
| 3 | 1.00 | "Beings we roast before the dawn" 0.83 | 1.00 | 1.00 | take 1 |
| 4 | "One more cut before you go" 0.83 | 1.00 | 1.00 | 1.00 | take 2 |
Five misses in 16 cells, picks 2, 3, 1, 2, and every picked line is right. The second-best take can sit under the pick as a vocal double.

Step 4: Generate a guide rap with ACE-Step 1.5
The guide singer is a stand-in: you want its timing, not its voice, so ask for a dry, plain vocal. Pebblewick's caption is "boom bap hip hop, solo male rapper, clear diction, dry upfront vocal", 90 BPM, 30 seconds, four lines of 10 words:
| Line | Lyric |
|---|---|
| 1 | Up at five with the drum turning beans to gold |
| 2 | Pebblewick roast got a caramel kick when it gets cold |
| 3 | We ship the beans the day we roast, never stale |
| 4 | Pour it slow, raise a cup, every sip tells the tale |
On 6 GB, ACE-Step 1.5 runs the 2B turbo DiT with no language model (thinking and chain of thought off), INT8 weight-only quantization and CPU offload. The 1.7B language model does not fit beside the DiT, and it rewrites your caption and lyrics anyway.
The same seed and words give a byte-identical guide. ACE-Step adds ad-libs and repeats or drops a hook line, so part of every score is structure.
Step 5: Re-sing the guide in your voice with SoulX SVC
SVC copies the performance exactly, mistakes included, so a clean guide matters more than the prompt. A mel-band roformer karaoke model splits the lead vocal from the beat, and RMVPE extracts pitch. Nothing removes reverb, so the dry vocal from Step 4 matters: SVC loses consonants under reverb. SoulX-Singer SVC with --auto_shift sings it again in your voice and range, with the same timing, back over ACE-Step's beat.

SVC keeps the flow and blurs hard consonants. It needs a real performance underneath: on a TTS clone of your voice reading the rap, it makes the words worse. A synthetic prompt is fine; a synthetic vocal to convert is not.
I build pipelines like this with AI agents on written workflows and show the work every week. See how we run it, from the inside, at neomanex.com/join.
Step 6: Score every seed, line by line
I score every seed against the lyric sheet. Whisper reads the converted vocal in 5-second windows, each edge snapped to the quietest 20 ms within 0.3 seconds so no cut lands inside a word. Heard words align to the sheet by edit distance, and each error is charged to its line. WER (word error rate) is errors divided by the 40 lyric words. A line is flagged when Whisper hears a word from an unsafe list (profanity and sexual words).
| Line | Seed 1 | Seed 2 | Seed 3 |
|---|---|---|---|
| 1 | 6 | 3 | 4 |
| 2 | 4 | 2 | 2 |
| 3 | 5 | 3, unsafe: "We sh*t the beans" | 3 |
| 4 | 5 | 2 | 3 |
| WER | 0.50 | 0.25 (lowest) | 0.30 |
| Verdict | ruled out by score | ruled out by line 3 | publish after listening |
Seed 2 scores best and still loses. I pick seed 3 by score and by ear.

Step 7: Mix to -14 LUFS and label it as AI-made
A mix that buries the words undoes the judging. Whisper judges the dry take; the mix adds only light effects. Both routes high-pass the vocal at 90 Hz and land at -14 LUFS, true peak -1.5 dBTP, loudness range 9. Route B adds a 2.5:1 compressor and +2 dB at 3.5 kHz, matching the guide's vocal level. Route A dips the band's 1 to 5 kHz range under the voice.
Keep the wav of every picked take: SoulX never sings the same input twice the same way. Listen to every flagged line. Then label the upload: YouTube asks creators to disclose AI-generated music(opens in new tab) with the "AI use" setting in YouTube Studio.
What goes wrong, and the rule each failure produced
Every failure here was quiet: the take sounded fine and the words were wrong.
| Failure (retold on Pebblewick) | Rule now |
|---|---|
| Seeds on the same words and settings ranged from clean to unusable | Run several seeds, score each line by line, pick by score, then by ear |
| SVC kept the timing and blurred "kick" | Ask for a dry, solo, clear-diction guide; judge after conversion |
| Whisper heard "ship" as profanity | Flag unsafe words per line; never publish a flagged take unheard |
| A fast rap transcribed whole lost its opening seconds | Judge in 5-second windows snapped to silence |
| A new lyric on a borrowed pitch curve came out unintelligible | Write the melody and use score mode |
| Another job took the card between the free check and the load: out of memory | One GPU job behind a lock, wait for an idle card; singing retries after an out-of-memory |
Turning failures into rules is its own loop: build a self-improving AI system.
Build your own: the minimal command set
Start with route A on four lines. Once one line comes back clean, the rest is repetition. These commands use the upstream projects at the versions above.
SoulX-Singer, one take in score mode (prompt metadata from SoulX's preprocessing, song/lead.json from your converter):
git clone https://github.com/Soul-AILab/SoulX-Singer.git && cd SoulX-Singer && git checkout 81aeb3a
python3.10 -m venv .venv && .venv/bin/pip install -r requirements.txt huggingface_hub
.venv/bin/hf download Soul-AILab/SoulX-Singer --local-dir pretrained_models/SoulX-Singer
.venv/bin/hf download Soul-AILab/SoulX-Singer-Preprocess --local-dir pretrained_models/SoulX-Singer-Preprocess
.venv/bin/python -m cli.inference --device cuda --control score \
--model_path pretrained_models/SoulX-Singer/model.pt --config soulxsinger/config/soulxsinger.yaml \
--prompt_wav_path prompt/vocal.wav --prompt_metadata_path prompt/metadata.json \
--target_metadata_path song/lead.json --phoneset_path soulxsinger/utils/phoneme/phone_set.json \
--save_dir out/take1
ACE-Step 1.5, one guide rap in the 6 GB tier:
cd .. && git clone https://github.com/ACE-Step/ACE-Step-1.5.git && cd ACE-Step-1.5 && git checkout ca1e85f && uv sync
Save this as guide.py inside ACE-Step-1.5, with your lyrics in rap.txt beside it, and run it with uv run python guide.py:
from acestep.handler import AceStepHandler
from acestep.llm_inference import LLMHandler
from acestep.inference import GenerationParams, GenerationConfig, generate_music
dit = AceStepHandler()
dit.initialize_service(project_root=".", config_path="acestep-v15-turbo", device="cuda",
offload_to_cpu=True, offload_dit_to_cpu=True, quantization="int8_weight_only")
p = GenerationParams(caption="boom bap hip hop, solo male rapper, clear diction, dry upfront vocal",
lyrics=open("rap.txt").read(), vocal_language="en", bpm=90, duration=30,
thinking=False, use_cot_metas=False, use_cot_caption=False, use_cot_language=False, seed=3)
generate_music(dit, LLMHandler(), p, GenerationConfig(batch_size=1, audio_format="wav",
use_random_seed=False, seeds=[3]), save_dir="guide")
Separation and pitch, with SoulX's own preprocessing tools: back in SoulX-Singer with its venv, copy the wav ACE-Step wrote to guide/seed3.wav (cd ../SoulX-Singer && mkdir -p guide, then copy the file from ../ACE-Step-1.5/guide/):
import soundfile as sf
from preprocess.tools import VocalSeparator, F0Extractor
P = "pretrained_models/SoulX-Singer-Preprocess/"
sep = VocalSeparator(
sep_model_path=P + "mel-band-roformer-karaoke/mel_band_roformer_karaoke_becruily.ckpt",
sep_config_path=P + "mel-band-roformer-karaoke/config_karaoke_becruily.yaml",
der_model_path=P + "dereverb_mel_band_roformer/dereverb_mel_band_roformer_anvuew_sdr_19.1729.ckpt",
der_config_path=P + "dereverb_mel_band_roformer/dereverb_mel_band_roformer_anvuew.yaml", device="cuda")
r = sep.process("guide/seed3.wav")
sf.write("guide/vocal.wav", r.vocals.T, r.sample_rate)
sf.write("guide/beat.wav", r.accompaniment.T, r.sample_rate)
f0 = F0Extractor(model_path=P + "rmvpe/rmvpe.pt", device="cuda")
for w in ("guide/vocal.wav", "prompt/vocal.wav"):
f0.process(w, f0_path=w.replace(".wav", "_f0.npy"))
SoulX SVC, the separated guide vocal in your voice:
.venv/bin/python -m cli.inference_svc --device cuda --auto_shift \
--model_path pretrained_models/SoulX-Singer/model-svc.pt --config soulxsinger/config/soulxsinger.yaml \
--prompt_wav_path prompt/vocal.wav --prompt_f0_path prompt/vocal_f0.npy \
--target_wav_path guide/vocal.wav --target_f0_path guide/vocal_f0.npy --save_dir out/svc
faster-whisper, one line or one window:
from faster_whisper import WhisperModel
m = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, _ = m.transcribe("line2.wav", language="en", beam_size=5, condition_on_previous_text=False)
heard = " ".join(s.text.strip() for s in segments)
Want to watch the next song get made, take by take? See how we run it, from the inside, at neomanex.com/join.
Steps
Record a 10-second voice prompt
Use about 10 seconds of clean speech in your voice, cut to the first 10 seconds. Build a 24 kHz mono prompt with a pitch file for SVC and a SoulX-preprocessed prompt for score mode.
Write the melody as notes
Write one token per note with a MIDI pitch and a length in beats. Join syllables across notes with ~ and end a held melisma with _, and keep a male voice between D3 and D4.
Sing four takes and let Whisper pick each line
Sing four takes with SoulX-Singer in score mode. Whisper large-v3 transcribes every line of every take, and the take closest to the lyric wins each line.
Generate a guide rap with ACE-Step 1.5
Write a caption that asks for a solo rapper, clear diction and a dry upfront vocal. Run the 2B turbo model with no language model, INT8 weights and CPU offload, one seed per guide.
Re-sing the guide in your voice with SoulX SVC
Split the guide vocal from the beat and extract pitch. Run SoulX-Singer SVC with auto shift and lay your new vocal back over the beat.
Score every seed, line by line
Transcribe the converted vocal in 5-second windows snapped to silence, align the words to the lyric sheet and count errors per line. Rule out any seed with an unsafe mishearing, then pick by score and by ear.
Mix to -14 LUFS and label it as AI-made
High-pass the vocal at 90 Hz, keep effects light and normalize to -14 LUFS with a true peak of -1.5 dBTP. Listen to every flagged line and disclose the upload as AI-generated music.
Frequently asked questions
What is AI voice clone singing?
AI voice clone singing is a technique that makes open-source models sing a melody or a guide song in your own cloned voice, using about 10 seconds of speech. A written melody goes to SoulX-Singer score mode, which sings your notes. A rap goes through an ACE-Step 1.5 guide song that SoulX-Singer SVC re-sings in your voice with the same timing. No per-voice training is needed.
Who is open-source AI voice clone singing for?
Open-source AI voice clone singing is for makers, creators and small studios with a consumer NVIDIA GPU of 6 GB or more who want jingles, intros or short songs in their own voice. SoulX-Singer, ACE-Step 1.5 and Whisper all run locally on that card, and Whisper checks every sung line against the lyric sheet before anything is published.
How do I make AI sing a song in my own voice for free?
Record about 10 seconds of clean speech in your voice. Then write the melody as notes, one token per note, and sing four takes with SoulX-Singer in score mode. Whisper large-v3 transcribes every line of every take, and the take closest to the lyric wins each line. Every model is open source and runs on your own GPU. Use only your own voice or one with written consent.
Do I need to train a model on my voice to make it sing?
No. SoulX-Singer is zero-shot: it reads a 10-second voice prompt at inference time and trains nothing per voice. Training an RVC model is the other route, and the RVC project recommends at least 10 minutes of low-noise speech plus a training run for each voice. Zero-shot is faster to set up, and a trained RVC model stays the alternative when you have the recordings.
Can AI rap in my own voice with open-source models?
Yes. ACE-Step 1.5 generates a guide rap from your lyrics and a style caption, and SoulX-Singer SVC sings that guide again in your voice with the same timing. SVC keeps the flow and blurs hard consonants, so every seed comes out differently. Generate several seeds, score each one line by line with Whisper large-v3, and pick by score and then by ear.
Can ACE-Step 1.5 and SoulX-Singer run on a 6 GB GPU?
Yes, on an RTX 3060 Laptop GPU with 6 GB. ACE-Step 1.5 runs the 2B turbo DiT with no language model, INT8 weight-only quantization and CPU offload. SoulX-Singer and Whisper large-v3 (float16) each load alone, because only one GPU job runs at a time. Before each load, the card stays under 600 MiB for 45 seconds. Other cards are untested.
Can I clone someone else's voice to make it sing?
Only with written consent from the voice's owner. AI voice clone singing works on any 10-second prompt, so consent is the rule that limits it: clone your own voice, or one whose owner agreed in writing, and never a public figure's voice without that agreement. When you publish the song, YouTube asks creators to disclose AI-generated music with the AI use setting at upload.
How do I know the AI sang the right words?
Transcribe every take with Whisper large-v3 and compare it to the lyric sheet. For a written melody, Whisper reads each line alone and the closest take wins the line. For a rap, Whisper reads 5-second windows snapped to silence, and every error is charged to its lyric line. Lines where Whisper hears profanity are flagged, so listen to each one before publishing.
