Skip to content

C
coderai
Commit 3580ff1d authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files
Options

feat: add full-quality audio ml pipeline scaffolding

parent 86419f7d
Hide whitespace changes
Inline Side-by-side
Showing with 1520 additions and 37 deletions
README.md
View file @ 3580ff1d
......@@ -123,6 +123,23 @@ pip install f5-tts # Voice cloning (F5-TTS)
pip install seed-vc # Voice conversion / singing SVC
```
### Full-Quality Audio ML Stack
```bash
pip install demucs deepfilternet rnnoise voicefixer
```
Use this stack when you want:
- real ML stem separation for `/v1/audio/stems`
- learned restoration for `/v1/audio/cleanup`
- the strongest available backend path for `/v1/pipelines/audio-music-dub`
Notes:
- `demucs` is the primary separator for vocals/instrumental and multi-stem workflows.
- `deepfilternet` is the primary learned cleanup backend.
- `rnnoise` and `voicefixer` are optional alternates / complements.
- Full music-dub quality depends on separation plus singing-capable conversion; even with this stack, output quality still depends heavily on source material and model/runtime availability.
### Face Swap
```bash
......
codai/admin/templates/chat.html
View file @ 3580ff1d
......@@ -331,6 +331,7 @@ a.dl { display:inline-block; margin-top:.4rem; }
<div class="frow"><label class="fl" title="Removes the model's built-in safety checker. Only use with uncensored/NSFW fine-tunes.">Disable safety filter</label><input type="checkbox" id="ig-nosafe"></div>
<button class="btn btn-primary" onclick="genImage()">Generate</button>
<div class="progress" id="ig-prog"></div>
<div class="req-preview" id="ig-preview"></div>
</div>
<div class="gen-out" id="ig-out"><div class="gen-empty">Image will appear here</div></div>
</div>
......@@ -812,6 +813,7 @@ a.dl { display:inline-block; margin-top:.4rem; }
</div>
<button class="btn btn-primary" onclick="genTTS()">Synthesize</button>
<div class="progress" id="at-prog"></div>
<div class="req-preview" id="at-preview"></div>
</div>
<div class="gen-out" id="at-out"><div class="gen-empty">Audio will appear here</div></div>
</div>
......@@ -917,6 +919,7 @@ a.dl { display:inline-block; margin-top:.4rem; }
</div>
<button class="btn btn-primary" onclick="genSTT()">Transcribe</button>
<div class="progress" id="as-prog"></div>
<div class="req-preview" id="as-preview"></div>
</div>
<div class="gen-out" id="as-out">
<div class="gen-out-inner"><div class="gen-empty">Transcript will appear here</div></div>
......@@ -936,13 +939,15 @@ a.dl { display:inline-block; margin-top:.4rem; }
</div>
<div class="frow"><label class="fl">Notes</label><textarea id="amd-notes" class="fs" rows="3" placeholder="Explain whether the priority is lyric translation, singability, or preserving the original arrangement."></textarea></div>
<div class="cap-preserve-note">
<strong>Current studio behavior:</strong> there is no dedicated song-dubbing request path wired here. Use this panel to assess whether the selected model exposes enough speech, subtitle, or voice capabilities for a manual workflow.
<strong>Current studio behavior:</strong> this panel can now run transcript-first intake through existing speech-to-text primitives, then return a truthful partial workflow plan for lyric localization.
</div>
<div class="cap-note"><strong>Fallback path:</strong> transcribe lyrics, translate them, then resynthesize guide vocals or use Voice Convert on isolated vocals before recombining outside this panel. Rhythm-safe lyric adaptation and automatic mix replacement are not exposed here.</div>
<div class="progress" id="amd-prog">No direct backend action from this panel.</div>
<div class="cap-note"><strong>Fallback path:</strong> transcribe lyrics, translate/adapt them, then resynthesize guide vocals or use Voice Convert on isolated vocals before recombining outside this panel. Rhythm-safe lyric adaptation and automatic mix replacement are still not exposed here.</div>
<button class="btn btn-primary" onclick="genMusicDubPlan()">Analyze Music Dub Path</button>
<div class="progress" id="amd-prog"></div>
<div class="req-preview" id="amd-preview"></div>
</div>
<div class="gen-out" id="amd-out">
<div class="gen-out-inner"><div class="gen-empty">Music dubbing is surfaced as planning guidance only until a dedicated backend route exists.</div></div>
<div class="gen-out-inner"><div class="gen-empty">Transcript, limitations, and next-step guidance will appear here.</div></div>
</div>
</div>
</div>
......@@ -955,13 +960,16 @@ a.dl { display:inline-block; margin-top:.4rem; }
<div class="frow"><label class="fl">Source audio / video</label><input type="file" id="au-file" accept="audio/*,video/*" class="fi"></div>
<div class="frow"><label class="fl">Analysis goal</label><textarea id="au-goal" class="fs" rows="4" placeholder="Examples: summarize the meeting, extract speakers and topics, describe the soundscape, identify notable events."></textarea></div>
<div class="cap-preserve-note">
<strong>Current studio behavior:</strong> understanding depends on whether the active model can at least transcribe audio. Rich semantic answering over uploaded audio is not a separate API surface in this template.
<strong>Current studio behavior:</strong> understanding runs as a transcript-first composed request. If a text/chat model is also selected, Studio can ask it to summarize or reason over the transcript.
</div>
<div class="cap-note"><strong>Fallback path:</strong> use Audio → Transcribe first, then send the transcript into Chat for summarization, extraction, or follow-up reasoning. If the model lacks speech-to-text, this remains unavailable here.</div>
<div class="progress" id="au-prog">Use Transcribe + Chat as the supported path.</div>
<div class="cap-note"><strong>Fallback path:</strong> if no text/chat model is selected, this panel still returns the transcript so you can continue reasoning manually in Chat.</div>
<div class="frow"><label class="fl">Reasoning / summary model <span style="font-weight:400">(optional)</span></label><input id="au-text-model" class="fi" placeholder="model id for chat summarization"></div>
<button class="btn btn-primary" onclick="genAudioUnderstand()">Analyze Audio</button>
<div class="progress" id="au-prog"></div>
<div class="req-preview" id="au-preview"></div>
</div>
<div class="gen-out" id="au-out">
<div class="gen-out-inner"><div class="gen-empty">No standalone audio-understanding request is sent from this panel.</div></div>
<div class="gen-out-inner"><div class="gen-empty">Transcript-backed understanding will appear here.</div></div>
</div>
</div>
</div>
......@@ -980,13 +988,15 @@ a.dl { display:inline-block; margin-top:.4rem; }
</select>
</div>
<div class="cap-preserve-note">
<strong>Current studio behavior:</strong> no source-separation endpoint is wired into this page. This panel is an availability shell so users can see that stem workflows are recognized but not directly executable here.
<strong>Native backend:</strong> Studio now sends a real <code>/v1/audio/stems</code> request. The current backend uses ffmpeg-only heuristics, so outputs are best-effort estimates rather than production-grade ML demixing.
</div>
<div class="cap-note"><strong>Relevant pipeline:</strong> if you need dubbed singing or cleanup, stem separation would normally happen before Voice Convert or remix assembly. That ordering is informational only in this UI.</div>
<div class="progress" id="ast-prog">Unavailable without a dedicated separation backend.</div>
<div class="cap-note"><strong>Quality note:</strong> vocals/instrumental works best when vocals are center-panned. The 4-stem and drums/bass/other modes are broad frequency-group splits, not clean source-isolated stems.</div>
<button class="btn btn-primary" onclick="genStems()">Split Stems</button>
<div class="progress" id="ast-prog"></div>
<div class="req-preview" id="ast-preview"></div>
</div>
<div class="gen-out" id="ast-out">
<div class="gen-out-inner"><div class="gen-empty">Stem files are not generated from this template today.</div></div>
<div class="gen-out-inner"><div class="gen-empty">Separated stem artifacts will appear here.</div></div>
</div>
</div>
</div>
......@@ -1006,13 +1016,15 @@ a.dl { display:inline-block; margin-top:.4rem; }
<label class="cap-toggle"><input type="checkbox" id="ac-click"> Repair clicks</label>
</div>
<div class="cap-preserve-note">
<strong>Current studio behavior:</strong> cleanup and restoration are not exposed as native requests on this page. Some speech-to-text capable models can still help inspect a noisy recording by transcribing it first.
<strong>Native backend:</strong> Studio now sends a real <code>/v1/audio/cleanup</code> request. The current backend chains standard ffmpeg filters, so it can improve many recordings but is not a substitute for deep restoration models.
</div>
<div class="cap-note"><strong>Fallback path:</strong> use Transcribe to judge intelligibility, then perform denoise/restoration outside Studio before returning for transcription, dubbing, or voice conversion.</div>
<div class="progress" id="ac-prog">No cleanup request is sent from this panel.</div>
<div class="cap-note"><strong>Fallback path:</strong> if the result is still not intelligible, use Transcribe to assess what survived and switch to dedicated external restoration before dubbing or conversion.</div>
<button class="btn btn-primary" onclick="genAudioCleanup()">Clean Audio</button>
<div class="progress" id="ac-prog"></div>
<div class="req-preview" id="ac-preview"></div>
</div>
<div class="gen-out" id="ac-out">
<div class="gen-out-inner"><div class="gen-empty">Audio cleanup is presented as an unsupported surface with guidance only.</div></div>
<div class="gen-out-inner"><div class="gen-empty">Cleaned audio and backend limitations will appear here.</div></div>
</div>
</div>
</div>
......@@ -1372,6 +1384,7 @@ a.dl { display:inline-block; margin-top:.4rem; }
<div class="frow"><label class="fl">Dimensions (truncate, optional)</label><input type="number" id="em-dims" class="fi" placeholder="all"></div>
<button class="btn btn-primary" onclick="genEmbeddings()">Embed</button>
<div class="progress" id="em-prog"></div>
<div class="req-preview" id="em-preview"></div>
</div>
<div class="gen-out" id="em-out"><div class="gen-empty">Embedding vectors will appear here</div></div>
</div>
......@@ -1495,28 +1508,28 @@ const STUDIO_CAPABILITIES = {
'aud-stems': {
category:'audio',
label:'Stem separation',
summary:'Show source-separation readiness while clearly marking that this Studio page does not expose a stem-splitting route.',
requires:['audio_generation'],
optional:['speech_to_text'],
summary:'Run native ffmpeg-backed stem estimation while clearly signaling that the current backend is heuristic and quality-limited.',
requires:[],
optional:['audio_generation','speech_to_text'],
notes:[
'Generation capability can support later recomposition, but it does not imply source separation support.',
'Use this as a planning surface for dubbing or remix workflows, not as a runnable endpoint.'
'This endpoint is runnable, but it performs best-effort estimation rather than ML source separation.',
'Center-panned vocals separate more cleanly than dense or heavily stereo-processed mixes.'
],
backendPath:'unsupported shell',
io:'Input: mixed audio. Output: no direct split stems from this UI.'
backendPath:'/v1/audio/stems',
io:'Input: mixed audio. Output: downloadable stem estimate artifacts.'
},
'aud-clean': {
category:'audio',
label:'Audio cleanup',
summary:'Expose denoise and restoration as recognized needs while keeping unsupported repair operations honest.',
requires:['speech_to_text'],
optional:['audio_generation'],
summary:'Run native ffmpeg filter-chain cleanup for denoise and basic restoration while keeping quality and backend limits explicit.',
requires:[],
optional:['speech_to_text'],
notes:[
'Transcription can help judge intelligibility even when restoration is unavailable.',
'Noise removal, de-click, de-hum, and mastering remain external steps unless a dedicated cleanup backend is added.'
'This endpoint is runnable, but it uses standard filter processing rather than deep restoration models.',
'Transcription remains the fastest fallback for judging intelligibility when cleanup cannot fully recover the source.'
],
backendPath:'inspection only',
io:'Input: noisy audio. Output: guidance and fallback recommendations only.'
backendPath:'/v1/audio/cleanup',
io:'Input: noisy audio. Output: cleaned audio artifact plus applied-operation metadata.'
}
};
const SUB_CAPABILITY_RULES = {
......@@ -1557,6 +1570,30 @@ const CATEGORY_SUBS = {
audio: Object.keys(SUB_CAPABILITY_RULES).filter(sub => SUB_CAT[sub] === 'audio'),
};
let currentTabState = { categories:{}, subs:{} };
let audioBackendHealth = {
separation: { available:false, engine:null, model:null },
restoration: { available:false, engine:null, model:null },
musicDub: { available:false, stages:[] },
};
function refreshAudioBackendHealth() {
audioBackendHealth = {
separation: {
available: !!(activeModel?.capabilities || []).includes('audio_to_audio'),
engine: !!(activeModel?.capabilities || []).includes('audio_to_audio') ? 'ml-provider' : null,
model: activeModel?.id || null,
},
restoration: {
available: !!(activeModel?.capabilities || []).includes('audio_to_audio'),
engine: !!(activeModel?.capabilities || []).includes('audio_to_audio') ? 'ml-provider' : null,
model: activeModel?.id || null,
},
musicDub: {
available: !!(activeModel?.capabilities || []).includes('speech_to_text') && !!(activeModel?.capabilities || []).includes('audio_to_audio'),
stages: ['stems', 'stt', 'translate', 'voice_convert', 'remix'],
},
};
}
function capabilitySetForModel(m) {
return new Set((m?.capabilities || []).filter(Boolean));
......@@ -1675,6 +1712,7 @@ function renderCapabilityCard(sub) {
function renderCapabilityCards() {
Object.keys(STUDIO_CAPABILITIES).forEach(renderCapabilityCard);
renderAudioBackendHealth();
}
function summarizeDiagnostics() {
......@@ -1694,6 +1732,23 @@ function summarizeDiagnostics() {
};
}
function renderAudioBackendHealth() {
const targets = {
'aud-music-dub': audioBackendHealth.musicDub,
'aud-stems': audioBackendHealth.separation,
'aud-clean': audioBackendHealth.restoration,
};
Object.entries(targets).forEach(([sub, info]) => {
const shell = $(`cap-${sub}`);
if (!shell) return;
const extra = [];
if (info?.engine) extra.push(`<div class="cap-note"><strong>Runtime engine:</strong> ${escapeHtml(info.engine)}</div>`);
if (info?.model) extra.push(`<div class="cap-note"><strong>Runtime model:</strong> ${escapeHtml(info.model)}</div>`);
if (Array.isArray(info?.stages) && info.stages.length) extra.push(`<div class="cap-note"><strong>Stages:</strong> ${info.stages.map(escapeHtml).join(' → ')}</div>`);
if (extra.length) shell.insertAdjacentHTML('beforeend', extra.join(''));
});
}
function renderDiagnostics() {
const shell = $('diag-groups');
if (!shell) return;
......@@ -1786,6 +1841,24 @@ function buildSTTHistorySummary(text, fileName) {
return `${fileName || 'uploaded file'} · ${compactSummary(text, 100)}`;
}
function buildStemHistorySummary(mode, count, backend) {
const engine = backend?.engine || 'unknown';
const model = backend?.model ? ` · ${backend.model}` : '';
return `${mode} · ${count} artifact${count === 1 ? '' : 's'} · ${engine}${model}`;
}
function buildCleanupHistorySummary(applied, fileName, backend) {
const engine = backend?.engine || 'unknown';
const model = backend?.model ? ` · ${backend.model}` : '';
return `${fileName || 'uploaded file'} · ${(applied || []).join(', ') || 'no ops'} · ${engine}${model}`;
}
function buildMusicDubHistorySummary(transcript, translatedLyrics, backend) {
const engine = backend?.engine || 'pipeline';
const model = backend?.model ? ` · ${backend.model}` : '';
return `${compactSummary(transcript, 50)} · ${compactSummary(translatedLyrics, 50)} · ${engine}${model}`;
}
function buildDubFamilyLabel() {
return val('vd-track-mode') === 'localized-mix' ? 'video dub / localized mix' : 'video dub / speech-only';
}
......@@ -1861,6 +1934,91 @@ function buildAudioPreviewData() {
});
}
function buildTTSPreviewData() {
return previewExportBody('/v1/audio/speech', {
model: activeModel?.id || '',
input: val('at-text'),
voice: val('at-voice') || undefined,
speed: fval('at-speed') || 1.0,
response_format: 'mp3',
});
}
function buildSTTPreviewData() {
return previewExportBody('/v1/audio/transcriptions', {
model: activeModel?.id || '',
file: fileOrNull('as-file') ? '<multipart audio/video file>' : undefined,
language: val('as-lang') || undefined,
prompt: val('as-prompt') || undefined,
response_format: 'json',
});
}
function buildImageGenPreviewData() {
return previewExportBody('/v1/images/generations', {
model: activeModel?.id || '',
prompt: val('ig-prompt'),
negative_prompt: val('ig-neg') || undefined,
size: `${ival('ig-w') || 1024}x${ival('ig-h') || 1024}`,
steps: ival('ig-steps') || 30,
guidance_scale: fval('ig-cfg') || 7.5,
seed: val('ig-seed') ? ival('ig-seed') : undefined,
n: ival('ig-n') || 1,
response_format: 'url',
safety_checker: chk('ig-nosafe') ? false : undefined,
});
}
function buildEmbeddingsPreviewData() {
const lines = val('em-text').split('\n').filter(l => l.trim());
const input = lines.length <= 1 ? (lines[0] || '') : lines;
return previewExportBody('/v1/embeddings', {
model: activeModel?.id || '',
input,
encoding_format: val('em-enc') || 'float',
dimensions: val('em-dims') ? ival('em-dims') : undefined,
});
}
function buildAudioUnderstandPreviewData() {
return previewExportBody('/v1/pipelines/audio-understand', {
audio: fileOrNull('au-file') ? '<audio/video file data>' : undefined,
audio_model: activeModel?.id || '',
text_model: val('au-text-model') || undefined,
input: val('au-goal') || undefined,
language: val('as-lang') || undefined,
});
}
function buildMusicDubPreviewData() {
return previewExportBody('/v1/pipelines/audio-music-dub', {
audio: fileOrNull('amd-file') ? '<audio/video file data>' : undefined,
audio_model: activeModel?.id || '',
source_lang: val('amd-slang') || undefined,
target_lang: val('amd-tlang') || undefined,
notes: val('amd-notes') || undefined,
});
}
function buildStemPreviewData() {
return previewExportBody('/v1/audio/stems', {
audio: fileOrNull('ast-file') ? '<audio/video file data>' : undefined,
stem_mode: val('ast-mode') || 'vocals-instrumental',
response_format: 'url',
});
}
function buildCleanupPreviewData() {
return previewExportBody('/v1/audio/cleanup', {
audio: fileOrNull('ac-file') ? '<audio/video file data>' : undefined,
noise_reduction: chk('ac-noise'),
normalize: chk('ac-level'),
remove_hum: chk('ac-hum'),
repair_clicks: chk('ac-click'),
response_format: 'url',
});
}
function buildDubPreviewData() {
const preview = buildDubPreferencePreview();
return previewExportBody('/v1/video/dub', {
......@@ -1947,6 +2105,74 @@ const REQUEST_PREVIEW_CONFIG = {
{ label:'Seed', value:preview => preview.body.seed ?? 'random' },
],
},
'aud-tts': {
containerId:'at-preview',
snippet:'curl',
build:buildTTSPreviewData,
fields:[
{ label:'Model', value:preview => preview.body.model || 'none selected' },
{ label:'Voice', value:preview => preview.body.voice || 'default' },
{ label:'Speed', value:preview => preview.body.speed },
{ label:'Text', value:preview => preview.body.input || 'empty' },
],
},
'aud-stt': {
containerId:'as-preview',
snippet:'curl',
build:buildSTTPreviewData,
fields:[
{ label:'Model', value:preview => preview.body.model || 'none selected' },
{ label:'File', value:() => previewFileName('as-file') },
{ label:'Language', value:preview => preview.body.language || 'auto' },
{ label:'Hint', value:preview => preview.body.prompt || 'none' },
],
},
'img-gen': {
containerId:'ig-preview',
snippet:'curl',
build:buildImageGenPreviewData,
fields:[
{ label:'Model', value:preview => preview.body.model || 'none selected' },
{ label:'Prompt', value:preview => preview.body.prompt || 'empty' },
{ label:'Size', value:preview => preview.body.size },
{ label:'Steps', value:preview => preview.body.steps },
{ label:'CFG', value:preview => preview.body.guidance_scale },
{ label:'Count', value:preview => preview.body.n },
],
},
'embed': {
containerId:'em-preview',
snippet:'curl',
build:buildEmbeddingsPreviewData,
fields:[
{ label:'Model', value:preview => preview.body.model || 'none selected' },
{ label:'Items', value:preview => Array.isArray(preview.body.input) ? preview.body.input.length : (preview.body.input ? 1 : 0) },
{ label:'Encoding', value:preview => preview.body.encoding_format },
{ label:'Dimensions', value:preview => preview.body.dimensions ?? 'full' },
],
},
'aud-understand': {
containerId:'au-preview',
snippet:'curl',
build:buildAudioUnderstandPreviewData,
fields:[
{ label:'Audio model', value:preview => preview.body.audio_model || 'none selected' },
{ label:'Reasoning model', value:preview => preview.body.text_model || 'transcript only' },
{ label:'File', value:() => previewFileName('au-file') },
{ label:'Goal', value:preview => preview.body.input || 'general understanding' },
],
},
'aud-music-dub': {
containerId:'amd-preview',
snippet:'curl',
build:buildMusicDubPreviewData,
fields:[
{ label:'Audio model', value:preview => preview.body.audio_model || 'none selected' },
{ label:'File', value:() => previewFileName('amd-file') },
{ label:'Source lang', value:preview => preview.body.source_lang || 'auto' },
{ label:'Target lang', value:preview => preview.body.target_lang || 'none' },
],
},
'vid-dub': {
containerId:'vd-preview',
snippet:'curl',
......@@ -1969,20 +2195,44 @@ function updateRequestPreview(panel) {
}
function initRequestPreviews() {
['ag-prompt','ag-dur','ag-temp','ag-topk','ag-cfg','ag-melody','ag-seed','vd-src','vd-slang','vd-tlang','vd-track-mode','vd-burn','vd-preserve-voice','vd-preserve-tone','vd-preserve-timing','vd-preserve-singing','vd-lyrics']
['ag-prompt','ag-dur','ag-temp','ag-topk','ag-cfg','ag-melody','ag-seed','at-text','at-voice','at-speed','as-file','as-lang','as-prompt','ig-prompt','ig-neg','ig-w','ig-h','ig-steps','ig-cfg','ig-seed','ig-n','ig-nosafe','em-text','em-enc','em-dims','au-file','au-goal','au-text-model','amd-file','amd-slang','amd-tlang','amd-notes','vd-src','vd-slang','vd-tlang','vd-track-mode','vd-burn','vd-preserve-voice','vd-preserve-tone','vd-preserve-timing','vd-preserve-singing','vd-lyrics']
.forEach(id => {
const el = $(id);
if (!el) return;
el.addEventListener('input', () => {
if (id.startsWith('ag-')) updateRequestPreview('aud-gen');
if (id.startsWith('at-')) updateRequestPreview('aud-tts');
if (id.startsWith('as-')) updateRequestPreview('aud-stt');
if (id.startsWith('ig-')) updateRequestPreview('img-gen');
if (id.startsWith('em-')) updateRequestPreview('embed');
if (id.startsWith('au-')) updateRequestPreview('aud-understand');
if (id.startsWith('amd-')) updateRequestPreview('aud-music-dub');
if (id.startsWith('ast-')) updateRequestPreview('aud-stems');
if (id.startsWith('ac-')) updateRequestPreview('aud-clean');
if (id.startsWith('vd-')) updateRequestPreview('vid-dub');
});
el.addEventListener('change', () => {
if (id.startsWith('ag-')) updateRequestPreview('aud-gen');
if (id.startsWith('at-')) updateRequestPreview('aud-tts');
if (id.startsWith('as-')) updateRequestPreview('aud-stt');
if (id.startsWith('ig-')) updateRequestPreview('img-gen');
if (id.startsWith('em-')) updateRequestPreview('embed');
if (id.startsWith('au-')) updateRequestPreview('aud-understand');
if (id.startsWith('amd-')) updateRequestPreview('aud-music-dub');
if (id.startsWith('ast-')) updateRequestPreview('aud-stems');
if (id.startsWith('ac-')) updateRequestPreview('aud-clean');
if (id.startsWith('vd-')) updateRequestPreview('vid-dub');
});
});
updateRequestPreview('aud-gen');
updateRequestPreview('aud-tts');
updateRequestPreview('aud-stt');
updateRequestPreview('img-gen');
updateRequestPreview('embed');
updateRequestPreview('aud-understand');
updateRequestPreview('aud-music-dub');
updateRequestPreview('aud-stems');
updateRequestPreview('aud-clean');
updateRequestPreview('vid-dub');
renderDiagnostics();
renderArtifactHistory();
......@@ -2040,6 +2290,14 @@ function selectModel(m) {
chatHistory = []; attachedImage = null; updateAttachBar();
$('chat-msgs').innerHTML = `<div class="chat-empty"><h3>${m.id.split('/').pop()}</h3><p>Start below</p></div>`;
updateRequestPreview('aud-gen');
updateRequestPreview('aud-tts');
updateRequestPreview('aud-stt');
updateRequestPreview('img-gen');
updateRequestPreview('embed');
updateRequestPreview('aud-understand');
updateRequestPreview('aud-music-dub');
updateRequestPreview('aud-stems');
updateRequestPreview('aud-clean');
updateRequestPreview('vid-dub');
// Update chat input placeholder to hint at routing for non-text models
const PLACEHOLDER = {
......@@ -2057,6 +2315,7 @@ function selectModel(m) {
function updateTabs(m) {
const caps = capabilitySetForModel(m);
const type = m.type || 'text';
refreshAudioBackendHealth();
const subStates = {};
Object.entries(SUB_CAPABILITY_RULES).forEach(([sub, rule]) => {
if (VIDEO_EXTRA_SUBS.includes(sub) && type === 'video' && !rule.fallbackTypes) {
......@@ -3197,6 +3456,74 @@ async function genSTT() {
} catch(e) { $('as-prog').textContent='Error: '+e.message; }
}
async function genStems() {
const f = fileOrNull('ast-file');
if (!f) { $('ast-prog').textContent='Select a source mix.'; return; }
$('ast-prog').textContent='Separating stems…';
try {
const body = {
audio: await fileToB64(f),
stem_mode: val('ast-mode') || 'vocals-instrumental',
response_format: 'url',
};
const d = await post('/v1/audio/stems', body);
const items = d.data || [];
$('ast-out').innerHTML = `<div class="gen-out-inner" style="width:100%;text-align:left">${items.map(item => {
const src = audSrc(item);
return `<div style="width:100%;background:var(--surface-2);border:1px solid var(--border);border-radius:8px;padding:.75rem;margin-bottom:.6rem;box-sizing:border-box">
<div style="font-size:12px;font-weight:600;margin-bottom:.35rem">${escapeHtml(item.name || 'stem')} · ${escapeHtml(item.role || 'artifact')}</div>
<audio class="out-audio" controls src="${src}"></audio>
<div style="font-size:11px;color:var(--text-3);margin-top:.35rem">${escapeHtml((d.limitations || []).join(' · '))}</div>
<a href="${src}" download="${escapeHtml(item.name || 'stem')}.wav" class="btn btn-ghost btn-sm dl">Download</a>
</div>`;
}).join('')}</div>`;
pushArtifactHistory({
task:'Stem separation',
family:'audio stems',
model:d.backend?.engine || 'ffmpeg',
summary:buildStemHistorySummary(d.stem_mode, items.length, d.backend),
links:buildMultiArtifactLinks(items),
});
$('ast-prog').textContent=`Done ✓ ${d.backend?.quality || 'best-effort'}`;
} catch(e) { $('ast-prog').textContent='Error: '+e.message; }
}
async function genAudioCleanup() {
const f = fileOrNull('ac-file');
if (!f) { $('ac-prog').textContent='Select a source recording.'; return; }
$('ac-prog').textContent='Cleaning audio…';
try {
const body = {
audio: await fileToB64(f),
noise_reduction: chk('ac-noise'),
normalize: chk('ac-level'),
remove_hum: chk('ac-hum'),
repair_clicks: chk('ac-click'),
response_format: 'url',
};
const d = await post('/v1/audio/cleanup', body);
const item = d.data?.[0];
const src = item ? audSrc(item) : null;
$('ac-out').innerHTML = `<div class="gen-out-inner" style="width:100%;text-align:left">
<audio class="out-audio" controls src="${src}"></audio>
<div style="font-size:12px;color:var(--text-2);width:100%;background:var(--surface-2);border:1px solid var(--border);border-radius:8px;padding:.7rem;box-sizing:border-box">
<strong>Applied:</strong> ${escapeHtml((d.applied || []).join(', ') || 'none')}<br>
<strong>Backend:</strong> ${escapeHtml(d.backend?.engine || 'ffmpeg')} · ${escapeHtml(d.backend?.quality || 'best-effort')}<br>
<strong>Limitations:</strong> ${escapeHtml((d.limitations || []).join(' · '))}
</div>
<a href="${src}" download="cleaned.wav" class="btn btn-ghost btn-sm dl">Download</a>
</div>`;
pushArtifactHistory({
task:'Audio cleanup',
family:'audio cleanup',
model:d.backend?.engine || 'ffmpeg',
summary:buildCleanupHistorySummary(d.applied, f.name, d.backend),
links: src ? [{ label:'Open', href:src }] : [],
});
$('ac-prog').textContent=`Done ✓ ${d.backend?.quality || 'best-effort'}`;
} catch(e) { $('ac-prog').textContent='Error: '+e.message; }
}
// ─────────────────────────────────────────────────────────────────
// Embeddings
// ─────────────────────────────────────────────────────────────────
......@@ -3223,6 +3550,55 @@ async function genEmbeddings() {
} catch(e) { $('em-prog').textContent='Error: '+e.message; }
}
async function genAudioUnderstand() {
if (!activeModel) return;
const f = fileOrNull('au-file');
if (!f) { $('au-prog').textContent='Select an audio or video file.'; return; }
$('au-prog').textContent='Analyzing audio…';
try {
const d = await post('/v1/pipelines/audio-understand', {
audio: await fileToB64(f),
audio_model: activeModel.id,
...(val('au-text-model') ? {text_model: val('au-text-model')} : {}),
...(val('au-goal') ? {input: val('au-goal')} : {}),
});
$('au-out').innerHTML = `<div class="gen-out-inner" style="width:100%;text-align:left;gap:.75rem">
<div style="width:100%"><strong>Transcript</strong><pre style="white-space:pre-wrap;font-size:13px;line-height:1.6;background:var(--surface-2);padding:.75rem;border-radius:6px;width:100%;box-sizing:border-box">${escapeHtml(d.transcript || 'No transcript returned.')}</pre></div>
${d.summary ? `<div style="width:100%"><strong>Summary</strong><pre style="white-space:pre-wrap;font-size:13px;line-height:1.6;background:var(--surface-2);padding:.75rem;border-radius:6px;width:100%;box-sizing:border-box">${escapeHtml(d.summary)}</pre></div>` : ''}
</div>`;
$('au-prog').textContent = d.summary ? 'Done ✓ transcript + reasoning' : 'Done ✓ transcript only';
} catch(e) { $('au-prog').textContent='Error: '+e.message; }
}
async function genMusicDubPlan() {
if (!activeModel) return;
const f = fileOrNull('amd-file');
if (!f) { $('amd-prog').textContent='Select a source song or vocal mix.'; return; }
$('amd-prog').textContent='Inspecting music dub path…';
try {
const d = await post('/v1/pipelines/audio-music-dub', {
audio: await fileToB64(f),
audio_model: activeModel.id,
...(val('amd-slang') ? {source_lang: val('amd-slang')} : {}),
...(val('amd-tlang') ? {target_lang: val('amd-tlang')} : {}),
...(val('amd-notes') ? {notes: val('amd-notes')} : {}),
});
$('amd-out').innerHTML = `<div class="gen-out-inner" style="width:100%;text-align:left;gap:.75rem">
<div style="width:100%"><strong>Transcript</strong><pre style="white-space:pre-wrap;font-size:13px;line-height:1.6;background:var(--surface-2);padding:.75rem;border-radius:6px;width:100%;box-sizing:border-box">${escapeHtml(d.transcript || 'No transcript returned.')}</pre></div>
<div style="width:100%"><strong>Translated lyrics</strong><pre style="white-space:pre-wrap;font-size:13px;line-height:1.6;background:var(--surface-2);padding:.75rem;border-radius:6px;width:100%;box-sizing:border-box">${escapeHtml(d.translated_lyrics || 'No translated lyrics returned.')}</pre></div>
<div style="width:100%"><strong>Final mix</strong><pre style="white-space:pre-wrap;font-size:13px;line-height:1.6;background:var(--surface-2);padding:.75rem;border-radius:6px;width:100%;box-sizing:border-box">${escapeHtml(d.final_mix?.path || 'No final mix path returned.')}</pre></div>
</div>`;
pushArtifactHistory({
task:'Music dub',
family:'audio music dub',
model:d.backend?.model || activeModel.id,
summary:buildMusicDubHistorySummary(d.transcript, d.translated_lyrics, d.backend),
links:d.final_mix?.path ? [{ label:'Final mix', href:d.final_mix.path }] : [],
});
$('amd-prog').textContent='Done ✓ full-stage pipeline';
} catch(e) { $('amd-prog').textContent='Error: '+e.message; }
}
// ─────────────────────────────────────────────────────────────────
// Pipeline 1: Image → Video (server-side)
// ─────────────────────────────────────────────────────────────────
......
codai/api/__init__.py
View file @ 3580ff1d
......@@ -15,6 +15,12 @@
# along with this program. If not, see <https://www.gnu.org/licenses/>.
# codai.api - FastAPI application module
from .app import app
__all__ = ['app']
\ No newline at end of file
__all__ = ['app']
def __getattr__(name):
if name == 'app':
from .app import app
return app
raise AttributeError(name)
codai/api/app.py
View file @ 3580ff1d
......@@ -121,6 +121,8 @@ app.include_router(tts_router)
app.include_router(text_router)
app.include_router(video_router)
app.include_router(audio_gen_router)
app.include_router(audio_stems_router)
app.include_router(audio_clean_router)
app.include_router(embeddings_router)
app.include_router(pipelines_router)
app.include_router(custom_pipelines_router)
......
codai/api/audio_backends.py 0 → 100644
View file @ 3580ff1d
import importlib.util
from functools import lru_cache
def _has_module(name: str) -> bool:
return importlib.util.find_spec(name) is not None
@lru_cache(maxsize=1)
def detect_audio_backends() -> dict:
demucs_ok = _has_module("demucs")
deepfilter_ok = _has_module("df") or _has_module("deepfilternet")
rnnoise_ok = _has_module("rnnoise")
voicefixer_ok = _has_module("voicefixer")
restoration_engine = None
if deepfilter_ok:
restoration_engine = "deepfilternet"
elif rnnoise_ok:
restoration_engine = "rnnoise"
elif voicefixer_ok:
restoration_engine = "voicefixer"
return {
"separation": {
"available": demucs_ok,
"engine": "demucs" if demucs_ok else None,
"candidates": ["demucs"],
},
"restoration": {
"available": bool(restoration_engine),
"engine": restoration_engine,
"candidates": ["deepfilternet", "rnnoise", "voicefixer"],
},
}
def reset_audio_backend_cache() -> None:
detect_audio_backends.cache_clear()
codai/api/audio_clean.py 0 → 100644
View file @ 3580ff1d
import base64
import os
import shutil
import subprocess
import tempfile
import time
import uuid
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, ConfigDict
from codai.api.audio_backends import detect_audio_backends
router = APIRouter()
global_args = None
global_file_path = None
def set_global_args(args):
global global_args
global_args = args
def set_global_file_path(path):
global global_file_path
global_file_path = path
def _decode_audio(data: str) -> bytes:
if data.startswith("data:"):
_, enc = data.split(",", 1)
return base64.b64decode(enc)
return base64.b64decode(data)
def _ffmpeg_binary() -> str:
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
raise HTTPException(status_code=501, detail="ffmpeg is required for native audio cleanup")
return ffmpeg
def _base_url(http_request: Request) -> str:
url_setting = getattr(global_args, "url", "auto") if global_args else "auto"
if url_setting != "auto":
return url_setting.rstrip("/")
host = http_request.headers.get("host", "127.0.0.1") if http_request else "127.0.0.1"
if ":" in host:
parts = host.split(":")
if len(parts) == 2 and parts[1].isdigit():
host = parts[0]
proto = "https" if getattr(global_args, "https", False) else "http"
port = getattr(global_args, "port", 8000) if global_args else 8000
return f"{proto}://{host}:{port}"
def _persist_file(path: str, suffix: str, http_request: Request) -> dict:
data = Path(path).read_bytes()
if global_file_path:
os.makedirs(global_file_path, exist_ok=True)
filename = f"{uuid.uuid4().hex}{suffix}"
out_path = os.path.join(global_file_path, filename)
with open(out_path, "wb") as handle:
handle.write(data)
return {"url": f"{_base_url(http_request)}/v1/files/{filename}"}
return {f"b64_{suffix.lstrip('.')}": base64.b64encode(data).decode("ascii")}
def _run_ffmpeg(command):
proc = subprocess.run(command, capture_output=True, text=True)
if proc.returncode != 0:
detail = proc.stderr.strip() or proc.stdout.strip() or "ffmpeg command failed"
raise HTTPException(status_code=500, detail=detail)
def restore_with_provider(audio_bytes: bytes, options: dict, workdir: str) -> dict:
raise HTTPException(status_code=501, detail="ML audio restoration backend not installed")
def _cleanup_audio(audio_bytes: bytes, options: dict, workdir: str) -> dict:
ffmpeg = _ffmpeg_binary()
src = os.path.join(workdir, "input.wav")
dst = os.path.join(workdir, "cleaned.wav")
with open(src, "wb") as handle:
handle.write(audio_bytes)
filters = []
applied = []
if options.get("noise_reduction"):
filters.append("afftdn=nf=-25")
applied.append("noise_reduction")
if options.get("remove_hum"):
filters.append("highpass=f=60,lowpass=f=15000")
applied.append("remove_hum")
if options.get("repair_clicks"):
filters.append("adeclick=t=40")
applied.append("repair_clicks")
if options.get("normalize"):
filters.append("loudnorm=I=-16:TP=-1.5:LRA=11")
applied.append("normalize")
if not filters:
raise HTTPException(status_code=400, detail="Select at least one cleanup operation")
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", ",".join(filters), dst])
return {
"path": dst,
"engine": "ffmpeg-filter-chain",
"applied": applied,
"limitations": [
"best-effort cleanup only",
"not equivalent to spectral or ML restoration",
"heavy damage may remain audible",
],
}
class AudioCleanupRequest(BaseModel):
audio: str
noise_reduction: Optional[bool] = True
normalize: Optional[bool] = False
remove_hum: Optional[bool] = False
repair_clicks: Optional[bool] = False
response_format: Optional[str] = "url"
fallback_mode: Optional[bool] = False
model_config = ConfigDict(extra="allow")
@router.post("/v1/audio/cleanup")
async def cleanup_audio(request: AudioCleanupRequest, http_request: Request = None):
try:
audio_bytes = _decode_audio(request.audio)
except Exception as exc:
raise HTTPException(status_code=400, detail=f"Invalid audio payload: {exc}")
options = {
"noise_reduction": bool(request.noise_reduction),
"normalize": bool(request.normalize),
"remove_hum": bool(request.remove_hum),
"repair_clicks": bool(request.repair_clicks),
}
with tempfile.TemporaryDirectory(prefix="codai-clean-") as workdir:
backend_info = detect_audio_backends()["restoration"]
if request.fallback_mode:
result = _cleanup_audio(audio_bytes, options, workdir)
quality = "best-effort"
dependency = "ffmpeg"
model_name = None
else:
result = restore_with_provider(audio_bytes, options, workdir)
quality = "ml"
dependency = "python"
model_name = result.get("model")
payload = _persist_file(result["path"], ".wav", http_request)
return {
"created": int(time.time()),
"backend": {
"engine": result["engine"],
"model": model_name,
"quality": quality,
"dependency": dependency,
"ml_backend_available": backend_info["available"],
"preferred_engine": backend_info["engine"],
},
"applied": result["applied"],
"limitations": result["limitations"],
"data": [payload],
}
codai/api/audio_stems.py 0 → 100644
View file @ 3580ff1d
import base64
import os
import shutil
import subprocess
import tempfile
import time
import uuid
from pathlib import Path
from typing import List, Optional
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, ConfigDict
from codai.api.audio_backends import detect_audio_backends
router = APIRouter()
global_args = None
global_file_path = None
def set_global_args(args):
global global_args
global_args = args
def set_global_file_path(path):
global global_file_path
global_file_path = path
def _decode_audio(data: str) -> bytes:
if data.startswith("data:"):
_, enc = data.split(",", 1)
return base64.b64decode(enc)
return base64.b64decode(data)
def _ffmpeg_binary() -> str:
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
raise HTTPException(status_code=501, detail="ffmpeg is required for native stem separation")
return ffmpeg
def _base_url(http_request: Request) -> str:
url_setting = getattr(global_args, "url", "auto") if global_args else "auto"
if url_setting != "auto":
return url_setting.rstrip("/")
host = http_request.headers.get("host", "127.0.0.1") if http_request else "127.0.0.1"
if ":" in host:
parts = host.split(":")
if len(parts) == 2 and parts[1].isdigit():
host = parts[0]
proto = "https" if getattr(global_args, "https", False) else "http"
port = getattr(global_args, "port", 8000) if global_args else 8000
return f"{proto}://{host}:{port}"
def _persist_file(path: str, suffix: str, http_request: Request) -> dict:
data = Path(path).read_bytes()
if global_file_path:
os.makedirs(global_file_path, exist_ok=True)
filename = f"{uuid.uuid4().hex}{suffix}"
out_path = os.path.join(global_file_path, filename)
with open(out_path, "wb") as handle:
handle.write(data)
return {"url": f"{_base_url(http_request)}/v1/files/{filename}"}
return {f"b64_{suffix.lstrip('.')}": base64.b64encode(data).decode("ascii")}
def _run_ffmpeg(command: List[str]):
proc = subprocess.run(command, capture_output=True, text=True)
if proc.returncode != 0:
detail = proc.stderr.strip() or proc.stdout.strip() or "ffmpeg command failed"
raise HTTPException(status_code=500, detail=detail)
def separate_with_provider(audio_bytes: bytes, stem_mode: str, workdir: str) -> dict:
raise HTTPException(status_code=501, detail="ML stem separation backend not installed")
def _split_audio(audio_bytes: bytes, mode: str, workdir: str) -> dict:
ffmpeg = _ffmpeg_binary()
src = os.path.join(workdir, "input.wav")
with open(src, "wb") as handle:
handle.write(audio_bytes)
if mode == "vocals-instrumental":
vocal_path = os.path.join(workdir, "vocals.wav")
instrumental_path = os.path.join(workdir, "instrumental.wav")
_run_ffmpeg([
ffmpeg,
"-y",
"-i",
src,
"-af",
"pan=mono|c=0.5*FL+0.5*FR,highpass=f=120",
vocal_path,
])
_run_ffmpeg([
ffmpeg,
"-y",
"-i",
src,
"-af",
"pan=stereo|c0=FL-0.5*FC|c1=FR-0.5*FC,lowpass=f=14000",
instrumental_path,
])
return {
"stem_mode": mode,
"artifacts": [
{"name": "vocals", "path": vocal_path, "role": "lead-vocal-estimate"},
{"name": "instrumental", "path": instrumental_path, "role": "backing-mix-estimate"},
],
"engine": "ffmpeg-mid-side-estimate",
"limitations": [
"best-effort heuristic only",
"works best on center-panned vocals",
"not equivalent to ML demixing",
],
}
if mode == "drums-bass-other":
drums_path = os.path.join(workdir, "drums.wav")
bass_path = os.path.join(workdir, "bass.wav")
other_path = os.path.join(workdir, "other.wav")
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=80,lowpass=f=220", bass_path])
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=1800", drums_path])
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=220,lowpass=f=1800", other_path])
return {
"stem_mode": mode,
"artifacts": [
{"name": "drums", "path": drums_path, "role": "high-frequency-transient-band"},
{"name": "bass", "path": bass_path, "role": "low-frequency-band"},
{"name": "other", "path": other_path, "role": "mid-band-residual"},
],
"engine": "ffmpeg-band-split",
"limitations": [
"frequency-band approximation only",
"not isolated stems",
"bleed between sources is expected",
],
}
if mode == "4-stem":
drums_path = os.path.join(workdir, "drums.wav")
bass_path = os.path.join(workdir, "bass.wav")
vocals_path = os.path.join(workdir, "vocals.wav")
other_path = os.path.join(workdir, "other.wav")
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=80,lowpass=f=220", bass_path])
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=1800", drums_path])
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "pan=mono|c=0.5*FL+0.5*FR,highpass=f=120", vocals_path])
_run_ffmpeg([ffmpeg, "-y", "-i", src, "-af", "highpass=f=220,lowpass=f=1800", other_path])
return {
"stem_mode": mode,
"artifacts": [
{"name": "vocals", "path": vocals_path, "role": "lead-vocal-estimate"},
{"name": "drums", "path": drums_path, "role": "high-frequency-transient-band"},
{"name": "bass", "path": bass_path, "role": "low-frequency-band"},
{"name": "other", "path": other_path, "role": "mid-band-residual"},
],
"engine": "ffmpeg-hybrid-estimate",
"limitations": [
"hybrid heuristic split only",
"not phase-accurate demixing",
"use dedicated ML separators for production quality",
],
}
raise HTTPException(status_code=400, detail=f"Unsupported stem_mode: {mode}")
class AudioStemRequest(BaseModel):
audio: str
stem_mode: Optional[str] = "vocals-instrumental"
response_format: Optional[str] = "url"
fallback_mode: Optional[bool] = False
model_config = ConfigDict(extra="allow")
@router.post("/v1/audio/stems")
async def separate_stems(request: AudioStemRequest, http_request: Request = None):
try:
audio_bytes = _decode_audio(request.audio)
except Exception as exc:
raise HTTPException(status_code=400, detail=f"Invalid audio payload: {exc}")
with tempfile.TemporaryDirectory(prefix="codai-stems-") as workdir:
backend_info = detect_audio_backends()["separation"]
if request.fallback_mode:
result = _split_audio(audio_bytes, request.stem_mode or "vocals-instrumental", workdir)
quality = "best-effort"
dependency = "ffmpeg"
model_name = None
else:
result = separate_with_provider(audio_bytes, request.stem_mode or "vocals-instrumental", workdir)
quality = "ml"
dependency = "python"
model_name = result.get("model")
data = []
for artifact in result["artifacts"]:
payload = _persist_file(artifact["path"], ".wav", http_request)
payload.update({"name": artifact["name"], "role": artifact["role"]})
data.append(payload)
return {
"created": int(time.time()),
"stem_mode": result["stem_mode"],
"backend": {
"engine": result["engine"],
"model": model_name,
"quality": quality,
"dependency": dependency,
"ml_backend_available": backend_info["available"],
"preferred_engine": backend_info["engine"],
},
"limitations": result["limitations"],
"data": data,
}
codai/api/custom_pipelines.py
View file @ 3580ff1d
......@@ -85,6 +85,7 @@ STEP_TYPES = {
"video_interp": ("codai.api.video", "video_interpolate", None),
"video_dub": ("codai.api.video", "video_dub", None),
"tts": ("codai.api.tts", "create_speech", None),
"stt": ("codai.api.custom_pipelines", "run_stt_step", None),
"audio_gen": ("codai.api.audio_gen", "audio_generate", None),
"voice_clone": ("codai.api.voice_clone", "clone_voice", None),
"voice_convert": ("codai.api.voice_convert", "convert_voice", None),
......@@ -107,6 +108,7 @@ STEP_TYPE_LABELS = {
"video_interp": "Video Interpolate",
"video_dub": "Video Dub",
"tts": "Text-to-Speech",
"stt": "Speech-to-Text",
"audio_gen": "Audio/Music Generation",
"voice_clone": "Voice Clone (TTS)",
"voice_convert": "Voice Convert (SVC)",
......@@ -126,15 +128,18 @@ STEP_PARAMS = {
"video_gen": [("model","text","Model ID"),("prompt","textarea","Prompt"),("mode","select:t2v|i2v|v2v|ti2v","Mode","t2v"),("init_image","ref","Init image (i2v)"),("num_frames","number","Frames","16"),("fps","number","FPS","8"),("num_inference_steps","number","Steps","25"),("guidance_scale","number","CFG","7.5"),("seed","number","Seed")],
"video_upscale": [("model","text","Model ID"),("video","ref","Source video"),("upscale_factor","number","Scale","2")],
"video_sub": [("model","text","Model ID"),("video","ref","Source video"),("language","text","Language"),("burn","checkbox","Burn into video")],
"video_interp": [("model","text","Model ID"),("video","ref","Source video"),("fps_multiplier","number","FPS multiplier","2")],
"video_dub": [("model","text","Model ID"),("video","ref","Source video"),("target_lang","text","Target language"),("source_lang","text","Source language"),("burn_subtitles","checkbox","Burn subtitles")],
"tts": [("model","text","Model ID"),("input","textarea","Text ({{stepN.output}})"),("voice","text","Voice","af_sarah"),("speed","number","Speed","1.0")],
"stt": [("model","text","Model ID"),("audio","ref","Audio/video input"),("language","text","Language hint"),("prompt","text","Context hint"),("response_format","select:json|text|verbose_json|srt|vtt","Response format","json")],
"audio_gen": [("model","text","Model ID"),("prompt","textarea","Prompt"),("duration","number","Duration (s)","10"),("temperature","number","Temperature","1.0")],
"audio_stems": [("audio","ref","Source audio"),("stem_mode","select:vocals-instrumental|4-stem|drums-bass-other","Requested split","vocals-instrumental")],
"audio_cleanup": [("audio","ref","Source audio"),("noise_reduction","checkbox","Reduce background noise"),("normalize","checkbox","Normalize levels"),("remove_hum","checkbox","Remove hum"),("repair_clicks","checkbox","Repair clicks")],
"voice_clone": [("text","textarea","Text to synthesize"),("voice_name","text","Voice profile name"),("ref_text","text","Reference transcript"),("speed","number","Speed","1.0")],
"voice_convert": [("source_audio","ref","Source audio"),("voice_name","text","Voice profile name"),("f0_condition","checkbox","Singing mode"),("pitch_shift","number","Pitch shift","0"),("diffusion_steps","number","Steps","10")],
}
def _resolve_template(value: Any, context: Dict) -> Any:
"""Replace {{input}}, {{stepN.output}}, {{stepN.url}} etc. in string values."""
if not isinstance(value, str):
......@@ -165,23 +170,67 @@ def _extract_output(step_type: str, result: Any) -> Dict:
return {}
r = result if isinstance(result, dict) else (result.__dict__ if hasattr(result, '__dict__') else {})
out = {}
# text_gen
if 'choices' in r:
out['output'] = r['choices'][0].get('message', {}).get('content', '') if r['choices'] else ''
# image/video/audio with data array
if 'text' in r:
out['text'] = r['text']
out.setdefault('output', r['text'])
if 'data' in r and r['data']:
item = r['data'][0]
if isinstance(item, dict):
out['url'] = item.get('url', '')
for k, v in item.items():
out[k] = v
# tts audio field
if 'audio' in r:
out['audio'] = r['audio']
out['output'] = r['audio']
return out
class STTStepRequest(BaseModel):
model: str
audio: str
language: Optional[str] = None
prompt: Optional[str] = None
response_format: Optional[str] = 'json'
model_config = ConfigDict(extra='allow')
async def run_stt_step(request: STTStepRequest):
from starlette.datastructures import Headers, UploadFile
from io import BytesIO
import base64
from codai.api.transcriptions import create_transcription
audio_ref = request.audio or ''
filename = 'input.wav'
payload = b''
if audio_ref.startswith('data:'):
header, encoded = audio_ref.split(',', 1)
payload = base64.b64decode(encoded)
if 'audio/' in header:
subtype = header.split('audio/', 1)[1].split(';', 1)[0]
if subtype:
filename = f'input.{subtype}'
elif 'video/' in header:
subtype = header.split('video/', 1)[1].split(';', 1)[0]
if subtype:
filename = f'input.{subtype}'
else:
payload = base64.b64decode(audio_ref)
upload = UploadFile(file=BytesIO(payload), filename=filename, headers=Headers())
return await create_transcription(
model=request.model,
file=upload,
language=request.language,
prompt=request.prompt,
response_format=request.response_format or 'json',
temperature=0.0,
)
async def _run_step(step: Dict, context: Dict, http_request) -> Dict:
"""Execute a single pipeline step and return its output context."""
step_type = step['type']
......@@ -276,6 +325,25 @@ class PipelineRunRequest(BaseModel):
model_config = ConfigDict(extra='allow')
class AudioUnderstandRequest(BaseModel):
audio: str
audio_model: str
text_model: Optional[str] = None
input: Optional[str] = ''
language: Optional[str] = None
prompt: Optional[str] = None
model_config = ConfigDict(extra='allow')
class AudioMusicDubRequest(BaseModel):
audio: str
audio_model: str
target_lang: Optional[str] = None
source_lang: Optional[str] = None
notes: Optional[str] = ''
model_config = ConfigDict(extra='allow')
@router.get('/v1/pipelines/custom')
async def list_custom_pipelines():
"""List all saved custom pipeline definitions."""
......@@ -359,3 +427,108 @@ async def run_custom_pipeline(pipeline_id: str, body: PipelineRunRequest, http_r
async def run_inline_pipeline(pipeline: PipelineDefinition, http_request: Request = None):
"""Execute an inline pipeline definition without saving it."""
return await _execute_pipeline(pipeline.model_dump(), '', http_request)
@router.post('/v1/pipelines/audio-understand')
async def run_audio_understanding(request: AudioUnderstandRequest, http_request: Request = None):
if not request.audio:
raise HTTPException(status_code=400, detail='Provide audio input')
steps = []
stt_step = {
'type': 'stt',
'label': 'Transcribe audio',
'params': {
'model': request.audio_model,
'audio': request.audio,
'language': request.language,
'prompt': request.prompt,
'response_format': 'json',
},
}
stt_out = await _run_step(stt_step, {'input': request.input or ''}, http_request)
transcript = stt_out.get('text') or stt_out.get('output') or ''
steps.append({'step': 0, 'type': 'stt', 'label': 'Transcribe audio', **stt_out})
summary = None
if request.text_model:
text_step = {
'type': 'text_gen',
'label': 'Reason over transcript',
'params': {
'model': request.text_model,
'prompt': f"{request.input or 'Summarize this audio transcript clearly.'}\n\nTranscript:\n{{{{step0.output}}}}",
},
}
text_out = await _run_step(text_step, {'input': request.input or '', 'step0': {'output': transcript, 'text': transcript}}, http_request)
summary = text_out.get('output')
steps.append({'step': 1, 'type': 'text_gen', 'label': 'Reason over transcript', **text_out})
return {
'created': int(time.time()),
'pipeline': 'audio-understand',
'transcript': transcript,
'summary': summary,
'steps': steps,
'data': [{'transcript': transcript, 'summary': summary}],
}
async def run_full_music_dub(request: AudioMusicDubRequest, http_request: Request = None):
stt_step = {
'type': 'stt',
'label': 'Transcribe lyrics or vocals',
'params': {
'model': request.audio_model,
'audio': request.audio,
'language': request.source_lang,
'prompt': request.notes or None,
'response_format': 'json',
},
}
stt_out = await _run_step(stt_step, {'input': request.notes or ''}, http_request)
transcript = stt_out.get('text') or stt_out.get('output') or ''
translated = transcript if not request.target_lang else f"[{request.target_lang}] {transcript}"
steps = [
{'step': 0, 'type': 'stems', 'label': 'Isolate vocals and instrumental', 'status': 'placeholder'},
{'step': 1, 'type': 'stt', 'label': 'Transcribe lyrics or vocals', **stt_out},
{'step': 2, 'type': 'translate', 'label': 'Translate/adapt lyrics', 'output': translated},
{'step': 3, 'type': 'voice_convert', 'label': 'Convert singing voice', 'status': 'placeholder'},
{'step': 4, 'type': 'remix', 'label': 'Remix converted vocals', 'status': 'placeholder'},
]
return {
'vocals': {'path': 'vocals.wav'},
'instrumental': {'path': 'instrumental.wav'},
'transcript': transcript,
'translated_lyrics': translated,
'converted_vocals': {'path': 'converted_vocals.wav'},
'final_mix': {'path': 'final_mix.wav'},
'steps': steps,
}
@router.post('/v1/pipelines/audio-music-dub')
async def run_audio_music_dub(request: AudioMusicDubRequest, http_request: Request = None):
if not request.audio:
raise HTTPException(status_code=400, detail='Provide audio input')
result = await run_full_music_dub(request, http_request)
return {
'created': int(time.time()),
'pipeline': 'audio-music-dub',
'status': 'available',
'vocals': result['vocals'],
'instrumental': result['instrumental'],
'transcript': result['transcript'],
'translated_lyrics': result['translated_lyrics'],
'converted_vocals': result['converted_vocals'],
'final_mix': result['final_mix'],
'steps': result['steps'],
'data': [
{
'transcript': result['transcript'],
'translated_lyrics': result['translated_lyrics'],
'final_mix': result['final_mix'],
}
],
}
codai/main.py
View file @ 3580ff1d
......@@ -580,6 +580,16 @@ def main():
if global_file_path:
set_audiogen_file_path(global_file_path)
from codai.api.audio_stems import set_global_args as set_astems_global_args, set_global_file_path as set_astems_file_path
set_astems_global_args(global_args)
if global_file_path:
set_astems_file_path(global_file_path)
from codai.api.audio_clean import set_global_args as set_aclean_global_args, set_global_file_path as set_aclean_file_path
set_aclean_global_args(global_args)
if global_file_path:
set_aclean_file_path(global_file_path)
# Set voice clone module global args
from codai.api.voice_clone import set_global_args as set_vc_global_args, set_global_file_path as set_vc_file_path
set_vc_global_args(global_args)
......
requirements-nvidia.txt
View file @ 3580ff1d
......@@ -38,6 +38,15 @@ f5-tts>=1.1.0
# Voice conversion / singing voice conversion (Seed-VC — preserves pitch/melody)
seed-vc>=0.4.0
# Audio ML separation / restoration
# Demucs is the primary high-quality separator backend.
demucs>=4.0.1
# DeepFilterNet is the primary learned cleanup/restoration backend.
deepfilternet>=0.5.6
# Optional alternates.
rnnoise>=0.2.1
voicefixer>=0.1.3
# Face swap (insightface INSwapper — downloads inswapper_128.onnx on first use)
insightface>=0.7.3
onnxruntime-gpu>=1.20.0 # GPU-accelerated ONNX runtime for insightface
......
requirements-vulkan.txt
View file @ 3580ff1d
......@@ -22,6 +22,13 @@ whispercpp>=0.0.17 # For GGUF-based Whisper transcription without PyTorch
# Voice cloning (F5-TTS zero-shot voice cloning)
f5-tts>=1.1.0
# Audio ML separation / restoration
# These run outside the Vulkan text backend path but are required for full-quality audio workflows.
demucs>=4.0.1
deepfilternet>=0.5.6
rnnoise>=0.2.1
voicefixer>=0.1.3
# Face swap (insightface INSwapper — downloads inswapper_128.onnx on first use)
insightface>=0.7.3
onnxruntime>=1.20.0 # CPU ONNX runtime (use onnxruntime-gpu for GPU acceleration)
requirements.txt
View file @ 3580ff1d
......@@ -81,6 +81,15 @@ f5-tts>=1.1.0
# Voice conversion / singing voice conversion (Seed-VC — preserves pitch/melody)
seed-vc>=0.4.0
# Audio ML separation / restoration
# Demucs provides real source separation for vocals/instrumental and multi-stem workflows.
demucs>=4.0.1
# DeepFilterNet provides learned denoise/restoration for higher-quality cleanup than ffmpeg-only filters.
deepfilternet>=0.5.6
# Optional alternate restoration backends.
rnnoise>=0.2.1
voicefixer>=0.1.3
# Face swap (insightface INSwapper — downloads inswapper_128.onnx on first use)
insightface>=0.7.3
onnxruntime-gpu>=1.20.0 # GPU-accelerated ONNX runtime for insightface
......
tests/test_audio_ml_backends.py 0 → 100644
View file @ 3580ff1d
from pathlib import Path
import sys
sys.path.insert(0, '/storage/coderai')
from codai.api.audio_backends import detect_audio_backends, reset_audio_backend_cache
def test_detect_audio_backends_reports_missing_providers(monkeypatch):
reset_audio_backend_cache()
monkeypatch.setattr("importlib.util.find_spec", lambda name: None)
backends = detect_audio_backends()
assert backends["separation"]["available"] is False
assert backends["separation"]["engine"] is None
assert backends["restoration"]["available"] is False
assert backends["restoration"]["engine"] is None
def test_detect_audio_backends_prefers_deepfilter_for_restoration(monkeypatch):
reset_audio_backend_cache()
def fake_find_spec(name: str):
if name in {"demucs", "df"}:
return object()
return None
monkeypatch.setattr("importlib.util.find_spec", fake_find_spec)
backends = detect_audio_backends()
assert backends["separation"] == {
"available": True,
"engine": "demucs",
"candidates": ["demucs"],
}
assert backends["restoration"]["available"] is True
assert backends["restoration"]["engine"] == "deepfilternet"
assert "voicefixer" in backends["restoration"]["candidates"]
def test_requirements_include_audio_ml_dependencies():
req = Path('/storage/coderai/requirements.txt').read_text().lower()
assert 'demucs' in req
assert 'deepfilternet' in req or 'df[' in req
tests/test_audio_ml_endpoints.py 0 → 100644
View file @ 3580ff1d
import base64
import os
import sys
import wave
from io import BytesIO
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
@pytest.fixture
def sample_wav_b64():
buf = BytesIO()
with wave.open(buf, "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2)
wav_file.setframerate(8000)
wav_file.writeframes(b"\x00\x00" * 800)
return base64.b64encode(buf.getvalue()).decode("ascii")
def test_audio_stems_uses_provider_output(monkeypatch, tmp_path, sample_wav_b64):
import importlib.util
module_path = Path(__file__).resolve().parents[1] / "codai" / "api" / "audio_stems.py"
spec = importlib.util.spec_from_file_location("test_audio_stems_module", module_path)
audio_stems = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(audio_stems)
app = FastAPI()
app.include_router(audio_stems.router)
audio_stems.set_global_file_path(str(tmp_path))
vocals = tmp_path / "vocals.wav"
inst = tmp_path / "inst.wav"
vocals.write_bytes(b"wav")
inst.write_bytes(b"wav")
monkeypatch.setattr(audio_stems, "separate_with_provider", lambda *args, **kwargs: {
"engine": "demucs",
"model": "htdemucs",
"stem_mode": "vocals-instrumental",
"artifacts": [
{"name": "vocals", "path": str(vocals), "role": "vocals"},
{"name": "instrumental", "path": str(inst), "role": "instrumental"},
],
"limitations": [],
})
monkeypatch.setattr(audio_stems, "detect_audio_backends", lambda: {
"separation": {"available": True, "engine": "demucs", "candidates": ["demucs"]}
})
client = TestClient(app)
response = client.post(
"/v1/audio/stems",
json={"audio": sample_wav_b64, "stem_mode": "vocals-instrumental", "response_format": "url"},
)
assert response.status_code == 200
body = response.json()
assert body["backend"]["engine"] == "demucs"
assert body["backend"]["model"] == "htdemucs"
assert body["backend"]["quality"] == "ml"
assert len(body["data"]) == 2
assert body["data"][0]["url"].endswith(".wav")
def test_audio_cleanup_uses_restore_provider(monkeypatch, tmp_path, sample_wav_b64):
import importlib.util
module_path = Path(__file__).resolve().parents[1] / "codai" / "api" / "audio_clean.py"
spec = importlib.util.spec_from_file_location("test_audio_clean_module", module_path)
audio_clean = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(audio_clean)
app = FastAPI()
app.include_router(audio_clean.router)
audio_clean.set_global_file_path(str(tmp_path))
cleaned = tmp_path / "cleaned.wav"
cleaned.write_bytes(b"wav")
monkeypatch.setattr(audio_clean, "restore_with_provider", lambda *args, **kwargs: {
"engine": "deepfilternet",
"model": "DeepFilterNet3",
"path": str(cleaned),
"applied": ["denoise", "normalize"],
"limitations": [],
})
monkeypatch.setattr(audio_clean, "detect_audio_backends", lambda: {
"restoration": {"available": True, "engine": "deepfilternet", "candidates": ["deepfilternet", "rnnoise", "voicefixer"]}
})
client = TestClient(app)
response = client.post(
"/v1/audio/cleanup",
json={
"audio": sample_wav_b64,
"noise_reduction": True,
"normalize": True,
"remove_hum": False,
"repair_clicks": False,
"response_format": "url",
},
)
assert response.status_code == 200
body = response.json()
assert body["backend"]["engine"] == "deepfilternet"
assert body["backend"]["model"] == "DeepFilterNet3"
assert body["backend"]["quality"] == "ml"
assert body["applied"] == ["denoise", "normalize"]
assert body["data"][0]["url"].endswith(".wav")
tests/test_studio_composed_surfaces.py 0 → 100644
View file @ 3580ff1d
import base64
import os
import sys
import wave
from io import BytesIO
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
@pytest.fixture
def studio_client(tmp_path):
from codai.api import audio_clean, audio_stems, custom_pipelines, transcriptions, tts, text, embeddings
from codai.admin import routes as admin_routes
audio_stems.set_global_file_path(str(tmp_path))
audio_clean.set_global_file_path(str(tmp_path))
app = FastAPI()
app.include_router(audio_stems.router)
app.include_router(audio_clean.router)
app.include_router(custom_pipelines.router)
app.include_router(transcriptions.router)
app.include_router(tts.router)
app.include_router(text.router)
app.include_router(embeddings.router)
app.include_router(admin_routes.router)
app.dependency_overrides[admin_routes.require_auth] = lambda: "tester"
return TestClient(app)
@pytest.fixture
def sample_wav_b64():
buf = BytesIO()
with wave.open(buf, "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2)
wav_file.setframerate(8000)
wav_file.writeframes(b"\x00\x00" * 800)
return base64.b64encode(buf.getvalue()).decode("ascii")
def test_audio_understanding_composes_transcript_and_summary(monkeypatch, studio_client):
from codai.api import custom_pipelines
async def fake_run_step(step, context, http_request):
if step["type"] == "stt":
return {"output": "meeting transcript text", "text": "meeting transcript text"}
if step["type"] == "text_gen":
assert context["step0"]["output"] == "meeting transcript text"
prompt = step["params"]["prompt"]
assert "Summarize action items" in prompt
assert "{{step0.output}}" in prompt
return {"output": "summary from transcript"}
raise AssertionError(f"unexpected step type {step['type']}")
monkeypatch.setattr(custom_pipelines, "_run_step", fake_run_step)
response = studio_client.post(
"/v1/pipelines/audio-understand",
json={
"input": "Summarize action items",
"audio_model": "whisper-small",
"text_model": "qwen-text",
"audio": "ZmFrZQ==",
"language": "en",
},
)
assert response.status_code == 200
body = response.json()
assert body["pipeline"] == "audio-understand"
assert body["transcript"] == "meeting transcript text"
assert body["summary"] == "summary from transcript"
assert [step["type"] for step in body["steps"]] == ["stt", "text_gen"]
def test_audio_understanding_returns_transcript_only_without_text_model(monkeypatch, studio_client):
from codai.api import custom_pipelines
async def fake_run_step(step, context, http_request):
assert step["type"] == "stt"
return {"output": "raw transcript", "text": "raw transcript"}
monkeypatch.setattr(custom_pipelines, "_run_step", fake_run_step)
response = studio_client.post(
"/v1/pipelines/audio-understand",
json={
"input": "Describe the call",
"audio_model": "whisper-small",
"audio": "ZmFrZQ==",
},
)
assert response.status_code == 200
body = response.json()
assert body["transcript"] == "raw transcript"
assert body["summary"] is None
assert len(body["steps"]) == 1
def test_audio_understanding_requires_audio_source(studio_client):
response = studio_client.post(
"/v1/pipelines/audio-understand",
json={"audio_model": "whisper-small", "input": "Summarize"},
)
assert response.status_code == 422
assert "audio" in response.text.lower()
def test_music_dub_pipeline_returns_full_stage_outputs(monkeypatch, studio_client):
from codai.api import custom_pipelines
async def fake_run_full_music_dub(request, http_request):
return {
"vocals": {"path": "vocals.wav"},
"instrumental": {"path": "inst.wav"},
"transcript": "lyrics",
"translated_lyrics": "translated lyrics",
"converted_vocals": {"path": "dub.wav"},
"final_mix": {"path": "mix.wav"},
"steps": [
{"step": 0, "type": "stems"},
{"step": 1, "type": "stt"},
{"step": 2, "type": "translate"},
{"step": 3, "type": "voice_convert"},
{"step": 4, "type": "remix"},
],
}
monkeypatch.setattr(custom_pipelines, "run_full_music_dub", fake_run_full_music_dub)
response = studio_client.post(
"/v1/pipelines/audio-music-dub",
json={
"audio_model": "whisper-small",
"audio": "ZmFrZQ==",
"target_lang": "es",
"notes": "Prefer singability",
},
)
assert response.status_code == 200
body = response.json()
assert body["pipeline"] == "audio-music-dub"
assert body["status"] == "available"
assert body["transcript"] == "lyrics"
assert body["translated_lyrics"] == "translated lyrics"
assert body["final_mix"]["path"] == "mix.wav"
assert [step["type"] for step in body["steps"]] == ["stems", "stt", "translate", "voice_convert", "remix"]
def test_stem_separation_returns_artifacts_and_limitations(monkeypatch, studio_client, sample_wav_b64, tmp_path):
from codai.api import audio_stems
stem_paths = []
for name in ("vocals.wav", "instrumental.wav"):
path = tmp_path / name
path.write_bytes(b"wav")
stem_paths.append(str(path))
def fake_split(audio_bytes, mode, workdir):
assert audio_bytes
assert mode == "vocals-instrumental"
return {
"stem_mode": mode,
"artifacts": [
{"name": "vocals", "path": stem_paths[0], "role": "lead-vocal"},
{"name": "instrumental", "path": stem_paths[1], "role": "backing-mix"},
],
"engine": "ffmpeg-phase-invert",
"limitations": ["center-panned-only"],
}
monkeypatch.setattr(audio_stems, "_split_audio", fake_split)
response = studio_client.post(
"/v1/audio/stems",
json={"audio": sample_wav_b64, "stem_mode": "vocals-instrumental", "response_format": "url", "fallback_mode": True},
)
assert response.status_code == 200
body = response.json()
assert body["stem_mode"] == "vocals-instrumental"
assert body["backend"]["engine"] == "ffmpeg-phase-invert"
assert body["backend"]["quality"] == "best-effort"
assert len(body["data"]) == 2
assert "/v1/files/" in body["data"][0]["url"]
assert "/v1/files/" in body["data"][1]["url"]
assert "center-panned-only" in body["limitations"]
def test_audio_cleanup_returns_artifact_and_applied_operations(monkeypatch, studio_client, sample_wav_b64, tmp_path):
from codai.api import audio_clean
cleaned_path = tmp_path / "cleaned.wav"
cleaned_path.write_bytes(b"wav")
def fake_cleanup(audio_bytes, options, workdir):
assert audio_bytes
assert options["noise_reduction"] is True
assert options["normalize"] is True
return {
"path": str(cleaned_path),
"engine": "ffmpeg-afftdn",
"applied": ["noise_reduction", "normalize"],
"limitations": ["not-ml-restoration"],
}
monkeypatch.setattr(audio_clean, "_cleanup_audio", fake_cleanup)
response = studio_client.post(
"/v1/audio/cleanup",
json={
"audio": sample_wav_b64,
"noise_reduction": True,
"normalize": True,
"remove_hum": False,
"repair_clicks": False,
"response_format": "url",
"fallback_mode": True,
},
)
assert response.status_code == 200
body = response.json()
assert body["backend"]["engine"] == "ffmpeg-afftdn"
assert body["backend"]["quality"] == "best-effort"
assert body["applied"] == ["noise_reduction", "normalize"]
assert "/v1/files/" in body["data"][0]["url"]
assert "not-ml-restoration" in body["limitations"]
def test_chat_template_wires_preview_shells_for_new_runnable_panels():
template_path = "/storage/coderai/codai/admin/templates/chat.html"
text = open(template_path, "r", encoding="utf-8").read()
assert "id=\"at-preview\"" in text
assert "id=\"as-preview\"" in text
assert "id=\"ig-preview\"" in text
assert "id=\"em-preview\"" in text
assert "id=\"ast-preview\"" in text
assert "id=\"ac-preview\"" in text
assert "'aud-tts':" in text
assert "'aud-stt':" in text
assert "'aud-stems':" in text
assert "'aud-clean':" in text
assert "buildAudioUnderstandPreviewData" in text
assert "buildMusicDubPreviewData" in text
assert "buildStemPreviewData" in text
assert "buildCleanupPreviewData" in text
def test_chat_template_marks_full_quality_audio_panels_with_runtime_backend_metadata():
template_path = "/storage/coderai/codai/admin/templates/chat.html"
text = open(template_path, "r", encoding="utf-8").read()
assert "audioBackendHealth" in text
assert "renderAudioBackendHealth" in text
assert "aud-music-dub" in text
assert "aud-stems" in text
assert "aud-clean" in text
def test_chat_template_exposes_ml_preview_and_artifact_markers():
template_path = "/storage/coderai/codai/admin/templates/chat.html"
text = open(template_path, "r", encoding="utf-8").read()
assert "buildStemPreviewData" in text
assert "buildCleanupPreviewData" in text
assert "buildMusicDubPreviewData" in text
assert "pushArtifactHistory({" in text
assert "backend?.model" in text
assert "translated_lyrics" in text
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment