• Stefy Lanza (nextime / spora )'s avatar
    audio: STT backends, diarization, speaker embeddings/ID, word timestamps · 7e5a925c
    Stefy Lanza (nextime / spora ) authored
    Transcription (/v1/audio/transcriptions):
    - Route multipart uploads by peeked model (front _peek_model) so they land on
      the model's assigned engine instead of scattering by capability.
    - New pluggable STT backends (codai/api/stt_backends.py): wav2vec2 (HF), vosk
      (CPU dirs), NVIDIA NeMo-Canary (isolated venv worker) + faster-whisper/whisper.
    - Per-model language selection + validation; Canary source->target translation.
    - OpenAI timestamp_granularities[]=word: top-level words[] in verbose_json
      (vosk/wav2vec2/faster-whisper/canary).
    - VRAM-eviction tracking (acquire_stt_backend) + keep_resident co-residency for
      the small set; /v1/models surfaces languages + backends.
    
    Diarization (/v1/audio/diarization + diarize=true):
    - pyannote.audio isolated venv worker (codai/api/pyannote_worker.py,
      tools/pyannote_service.py) on the 3090; speaker-labeled segments; graceful
      degradation. Pinned torch/pyannote/hf_hub/matplotlib in requirements-pyannote.txt.
    
    Speaker embeddings + recognition:
    - /v1/audio/speaker-embeddings (ecapa via speechbrain, pyannote/wespeaker).
    - Enroll/list/delete + identify/verify (codai/api/speaker_registry.py); persistent
      voiceprint store; diarization identify=true relabels turns with enrolled names.
    
    New isolated-venv requirements: requirements-nemo.txt, requirements-pyannote.txt
    (+ vosk in requirements.txt).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    7e5a925c
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...