keep the sound, the shape and the uploaded file when editing media (v0.4.86)

Four ways an edit could answer 200 while quietly doing the wrong thing.

The enhancement paths decode a clip to frames and encode a new video, so the
audio was left behind: a clip that arrived with speech came back silent. Only
the ffmpeg minterpolate fallback -- the path nobody takes by default -- ever
kept it. _mux_original_audio() puts the track back on all three in-process
paths, stream-copied where MP4 accepts the codec and re-encoded to AAC where it
does not. Both paths already preserve the source duration, so nothing is
stretched. A silent source, or a remux PyAV cannot do, returns the frames
unchanged rather than losing a finished GPU job.

The image and video editing endpoints took base64 (and, for video, an http
URL), so a caller that had already uploaded a file had to re-send it on every
edit -- megabytes a call for a file on the same disk. They now resolve through
the compose resolver, so a /v1/files path or a sha256: upload id works
everywhere, and our own files are read off disk instead of fetched back over a
loopback that would need a token. Responses carry the relative path beside the
url, as compose does, so a caller need not depend on the public hostname.

Routing through that resolver would have tightened what callers may send: it
validates base64 strictly, which rejects the 76-column wrapping that MIME, the
base64 tool and encodebytes all produce. Whitespace carries no meaning in
base64, so it is now stripped rather than refused.

/v1/images/edits defaulted size to the string "1024x1024", so omitting it
squashed every photo that was not square. Omitted now keeps the source's shape
at the ~1 MP a diffusion model wants, on the 64-grid it needs.

/v1/images/upscale fell through to PIL LANCZOS for an unrecognised model and
still answered 200, charging for an AI upscale that never ran and returning
something merely soft. That is a 400 naming the models that would work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.85"
+__version__ = "0.4.86"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/images.py

@@ -234,6 +234,10 @@ def save_image_response(img, request_format="base64", http_request=None):
         # Add URL to response
         from codai.api.urlutils import build_file_url
         result["url"] = build_file_url(filename, http_request)
+        # The relative path as well, as /v1/compose returns it: a caller that
+        # feeds the result into another call should not have to know, or depend
+        # on, this install's public hostname.
+        result["path"] = f"/v1/files/{filename}"
         
         # If client explicitly requested base64, include it
         # Otherwise, only return URL when file-path is set
@@ -1726,10 +1730,12 @@ async def create_image_generation(request: ImageGenerationRequest, http_request:
 class ImageEditRequest(BaseModel):
     model: str
     prompt: str
-    image: str              # base64-encoded PNG or "data:image/...;base64,..."
-    mask: Optional[str] = None   # optional inpaint mask (base64 PNG)
+    image: str              # /v1/files path, sha256: id, data URI, URL or base64
+    mask: Optional[str] = None   # accepted but unused here -- see /v1/images/inpaint
     n: int = 1
-    size: Optional[str] = "1024x1024"
+    #: Omitted means "keep the source's shape" (see _default_edit_size), NOT a
+    #: square: defaulting to 1024x1024 silently distorted every non-square photo.
+    size: Optional[str] = None
     response_format: Optional[str] = "url"
     strength: Optional[float] = 0.75    # denoising strength (0=keep original, 1=ignore)
     steps: Optional[int] = None
@@ -1742,15 +1748,44 @@ class ImageEditRequest(BaseModel):
         extra = "allow"
 
 
-def _decode_b64_image(data: str):
-    """Decode base64 image string to PIL Image."""
+def _default_edit_size(width: int, height: int, multiple: int = 64,
+                       target_pixels: int = 1024 * 1024) -> tuple:
+    """(w, h) keeping the source aspect at ~``target_pixels``, on a grid.
+
+    An omitted ``size`` used to mean 1024x1024, which stretched every photo that
+    was not square -- a 3:2 holiday snap came back visibly squashed. Diffusion
+    models need each side on a multiple (usually 64), so the aspect is preserved
+    as closely as that grid allows rather than exactly.
+    """
+    width, height = max(1, int(width)), max(1, int(height))
+    scale = (target_pixels / float(width * height)) ** 0.5
+    w = int(round(width * scale / multiple)) * multiple
+    h = int(round(height * scale / multiple)) * multiple
+    return max(multiple, w), max(multiple, h)
+
+
+def _image_bytes(ref: str, field: str = "image") -> bytes:
+    """The bytes behind any accepted image reference.
+
+    Accepts everything the compose endpoints accept -- a ``/v1/files/...`` path
+    or a ``sha256:`` id from ``/v1/files/upload``, a ``data:`` URI, an http(s)
+    URL, or raw base64 -- so a caller that uploaded the photo once can name it
+    on each edit instead of re-sending it. A file this install already holds is
+    read from disk, never fetched back over HTTP.
+    """
+    if not ref:
+        raise HTTPException(status_code=400, detail=f"{field} is required")
+    from codai.compose.media import MediaError, fetch_bytes
+    try:
+        return fetch_bytes(ref)
+    except MediaError as e:
+        raise HTTPException(status_code=400, detail=f"{field}: {e}")
+
+
+def _decode_b64_image(data: str, field: str = "image"):
+    """Any accepted image reference → an RGB PIL image."""
     from PIL import Image as PILImage
-    if data.startswith("data:"):
-        _, encoded = data.split(",", 1)
-        raw = base64.b64decode(encoded)
-    else:
-        raw = base64.b64decode(data)
-    return PILImage.open(io.BytesIO(raw)).convert("RGB")
+    return PILImage.open(io.BytesIO(_image_bytes(data, field))).convert("RGB")
 
 
 def _load_img2img_pipeline(model_name: str, global_args):
@@ -1844,6 +1879,11 @@ async def create_image_edit(request: ImageEditRequest, http_request: Request = N
                     source_img = source_img.resize((width, height))
                 except ValueError:
                     pass
+        else:
+            # No size asked for: keep the photo's own shape at ~1 MP.
+            width, height = _default_edit_size(*source_img.size)
+            if (width, height) != source_img.size:
+                source_img = source_img.resize((width, height))
 
         seed = request.seed or getattr(global_args, 'image_seed', None)
         generator = torch.Generator(device=pipe.device).manual_seed(seed) if seed else None
@@ -2204,9 +2244,29 @@ def _run_upscale(upscaler, image_bytes: bytes, scale: int):
     return img.resize((w * scale, h * scale), PILImage.LANCZOS)
 
 
+def _configured_upscalers() -> list:
+    """Image models this install could actually upscale with."""
+    names = []
+    try:
+        from codai.models.capabilities import detect_model_capabilities
+        for m in multi_model_manager.image_models:
+            try:
+                if detect_model_capabilities(m).image_upscaling:
+                    names.append(m)
+            except Exception:
+                continue
+    except Exception:
+        pass
+    return names
+
+
 @router.post("/v1/images/upscale", summary="Upscale an image")
 async def create_image_upscale(request: ImageUpscaleRequest, http_request: Request = None):
-    """Upscale an image using Real-ESRGAN or PIL LANCZOS fallback."""
+    """Upscale an image with a real super-resolution model.
+
+    There is no PIL/LANCZOS fallback: a plain resize is not an upscale, and
+    returning one silently charged for work that did not happen.
+    """
     global global_args
     _progress_loading(request.model or "image")
     model_info = await asyncio.to_thread(
@@ -2221,7 +2281,19 @@ async def create_image_upscale(request: ImageUpscaleRequest, http_request: Reque
             raise HTTPException(status_code=500, detail=f"Failed to load upscaler: {e}")
         if upscaler[0] != 'pil':
             _UPSCALER_CACHE[model_name] = upscaler
-    raw = base64.b64decode(request.image.split(',', 1)[-1] if ',' in request.image else request.image)
+    if upscaler[0] == 'pil':
+        # _load_upscaler fell through to PIL LANCZOS, which is a plain resize:
+        # answering 200 here bills the caller for an "AI upscale" that never
+        # ran, and the result looks merely soft rather than wrong. Say so.
+        usable = _configured_upscalers()
+        raise HTTPException(
+            status_code=400,
+            detail=(f"'{model_name}' is not an upscaling model"
+                    + (f" — configured upscalers: {', '.join(usable)}"
+                       if usable else
+                       " — no upscaling model is configured; add a Real-ESRGAN "
+                       "/ SwinIR / diffusers upscaler on the model page")))
+    raw = _image_bytes(request.image)
     try:
         out_img = await asyncio.get_event_loop().run_in_executor(
             None, _run_upscale, upscaler, raw, request.scale or 4)
@@ -2492,7 +2564,7 @@ def _run_deblur(image_bytes: bytes, strength: float) -> "PILImage.Image":
 @router.post("/v1/images/deblur", summary="Deblur an image")
 async def create_image_deblur(request: ImageDeblurRequest, http_request: Request = None):
     """Remove blur from an image using Wiener deconvolution and unsharp masking."""
-    raw = base64.b64decode(request.image.split(',', 1)[-1] if ',' in request.image else request.image)
+    raw = _image_bytes(request.image)
     try:
         result_img = await asyncio.get_event_loop().run_in_executor(
             None, _run_deblur, raw, request.strength or 0.5)
@@ -2550,7 +2622,7 @@ def _run_unpixelate(image_bytes: bytes, scale: int, model_path: Optional[str]) -
 @router.post("/v1/images/unpixelate", summary="Restore a pixelated image")
 async def create_image_unpixelate(request: ImageUnpixelateRequest, http_request: Request = None):
     """Remove pixelation / upscale with detail recovery using Real-ESRGAN."""
-    raw = base64.b64decode(request.image.split(',', 1)[-1] if ',' in request.image else request.image)
+    raw = _image_bytes(request.image)
     try:
         result_img = await asyncio.get_event_loop().run_in_executor(
             None, _run_unpixelate, raw, request.scale or 4, request.model)

codai/api/video.py

@@ -127,16 +127,22 @@ def _derive_device() -> str:
 
 
 def _decode_b64_or_url(data: str) -> bytes:
+    """The bytes behind any accepted media reference.
+
+    Accepts everything the compose endpoints accept -- a ``/v1/files/...`` path
+    or a ``sha256:`` id from ``/v1/files/upload``, a ``data:`` URI, an http(s)
+    URL, or raw base64 -- so a caller that has already uploaded a file can name
+    it instead of re-sending the whole clip on every edit. A file this install
+    holds is read from disk rather than fetched over HTTP, which would need a
+    token and would fail whenever the front is bound somewhere else.
+    """
     if not data:
         return b''
-    if data.startswith("data:"):
-        _, enc = data.split(",", 1)
-        return base64.b64decode(enc)
-    if data.startswith("http://") or data.startswith("https://"):
-        import urllib.request
-        with urllib.request.urlopen(data, timeout=60) as r:
-            return r.read()
-    return base64.b64decode(data)
+    from codai.compose.media import MediaError, fetch_bytes
+    try:
+        return fetch_bytes(data)
+    except MediaError as e:
+        raise HTTPException(status_code=400, detail=str(e))
 
 
 def _as_b64(ref: str) -> str:
@@ -168,7 +174,8 @@ def _save_file(data: bytes, ext: str, http_request) -> dict:
         out_path = os.path.join(global_file_path, filename)
         with open(out_path, 'wb') as f:
             f.write(data)
-        return {"url": _build_url(filename, http_request)}
+        return {"url": _build_url(filename, http_request),
+                "path": f"/v1/files/{filename}"}
     else:
         return {f"b64_{ext}": base64.b64encode(data).decode()}
 
@@ -208,6 +215,98 @@ def _encode_mp4_pyav(frames, fps: int, crf: int) -> bytes:
     return data
 
 
+def _has_audio(path: str) -> bool:
+    """True when ``path`` carries at least one audio stream."""
+    try:
+        import av
+        with av.open(path) as c:
+            return bool(c.streams.audio)
+    except Exception:
+        return False
+
+
+def _mux_original_audio(video_path: str, source_path: str,
+                        temps: Optional[list] = None) -> str:
+    """Put the source clip's audio back onto a re-encoded video.
+
+    The enhancement paths decode a clip to frames and encode a brand new video,
+    which leaves the audio behind: a clip that arrived with speech or music came
+    back silent, and only the ffmpeg fallback ever kept it. Both paths preserve
+    the source duration -- upscaling keeps the frame count and rate, and
+    interpolation compensates with ``output_fps`` -- so the track lines up as it
+    is, with no stretching.
+
+    The track is stream-copied when the output container accepts its codec and
+    re-encoded to AAC when it does not. Returns the new path, or ``video_path``
+    unchanged when the source has no audio or the remux fails: a silent result
+    is better than failing an enhancement that otherwise succeeded, and the
+    caller gets the frames it paid the GPU for either way.
+    """
+    import logging
+    _log = logging.getLogger(__name__)
+    if not source_path or not os.path.exists(source_path):
+        return video_path
+    if not _has_audio(source_path):
+        return video_path
+    out = tempfile.mktemp(suffix='_aud.mp4')
+    try:
+        import av
+        with av.open(video_path) as vin, av.open(source_path) as ain, \
+                av.open(out, mode='w') as outc:
+            v_in = vin.streams.video[0]
+            a_in = ain.streams.audio[0]
+            v_out = outc.add_stream_from_template(v_in)
+            try:
+                a_out = outc.add_stream_from_template(a_in)
+                stream_copy = True
+            except Exception:
+                # The source codec cannot live in MP4 (opus/vorbis from a webm,
+                # say) -- re-encode rather than drop the track.
+                a_out = outc.add_stream('aac', rate=a_in.rate)
+                try:
+                    a_out.bit_rate = 192000
+                except Exception:
+                    pass
+                stream_copy = False
+            for pkt in vin.demux(v_in):
+                if pkt.dts is None:
+                    continue
+                pkt.stream = v_out
+                outc.mux(pkt)
+            if stream_copy:
+                for pkt in ain.demux(a_in):
+                    if pkt.dts is None:
+                        continue
+                    pkt.stream = a_out
+                    outc.mux(pkt)
+            else:
+                # An encoder stream may not have its format/layout filled in
+                # until it is configured, so fall back to what AAC always takes.
+                _fmt = getattr(getattr(a_out, "format", None), "name", None) or "fltp"
+                _layout = getattr(getattr(a_out, "layout", None), "name", None) \
+                    or getattr(getattr(a_in, "layout", None), "name", None) or "stereo"
+                resampler = av.AudioResampler(
+                    format=_fmt, layout=_layout, rate=a_out.rate or a_in.rate)
+                for frame in ain.decode(a_in):
+                    for rs in resampler.resample(frame):
+                        rs.pts = None
+                        for pkt in a_out.encode(rs):
+                            outc.mux(pkt)
+                for pkt in a_out.encode():
+                    outc.mux(pkt)
+    except Exception as e:
+        _log.warning("could not carry the source audio into the result: %s", e)
+        try:
+            if os.path.exists(out):
+                os.unlink(out)
+        except Exception:
+            pass
+        return video_path
+    if temps is not None:
+        temps.append(out)
+    return out
+
+
 class _Mp4Writer:
     """Streaming H.264 MP4 writer via PyAV — add(frame) one at a time so a large
     (e.g. 4×-upscaled) sequence never has to be held fully in memory. In-process,
@@ -2816,8 +2915,9 @@ def _model_upscale_video(model_name: str, in_path: str, factor: int,
     except Exception:
         _wait_until_safe = None
     # Stream upscaled frames straight into the encoder so the (large, ×factor)
-    # output never has to be fully resident in memory. Audio is not carried in the
-    # pure in-process path (the enhancement targets are silent clips).
+    # output never has to be fully resident in memory. The source audio is muxed
+    # back on afterwards -- the frames are re-encoded, so it cannot survive the
+    # write itself.
     out = tempfile.mktemp(suffix='_up.mp4')
     temps.append(out)
     writer = _Mp4Writer(out, fps)
@@ -2847,7 +2947,7 @@ def _model_upscale_video(model_name: str, in_path: str, factor: int,
     finally:
         writer.close()
         _vid_progress_done()
-    return out
+    return _mux_original_audio(out, in_path, temps)
 
 
 def _count_video_frames(path: str) -> int:
@@ -3022,7 +3122,7 @@ def _interpolate_inprocess(path, multiplier, model_name, temps, tid=None, output
     temps.append(out_path)
     with open(out_path, "wb") as f:
         f.write(data)
-    return out_path
+    return _mux_original_audio(out_path, path, temps)
 
 
 def _cpu_thread_limit() -> int:
@@ -3383,7 +3483,8 @@ def _rife_ncnn_interpolate(path: str, multiplier: int, temps: list) -> str:
             raise RuntimeError(
                 f"reassembly failed: {r.stderr.decode(errors='replace')}")
         task_registry.finish(_tid, "done")
-        return out
+        # Reassembled from PNG frames, so the source audio is not in `out` yet.
+        return _mux_original_audio(out, path, temps)
     except TaskCancelled:
         task_registry.finish(_tid, "cancelled")
         raise

codai/compose/media.py

@@ -246,7 +246,12 @@ def fetch(ref: str, timeout: float = 120.0, max_bytes: int = 2 << 30) -> bytes:
         # this machine.
         raise MediaError(f"{ref}: not a URL, a /v1/files path, or base64 data")
     try:
-        return base64.b64decode(ref, validate=True)
+        # Whitespace is not part of base64 but is everywhere in practice: MIME
+        # wrapping at 76 columns, `base64` from a shell, Python's encodebytes.
+        # Strict validation is still what rejects a reference that is not base64
+        # at all -- it just must not reject a correctly encoded one for its line
+        # breaks, which is what every wrapped payload has.
+        return base64.b64decode("".join(ref.split()), validate=True)
     except Exception:
         raise MediaError("media reference is not a URL, a /v1/files path, a data: URI or base64")