engine: route hot-path logs through os.write (fast_print) instead of print()

The engine emits a lot of debug output during generation. Builtin print() goes
through a buffered, lock-guarded text stream — per-call overhead plus a lock that
can serialize against other threads on the hot path. New codai/api/fastlog.py
fast_print writes each line with a single os.write(1) syscall: unbuffered, no
Python-level lock, atomic for pipe writes up to PIPE_BUF (log lines don't
interleave), with a fallback to real print() for stderr/failures. Imported as
print() in the generation hot path: api/text.py, backends/vulkan.py, backends/
cuda.py.

Verified: fast_print mirrors print's signature (sep/end/file/flush), writes to
fd 1, falls back for stderr; all three modules compile.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
parent 1b8c1c33
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
"""Fast, low-overhead stdout for engine logs.
The engine emits a lot of debug output from the generation hot path. Python's
``print()`` goes through a buffered, lock-guarded text stream — extra overhead per
call and a lock that can serialize against other threads. ``fast_print`` writes the
formatted line with a single ``os.write(1)`` syscall instead: unbuffered, no
Python-level lock, and atomic for pipe writes up to PIPE_BUF (so engine log lines
don't interleave). Import it as ``print`` in a hot module:
from codai.api.fastlog import fast_print as print
It mirrors ``print``'s signature and falls back to the real ``print`` for non-stdout
targets (e.g. ``file=sys.stderr``) or if the raw write fails.
"""
import builtins
import os
import sys
def fast_print(*args, sep=" ", end="\n", file=None, flush=False):
if file is not None and file is not sys.stdout:
return builtins.print(*args, sep=sep, end=end, file=file, flush=flush)
try:
data = (sep.join(str(a) for a in args) + end).encode("utf-8", "replace")
os.write(1, data)
except Exception:
try:
builtins.print(*args, sep=sep, end=end, flush=flush)
except Exception:
pass
...@@ -25,6 +25,11 @@ import time ...@@ -25,6 +25,11 @@ import time
import uuid import uuid
from typing import AsyncGenerator, Dict, List, Optional from typing import AsyncGenerator, Dict, List, Optional
# Engine debug/log lines go out via a single os.write(1) syscall instead of the
# buffered, lock-guarded builtin print() — lower overhead on the generation hot
# path. Shadows print() for this whole module.
from codai.api.fastlog import fast_print as print # noqa: A004
from fastapi import APIRouter, HTTPException, Request from fastapi import APIRouter, HTTPException, Request
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
......
...@@ -23,6 +23,10 @@ from typing import Optional, List, Dict ...@@ -23,6 +23,10 @@ from typing import Optional, List, Dict
from threading import Thread from threading import Thread
from abc import ABC from abc import ABC
# Low-overhead stdout for the generation hot path: a single os.write(1) per line
# instead of buffered, lock-guarded print(). Shadows print() for this module.
from codai.api.fastlog import fast_print as print # noqa: A004
# Import from codai modules # Import from codai modules
from codai.backends.base import ModelBackend from codai.backends.base import ModelBackend
from codai.models.capabilities import detect_model_capabilities from codai.models.capabilities import detect_model_capabilities
......
...@@ -24,6 +24,10 @@ import time ...@@ -24,6 +24,10 @@ import time
from typing import AsyncIterator, Optional, Union, List, Dict, Any from typing import AsyncIterator, Optional, Union, List, Dict, Any
from pathlib import Path from pathlib import Path
# Low-overhead stdout for the generation hot path: a single os.write(1) per line
# instead of buffered, lock-guarded print(). Shadows print() for this module.
from codai.api.fastlog import fast_print as print # noqa: A004
from codai.backends.base import ModelBackend from codai.backends.base import ModelBackend
from codai.models.utils import ( from codai.models.utils import (
check_hf_chat_template, check_hf_chat_template,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment