-
Stefy Lanza (nextime / spora ) authored
training failed: Command '[…py310…, '-m', 'venv', '/AI/guffcache/longcat_venv-train']' returned non-zero exit status 1. An exit status is not a diagnosis. The actual reason was in the log only because the child's stderr happened to land there: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding ModuleNotFoundError: No module named 'encodings' Every step of the venv build went through subprocess.run(check=True), which reports a number and discards what the process said. They now go through a runner that raises with the child's output, truncated at both ends if it is long. The next failure here names itself instead of needing a log archaeologist. The environment is also stripped for anything that spawns the 3.10 interpreter — PYTHONHOME, PYTHONPATH, VIRTUAL_ENV, PYTHONEXECUTABLE and friends — because that is exactly the class of variable that produces the error above, and the failure is fatal and silent in the parent. HF cache paths, CUDA_VISIBLE_DEVICES and everything else still reach the child. This covers the venv build, the pip installs, the service launch and the train/quantise workers; the service used dict(os.environ) directly before. Worth being straight about: I could NOT reproduce the failure. The same command from a throwaway container on the same image, same user, same mount, exits 0 and writes a complete venv, and the engine process has no PYTHONHOME, PYTHONPATH or VIRTUAL_ENV in its environment. So this is not a confirmed fix for that specific failure — it is the removal of the reason I cannot tell you what went wrong, plus hardening against the one cause the error text points at. The retry will either work or finally say why. 12 tests. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
0e33dc3c