• Stefy Lanza (nextime / spora )'s avatar
    embeddings: multi-vector on the image that can serve it, and a 400 where it cannot (v0.3.73) · da2859f0
    Stefy Lanza (nextime / spora ) authored
    bge-m3's sparse and colbert come from linear heads (sparse_linear.pt /
    colbert_linear.pt) applied to the last hidden state -- a transformers forward
    pass, not an embedding server's output. Two consequences that were both silent:
    
    A DENSE-ONLY BACKEND NOW SAYS SO. `embedding_types` was simply ignored outside
    the native path: a caller asking for sparse got a normal-looking response with
    only `embedding` in it, its hybrid retrieval then ran on half the signal, and
    the result was merely a worse score -- which reads as a model problem, not a
    configuration one. Now a 400 that names the cause, where it DOES work
    (coderai-embeddings), and why the vLLM image cannot: that image is LIGHT core
    with no torch in its main venv BY DESIGN, so _load_bge_m3 could not even import.
    Dense-only requests are untouched.
    
    MULTI-VECTOR LONG INPUTS ARE WINDOWED. 8192 tokens is ~6000 words, so this bites
    on whole documents rather than passages -- and when it bit, the tokenizer's
    `truncation=True` kept the opening and said nothing. Each type is merged the way
    the encoder already merges WITHIN a window, which is what makes these choices
    defensible rather than invented:
    
      dense    mean, RE-NORMALIZED. It is an L2-normalized CLS vector and a mean of
               unit vectors is not one; cosine similarity on a non-unit vector is
               quietly wrong.
      sparse   MAX per token id -- the encoder max-pools per token id already
               (scatter_reduce amax), and a term's weight should be its strongest
               occurrence, not an average diluted by the pages that never mention
               it. A test guards against someone later tidying max into mean.
      colbert  CONCATENATED: max-sim scores each query token against ALL document
               token vectors, so every content token of every window belongs.
    
    `embed_long_input=truncate` keeps the old behaviour explicitly, and `first`
    keeps only the opening window, for an operator who would rather have that than a
    merge.
    
    Also removes three customer references I had put in shipped profile comments,
    two of which went out in 0.3.69. A capability must be scoped and justified by
    what it does, never by one install's needs, and a customer name does not belong
    in a released image. The measurements keep their force without the attribution:
    "hybrid retrieval wants dense and sparse from one pass", "a production fleet",
    "a GPU-less orchestrator".
    
    39 tests. Suite at the 25 pre-existing failures.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    da2859f0
Name
Last commit
Last update
..
conftest.py Loading commit data...
ocr_lane_stub_worker.py Loading commit data...
test_admin_auth.py Loading commit data...
test_admin_forward_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_adoption_matches_image_and_card.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_autoupgrade.py Loading commit data...
test_autoupgrade_midnight_force.py Loading commit data...
test_autoupgrade_recovers_a_down_server.py Loading commit data...
test_autoupgrade_sees_training.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_shutdown.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_character_studio_fetch.py Loading commit data...
test_chat_passthrough.py Loading commit data...
test_cluster.py Loading commit data...
test_compose.py Loading commit data...
test_embed_long_input.py Loading commit data...
test_forward_auth_peer_trust.py Loading commit data...
test_front_api_auth.py Loading commit data...
test_front_inflight_leak.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_image_has_c_compiler.py Loading commit data...
test_ledger_out_of_band.py Loading commit data...
test_longcat_avatar_layout.py Loading commit data...
test_longcat_bsa.py Loading commit data...
test_longcat_cancellable.py Loading commit data...
test_longcat_checkpoint_resolve.py Loading commit data...
test_longcat_component_quant.py Loading commit data...
test_longcat_config_ui.py Loading commit data...
test_longcat_external_training.py Loading commit data...
test_longcat_int8_load.py Loading commit data...
test_longcat_lora.py Loading commit data...
test_longcat_orchestration.py Loading commit data...
test_longcat_packaging.py Loading commit data...
test_longcat_placement.py Loading commit data...
test_longcat_progress.py Loading commit data...
test_longcat_prompt_cache.py Loading commit data...
test_longcat_quantize.py Loading commit data...
test_longcat_scale.py Loading commit data...
test_longcat_tasks.py Loading commit data...
test_longcat_variants.py Loading commit data...
test_longcat_vendored_source.py Loading commit data...
test_longcat_venv_env.py Loading commit data...
test_longcat_vram_estimate.py Loading commit data...
test_longcat_worker.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_model_scan_cache.py Loading commit data...
test_models_context_window.py Loading commit data...
test_ocr_accent_repair.py Loading commit data...
test_ocr_blank_page.py Loading commit data...
test_ocr_chandra.py Loading commit data...
test_ocr_confidence.py Loading commit data...
test_ocr_detector_venv.py Loading commit data...
test_ocr_dual_mode.py Loading commit data...
test_ocr_fanout.py Loading commit data...
test_ocr_language.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_raster_pool.py Loading commit data...
test_ocr_request_budgets.py Loading commit data...
test_ocr_resident_budget.py Loading commit data...
test_ocr_resume_engines.py Loading commit data...
test_ocr_resume_shared.py Loading commit data...
test_ocr_served_model.py Loading commit data...
test_ocr_token_trace.py Loading commit data...
test_ocr_unreadable_and_dpi.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_ocr_window_published.py Loading commit data...
test_ocr_worker_lanes.py Loading commit data...
test_olmocr_pod_tls.py Loading commit data...
test_oom_priority.py Loading commit data...
test_pod_log_drain.py Loading commit data...
test_pod_token_adoption_race.py Loading commit data...
test_pool_growth_adopts.py Loading commit data...
test_proxy_logprobs.py Loading commit data...
test_remote_execution.py Loading commit data...
test_runpod_client_scale.py Loading commit data...
test_runpod_fleet_status.py Loading commit data...
test_runpod_ocr_remotization.py Loading commit data...
test_runpod_placement.py Loading commit data...
test_runpod_pod_extra_args.py Loading commit data...
test_runpod_reaper_grace.py Loading commit data...
test_runpod_regional_volumes.py Loading commit data...
test_runpod_registry_heartbeat.py Loading commit data...
test_runpod_request_gpu_choice.py Loading commit data...
test_runpod_restart_orphans.py Loading commit data...
test_runpod_scale_down.py Loading commit data...
test_runpod_scale_out.py Loading commit data...
test_runpod_schedule_region_spend.py Loading commit data...
test_runpod_warm_floor_parallel.py Loading commit data...
test_runpod_warm_floor_rerent.py Loading commit data...
test_runpod_warmup.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_surya_item_cap.py Loading commit data...
test_surya_loop_reported.py Loading commit data...
test_surya_retry_budget.py Loading commit data...
test_surya_runpod_fixes.py Loading commit data...
test_surya_thread_boundary.py Loading commit data...
test_text_lora_to_pod.py Loading commit data...
test_township_api_key.py Loading commit data...
test_township_clip_geometry.py Loading commit data...
test_township_longcat.py Loading commit data...
test_township_page_js.py Loading commit data...
test_township_templates.py Loading commit data...
test_township_video_dataset.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_video_as_b64_scope.py Loading commit data...
test_video_geometry.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_vram_releasers.py Loading commit data...
test_weight_stores.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...