• Stefy Lanza (nextime / spora )'s avatar
    feat: serve GeoCLIP visual geolocation via /v1/embeddings (geoclip + geoclip-location) · fd3221c6
    Stefy Lanza (nextime / spora ) authored
    Implements the HomeHunter request (REQUEST-geoclip-embeddings.md): two model ids
    on the existing embeddings endpoint, both returning 512-d L2-normalised vectors in
    ONE shared space so a property photo can be scored directly against candidate GPS
    coordinates.
    
      geoclip           — image encoder (CLIP ViT-L/14 + projection MLP), on GPU.
                          Accepts the image data URI in EITHER `input` (per the request)
                          or the `image` field (like dinov2).
      geoclip-location  — GPS encoder (equal-earth + RFF MLP), pinned to CPU: it is a
                          tiny MLP, so CPU is fast, frees VRAM, and is bit-exact across
                          batch sizes (GPU reduction order otherwise perturbs a cached
                          coordinate by ~1e-7). Batches an array of "lat,lon" strings to
                          one vector per element, in input order.
    
    Both sides load from the SAME bundled GeoCLIP checkpoint, so the spaces coincide by
    construction — the mismatched-checkpoint failure the request warns about cannot
    happen. Loaded as separate _EmbeddingModel('geoclip', ...) instances so each is
    independently evictable.
    
    transformers>=5 compat: CLIPModel.get_image_features now returns a
    BaseModelOutputWithPooling (768-d projected features in pooler_output) instead of a
    bare tensor, which breaks GeoCLIP's internal mlp() call. The image path extracts the
    768-d tensor itself (tolerant of both the old tensor and the new object) before the
    projection MLP.
    
    geoclip (+geopy, geographiclib) added to requirements.txt; its heavy deps (torch,
    transformers, Pillow, pandas, numpy) are already pinned, so no ML-stack churn.
    
    Verified end-to-end: 512-d L2-normalised both sides; batching + index order; same
    request bit-exact; both ids in GET /v1/models; image·location scoring varies by image.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01LoSpEthysqmseCc6Geizty
    fd3221c6
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...