• Stefy Lanza (nextime / spora )'s avatar
    feat: serve GeoCLIP visual geolocation via /v1/embeddings (geoclip + geoclip-location) · fd3221c6
    Stefy Lanza (nextime / spora ) authored
    Implements the HomeHunter request (REQUEST-geoclip-embeddings.md): two model ids
    on the existing embeddings endpoint, both returning 512-d L2-normalised vectors in
    ONE shared space so a property photo can be scored directly against candidate GPS
    coordinates.
    
      geoclip           — image encoder (CLIP ViT-L/14 + projection MLP), on GPU.
                          Accepts the image data URI in EITHER `input` (per the request)
                          or the `image` field (like dinov2).
      geoclip-location  — GPS encoder (equal-earth + RFF MLP), pinned to CPU: it is a
                          tiny MLP, so CPU is fast, frees VRAM, and is bit-exact across
                          batch sizes (GPU reduction order otherwise perturbs a cached
                          coordinate by ~1e-7). Batches an array of "lat,lon" strings to
                          one vector per element, in input order.
    
    Both sides load from the SAME bundled GeoCLIP checkpoint, so the spaces coincide by
    construction — the mismatched-checkpoint failure the request warns about cannot
    happen. Loaded as separate _EmbeddingModel('geoclip', ...) instances so each is
    independently evictable.
    
    transformers>=5 compat: CLIPModel.get_image_features now returns a
    BaseModelOutputWithPooling (768-d projected features in pooler_output) instead of a
    bare tensor, which breaks GeoCLIP's internal mlp() call. The image path extracts the
    768-d tensor itself (tolerant of both the old tensor and the new object) before the
    projection MLP.
    
    geoclip (+geopy, geographiclib) added to requirements.txt; its heavy deps (torch,
    transformers, Pillow, pandas, numpy) are already pinned, so no ML-stack churn.
    
    Verified end-to-end: 512-d L2-normalised both sides; batching + index order; same
    request bit-exact; both ids in GET /v1/models; image·location scoring varies by image.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01LoSpEthysqmseCc6Geizty
    fd3221c6
This project manages its dependencies using pip. Learn more
requirements.txt 4.86 KB