Object detection for AI agents, bounding boxes as JSON

Your agents need to see. Give them vision with a single call: upload an image, specify the objects you care about, get bounding boxes back as strict JSON. The included client is importable as a Python module, returns machine-parseable output on every call, and is designed to be wrapped as an agent tool (e.g. an MCP server) so LLM agents can discover and invoke it autonomously.

An unofficial, honest bundle of two NVIDIA research tools:

1. LocateAnything — public HuggingFace Space client for visual grounding
2. PiD — local Pixel Diffusion Decoder for high-resolution latent-to-image generation

• ✅ LocateAnything : No GPU required, no model download, public API
• ⚠️ PiD : Requires local NVIDIA GPU (~12-16 GB VRAM), checkpoints downloaded separately
• ⚠️ Both : Unofficial clients — we do not redistribute the models.

Table of contents

• Part A — LocateAnything Client
• Part B — PiD Local Setup
• Honest limitations
• Files in this pack

Part A — LocateAnything Client

A production-ready Python client (client.py) + prompts + specs to locate any object in images or videos using natural language.

Use cases covered:

• Open-set object detection (box or point)
• Referring-expression grounding ("the red car on the left")
• Multi-category batch processing
• JSON export of every detection with coordinates

Architecture

┌─────────────────────────────────────────────────────────────┐
│                 LOCATEANYTHING CLIENT v1.0                   │
├─────────────────────────────────────────────────────────────┤
│  Your Machine          │   HuggingFace Space (nvidia)       │
│  ──────────────────────┼──────────────────────────────────  │
│  client.py             │   LocateAnything-3B model          │
│  ├─ image/video input  │   ├─ vision encoder                │
│  ├─ prompt / category  │   ├─ Parallel Box Decoding (PBD)   │
│  ├─ retry / timeout    │   └─ text decoder (Qwen 2.5 3B)    │
│  └─ JSON / annotated   │                                    │
│      media output      │   ZeroGPU (free tier)              │
└─────────────────────────────────────────────────────────────┘

One-shot install (LocateAnything)

unzip locateanything-v1.zip && cd locateanything-v1
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Quick start

1. Single image

python client.py image photo.jpg --category "car,person,dog" --mode box

Output:

Input : photo.jpg
Task  : Detection | Mode: hybrid | Category: car,person,dog
Found : 3 detection(s)
  - car                  box    coords=[512.0, 312.0, 892.0, 498.0]
  - person               box    coords=[120.0, 200.0, 220.0, 580.0]
  - dog                  box    coords=[400.0, 450.0, 520.0, 560.0]
Annotated image: /tmp/gradio/.../annotated.jpg

2. Single video

python client.py video clip.mp4 --category "cat" --mode point --max-video-frames 8

3. Batch folder

mkdir input output
cp *.jpg *.png input/
python client.py batch input/ -o output/

Every result is saved as {stem}_result.json and annotated media is copied to output/.

4. Custom prompt (grounding)

python client.py image scene.jpg \
  --task Grounding \
  --prompt "Locate the small blue cup on the top shelf"

Part B — PiD Local Setup

PiD (Pixel Diffusion Decoder) is a plug-and-play diffusion decoder that replaces VAE/RAE decoders, turning latent representations directly into super-resolved pixels in a single pass.

Supported backbones: FLUX, FLUX.2, SD3, SDXL, Z-Image, Z-Image-Turbo, DINOv2, SigLIP.

Hardware requirements (PiD)

CPU-only is not supported for PiD inference.

One-shot install (PiD)

bash install_pid.sh

This script will:

1. Check for CUDA availability
2. Clone the official nv-tlabs/PiD repository
3. Create a conda environment (pid)
4. Install dependencies
5. Download checkpoints from HuggingFace (nvidia/PiD)

Quick start (PiD)

Decode a latent to 2K image (FLUX backbone)

python pid_runner.py \
  --backbone flux \
  --pid_ckpt_type 2k \
  --input latent.pt \
  --output output_2k.png

Upsample an image through VAE → PiD (SDXL backbone)

python pid_runner.py \
  --backbone sdxl \
  --pid_ckpt_type 2kto4k \
  --input photo_1024.png \
  --output photo_4k.png

See SPEC.md for the full backbone/resolution matrix and PROMPTS.md for recommended parameter sets.

Honest limitations

Files in this pack

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.