NVIDIA Vision Bundle v2.1

Your agents need to see. Give them vision with a single call: upload an image, specify the objects you care about, get bounding boxes back as strict JSON. The included client is importable as a Python module, returns machine-parseable output on every call, and is designed to be wrapped as an agent tool (e.g. an MCP server) so LLM agents can discover and invoke it autonomously.

An unofficial, honest bundle of two NVIDIA research tools:

1. LocateAnything — public HuggingFace Space client for visual grounding
2. PiD — local Pixel Diffusion Decoder for high-resolution latent-to-image generation

• ✅ LocateAnything : No GPU required, no model download, public API
• ⚠️ PiD : Requires local NVIDIA GPU (~12-16 GB VRAM), checkpoints downloaded separately
• ⚠️ Both : Unofficial clients — we do not redistribute the models.

Table of contents

• Part A — LocateAnything Client
• Part B — PiD Local Setup
• Honest limitations
• Files in this pack

Part A — LocateAnything Client

A production-ready Python client (client.py) + prompts + specs to locate any object in images or videos using natural language.

Use cases covered:

• Open-set object detection (box or point)
• Referring-expression grounding ("the red car on the left")
• Multi-category batch processing
• JSON export of every detection with coordinates

Architecture

┌─────────────────────────────────────────────────────────────┐
│                 LOCATEANYTHING CLIENT v1.0                   │
├─────────────────────────────────────────────────────────────┤
│  Your Machine          │   HuggingFace Space (nvidia)       │
│  ──────────────────────┼──────────────────────────────────  │
│  client.py             │   LocateAnything-3B model          │
│  ├─ image/video input  │   ├─ vision encoder                │
│  ├─ prompt / category  │   ├─ Parallel Box Decoding (PBD)   │
│  ├─ retry / timeout    │   └─ text decoder (Qwen 2.5 3B)    │
│  └─ JSON / annotated   │                                    │
│      media output      │   ZeroGPU (free tier)              │
└─────────────────────────────────────────────────────────────┘

One-shot install (LocateAnything)

unzip locateanything-v1.zip && cd locateanything-v1
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Quick start

1. Single image

python client.py image photo.jpg --category "car,person,dog" --mode box

Output:

Input : photo.jpg
Task  : Detection | Mode: hybrid | Category: car,person,dog
Found : 3 detection(s)
  - car                  box    coords=[512.0, 312.0, 892.0, 498.0]
  - person               box    coords=[120.0, 200.0, 220.0, 580.0]
  - dog                  box    coords=[400.0, 450.0, 520.0, 560.0]
Annotated image: /tmp/gradio/.../annotated.jpg

2. Single video

python client.py video clip.mp4 --category "cat" --mode point --max-video-frames 8

3. Batch folder

mkdir input output
cp *.jpg *.png input/
python client.py batch input/ -o output/

Every result is saved as {stem}_result.json and annotated media is copied to output/.

4. Custom prompt (grounding)

python client.py image scene.jpg \
  --task Grounding \
  --prompt "Locate the small blue cup on the top shelf"

Part B — PiD Local Setup

PiD (Pixel Diffusion Decoder) is a plug-and-play diffusion decoder that replaces VAE/RAE decoders, turning latent representations directly into super-resolved pixels in a single pass.

Supported backbones: FLUX, FLUX.2, SD3, SDXL, Z-Image, Z-Image-Turbo, DINOv2, SigLIP.

Hardware requirements (PiD)

CPU-only is not supported for PiD inference.

One-shot install (PiD)

bash install_pid.sh

This script will:

1. Check for CUDA availability
2. Clone the official nv-tlabs/PiD repository
3. Create a conda environment (pid)
4. Install dependencies
5. Download checkpoints from HuggingFace (nvidia/PiD)

Quick start (PiD)

Decode a latent to 2K image (FLUX backbone)

python pid_runner.py \
  --backbone flux \
  --pid_ckpt_type 2k \
  --input latent.pt \
  --output output_2k.png

Upsample an image through VAE → PiD (SDXL backbone)

python pid_runner.py \
  --backbone sdxl \
  --pid_ckpt_type 2kto4k \
  --input photo_1024.png \
  --output photo_4k.png

See SPEC.md for the full backbone/resolution matrix and PROMPTS.md for recommended parameter sets.

Honest limitations

Files in this pack