更新代码
This commit is contained in:
435
README.md
435
README.md
@ -1,395 +1,96 @@
|
|||||||
# SAM 3: Segment Anything with Concepts
|
# water_V5.py：超大遥感影像水体分割（SAM3 + Rasterio 分块）
|
||||||
|
|
||||||
Meta Superintelligence Labs
|
这个脚本用于在超大 GeoTIFF/遥感影像上分割水体（如 river/water body），重点在于在 16GB 显存条件下平衡速度与显存：
|
||||||
|
- 影像按“子区域”切分处理，避免一次性处理整图
|
||||||
|
- 每个子区域先做一次低分辨率粗分割定位水体，再只对“边缘带”进行高分辨率分块精修
|
||||||
|
- 精修的概率融合与大数组操作尽量放在 CPU（float16），显著降低显存占用
|
||||||
|
|
||||||
[Nicolas Carion](https://www.nicolascarion.com/)\*,
|
对应脚本：[water_V5.py](file:///e:/code/sam3-main/sam3-main/water_V5.py)
|
||||||
[Laura Gustafson](https://scholar.google.com/citations?user=c8IpF9gAAAAJ&hl=en)\*,
|
|
||||||
[Yuan-Ting Hu](https://scholar.google.com/citations?user=E8DVVYQAAAAJ&hl=en)\*,
|
|
||||||
[Shoubhik Debnath](https://scholar.google.com/citations?user=fb6FOfsAAAAJ&hl=en)\*,
|
|
||||||
[Ronghang Hu](https://ronghanghu.com/)\*,
|
|
||||||
[Didac Suris](https://www.didacsuris.com/)\*,
|
|
||||||
[Chaitanya Ryali](https://scholar.google.com/citations?user=4LWx24UAAAAJ&hl=en)\*,
|
|
||||||
[Kalyan Vasudev Alwala](https://scholar.google.co.in/citations?user=m34oaWEAAAAJ&hl=en)\*,
|
|
||||||
[Haitham Khedr](https://hkhedr.com/)\*, Andrew Huang,
|
|
||||||
[Jie Lei](https://jayleicn.github.io/),
|
|
||||||
[Tengyu Ma](https://scholar.google.com/citations?user=VeTSl0wAAAAJ&hl=en),
|
|
||||||
[Baishan Guo](https://scholar.google.com/citations?user=BC5wDu8AAAAJ&hl=en),
|
|
||||||
Arpit Kalla, [Markus Marks](https://damaggu.github.io/),
|
|
||||||
[Joseph Greer](https://scholar.google.com/citations?user=guL96CkAAAAJ&hl=en),
|
|
||||||
Meng Wang, [Peize Sun](https://peizesun.github.io/),
|
|
||||||
[Roman Rädle](https://scholar.google.com/citations?user=Tpt57v0AAAAJ&hl=en),
|
|
||||||
[Triantafyllos Afouras](https://www.robots.ox.ac.uk/~afourast/),
|
|
||||||
[Effrosyni Mavroudi](https://scholar.google.com/citations?user=vYRzGGEAAAAJ&hl=en),
|
|
||||||
[Katherine Xu](https://k8xu.github.io/)°,
|
|
||||||
[Tsung-Han Wu](https://patrickthwu.com/)°,
|
|
||||||
[Yu Zhou](https://yu-bryan-zhou.github.io/)°,
|
|
||||||
[Liliane Momeni](https://scholar.google.com/citations?user=Lb-KgVYAAAAJ&hl=en)°,
|
|
||||||
[Rishi Hazra](https://rishihazra.github.io/)°,
|
|
||||||
[Shuangrui Ding](https://mark12ding.github.io/)°,
|
|
||||||
[Sagar Vaze](https://sgvaze.github.io/)°,
|
|
||||||
[Francois Porcher](https://scholar.google.com/citations?user=LgHZ8hUAAAAJ&hl=en)°,
|
|
||||||
[Feng Li](https://fengli-ust.github.io/)°,
|
|
||||||
[Siyuan Li](https://siyuanliii.github.io/)°,
|
|
||||||
[Aishwarya Kamath](https://ashkamath.github.io/)°,
|
|
||||||
[Ho Kei Cheng](https://hkchengrex.com/)°,
|
|
||||||
[Piotr Dollar](https://pdollar.github.io/)†,
|
|
||||||
[Nikhila Ravi](https://nikhilaravi.com/)†,
|
|
||||||
[Kate Saenko](https://ai.bu.edu/ksaenko.html)†,
|
|
||||||
[Pengchuan Zhang](https://pzzhang.github.io/pzzhang/)†,
|
|
||||||
[Christoph Feichtenhofer](https://feichtenhofer.github.io/)†
|
|
||||||
|
|
||||||
\* core contributor, ° intern, † project lead, order is random within groups
|
## 功能概览
|
||||||
|
|
||||||
[[`Paper`](https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/)]
|
- 读取 GeoTIFF 子区域（Rasterio Window/MemoryFile）
|
||||||
[[`Project`](https://ai.meta.com/sam3)]
|
- overview 粗分割（低分辨率一次推理）
|
||||||
[[`Demo`](https://segment-anything.com/)]
|
- 由粗分割生成边缘带（ring/band）
|
||||||
[[`Blog`](https://ai.meta.com/blog/segment-anything-model-3/)]
|
- 仅对边缘带覆盖到的 tile 做精修推理（重叠窗口融合）
|
||||||
[[`BibTeX`](#citing-sam-3)]
|
- 后处理：内部 NoData 空洞填充、连通域面积过滤
|
||||||
|
- 输出单通道 uint8 掩膜 GeoTIFF（像元值 0/1）
|
||||||
|
|
||||||
 SAM 3 is a unified foundation model for promptable segmentation in images and videos. It can detect, segment, and track objects using text or visual prompts such as points, boxes, and masks. Compared to its predecessor [SAM 2](https://github.com/facebookresearch/sam2), SAM 3 introduces the ability to exhaustively segment all instances of an open-vocabulary concept specified by a short text phrase or exemplars. Unlike prior work, SAM 3 can handle a vastly larger set of open-vocabulary prompts. It achieves 75-80% of human performance on our new [SA-CO benchmark](https://github.com/facebookresearch/sam3?tab=readme-ov-file#sa-co-dataset) which contains 270K unique concepts, over 50 times more than existing benchmarks.
|
## 环境依赖
|
||||||
|
|
||||||
This breakthrough is driven by an innovative data engine that has automatically annotated over 4 million unique concepts, creating the largest high-quality open-vocabulary segmentation dataset to date. In addition, SAM 3 introduces a new model architecture featuring a presence token that improves discrimination between closely related text prompts (e.g., “a player in white” vs. “a player in red”), as well as a decoupled detector–tracker design that minimizes task interference and scales efficiently with data.
|
- Python + PyTorch（GPU 可选但推荐）
|
||||||
|
- rasterio
|
||||||
|
- numpy
|
||||||
|
- PIL (Pillow)
|
||||||
|
- scipy
|
||||||
|
- tqdm
|
||||||
|
|
||||||
<p align="center">
|
提示：运行时可能看到 `pkg_resources is deprecated` 警告，不影响推理结果。
|
||||||
<img src="assets/dog.gif" width=380 />
|
|
||||||
<img src="assets/player.gif" width=380 />
|
|
||||||
</p>
|
|
||||||
|
|
||||||
## Installation
|
## 运行方式
|
||||||
|
|
||||||
### Prerequisites
|
1. 打开 [water_V5.py](file:///e:/code/sam3-main/sam3-main/water_V5.py)，修改下面参数：
|
||||||
|
- `image_path`：输入 GeoTIFF
|
||||||
- Python 3.12 or higher
|
- `mask_output_path`：输出掩膜 GeoTIFF
|
||||||
- PyTorch 2.7 or higher
|
- `prompt`：文本提示（例如 `"water body"` / `"river"` / `"water"`）
|
||||||
- CUDA-compatible GPU with CUDA 12.6 or higher
|
2. 运行：
|
||||||
|
|
||||||
1. **Create a new Conda environment:**
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
conda create -n sam3 python=3.12
|
python e:\code\sam3-main\sam3-main\water_V5.py
|
||||||
conda deactivate
|
|
||||||
conda activate sam3
|
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Install PyTorch with CUDA support:**
|
输出 `mask_output_path` 为 uint8 单通道，像元值为 0/1，并保留输入影像的空间参考（transform/CRS）。
|
||||||
|
|
||||||
```bash
|
## 关键参数说明（建议优先调这几个）
|
||||||
pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Clone the repository and install the package:**
|
在脚本“参数设置”区域：
|
||||||
|
|
||||||
```bash
|
- `coarse_read_max_side`：overview 粗分割输入的最大边长（越大越准/越慢，越小越快/越粗）
|
||||||
git clone https://github.com/facebookresearch/sam3.git
|
- 建议：`1024 ~ 1536`（16GB 显存一般可从 1200 开始）
|
||||||
cd sam3
|
- `tile_size`：精修分块大小（原图像素）
|
||||||
pip install -e .
|
- 建议：`1536`（更稳）或 `2048`（更快但更吃内存/更容易慢）
|
||||||
```
|
- `overlap`：精修分块重叠（减少拼接缝）
|
||||||
|
- 建议：`128`（更快）或 `256`（边缘更稳但慢）
|
||||||
|
- `band_radius`：边缘带宽度（越大精修范围越大，速度越慢）
|
||||||
|
- 建议：`48 ~ 96`
|
||||||
|
- `coarse_threshold` / `final_threshold`：粗分割/最终阈值
|
||||||
|
- 通常 `0.5` 可用；若漏检多，可适当降低阈值（例如 `0.4`）
|
||||||
|
|
||||||
4. **Install additional dependencies for example notebooks or development:**
|
分区域参数：
|
||||||
|
- `num_splits_y`、`num_splits_x`：将整图拆成多少个子区域处理
|
||||||
|
- `region_overlap`：子区域之间重叠，避免区域边界出现断裂
|
||||||
|
|
||||||
```bash
|
后处理参数：
|
||||||
# For running example notebooks
|
- `min_area`：移除小碎片的最小连通域面积（像素）
|
||||||
pip install -e ".[notebooks]"
|
- `keep_largest_only`：只保留最大连通域（适合只想要“最大水体”的场景）
|
||||||
|
|
||||||
# For development
|
## 性能与显存的取舍建议（16GB）
|
||||||
pip install -e ".[train,dev]"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Getting Started
|
优先级从高到低：
|
||||||
|
|
||||||
⚠️ Before using SAM 3, please request access to the checkpoints on the SAM 3
|
- 优先调小 `tile_size`（例如 2048 → 1536），通常能显著降低显存与峰值计算量
|
||||||
Hugging Face [repo](https://huggingface.co/facebook/sam3). Once accepted, you
|
- `overlap` 从 256 降到 128，可明显加速（代价是拼接缝更可能出现）
|
||||||
need to be authenticated to download the checkpoints. You can do this by running
|
- `band_radius` 适当减小，让精修只发生在更窄的边缘范围
|
||||||
the following [steps](https://huggingface.co/docs/huggingface_hub/en/quick-start#authentication)
|
- `coarse_read_max_side` 适度提高可减少误分/漏分，但会更慢
|
||||||
(e.g. `hf auth login` after generating an access token.)
|
|
||||||
|
|
||||||
### Basic Usage
|
脚本已做的关键优化：
|
||||||
|
- 精修 tile 的“是否需要处理”的判断在 CPU 上完成，避免每个 tile 强制 GPU 同步
|
||||||
|
- per-tile 的强度拉伸（percentile）改为“子区域级别一次计算并复用”
|
||||||
|
- 推理使用 autocast（float16），通常更省显存
|
||||||
|
|
||||||
```python
|
## 输出说明
|
||||||
import torch
|
|
||||||
#################################### For Image ####################################
|
|
||||||
from PIL import Image
|
|
||||||
from sam3.model_builder import build_sam3_image_model
|
|
||||||
from sam3.model.sam3_image_processor import Sam3Processor
|
|
||||||
# Load the model
|
|
||||||
model = build_sam3_image_model()
|
|
||||||
processor = Sam3Processor(model)
|
|
||||||
# Load an image
|
|
||||||
image = Image.open("<YOUR_IMAGE_PATH.jpg>")
|
|
||||||
inference_state = processor.set_image(image)
|
|
||||||
# Prompt the model with text
|
|
||||||
output = processor.set_text_prompt(state=inference_state, prompt="<YOUR_TEXT_PROMPT>")
|
|
||||||
|
|
||||||
# Get the masks, bounding boxes, and scores
|
- 输出掩膜为 `uint8` 单通道：
|
||||||
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]
|
- 0：非水体
|
||||||
|
- 1：水体
|
||||||
|
- 输出 GeoTIFF 继承输入的 profile，并设置 `compress='lzw'`
|
||||||
|
|
||||||
#################################### For Video ####################################
|
## 常见问题
|
||||||
|
|
||||||
from sam3.model_builder import build_sam3_video_predictor
|
- 运行报 `AssertionError`（RoPE 相关）
|
||||||
|
- 该模型对输入分辨率有约束。脚本默认 `coarse_resolution = 1008`、`fine_resolution = 1008`，不要随意改成 768/512 等。
|
||||||
|
|
||||||
video_predictor = build_sam3_video_predictor()
|
- 输出全黑/全白
|
||||||
video_path = "<YOUR_VIDEO_PATH>" # a JPEG folder or an MP4 video file
|
- 先换 `prompt`（如 `"water"` / `"river"`），再调 `coarse_threshold/final_threshold`
|
||||||
# Start a session
|
- 若影像波段不是 RGB，脚本会取前三个波段做可视输入；需要更合适的波段组合时应自行调整 `_bands_to_pil`
|
||||||
response = video_predictor.handle_request(
|
|
||||||
request=dict(
|
|
||||||
type="start_session",
|
|
||||||
resource_path=video_path,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
response = video_predictor.handle_request(
|
|
||||||
request=dict(
|
|
||||||
type="add_prompt",
|
|
||||||
session_id=response["session_id"],
|
|
||||||
frame_index=0, # Arbitrary frame index
|
|
||||||
text="<YOUR_TEXT_PROMPT>",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
output = response["outputs"]
|
|
||||||
```
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
The `examples` directory contains notebooks demonstrating how to use SAM3 with
|
|
||||||
various types of prompts:
|
|
||||||
|
|
||||||
- [`sam3_image_predictor_example.ipynb`](examples/sam3_image_predictor_example.ipynb)
|
|
||||||
: Demonstrates how to prompt SAM 3 with text and visual box prompts on images.
|
|
||||||
- [`sam3_video_predictor_example.ipynb`](examples/sam3_video_predictor_example.ipynb)
|
|
||||||
: Demonstrates how to prompt SAM 3 with text prompts on videos, and doing
|
|
||||||
further interactive refinements with points.
|
|
||||||
- [`sam3_image_batched_inference.ipynb`](examples/sam3_image_batched_inference.ipynb)
|
|
||||||
: Demonstrates how to run batched inference with SAM 3 on images.
|
|
||||||
- [`sam3_agent.ipynb`](examples/sam3_agent.ipynb): Demonsterates the use of SAM
|
|
||||||
3 Agent to segment complex text prompt on images.
|
|
||||||
- [`saco_gold_silver_vis_example.ipynb`](examples/saco_gold_silver_vis_example.ipynb)
|
|
||||||
: Shows a few examples from SA-Co image evaluation set.
|
|
||||||
- [`saco_veval_vis_example.ipynb`](examples/saco_veval_vis_example.ipynb) :
|
|
||||||
Shows a few examples from SA-Co video evaluation set.
|
|
||||||
|
|
||||||
There are additional notebooks in the examples directory that demonstrate how to
|
|
||||||
use SAM 3 for interactive instance segmentation in images and videos (SAM 1/2
|
|
||||||
tasks), or as a tool for an MLLM, and how to run evaluations on the SA-Co
|
|
||||||
dataset.
|
|
||||||
|
|
||||||
To run the Jupyter notebook examples:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Make sure you have the notebooks dependencies installed
|
|
||||||
pip install -e ".[notebooks]"
|
|
||||||
|
|
||||||
# Start Jupyter notebook
|
|
||||||
jupyter notebook examples/sam3_image_predictor_example.ipynb
|
|
||||||
```
|
|
||||||
|
|
||||||
## Model
|
|
||||||
|
|
||||||
SAM 3 consists of a detector and a tracker that share a vision encoder. It has 848M parameters. The
|
|
||||||
detector is a DETR-based model conditioned on text, geometry, and image
|
|
||||||
exemplars. The tracker inherits the SAM 2 transformer encoder-decoder
|
|
||||||
architecture, supporting video segmentation and interactive refinement.
|
|
||||||
|
|
||||||
## Image Results
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<table style="min-width: 80%; border: 2px solid #ddd; border-collapse: collapse">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th rowspan="3" style="border-right: 2px solid #ddd; padding: 12px 20px">Model</th>
|
|
||||||
<th colspan="3" style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">Instance Segmentation</th>
|
|
||||||
<th colspan="5" style="text-align: center; padding: 12px 20px">Box Detection</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVIS</th>
|
|
||||||
<th style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">SA-Co/Gold</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVIS</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">COCO</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">SA-Co/Gold</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP<sub>o</sub>
|
|
||||||
</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Human</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">72.8</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">74.0</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">OWLv2*</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px; color: #999">29.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px; color: #999">43.4</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">24.6</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px; color: #999">30.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px; color: #999">45.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">46.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">23.9</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">24.5</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">DINO-X</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">38.5</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">21.3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">52.4</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">56.0</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">22.5</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Gemini 2.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">13.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">13.0</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">16.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">14.4</td>
|
|
||||||
</tr>
|
|
||||||
<tr style="border-top: 2px solid #b19c9cff">
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">SAM 3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">37.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">48.5</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">54.1</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">40.6</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">53.6</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">56.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">55.7</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">55.7</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<p style="text-align: center; margin-top: 10px; font-size: 0.9em; color: #ddd;">* Partially trained on LVIS, AP<sub>o</sub> refers to COCO-O accuracy</p>
|
|
||||||
|
|
||||||
</div>
|
|
||||||
|
|
||||||
## Video Results
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<table style="min-width: 80%; border: 2px solid #ddd; border-collapse: collapse">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th rowspan="2" style="border-right: 2px solid #ddd; padding: 12px 20px">Model</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">SA-V test</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">YT-Temporal-1B test</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">SmartGlasses test</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVVIS test</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">BURST test</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">mAP</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">HOTA</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Human</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">53.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">70.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">71.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">78.4</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">58.5</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">72.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
</tr>
|
|
||||||
<tr style="border-top: 2px solid #b19c9cff">
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">SAM 3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">30.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">58.0</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">50.8</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">69.9</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">36.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">63.6</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">36.3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">44.5</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
## SA-Co Dataset
|
|
||||||
|
|
||||||
We release 2 image benchmarks, [SA-Co/Gold](scripts/eval/gold/README.md) and
|
|
||||||
[SA-Co/Silver](scripts/eval/silver/README.md), and a video benchmark
|
|
||||||
[SA-Co/VEval](scripts/eval/veval/README.md). The datasets contain images (or videos) with annotated noun phrases. Each image/video and noun phrase pair is annotated with instance masks and unique IDs of each object matching the phrase. Phrases that have no matching objects (negative prompts) have no masks, shown in red font in the figure. See the linked READMEs for more details on how to download and run evaluations on the datasets.
|
|
||||||
|
|
||||||
* HuggingFace host: [SA-Co/Gold](https://huggingface.co/datasets/facebook/SACo-Gold), [SA-Co/Silver](https://huggingface.co/datasets/facebook/SACo-Silver) and [SA-Co/VEval](https://huggingface.co/datasets/facebook/SACo-VEval)
|
|
||||||
* Roboflow host: [SA-Co/Gold](https://universe.roboflow.com/sa-co-gold), [SA-Co/Silver](https://universe.roboflow.com/sa-co-silver) and [SA-Co/VEval](https://universe.roboflow.com/sa-co-veval)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Development
|
|
||||||
|
|
||||||
To set up the development environment:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -e ".[dev,train]"
|
|
||||||
```
|
|
||||||
|
|
||||||
To format the code:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ufmt format .
|
|
||||||
```
|
|
||||||
|
|
||||||
## Contributing
|
|
||||||
|
|
||||||
See [contributing](CONTRIBUTING.md) and the
|
|
||||||
[code of conduct](CODE_OF_CONDUCT.md).
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
This project is licensed under the SAM License - see the [LICENSE](LICENSE) file
|
|
||||||
for details.
|
|
||||||
|
|
||||||
## Acknowledgements
|
|
||||||
|
|
||||||
We would like to thank the following people for their contributions to the SAM 3 project: Alex He, Alexander Kirillov,
|
|
||||||
Alyssa Newcomb, Ana Paula Kirschner Mofarrej, Andrea Madotto, Andrew Westbury, Ashley Gabriel, Azita Shokpour,
|
|
||||||
Ben Samples, Bernie Huang, Carleigh Wood, Ching-Feng Yeh, Christian Puhrsch, Claudette Ward, Daniel Bolya,
|
|
||||||
Daniel Li, Facundo Figueroa, Fazila Vhora, George Orlin, Hanzi Mao, Helen Klein, Hu Xu, Ida Cheng, Jake Kinney,
|
|
||||||
Jiale Zhi, Jo Sampaio, Joel Schlosser, Justin Johnson, Kai Brown, Karen Bergan, Karla Martucci, Kenny Lehmann,
|
|
||||||
Maddie Mintz, Mallika Malhotra, Matt Ward, Michelle Chan, Michelle Restrepo, Miranda Hartley, Muhammad Maaz,
|
|
||||||
Nisha Deo, Peter Park, Phillip Thomas, Raghu Nayani, Rene Martinez Doehner, Robbie Adkins, Ross Girshik, Sasha
|
|
||||||
Mitts, Shashank Jain, Spencer Whitehead, Ty Toledano, Valentin Gabeur, Vincent Cho, Vivian Lee, William Ngan,
|
|
||||||
Xuehai He, Yael Yungster, Ziqi Pang, Ziyi Dou, Zoe Quake.
|
|
||||||
|
|
||||||
## Citing SAM 3
|
|
||||||
|
|
||||||
If you use SAM 3 or the SA-Co dataset in your research, please use the following BibTeX entry.
|
|
||||||
|
|
||||||
```bibtex
|
|
||||||
@misc{carion2025sam3segmentconcepts,
|
|
||||||
title={SAM 3: Segment Anything with Concepts},
|
|
||||||
author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
|
|
||||||
year={2025},
|
|
||||||
eprint={2511.16719},
|
|
||||||
archivePrefix={arXiv},
|
|
||||||
primaryClass={cs.CV},
|
|
||||||
url={https://arxiv.org/abs/2511.16719},
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|||||||
@ -1,96 +0,0 @@
|
|||||||
# water_V5.py：超大遥感影像水体分割（SAM3 + Rasterio 分块）
|
|
||||||
|
|
||||||
这个脚本用于在超大 GeoTIFF/遥感影像上分割水体（如 river/water body），重点在于在 16GB 显存条件下平衡速度与显存：
|
|
||||||
- 影像按“子区域”切分处理，避免一次性处理整图
|
|
||||||
- 每个子区域先做一次低分辨率粗分割定位水体，再只对“边缘带”进行高分辨率分块精修
|
|
||||||
- 精修的概率融合与大数组操作尽量放在 CPU（float16），显著降低显存占用
|
|
||||||
|
|
||||||
对应脚本：[water_V5.py](file:///e:/code/sam3-main/sam3-main/water_V5.py)
|
|
||||||
|
|
||||||
## 功能概览
|
|
||||||
|
|
||||||
- 读取 GeoTIFF 子区域（Rasterio Window/MemoryFile）
|
|
||||||
- overview 粗分割（低分辨率一次推理）
|
|
||||||
- 由粗分割生成边缘带（ring/band）
|
|
||||||
- 仅对边缘带覆盖到的 tile 做精修推理（重叠窗口融合）
|
|
||||||
- 后处理：内部 NoData 空洞填充、连通域面积过滤
|
|
||||||
- 输出单通道 uint8 掩膜 GeoTIFF（像元值 0/1）
|
|
||||||
|
|
||||||
## 环境依赖
|
|
||||||
|
|
||||||
- Python + PyTorch（GPU 可选但推荐）
|
|
||||||
- rasterio
|
|
||||||
- numpy
|
|
||||||
- PIL (Pillow)
|
|
||||||
- scipy
|
|
||||||
- tqdm
|
|
||||||
|
|
||||||
提示：运行时可能看到 `pkg_resources is deprecated` 警告，不影响推理结果。
|
|
||||||
|
|
||||||
## 运行方式
|
|
||||||
|
|
||||||
1. 打开 [water_V5.py](file:///e:/code/sam3-main/sam3-main/water_V5.py)，修改下面参数：
|
|
||||||
- `image_path`：输入 GeoTIFF
|
|
||||||
- `mask_output_path`：输出掩膜 GeoTIFF
|
|
||||||
- `prompt`：文本提示（例如 `"water body"` / `"river"` / `"water"`）
|
|
||||||
2. 运行：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python e:\code\sam3-main\sam3-main\water_V5.py
|
|
||||||
```
|
|
||||||
|
|
||||||
输出 `mask_output_path` 为 uint8 单通道，像元值为 0/1，并保留输入影像的空间参考（transform/CRS）。
|
|
||||||
|
|
||||||
## 关键参数说明（建议优先调这几个）
|
|
||||||
|
|
||||||
在脚本“参数设置”区域：
|
|
||||||
|
|
||||||
- `coarse_read_max_side`：overview 粗分割输入的最大边长（越大越准/越慢，越小越快/越粗）
|
|
||||||
- 建议：`1024 ~ 1536`（16GB 显存一般可从 1200 开始）
|
|
||||||
- `tile_size`：精修分块大小（原图像素）
|
|
||||||
- 建议：`1536`（更稳）或 `2048`（更快但更吃内存/更容易慢）
|
|
||||||
- `overlap`：精修分块重叠（减少拼接缝）
|
|
||||||
- 建议：`128`（更快）或 `256`（边缘更稳但慢）
|
|
||||||
- `band_radius`：边缘带宽度（越大精修范围越大，速度越慢）
|
|
||||||
- 建议：`48 ~ 96`
|
|
||||||
- `coarse_threshold` / `final_threshold`：粗分割/最终阈值
|
|
||||||
- 通常 `0.5` 可用；若漏检多，可适当降低阈值（例如 `0.4`）
|
|
||||||
|
|
||||||
分区域参数：
|
|
||||||
- `num_splits_y`、`num_splits_x`：将整图拆成多少个子区域处理
|
|
||||||
- `region_overlap`：子区域之间重叠，避免区域边界出现断裂
|
|
||||||
|
|
||||||
后处理参数：
|
|
||||||
- `min_area`：移除小碎片的最小连通域面积（像素）
|
|
||||||
- `keep_largest_only`：只保留最大连通域（适合只想要“最大水体”的场景）
|
|
||||||
|
|
||||||
## 性能与显存的取舍建议（16GB）
|
|
||||||
|
|
||||||
优先级从高到低：
|
|
||||||
|
|
||||||
- 优先调小 `tile_size`（例如 2048 → 1536），通常能显著降低显存与峰值计算量
|
|
||||||
- `overlap` 从 256 降到 128，可明显加速（代价是拼接缝更可能出现）
|
|
||||||
- `band_radius` 适当减小，让精修只发生在更窄的边缘范围
|
|
||||||
- `coarse_read_max_side` 适度提高可减少误分/漏分，但会更慢
|
|
||||||
|
|
||||||
脚本已做的关键优化：
|
|
||||||
- 精修 tile 的“是否需要处理”的判断在 CPU 上完成，避免每个 tile 强制 GPU 同步
|
|
||||||
- per-tile 的强度拉伸（percentile）改为“子区域级别一次计算并复用”
|
|
||||||
- 推理使用 autocast（float16），通常更省显存
|
|
||||||
|
|
||||||
## 输出说明
|
|
||||||
|
|
||||||
- 输出掩膜为 `uint8` 单通道：
|
|
||||||
- 0：非水体
|
|
||||||
- 1：水体
|
|
||||||
- 输出 GeoTIFF 继承输入的 profile，并设置 `compress='lzw'`
|
|
||||||
|
|
||||||
## 常见问题
|
|
||||||
|
|
||||||
- 运行报 `AssertionError`（RoPE 相关）
|
|
||||||
- 该模型对输入分辨率有约束。脚本默认 `coarse_resolution = 1008`、`fine_resolution = 1008`，不要随意改成 768/512 等。
|
|
||||||
|
|
||||||
- 输出全黑/全白
|
|
||||||
- 先换 `prompt`（如 `"water"` / `"river"`），再调 `coarse_threshold/final_threshold`
|
|
||||||
- 若影像波段不是 RGB，脚本会取前三个波段做可视输入；需要更合适的波段组合时应自行调整 `_bands_to_pil`
|
|
||||||
|
|
||||||
Reference in New Issue
Block a user