Text-Guided Geospatial Segmentation (LangSAM → FastSAM)

Convert satellite / aerial GeoTIFF imagery into georeferenced vector polygons (GeoJSON) of a class you describe in plain English — "building", "tree", "car" — with no training data and no manual digitizing.

The project builds a two-stage segmentation pipeline on top of segment-geospatial (samgeo):

1. LangSAM (GroundingDINO + SAM) turns a text prompt into coarse object masks.
2. Those masks are reduced to bounding boxes, subdivided into a grid, and fed to FastSAM as box prompts to recover crisper, per-object boundaries.
3. The final raster mask is written back as a georeferenced GeoTIFF and polygonized to GeoJSON, ready for QGIS / ArcGIS / PostGIS.

THE_END.py is the final, working pipeline. Every other .py file in the repo is a preserved development step — see Repository Tour.

---

Table of Contents

• Why two models?
• How the pipeline works
• Repository Tour
• Installation
• Usage
• Output files
• Key concepts
• Tuning
• Known limitations & gotchas
• Troubleshooting
• Credits

---

Why two models?

	LangSAM (GroundingDINO + SAM ViT-H)	FastSAM (YOLOv8-seg backbone)
Prompt type	Natural-language text	Points / boxes / "everything"
Strength	Finds the right objects semantically	Delineates boundaries quickly
Weakness	Slow; masks can bleed into neighbours	No semantic understanding — needs to be told where to look

Neither model alone gives a clean result on remote-sensing imagery. LangSAM answers "where are the trees?", and FastSAM answers "what exactly is the outline of the thing in this box?" Chaining them gives semantic targeting with sharper geometry.

---

How the pipeline works

                    image.tif  (GeoTIFF, georeferenced)
                         │
                         ▼
        ┌────────────────────────────────────┐
        │ rasterio.open()                    │   read affine transform,
        │   → src.transform, width, height   │   width/height, CRS
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [1]  │ LangSAM.predict(image, "tree",     │   text-prompted detection
        │   box_threshold=.24,               │   + segmentation
        │   text_threshold=.24)              │
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [2]  │ show_anns(cmap="Greys_r", blend=F) │   → buildings_langsam.tif
        │ raster_to_vector(tif, geojson)     │   → segmentation_result_langsam.geojson
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [3]  │ geopandas.read_file(geojson)       │   polygon.bounds →
        │   → [minx, miny, maxx, maxy] …     │   geographic bboxes
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [4]  │ geographic_to_image_coords()       │   ~affine  (inverse transform)
        │   geo bbox → pixel bbox            │   world coords → row/col
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [5]  │ split_bounding_box(bbox, 4, 4)     │   one loose box → 16 tight
        │                                    │   sub-boxes (finer prompts)
        └────────────────────────────────────┘
                         │
                         ├──────────────► draw_bounding_boxes()
                         │                → bounding_boxes_debug.png  (visual QA)
                         ▼
        ┌────────────────────────────────────┐
   [6]  │ SamGeo(FastSAM-x.pt).set_image()   │   FastSAM inference once,
        │ .box_prompt(bboxes=all_sub_bboxes) │   then prompt with all boxes
        └────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────┐
   [7]  │ save_masks()                       │   morphological CLOSE(3×3)
        │   morphologyEx → binarise → resize │   + OPEN(8×8) cleanup,
        │   array_to_image(..., self.source) │   copies georeferencing
        └────────────────────────────────────┘   from the source raster
                         │
                         ▼
              fastsam_box_prompt_result.tif   (georeferenced mask)
                         │
                         ▼  raster_to_vector / gdal_polygonize.py
                    result.geojson             (vector polygons)

The custom SamGeo class

samgeo's stock FastSAM wrapper does not expose everything the pipeline needs, so THE_END.py subclasses samgeo.fast_sam.FastSAM:

Method	What it adds
__init__(model="FastSAM-x.pt")	Validates the model name and auto-downloads the checkpoint to $TORCH_HOME (default ~/.cache/torch/hub/checkpoints) if missing.
set_image(image, device=None)	Accepts a local path or an HTTP URL, picks cuda if available (clearing the cache first) else cpu, runs "everything" inference once, and stores a reusable FastSAMPrompt.
box_prompt(bbox, bboxes, output)	Runs the box prompt and either returns the annotations or writes them straight to disk.
save_masks(output, better_quality=True, mask_multiplier=255)	Morphological cleanup (close 3×3 then open 8×8), flattens all masks into one binary layer, resizes to the original raster size with nearest-neighbour, picks uint8/uint16/uint32 by mask count, and writes a georeferenced TIFF by inheriting the source raster's CRS + transform.

Coordinate helpers

geographic_to_image_coords(geo_bbox, src_transform)  # world → pixel, via ~affine
image_to_geographic_coords(image_bbox, src_transform)  # pixel → world
split_bounding_box(bbox, rows, cols)                 # one box → rows×cols sub-boxes
draw_bounding_boxes(image_path, bboxes, output_path) # green rectangles, for QA

The round-trip through image_to_geographic_coords is what lets you take a mask produced in pixel space and hand real-world coordinates back to a GIS.

---

Repository Tour

The repo is deliberately kept as a worklog — each file is a rung on the ladder to THE_END.py.

Final pipeline

File	Description
THE_END.py	⭐ The final code. Full LangSAM → bbox → 4×4 split → FastSAM box-prompt pipeline, with georeferenced GeoTIFF output and a bounding-box debug image.

Development history (kept for reference)

File	Stage	What it explores
segmentation_script.py	1	Minimal starting point: SamGeo(vit_h) + sam.predict(boxes=…, point_crs="EPSG:4326") → mask.tif, then gdal_polygonize.py → GeoJSON.
segmentation.py	1	Wraps both prompt styles as functions and pulls basemap tiles with tms_to_geotiff(zoom=19, source="Satellite").
kutu_prompt_full_kordinat_sorgusu.py	2	Box prompt + coordinate validation. Uses pyproj.Transformer to convert EPSG:4326 boxes to the image CRS and rejects boxes outside the raster bounds before inference. Ends with an extensive Turkish primer on what a GIS polygon actually stores.
kutu_prompt_w_annotations.py	2	Same as above plus a matplotlib overlay of the mask on the true-colour RGB bands.
sam_tree_full.py	3	Pure LangSAM text prompt ("tree"), saving three renderings: with boxes, without boxes, and a flat Greys_r mask suitable for vectorising.
sam_building_fulll.py	3	Same for "building" (box_threshold=0.19), imports osgeo.gdal directly.
oto_full.py	3	Automatic (unprompted) mask generation, run twice — once with defaults, once with tuned sam_kwargs (points_per_side, pred_iou_thresh, stability_score_thresh, crop_n_layers, min_mask_region_area) — to compare hyperparameters.
fast_sam.py	4	First standalone SamGeo(FastSAM) subclass with box_prompt + save_masks, driven by hardcoded pixel boxes.
fastsam_box.py	4	Trimmed version of the same, used to sanity-check box prompting.
deneme_fast.py	4	Exercises all three FastSAM prompt modes — everything_prompt, text_prompt, box_prompt — each written to its own results folder as TIFF + GeoJSON + annotated PNG.
firstlang_afterfast_yolomsu.py	5	First LangSAM→FastSAM chain, but using FastSAM's raw YOLO predict() and converting results[0].boxes.xyxy into shapely polygons.
Internship_final.py	6	The chain in its recognisable form: LangSAM → GeoJSON → geographic bboxes → pixel bboxes → 4×4 split → FastSAM. Output is PNG.
combined.py	6	Fullest SamGeo subclass (everything_prompt, point_prompt, box_prompt, text_prompt, fast_show_mask, raster_to_vector). Passes geographic bounds straight to FastSAM — the bug that the coordinate conversion in later files fixes.
image_dim_coordinates.py	7	Adds rasterio georeferencing awareness and the geographic_to_image_coords helper, without the 4×4 split.
info.txt	—	Original install notes (Turkish), plus a comparison of gdal_polygonize vs sam.raster_to_vector for raster→vector conversion.

Sample data & artifacts

File	Description
enhanced_image.tif	Sample input raster (contrast-enhanced).
mask.tif	Sample binary segmentation mask.
mask.geojson, maskk.geojson	Vectorised masks.
segmentation_result.geojson	Polygonized output, EPSG:3857.
buildings.geojson, filtered_buildings.geojson	Empty FeatureCollection skeletons (CRS84) from a run that produced no detections — useful as a reminder to check thresholds.
runs/segment/predict/	Ultralytics YOLO run directory with normalised polygon labels (labels/image.txt).

---

Installation

Requirements

• Python 3.12 (the checked-in venv was built with 3.12; 3.9+ should work)
• ~8 GB RAM minimum. A CUDA GPU is optional — the code falls back to CPU automatically, but SAM ViT-H on CPU is slow.
• GDAL available on your system if you want to use gdal_polygonize.py.

Setup

git clone https://github.com/<your-username>/<your-repo>.git
cd <your-repo>

python -m venv venv
source venv/bin/activate          # macOS / Linux
# venv\Scripts\Activate.ps1       # Windows PowerShell

pip install -r requirements.txt

If requirements.txt gives you trouble (GDAL is notoriously fussy), install the core stack manually — this is the sequence from info.txt that is known to work:

pip install segment-geospatial groundingdino-py leafmap localtileserver
pip install segment-anything-fast
pip install gdal            # or: pip install GDAL

On macOS, Tkinter is needed by some of the older scripts:

brew install python-tk

The final pipeline sidesteps this entirely with matplotlib.use('Agg'), a headless backend — which is also what makes it safe to run on a server with no display.

Model checkpoints

Nothing needs to be downloaded by hand:

Model	Size	Downloaded to
FastSAM-x.pt	~145 MB	$TORCH_HOME (default ~/.cache/torch/hub/checkpoints) — fetched by SamGeo.__init__
sam_vit_h_4b8939.pth	~2.4 GB	Working directory — fetched by samgeo on first SamGeo(model_type="vit_h")
GroundingDINO weights	~700 MB	Hugging Face cache — fetched by LangSAM()

FastSAM-x.pt is not committed to this repo (it exceeds GitHub's 100 MB file limit) and does not need to be — the code downloads it on first run.

---

Usage

Open THE_END.py and edit the three settings at the bottom, plus the output directory:

# THE_END.py — line 164
output_folder = Path("/Users/korhanerdogdu/Desktop") / output_folder_name   # ← change this

# THE_END.py — lines 254-256
image_path  = '/Users/korhanerdogdu/Desktop/staj_belge/image.tif'           # ← your GeoTIFF
text_prompt = "tree"                                                        # ← what to find
text_prompt_segmentation(image_path, text_prompt)

Then:

python THE_END.py

Console output walks you through each stage:

Image dimensions: 1024x1024
Affine transformation: | 0.30, 0.00, -5705538.45| ...
Bounding boxes (geographic): [[-51.2546, -22.1771, -51.2541, -22.1767], ...]
Bounding boxes (image): [[120, 340, 260, 470], ...]
Bounding boxes drawn for verification: .../bounding_boxes_debug.png
Sub-bounding boxes (image coordinates): [[120, 340, 155, 372], ...]
Sub-bounding boxes (geographic coordinates): [...]
FastSAM segmentation results saved to .../fastsam_box_prompt_result.tif
All segmentation results are saved in .../Results_Buildings_Verified_2

Making it portable (recommended first change)

The hardcoded macOS paths are the one thing you'll want to replace. A minimal edit:

def text_prompt_segmentation(image_path, text_prompt, output_folder="results"):
    output_folder = Path(output_folder)
    output_folder.mkdir(parents=True, exist_ok=True)
    ...

if __name__ == "__main__":
    import sys
    text_prompt_segmentation(sys.argv[1], sys.argv[2])

python THE_END.py path/to/image.tif "building"

---

Output files

Written to <output_folder>/:

File	Produced by	Description
buildings_langsam.tif	sam_lang.show_anns(cmap="Greys_r", blend=False)	Stage-1 LangSAM mask, flat greyscale so it vectorises cleanly.
segmentation_result_langsam.geojson	sam_lang.raster_to_vector(...)	Stage-1 polygons — the source of the bounding boxes.
bounding_boxes_debug.png	draw_bounding_boxes(...)	Input image with every sub-box drawn in green. Check this first when results look wrong.
fastsam_box_prompt_result.tif	sam_geo.box_prompt(output=...)	Final georeferenced binary mask (0 / 255), same CRS + transform as the input.

To vectorise the final mask:

gdal_polygonize.py fastsam_box_prompt_result.tif -f GeoJSON result.geojson

or in Python:

from samgeo import SamGeo
SamGeo().raster_to_vector("fastsam_box_prompt_result.tif", "result.geojson")

---

Key concepts

Affine transform. A GeoTIFF carries a 6-parameter affine that maps pixel (col, row) → world (x, y). Inverting it (~src_transform) maps world coordinates back to pixels — this is exactly what geographic_to_image_coords does, and it is the hinge the whole pipeline turns on: LangSAM returns geography, FastSAM wants pixels.

Why subdivide the boxes (4×4)? LangSAM's boxes are generous — a box around "trees" often swallows roof, road, and shadow. Slicing each box into 16 sub-boxes gives FastSAM many tight, local prompts instead of one loose one, so it latches onto individual objects rather than the whole neighbourhood. Fewer splits = faster and coarser; more splits = slower and more fragmented.

Raster → vector, two ways (from info.txt):

• gdal_polygonize.py — traces polygons straight off the mask TIFF with no visualization step. Faster, fewer moving parts.
• sam.raster_to_vector() — lets you first render the mask with a colormap/alpha via show_anns(), then vectorise that. Slower, but you get a human-viewable mask on the way through.

Both are used in this repo; the final pipeline writes the georeferenced TIFF and leaves the choice to you.

box_threshold vs text_threshold (LangSAM):

• box_threshold — confidence needed for GroundingDINO to keep a detection box. Lower ⇒ more objects, more false positives.
• text_threshold — how strongly a detection must match the text prompt. Lower ⇒ looser semantic matching.

The scripts here range from 0.12 (permissive, combined.py) to 0.24 (balanced, THE_END.py).

---

Tuning

Knob	Where	Effect
box_threshold	THE_END.py:180	↓ finds more objects, more noise. Start at 0.24; drop to 0.15 if nothing is detected.
text_threshold	THE_END.py:180	↓ looser text↔object matching.
rows, cols	THE_END.py:223	Sub-box grid. 4×4 default; 2×2 for large objects (buildings), 6×6+ for small dense ones (cars).
MORPH_OPEN kernel	THE_END.py:93	np.ones((8, 8)) removes speckle. Shrink to (3,3) to keep thin structures.
mask_multiplier	THE_END.py:75	Mask value written to the TIFF (default 255). Use 1 for a true 0/1 mask.
text_prompt	THE_END.py:255	Singular nouns work best ("tree", not "trees").

---

Known limitations & gotchas

These are real, and worth knowing before you file a bug against yourself:

1. Hardcoded absolute macOS paths. output_folder (/Users/korhanerdogdu/Desktop) and image_path are baked in across every script. Edit them, or apply the portability patch above.

2. CRS mismatch risk. THE_END.py:203 forces the LangSAM vector output to EPSG:4326 with set_crs(..., allow_override=True), but src_transform comes from the input raster's native CRS. If your GeoTIFF is in a projected CRS (the sample segmentation_result.geojson is EPSG:3857), the inverse-affine step will produce meaningless pixel coordinates. Reproject the input to EPSG:4326 first, or reproject the GeoDataFrame with gdf.to_crs(src.crs) before extracting bounds. bounding_boxes_debug.png will show this immediately — boxes will be off-image or clustered in one corner.

3. MultiPolygon iteration is Shapely-1.x style. for poly in geom (THE_END.py:209) raises TypeError on Shapely 2.x. Use for poly in geom.geoms.

4. No CLI. Every script runs its example at import time. There is no if __name__ == "__main__": guard in THE_END.py, so importing it triggers a full inference run.

5. Memory. FastSAM runs everything inference over the whole raster before box prompting. Very large GeoTIFFs should be tiled first.

6. Sub-boxes are not deduplicated. A 4×4 split of N detections means 16N prompts, all merged into one flat mask. Individual object identity is lost — the output is a single binary layer, not instance polygons.

7. geographic_to_image_coords assumes north-up imagery (maxy → top). Rotated/skewed rasters will need the full affine treatment.

---

Troubleshooting

Symptom	Cause / Fix
An error occurred during LangSAM segmentation	GroundingDINO weights failed to download, or CUDA OOM. Force CPU: LangSAM(device="cpu").
Empty GeoJSON ("features": [])	Nothing detected. Lower box_threshold to 0.15. See buildings.geojson in this repo — that's what an empty run looks like.
TypeError: 'MultiPolygon' object is not iterable	Shapely 2.x — use geom.geoms (gotcha #3).
Boxes in the wrong place in bounding_boxes_debug.png	CRS mismatch (gotcha #2).
TkAgg/display errors	Already handled by matplotlib.use('Agg') — make sure it is set before import matplotlib.pyplot.
gdal_polygonize.py: command not found	pip install gdal installs the script into the venv's bin/. Activate the venv, or call python venv/bin/gdal_polygonize.py.
ValueError: Model must be one of ['FastSAM-x.pt', 'FastSAM-s.pt']	Only these two are whitelisted in SamGeo.__init__. FastSAM-s.pt is much faster and lighter if you're on CPU.

---

Credits

Built on:

• segment-geospatial (samgeo) — Qiusheng Wu
• Segment Anything (SAM) — Meta AI
• FastSAM — CASIA-IVA-Lab
• GroundingDINO / LangSAM
• rasterio, GeoPandas, Shapely, GDAL

Developed as an internship project on automated feature extraction from remote-sensing imagery.