Bridge Inspection AI

English | 中文

Stage 2 of the embodied-AI portfolio plan: a simulated inspection robot flies a camera around a structure (built from the companion AutoBridge-CAD tool's generated bridge geometry) and a trained vision model flags surface defects — the perception half of "perception + control", tied to an actual civil-engineering application instead of a generic benchmark.

All four pieces are working now: a crack classifier fine-tuned on real photographs, a MuJoCo scene built from an actual AutoBridge-CAD design with an inspection flight path, an LLM that turns raw detections into a report, and (as of the most recent milestone) a PPO policy that learned to choose a better visiting order over the inspection viewpoints than a random or naively-ordered baseline. Every piece is measured against a baseline or against each other, not assumed to work.

What's in here

• train_classifier.py — fine-tunes a ResNet18 (ImageNet-pretrained) on crack / no-crack photos, with a fixed 80/10/10 train/val/test split and the held-out test file list written to disk so evaluation can never accidentally touch training data.
• evaluate_classifier.py — loads a checkpoint, scores it on the held-out test split, prints a classification report, and saves a confusion matrix.
• visualize_predictions.py — saves a grid of held-out predictions, correct and incorrect both, so the model's actual mistakes are visible instead of just a summary accuracy number.
• plot_training_log.py — turns a captured training log into a loss/accuracy curve plot (for runs without --wandb).
• tests/test_data_smoke.py — sanity checks (dataset shape, split determinism and non-overlap, model forward pass) that skip cleanly if data/ hasn't been downloaded yet, rather than failing confusingly.
• sim/export_bridge_mesh.py — calls AutoBridge-CAD's own mesh-building functions (read-only, never modifies that repo) to export any design (spans, beam height, pier height, deck width, girder count all as CLI args) to STL, plus a bridge_design.json recording the parameters used. Must run under AutoBridge-CAD's own venv, not this project's — see the script's docstring.
• sim/convert_stl_binary.py — AutoBridge-CAD exports ASCII STL, MuJoCo's loader requires binary; converts via numpy-stl.
• sim/bridge_geometry.py — reads the actual compiled MuJoCo mesh back (world-space bounding box, per-pier footprint) instead of trusting the raw STL coordinates or a hand-measured constant, since MuJoCo recenters and reorients meshes internally.
• sim/scene_template.xml / sim/fly_inspection.py — the MuJoCo scene template and a scripted inspection camera path. Ground placement, pier count and position, and per-pier fly-around radius are all derived from bridge_design.json and the loaded mesh, not hardcoded — verified against a second, differently-shaped design (3 spans, different width and pier size) before trusting it on the real one.
• sim/composite_defects.py — composites real crack photos onto a color-masked patch of the rendered frames (image-space, not a 3D mesh texture — see "Simulation and sim-to-real results" below for why).
• sim/evaluate_on_renders.py — runs the trained classifier on those composited patches and checks it against the ground truth the compositing step already knows, no extra annotation needed.
• report.py — turns eval_results.json into a natural-language report. Deliberately never sees the ground-truth has_crack field (a real deployment wouldn't have it either) and is required to state the measured sim-to-real reliability numbers as an explicit caveat, not present detections with more confidence than they've earned.
• app.py / config.py / llm/client.py — the Streamlit front end for report.py, same "use .env or fill in a key for this session only" pattern as weekend-getaway and AutoBridge-CAD.
• sim/bridge_scene.py — builds a MuJoCo model from bridge_design.json + bridge.stl, shared by both fly_inspection.py and inspection_env.py so there's one source for "how a design becomes a loaded model."
• sim/inspection_env.py — a Gymnasium environment: choose an efficient visiting order over the same safe waypoints fly_inspection.py uses (not a flight-control problem — see "Autonomous navigation" below for why that's a deliberate scope cut).
• sim/train_navigation.py / sim/evaluate_navigation.py — trains PPO on that environment (self-contained in this repo, doesn't import or modify rl-locomotion-lab) and compares it against random and fixed-order baselines on steps-to-full-coverage.
• tests/test_inspection_env.py — includes a regression test for the exact bug that made the first training attempt fail (see below).

Quick start

python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Data (Concrete Crack Images for Classification, Özgenel 2019 via Kaggle — 40,000 images, 20,000 crack / 20,000 no-crack, requires a free Kaggle account and API token):

kaggle datasets download -d arunrk7/surface-crack-detection -p data --unzip
pytest   # confirms the dataset and model pipeline actually work before a real run

Two more datasets are used for evaluation only (not training), both third-party:

• SDNET2018 (Maguire, Dorafshan, and Thomas, 2018, Utah State University, DOI: 10.15142/T3TD19) — used for the cross-dataset generalization check in run_all_benchmarks.py --module testbed1_classification_crossdataset. Download and reorganize into Positive/Negative subfolders (see reorganize_sdnet2018.sh under the paper-strategy scripts) before running.
• real_world_eval/ (this repo) — 19 hand-verified concrete-surface photos from Wikimedia Commons, openly licensed (CC0/CC-BY/CC-BY-SA/Public Domain), used for the small real-photo reliability probe in evaluate_real_world_photos.py. Per-image title, license, and author are listed in real_world_eval/ATTRIBUTION.md.

Train and evaluate:

python train_classifier.py --epochs 5
python evaluate_classifier.py
python visualize_predictions.py

--wandb on train_classifier.py streams training curves to Weights & Biases; without it, capture the log and plot it after the fact:

mkdir -p runs
python train_classifier.py --epochs 5 2>&1 | tee runs/train_classifier_5epoch.log
python plot_training_log.py runs/train_classifier_5epoch.log

data/, models/*.pt, and runs/ are gitignored (the dataset alone is ~300MB); the "Results" section below is how the actual output gets surfaced instead.

Simulation (needs the companion AutoBridge-CAD design tool checked out as a sibling directory, with its own venv set up per its own README — set AUTOBRIDGE_CAD_DIR if it lives somewhere else):

../AutoBridge-CAD/.venv/bin/python sim/export_bridge_mesh.py \
    --spans 18 18 18 18 18 --beam-height 2 --pier-height 10 --bridge-width 12 --num-girders 10
python sim/convert_stl_binary.py
python sim/fly_inspection.py
python sim/composite_defects.py
python sim/evaluate_on_renders.py

Reporting layer — copy .env.example to .env and fill in an API key (or fill one into the sidebar directly), then:

streamlit run app.py

Results

• Training: 5 epochs, ResNet18 fine-tuned from ImageNet weights, Adam, lr=1e-4, batch size 64, on Apple Silicon GPU (MPS). Validation accuracy peaks at epoch 2 (99.98%) and the checkpoint saved there is what gets evaluated below, not the final epoch — by epoch 5, training loss is still falling (0.0017) while validation loss has climbed back up (0.0055), the standard shape of mild overfitting once the easy part of the task is learned.
• Held-out test set (4,000 images, never seen during training): 99.825% accuracy — 7 misclassified out of 4,000 (2 false positives, 5 false negatives), no meaningful bias toward either class.

The prediction grid above is the more useful artifact than the accuracy number: several of the false negatives are images with a genuinely thin, faint crack that's easy to miss even by eye. That's the model's actual failure mode, not "wrong 0.175% of the time" in the abstract. It's directly relevant later too, since a real inspection camera won't always get a clean, well-lit, in-focus shot of a defect either.

Simulation and sim-to-real results (milestones A + C)

A fixed design (5 spans x 18m, 2m beam height, 10m piers, 12m deck width, 10 T-girders — chosen and capacity-checked inside AutoBridge-CAD itself) gets exported through AutoBridge-CAD's existing export_mesh_to_stl (sim/export_bridge_mesh.py only imports that function, never modifies AutoBridge-CAD) and converted from ASCII to binary STL (sim/convert_stl_binary.py via numpy-stl — MuJoCo's loader rejects ASCII STL outright). A scripted camera (sim/fly_inspection.py, a MuJoCo mocap body, no flight dynamics) then flies a 44-waypoint path: 12 shots along the deck underside, 8 shots looped around each of the 4 intermediate piers.

Two things worth recording honestly rather than glossing over:

• A real bug, found by testing before trusting the output: the first pier fly-around used a 5m loop radius and produced several solid-black frames. The cause: these piers turned out to be wide slab piers, ~9.6m across (verified against the loaded mesh, not guessed), not slender columns — a 5m loop clips straight into the pier body at some angles. Widened to 9m and re-verified.
• MuJoCo recenters and reorients meshes internally. The raw STL's own coordinates don't survive into the scene as-is; only after combining the geom's compiled position and rotation does the world-space bounding box match the original design (verified: 12m x 95m x 12m, matching width x length x height). Camera waypoints are computed from that verified world-space geometry, not from an assumption about the file's raw coordinates.

Defect compositing (sim/composite_defects.py): the STL carries no UV coordinates, so proper 3D texture-mapping onto the mesh was out of scope for a first pass. Instead, defects are composited in image space — a color mask finds contiguous patches of the bridge's flat gray material in each rendered frame, and a real crack photo (from the same training data as the classifier) gets multiply-blended into a random valid patch, so it inherits the render's tone instead of looking pasted on. 24 of the 44 frames had a large enough clean surface for a 224px patch; the rest were skipped rather than forced.

Running the existing classifier on these patches, unmodified, no retraining is the actual sim-to-real measurement, not a demo: 62.5% accuracy (15/24). But that number hides the real story: 100% recall (15/15 crack patches correctly flagged) versus 0% specificity (0/9 clean patches correctly cleared — every single one got flagged). The false positives aren't random:

They're flat, near-uniform gray with essentially no texture. Real "no crack" photos still have natural surface noise — aggregate, staining, subtle roughness — that MuJoCo's flat-shaded material doesn't produce. The model has never seen anything this visually "clean" during training, and an anomalously uniform patch reads as suspicious rather than safe. The gap here isn't really about crack detection at all. Flat simulation rendering is itself out-of-distribution for a photo-trained model, which is the actual problem. Two follow-ups, not done yet: add procedural surface noise to the MuJoCo material to close some of that gap, or fine-tune the classifier on composited renders (labels are free — the patch location and class were chosen when compositing, no annotation needed).

Reporting layer (milestone D)

report.py reads docs/composited/eval_results.json and asks an LLM to write up a short Markdown inspection report. Two things enforced by design, not just by prompt wording:

• The ground-truth has_crack field never reaches the prompt (see tests/test_report.py::test_ground_truth_fields_never_reach_the_prompt) — a real deployment wouldn't have it, so the report is generated from the model's predictions alone, the same way it would be for real.
• The measured sim-to-real reliability numbers (recall / specificity from the section above) are passed in as explicit context the system prompt requires be stated as a caveat — the report can't quietly present itself as more trustworthy on this kind of imagery than it's actually measured to be.

Tested against a fake LLM (no real API calls) for the prompt-construction logic; generating an actual report needs a real key in app.py's sidebar or .env — not run here to avoid spending the project owner's API quota on something they can trivially run themselves.

Autonomous navigation (milestone E)

A deliberate scope cut, stated up front: this does not model real flight dynamics (thrust, drag, a policy that has to learn not to crash). That's a different research question (low-level flight control) from the one actually being asked here: given a fixed set of already-safe inspection viewpoints (the same ones fly_inspection.py uses), what order should they be visited in to cover the structure in as few moves as possible? inspection_env.py frames it as a graph problem: from its current waypoint, the agent picks one of its 6 nearest neighbors to move to next.

The first training run converged on a broken policy, and it's worth recording exactly how, not just that it got fixed. With 50k PPO timesteps, eval reward stayed negative and flat. Tracing the trained policy's actual moves showed why: after visiting 8 distinct waypoints, it got stuck oscillating between two nodes for the rest of the episode — final coverage 16%, worse than moving randomly (63%). The observation at the time was a current-position one-hot concatenated with the full 44-entry visited mask; the network had to infer which mask entries corresponded to which of its 6 available actions from a given node, and in practice it didn't learn that reliably. Adding one direct feature (for each of the 6 reachable neighbors, is it already visited) removed the need to infer that correspondence at all. Retrained from scratch with the same 50k timesteps, same everything else:

policy	mean coverage (10 episodes, 88-step budget)	full coverage rate
trained PPO	81.8%	0%
random	63.4%	0%
fixed label order (deck scan, then each pier loop)	30.2%	0%

The trained policy clearly beats both baselines. Stated plainly, though: it never reaches 100% coverage within the step budget in any of the 10 evaluation episodes. The fixed-order baseline scoring worst isn't really about the order being bad; it's that the graph only connects each waypoint to its 6 nearest neighbors, and the original script's order jumps between geometrically distant points that aren't directly reachable, forcing long detours through the graph. test_inspection_env.py includes a regression test for the specific observation-design bug above, so a future change to the environment can't silently reintroduce it.

The path itself (sim/visualize_navigation_path.py) shows the same story the numbers do: the policy clearly groups nearby waypoints (it loops a pier fully before moving to the next cluster, not random jumps around the structure), but also visibly lingers — some regions get revisited several times while 7 waypoints (the gray dots) never get visited at all within the step budget.

Actual camera frames along that path make the same thing concrete rather than abstract: the first few show real variety (deck underside, a transition, a pier), but roughly half the sampled frames end up looking at the same pier from similar angles — a direct, visual confirmation of the lingering behavior above, not just an artifact of how the sampling was done.

Not done: closing the remaining ~18% coverage gap (more training, reward reshaping, or a fully-connected action space instead of k-nearest neighbors are the obvious next things to try), and closing the sim-to-real gap from milestone C (material noise texture, or fine-tuning the classifier on composited renders).

Security notes

• Never commit .env; it's already in .gitignore. .env.example only has a placeholder.
• The manually-entered API key field in app.py's sidebar lives only in that session's memory, never written to disk or logged.

License

MIT