Spotlight LLM Bridge (spotlight-llm)

A high-level Rust bridge for Apple's WWDC26 on-device language models (FoundationModels.framework) and local Core Spotlight retrieval-augmented generation (RAG).

This library allows you to build completely local, private, and secure RAG applications on macOS. The LLM runs entirely on-device using the Apple Silicon Neural Engine, and can automatically invoke a custom Core Spotlight tool to fetch and reason over locally indexed items when answering prompts.

---

⚠️ Platform Requirements & Restrictions

Because this library binds directly to macOS system frameworks, it has strict compilation and runtime constraints:

• Operating System: macOS 17.0+ (macOS Sequoia, Build 26A5353q or higher)
• Architecture: Apple Silicon (ARM64 / aarch64-apple-darwin). Intel (x86_64) is not supported.
• Build toolchain: Xcode 18.0+ / Command Line Tools (targeting macOS SDK 26.0+).
• Entitlements (Production Apps): If deployed in a sandboxed application (e.g., App Store), you must add Spotlight search entitlements. CLI tools running unsandboxed do not require entitlements.

🔍 Troubleshooting: dyld: Library not loaded: @rpath/libswift_Concurrency.dylib

When compiling a binary project that depends on this crate, you might encounter a runtime crash on startup:

dyld[30416]: Library not loaded: @rpath/libswift_Concurrency.dylib
  Referenced from: .../target/debug/my-app
  Reason: tried: ... (no such file)

Why this happens: Rust's Cargo does not automatically add Swift runtime search paths to your final binary's RPATH.

How to fix it: Add a .cargo/config.toml file to the root of your binary project (the project consuming spotlight-llm) with the following settings to tell the linker to embed the Swift runtime search path:

[target.aarch64-apple-darwin]
rustflags = [
    "-C", "link-arg=-Wl,-rpath,/usr/lib/swift"
]

Alternatively, you can build/run your project by prefixing the RUSTFLAGS environment variable:

RUSTFLAGS="-C link-arg=-Wl,-rpath,/usr/lib/swift" cargo run

---

Installation

Add this to your Cargo.toml:

[dependencies]
spotlight-llm = { git = "https://github.com/RickyDane/macOS-Core-Spotlight-Rust-Bridge.git" }
tokio = { version = "1.0", features = ["full"] }

---

Quick Start

1. Initialize and Ask a Question (Direct LLM)

use spotlight_llm::SpotlightLlmSession;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize session (loads system model)
    let session = SpotlightLlmSession::new();
    
    // Prompt the on-device model directly
    let response = session.respond("What is on-device AI?").await?;
    println!("Response: {}", response);
    
    Ok(())
}

2. Retrieval-Augmented Generation (RAG) with Core Spotlight

You can index documents into Core Spotlight, and when you ask the LLM a question, it will automatically formulate queries, call the local index tool, and synthesize a response:

use spotlight_llm::SpotlightLlmSession;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let session = SpotlightLlmSession::new();

    // 1. Index a document in Core Spotlight
    session.index_item(
        "proj-001",
        "On-Device LLM Spotlight Search Bridge",
        "This project implements a Rust crate to bridge Apple's on-device models with Core Spotlight."
    )?;

    // 2. Ask a question that requires that document
    let prompt = "Find the project description for 'Spotlight Search Bridge' in Spotlight and summarize its purpose.";
    let response = session.respond(prompt).await?;
    
    println!("LLM RAG Response:\n{}", response);
    Ok(())
}

3. Real-Time Token Streaming with TPS Tracking

Stream responses word-by-word with dynamic Tokens Per Second (TPS) statistics in your terminal:

use spotlight_llm::SpotlightLlmSession;
use std::io::{self, Write};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let session = SpotlightLlmSession::new();
    
    let mut rx = session.respond_stream("Describe the lifecycle of a star.")?;
    
    while let Some(delta) = rx.recv().await {
        print!("{}", delta);
        io::stdout().flush()?;
    }
    
    Ok(())
}

---

Running Examples

The repository includes several ready-to-run examples. You can test them with Cargo:

# Basic RAG search demo
cargo run --example demo

# Multi-tool comparison RAG query (Spaghetti vs Tacos)
cargo run --example recipes

# Light direct prompt (no RAG)
cargo run --example minimal

# Real-time token streaming with live TPS tracking
cargo run --example stream

---

License

This project is licensed under either of:

• MIT License (LICENSE-MIT)