Production-Ready GenAI with Open Models for Java Teams

JavaOne 2026 Demo Project — Local inference with Ollama + LangChain4j, RAG pipeline with evaluation.

Prerequisites

• Java 21+ (java -version)
• Maven 3.9+ (mvn -version)
• Ollama installed and running (ollama.com/download)

Quick Start

1. Pull the required models

ollama pull llama3.1:8b
ollama pull mistral:7b
ollama pull nomic-embed-text

2. Verify Ollama is running

ollama list

3. Build and run

mvn clean package -DskipTests
mvn spring-boot:run

Demo 1: Local Inference + Tool Calling

Chat with the model:

curl "http://localhost:8080/chat?message=What+is+the+Java+record+keyword?"

Tool calling (inventory lookup — uses /chat/tools endpoint):

curl "http://localhost:8080/chat/tools?message=How+many+units+of+JDK-21+do+we+have+in+stock?"

To swap models, change ollama.chat-model in src/main/resources/application.yml to mistral:7b and restart.

Demo 2: RAG Pipeline + Evaluation

Ingest documents into the vector store:

curl -X POST "http://localhost:8080/ingest"

Ask questions grounded in your documents:

curl "http://localhost:8080/ask?question=What+are+virtual+threads+in+Java+21?"

Run quality evaluation:

curl -X POST "http://localhost:8080/evaluate"

Project Structure

├── pom.xml
├── docs/                              # Sample documents for RAG
│   ├── java21-features.txt
│   ├── spring-boot-config.txt
│   └── kubernetes-java-deploy.txt
└── src/main/java/com/javaone/openmodels/
    ├── OpenModelsDemoApplication.java  # Spring Boot entry point
    ├── config/
    │   ├── OllamaConfig.java          # Chat model + chat assistant bean
    │   └── RagConfig.java             # Embedding model, store, RAG assistant
    ├── controller/
    │   ├── ChatController.java        # GET /chat, GET /chat/tools
    │   └── RagController.java         # POST /ingest, GET /ask, POST /evaluate
    └── service/
        ├── Assistant.java             # AI Service interface
        ├── DocumentIngestor.java      # Document loading, splitting, embedding
        ├── InventoryTools.java        # @Tool example for Demo 1
        └── RagEvaluator.java          # Golden test set evaluation

Endpoints

Method	Path	Description
GET	/chat?message=...	Chat with LLM (Demo 1)
GET	/chat/tools?message=...	Chat with tool calling (Demo 1)
POST	/ingest	Ingest docs into vector store (Demo 2)
GET	/ask?question=...	RAG query (Demo 2)
POST	/evaluate	Run eval test set (Demo 2)

Demo Script References

• Demo 1: Local Inference with Ollama + LangChain4j
• Demo 2: RAG Pipeline with Quality Evaluation