paper.tex

\documentclass[10pt,a4paper]{article}

% ── Packages ──
\usepackage[margin=0.9in]{geometry}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{listings}
\usepackage{enumitem}
\usepackage[numbers]{natbib}
\usepackage{caption}

\hypersetup{colorlinks=true, linkcolor=blue, citecolor=blue, urlcolor=blue}

\lstset{
  basicstyle=\small\ttfamily,
  breaklines=true,
  frame=single,
  backgroundcolor=\color{gray!8},
  columns=fullflexible,
}

\title{OptiSkill: Bridging Operational Research Analysis\\
       and Practitioners with Agentic Skills}

\date{}

\begin{document}
\maketitle
\vspace{-2em}

% ══════════════════════════════════════════════════════════════════════
\begin{abstract}
OptiChat~\cite{chen2025optichat} is a multi-agent dialogue system that enables
practitioners to query and analyse Pyomo optimisation models through natural
language.  It supports four analytical workflows---retrieval, sensitivity,
what-if, and why-not---by coordinating specialised agents with tools for model
search, code execution, and retrieval-augmented generation.  In this work, we
introduce \textbf{OptiSkill}, a distillation of OptiChat's capabilities into a
single Markdown skill document (\texttt{SKILL.md}) that any LLM agent can read
and execute without specialised tooling.  We benchmark OptiSkill against the
full OptiChat system on 133~questions spanning 24~optimisation models.
OptiSkill achieves 82.1\% accuracy on retrieval and 84.6\% on structured
what-if queries, but drops to 27.8\% on sensitivity analysis and 19.2\% on
why-not reasoning, yielding an overall accuracy of 58.6\% versus OptiChat's
94.7\%.  We evaluate using two complementary test suites: quantitative
verification (numeric and component-level answers checked against ground truth)
and qualitative evaluation (natural-language analytical responses judged
semantically).  These results show that a single skill document can capture
template-driven analytical patterns effectively, while highlighting that tasks
requiring interactive tool use and multi-step domain reasoning remain difficult
to encode declaratively.
\end{abstract}

% ══════════════════════════════════════════════════════════════════════
\section{Introduction}

Optimisation models are widely used in operations research, yet interpreting
their solutions, understanding parameter sensitivities, and exploring
counterfactual scenarios typically require domain expertise and fluency with
modelling APIs.  OptiChat~\cite{chen2025optichat} addresses this gap by
providing a natural-language interface to Pyomo models, allowing practitioners
to ask questions and receive explanations without writing code.

OptiChat supports four core query types:

\begin{itemize}[nosep]
  \item \textbf{Retrieval}: Extract and report specific model components---parameter values, variable solutions, constraint expressions, or
        objective function values (e.g., ``What is the optimal production
        quantity for product~A?'').
  \item \textbf{Sensitivity}: Rank parameters by their marginal impact on the
        objective value using dual information such as shadow prices and reduced
        costs (e.g., ``Which constraint is most binding?'').
  \item \textbf{What-if}: Modify one or more parameter values, re-solve the
        model, and compare the new solution against the original (e.g., ``What
        happens to total cost if demand increases by 20\%?'').
  \item \textbf{Why-not}: Force a decision variable to a user-specified value
        or range, re-solve, and explain the resulting change in the objective
        and any constraint violations (e.g., ``Why can't we assign all jobs to
        machine~1?'').
\end{itemize}

The original system implements these workflows through a multi-agent
architecture with specialised tools.  In this work, we ask whether these
capabilities can be compressed into a \emph{single skill document}---a
self-contained Markdown file that a general-purpose LLM agent reads and follows
to perform the same analyses.  We call this distilled system
\textbf{OptiSkill}.

% ══════════════════════════════════════════════════════════════════════
\section{System Design}

\subsection{OptiChat: Multi-Agent Baseline}

OptiChat uses a two-tier architecture built on Google's Agent Development Kit
(ADK).  A lightweight \emph{root agent} receives the user's natural-language
query and delegates technical work to an \emph{expert agent} that has access to
specialised tools for model component search, Python code execution, and
retrieval-augmented generation over source code and research papers.  The expert
agent selects and sequences tools dynamically, enabling it to handle complex,
multi-step analyses---such as iteratively searching for relevant parameters,
executing modifications, and interpreting solver output.  The root agent then
translates the expert's technical analysis into a user-friendly response.

\subsection{OptiSkill: Skill-Based System}

OptiSkill replaces both agents and all tools with a single \texttt{SKILL.md}
file (\(\sim\!\)750 lines).  When a general-purpose LLM agent receives a
user query, it reads the skill document and generates a self-contained Python
script that performs the requested analysis.  The skill is structured as
follows:

\begin{enumerate}[nosep]
  \item \textbf{Prerequisites:} Install dependencies, write the model source as
        an embedded heredoc, build/solve/pickle the model, and extract a JSON
        component dictionary.
  \item \textbf{Query Classification:} A decision table mapping question
        patterns to one of four analytical workflows (retrieval, sensitivity,
        what-if, why-not).
  \item \textbf{Shared Utilities:} A solver fallback chain
        (\texttt{gurobi}~$\to$~\texttt{appsi\_highs}~$\to$~\texttt{glpk}) and
        Pyomo best-practice patterns (e.g., using \texttt{value()} for
        evaluation, \texttt{deepcopy} before modification).
  \item \textbf{Workflow Templates:} Each workflow contains a complete, runnable
        code template with inline comments explaining what to adapt per query.
  \item \textbf{Generalisability:} Instructions for adapting the skill to
        arbitrary Pyomo models and handling edge cases such as MILP duals and
        infeasible re-solves.
\end{enumerate}

The skill embeds the Stigler diet problem as a working demonstration and all
code needed to bootstrap, solve, and query it.  For new models, the agent
replaces the embedded source and follows the same workflows.

\begin{figure}[t]
  \centering
  \small
  \begin{tabular}{c}
  \textbf{OptiChat (Multi-Agent)} \\[2pt]
  User $\to$ Root Agent $\to$ Expert Agent $\to$ \{Tools\} $\to$ Answer \\[8pt]
  \textbf{OptiSkill (Single Document)} \\[2pt]
  User $\to$ LLM + \texttt{SKILL.md} $\to$ Generated Python $\to$ Answer
  \end{tabular}
  \caption{Architecture comparison.  OptiSkill replaces two coordinating agents
  and specialised tools with a single Markdown document that guides code
  generation.}
  \label{fig:arch}
\end{figure}

% ══════════════════════════════════════════════════════════════════════
\section{Evaluation}

\subsection{Setup}

We evaluate both systems on 133~questions across 24~Pyomo models.  The test set
is divided into two suites: \textbf{quantitative verification} (13 models,
96~questions) containing retrieval, sensitivity, and what-if queries whose
answers are checked against concrete numeric or component-level ground truth;
and \textbf{qualitative evaluation} (11 models, 37~questions) containing
complex what-if and why-not analyses whose natural-language responses are judged
semantically against reference answers.
We exclude 39~feasibility-restoration questions because OptiSkill does not
include the Gurobi IIS-based infeasibility diagnosis tool available to
OptiChat.

OptiSkill uses GPT-5-mini for query classification and code generation.
OptiChat uses GPT-5 with the full ADK agent infrastructure.  Both systems use
Gurobi as the primary solver.

\subsection{Results}

Table~\ref{tab:scope} shows overall results by test suite, and
Table~\ref{tab:pertype} provides the per-query-type breakdown.

\begin{table}[t]
  \centering
  \caption{Overall results by test suite.}
  \label{tab:scope}
  \small
  \begin{tabular}{lrrr}
    \toprule
    & \textbf{Questions} & \textbf{OptiChat} & \textbf{OptiSkill} \\
    \midrule
    Quantitative verification  & 96  & 96/96 (100.0\%) & 70/96 (72.9\%)  \\
    Qualitative evaluation     & 37  & 30/37 (81.1\%)  & 8/37 (21.6\%)   \\
    \midrule
    \textbf{Total}           & 133 & 126/133 (94.7\%)& 78/133 (58.6\%) \\
    \bottomrule
  \end{tabular}
\end{table}

\begin{table}[t]
  \centering
  \caption{Per-query-type accuracy.  Retrieval, sensitivity, and structured
  what-if are evaluated quantitatively; why-not is evaluated qualitatively;
  what-if spans both suites.}
  \label{tab:pertype}
  \small
  \begin{tabular}{llrrr}
    \toprule
    \textbf{Query Type} & \textbf{Evaluation} & \textbf{Questions}
      & \textbf{OptiChat} & \textbf{OptiSkill} \\
    \midrule
    Retrieval    & Quantitative & 39 & 39/39 (100.0\%) & 32/39 (82.1\%) \\
    Sensitivity  & Quantitative & 18 & 18/18 (100.0\%) & 5/18 (27.8\%)  \\
    What-if      & Quantitative & 39 & 39/39 (100.0\%) & 33/39 (84.6\%) \\
    What-if      & Qualitative  & 11 & 9/11 (81.8\%)   & 3/11 (27.3\%)  \\
    Why-not      & Qualitative  & 26 & 21/26 (80.8\%)  & 5/26 (19.2\%)  \\
    \midrule
    \textbf{All} &      & 133& 126/133 (94.7\%)& 78/133 (58.6\%)\\
    \bottomrule
  \end{tabular}
\end{table}

\paragraph{Retrieval and structured what-if perform well.}
OptiSkill achieves 82.1\% on retrieval and 84.6\% on structured what-if
queries.  These workflows are highly template-driven: the agent loads the model,
looks up a component or modifies a parameter, re-solves, and reports the result.
The skill's code templates transfer reliably across different models for these
patterns.

\paragraph{Sensitivity analysis underperforms.}
Sensitivity drops to 27.8\%.  This workflow requires correctly configuring Pyomo
\texttt{Suffix} components for dual extraction \emph{before} solving, then
ranking parameters by shadow price magnitude.  Errors typically stem from
incorrect suffix setup or misidentifying which components carry dual
information---steps that OptiChat's expert agent handles through interactive
trial and error.

\paragraph{Why-not and complex what-if require multi-step reasoning.}
Why-not questions (19.2\%) and qualitative what-if (27.3\%) require
understanding the semantic structure of models---for example, reasoning about
which chemical reactions produce a compound, or why certain binary variables
cannot coexist.  These cannot be reduced to code templates and demand the kind
of iterative exploration that OptiChat's tool-augmented agent provides.

\paragraph{Classification is not the bottleneck.}
Workflow classification accuracy is high: OptiSkill correctly identifies which
workflow to use in most cases.  Failures occur at code generation, where the
agent produces syntactically valid but semantically incorrect scripts---for
instance, modifying the wrong parameter or misinterpreting relationships
between model components.

% ══════════════════════════════════════════════════════════════════════
\section{Discussion}

\subsection{What the Skill Captures}

The skill document effectively encodes procedural patterns (bootstrap sequence,
solver fallback, deepcopy-before-modify), Pyomo API best practices, and query
classification logic.  For template-driven tasks where the analysis pattern is
fixed and only the queried data varies, a single document provides sufficient
guidance to achieve over 80\% accuracy.

\subsection{Why OptiSkill Falls Short}

The 36-point accuracy gap reflects fundamental differences between the two
architectures.  OptiChat's expert agent can search model components
interactively, execute Python incrementally, inspect intermediate results, and
retrieve relevant code snippets via RAG---enabling it to self-correct and adapt
its approach.  OptiSkill must generate a complete, correct script in a single
pass with no opportunity for error recovery.  Additionally, the bootstrapped
JSON dictionary provides component names and values but does not capture the
semantic relationships between components that OptiChat's expert agent discovers
dynamically.  The model gap (GPT-5-mini vs.\ GPT-5) also contributes,
particularly on tasks requiring complex domain reasoning such as analysing
binary constraint interactions.

\subsection{Practical Implications}

Skill documents are most effective for well-structured, template-driven tasks.
A hybrid approach---using a skill for common structured queries and delegating
complex questions to a richer agent with interactive tools---could capture the
best of both paradigms.

% ══════════════════════════════════════════════════════════════════════
\section{Conclusion}

We introduced OptiSkill, a distillation of the OptiChat multi-agent system into
a single \texttt{SKILL.md} document.  OptiSkill performs well on retrieval
(82.1\%) and structured what-if (84.6\%) queries, demonstrating that
document-guided agents can perform meaningful optimisation analysis.  However,
sensitivity analysis (27.8\%) and why-not reasoning (19.2\%) remain
challenging, as these tasks require the interactive tool use and multi-step
domain reasoning that characterise the full multi-agent system.  These findings
suggest that skill documents are a viable delivery mechanism for structured
analytical workflows, while more open-ended reasoning continues to benefit from
richer agent architectures.

% ══════════════════════════════════════════════════════════════════════
\section*{Reproducibility}

The complete \texttt{SKILL.md}, test harness (\texttt{test\_skill\_agent.py}),
all 24~model source files, and both result sets are included in the submission
archive.  OptiSkill requires only open-source dependencies (\texttt{pyomo},
\texttt{cloudpickle}, \texttt{highspy}) and runs without commercial solver
licences when Gurobi is unavailable.

% ══════════════════════════════════════════════════════════════════════
\bibliographystyle{plainnat}
\begin{thebibliography}{1}

\bibitem[Chen et~al.(2025)]{chen2025optichat}
H.~Chen, G.~E. Constante-Flores, K.~S.~I. Mantri, S.~M. Kompalli,
  A.~S. Ahluwalia, and C.~Li.
\newblock {OptiChat}: Bridging optimization models and practitioners with large
  language models.
\newblock \emph{INFORMS Journal on Data Science}, 2025.
\newblock \url{https://pubsonline.informs.org/doi/abs/10.1287/ijds.2025.0074}.

\end{thebibliography}

\end{document}