Results of a bake-off between OpenSpec and pure Claude Code

[jtlapp]

Hey everyone,

To get a sense of how well OpenSpec performs relative to implementing purely with Claude Code for a complicated project, I held a bake-off. I created a product requirements document, divided it into phases, and had both configurations implement the phases. Here are some general findings:

• OpenSpec was much more pleasurable to interact with, involving me more in design.
• OpenSpec surfaced 3 more gaps and gave me control over resolving those gaps.
• OpenSpec produced 50% more code having 50% more cyclomatic complexity.
• OpenSpec took twice as long and cost three times as much (API usage costs).

Here are some pointers that may be of interest:

• The planning and results repo: https://github.com/jtlapp/sdd-cmp-plan
• The details results: https://github.com/jtlapp/sdd-cmp-plan/blob/main/RESULTS.md
• The Claude-Code-only repo: https://github.com/jtlapp/sdd-cmp-cc-only
• The OpenSpec-developed repo: https://github.com/jtlapp/sdd-cmp-cc-openspec
• An analysis of code differences: https://github.com/jtlapp/sdd-cmp-plan/blob/main/results/code-analysis.txt

There are many caveats that may render my comparison meaningless, but I do wonder if the analysis given at the end of the analysis of code difference might provide some insight for improving OpenSpec.

Here are the caveats:

• I didn't review OpenSpec specs. Although I provided the feedback requested during OpenSepc exploration, I did not review the specs generated for each change or the specs generated for the system as a whole.
• Initial/final test plans. I had the workflows create anticipated test cases in advance of implementation, which certainly affected outcomes. The workflows also maintained summaries of the final test cases and the diffs from the initial test cases. This also increased usage.
• Overtaxed OpenSpec. OpenSpec is not intended for more than 10 tasks at a time, but several phases had more than 10.
• Didn't spec from scratch. I started with a PRD detailing observable behavior. Spec-driven development is geared toward working with the LLM to define the specification from scratch.
• Assisted with surfaced problems. The workflows gave opportunities for me to explore aspects of implementation during planning, but I mainly explored the aspects that the LLM appeared to be indicating were potentially problematic.

I plan to explore continuing with OpenSpec specifically trying to moderate these two issues:

• A lack of awareness across spec domains resulted in code duplication to address local needs instead of abstracting existing code to serve multiple needs.
• Some SHALL statements dictated the presence of code that was never used.

[m9dfukc]

Really interesting comparison!
Hadn't come across OpenLore before — I've built my own little toolset for detecting spec drift, so I'm curious to see how others approach it.
I've long admired the work coming out of JUXT, so I'll definitely have to take a proper look at Allium at some point. Thanks for putting it on my radar.