model-thought-process.md

Evaluating Language Ecosystems in the AI Era - A Journey of Thought

# Introduction

This document chronicles the thought process behind developing an "Evaluation Model for Language Ecosystems in the AI Era." It's written for those who already have some familiarity with the topic, hoping it might serve as a reference for anyone thinking through similar challenges.

Purpose:

• To share the background of this framework's development
• To document the process of trial and error
• To serve as a reference for those tackling similar problems

Disclaimers:

• This document represents my personal views and does not represent the views of my organization
• The analysis is based on circumstances as of late December 2025
• The content may become outdated as technology evolves

## My Background and Perspective

The Evolution of My Relationship with Programming

• I love programming and mathematics.
• My journey with programming began as a childhood hobby.
• From "personal enjoyment" to "creating things others can use"
• From "creating useful things" to "building systems that endure in production"

What I learned through this journey:

• The most important realization — Maintainability matters
• Requirements never stop changing — Initial specifications are never the final word
• Code lives longer than you expect — Years, sometimes decades
• Others will maintain your code — Someone in the future will read and modify it

In other words, what really matters in programming is:

Not "writing it correctly the first time," but "making it safely changeable later" Not "showing off your skills," but "continuing to solve problems"

A Shift in Perspective: From "Can I?" to "Can I Keep Doing It?"

Through years of experience, my evaluation criteria naturally evolved:

• Does it work? → Will it keep working?
• Can I write it? → Can we maintain it?

Think in terms of time, not moments.

This realization became the origin of the concept of "Fixability."

And Then, Into the Enterprise Realm

• Organizational decision-making
• Contracts, audits, accountability
• Risk management and practical constraints

This evolution shaped the perspective of this framework.

---

Practical Experience as a Technical Consultant

• Technology selection and architecture design for enterprise system development
• Practical use and evaluation of AI coding tools
• Addressing real-world challenges in organizational technology adoption

Development Experience Across Diverse Language Ecosystems

Multiple Generations of Technology:

• Started with MS-DOS and BASIC
• Learned communication through RS-232C and NetWare
• Once worked with Fortran, COBOL, Turbo Pascal, Delphi, and Objective-C
• Built applications with Visual Basic, C/C++, Java, and C#
• Developed web applications with Perl, PHP, Ruby, JavaScript, and TypeScript
• Now primarily using C#, TypeScript, and Python for work, and Kotlin, Swift, and Rust for personal projects

What I've Learned from Technological Evolution:

• Technology keeps changing dramatically, but fundamental principles remain relatively stable
• The answer stays the same: "Leverage existing assets and evolve what needs to evolve"

Balancing Field Work with Consulting

Even after transitioning from business system developer to technical consultant, I continue to work on actual implementations.

From a consultant's perspective:

• Evaluation criteria needed for client decision-making
• Real constraints of enterprise organizations
• Long-term technology strategy planning

From an developer's perspective:

• Daily experience with AI coding tools and the correction loop
• Feeling the gap between theory and reality
• Keeping up with the evolution of languages and libraries

I speak as a consultant and validate as a developer.
Hypotheses must be falsifiable through real-world correction loops.

This framework emerged from experiences spanning hobby to enterprise, from balancing consulting and implementation, and from observations across diverse languages.

---

# 0. The Beginning of a Thought Experiment: "Do We Need Languages Optimized for AI?"

## The Initial Question

How should programming languages evolve in the age of AI coding?
Should we create new languages specifically for AI? Let me formulate my own answer as of late December 2025.

## Background

• The age of AI-generated code has arrived
• Existing languages were designed for humans
• What would a language that's "easy to write" and "easy to read" for AI look like?
• Should AI generation ease take priority over human readability?

## Initial Hypotheses

• Do we need AI-specific concise syntax?
  • Optimal for AI likely varies by model, doesn't it?
  • Different vendors have different preferences, so it's unlikely to become an industry standard
  • If we're converging toward intermediate representations, existing languages might work just as well
• Ultra-strict grammar that eliminates ambiguity?
  • Might work initially, but evolution requirements could break it
  • Ambiguities in long-lived languages often reflect trade-offs between evolution and values
• Optimizing for AI at the expense of human readability?
  • In a zero-trust era, accepting things based solely on black-box testing?
    • Logic that only activates under untested conditions
    • Backdoors that activate under specific circumstances
    • Behaviors AI added "helpfully"
    • These can't be verified if invisible = organizations can't take responsibility, can they?
  • Currently, human readability serves as a safety mechanism
  • I don't think enterprises would accept this at present
• Even if machine-validated for correctness, would it be accepted?
  • Impossible for my generation, but perhaps when the first generation grows up with it as normal?
  • Based on my experience observing the Internet, smartphones, and cloud evolution

## A Pragmatic Direction: "Making Existing Languages AI-Friendly"

What does "evolving existing languages for AI" mean? For example:

• More expressive type systems
• Errors that clearly indicate causes and fixes
• Mechanisms to access AST and type information via APIs
• Standardization of runtime tracing and observability

From this perspective:

• C# with its Roslyn API is in a strong position
• TypeScript's type system continues to evolve
• Python's type hints and static analysis tools are maturing

What matters is: "Providing these as part of the standard language ecosystem"

This enables:

• Avoiding dependence on specific AI models
• Aggregating community-wide knowledge
• Maintaining compatibility across AI tools

## My Personal Conclusion as of Late December 2025

AI-optimized languages might succeed in the future. However, at present, the likelihood of new AI-optimized general-purpose languages gaining acceptance is low. I've concluded they're currently unnecessary. Optimizing existing languages for AI is a better approach.

I believe this direction is more practically acceptable and realistic.

---

# 1. The Next Thought Experiment: "So How Well-Suited Are Current Languages for AI?"

## From One Question to Another

The previous conclusion sparked a new question:

"So, how well-suited are commonly used languages for correcting AI-generated code?"

"Where are the optimization opportunities for each language?"

From abstract to concrete:

• "Ecosystems matter"—I get that
• But what specifically should we evaluate?
• What are the differences between C#, Python, and TypeScript?

Application to decision-making:

• When clients ask "which language is better?"
• When selecting a language for a project
• Specific evaluation criteria are needed

Direction for optimization:

• First, understand each language's "strengths" and "weaknesses" from an AI perspective
• Also need to verify AI generation accuracy for each language
• Initial generation quality (new creation) matters, but editing existing code is crucial for continued use

New questions:

• What are the AI-perspective "strengths" and "weaknesses" of existing languages?
• Where can we improve existing languages to increase their AI compatibility?
• What actually influences AI code generation quality?

---

# 2. The First Realization: "The Problem Isn't Generation—It's Correction"

## Observation: Coding with AI

• AI-generated code appears to work at first glance
• But it breaks with minor changes
• Passing error messages back to AI leads to the same errors repeating
• "Ease of correction" varies by language
• Adding comments as supplementary explanations in source code sometimes improves accuracy

## A Paradigm Shift

The question of "AI code generation quality" was looking at it from the wrong angle

What matters is: Not whether AI can write correct code on the first try (initial generation quality), But whether AI can correctly fix mistakes (fixability)

## The Fundamental Structure of AI Coding

An intuitive realization:

Without concrete evidence yet, but in a vague state, I thought the essence of AI coding might be whether mistakes can be corrected properly.

For AI to fix mistakes, it needs external input on "what's wrong" and "how to fix it"

Why this structure?

• AI models learn from training data
• But the model doesn't possess all information
• Specifically:
  • Latest information after knowledge cutoff
  • Project-specific context
  • Domain-specific rules
  • Runtime errors and performance profile information

In other words, for AI to correct properly:

1. Accurate external information is needed
2. When that information comes from multiple sources, consistency is needed
3. That information must be provided in a format AI can understand
4. AI must be able to use that information to make corrections

## What This Structure Means

A "good language ecosystem" for AI is:

• Not an environment where AI generates perfect code on the first try
• But an environment that can accurately tell AI "where and what is wrong" and "how to fix it"

Specifically:

• Clear type error messages — Tell what's wrong
  • Updating dependency libraries might compile but cause runtime errors
  • Tests might fail, requiring corrections to meet specifications
  • Some issues can't be detected at implementation (compile) time (such as parallel execution issues)
• Detailed runtime traces — Tell where failures occurred
• API metadata — Tell correct usage
• Compiler diagnostics — Provide hints on how to fix
• Implementation-runtime consistency - Inconsistency confuses AI

## Similarity to Maintainability

This structure mirrors maintainability:

Human coding:

• Requirements keep changing → Easy-to-change code matters
• Future someone maintains it → Structure that conveys intent matters
• Bug cause identification → Clear error information, runtime observability, test code matter

AI coding:

• Generated results aren't perfect → Easy-to-correct environment matters
  • Humans also implement through trial and error, improving accuracy through reviews
• AI corrects as an "other" → Semantic information clarity matters
• Recovery from errors → Accurate feedback matters

## Clarifying the Direction of Thought

• The correction loop repeats from initial generation through validation until all mistakes are fixed
• Mistakes happen not just at implementation time, but also at runtime
• For the validation loop:
  • At what timing
  • What types of mistakes exist
  • What's needed to correct them
• This is a language-agnostic problem, requiring generalized thinking.

---

# Aside: I Don't Consider "Same Input ⇒ Same Output" a Requirement

## A Common Debate

When discussing AI coding, debates often arise about "whether identical input should produce identical output."

Personally, I don't think this point is that important.

## Why I Think This

The reason is simple: human programmers also write different implementations for the same specifications.

In practice:

• Same requirements
• Same constraints
• Same design principles

Yet:

• Implementation structure
• Coding style
• Optimization direction

Naturally differ by developer.

## What Actually Matters

What matters is:

• Does it meet specifications?
• Can it be verified through testing?
• Can we explain why it's correct?

Not whether "the written code is identical every time."

## As Long as AI Outputs in General-Purpose Languages

As long as AI outputs code in general-purpose languages readable by humans, even with some variation in generation results:

• White-box testing
• Black-box testing
• Human review

can ensure the validity and accountability of deliverables on the human side.

As long as this premise holds, the non-deterministic nature of AI's generation process shouldn't be a significant practical problem.

## The Importance of Layer Separation

However, the process of converting generated source code into executables must be deterministic.

Because compilation and builds produce consistent results:

• Test result reproducibility
• Release reliability
• Audits and incident response

remain viable.

## Summary

In other words, what matters is:

"Generation can be trial-and-error (non-deterministic), but execution and verification must be deterministic"

—this layer separation

This perspective leads to emphasizing "implementation-time" and "runtime" consistency in the evaluation model construction that follows.

## Yet, a New Question

This consideration raised a new question:

"If AI output isn't stable, can we achieve stable utilization?"

How to resolve this contradiction:

While saying AI generation can be non-deterministic, practice demands "stable usability."

The answer lies in defining stability:

• ❌ Not stability as "generating identical code every time"
• ✅ But stability as "correction loop converges"

In other words:

1. Initial generation can be diverse
2. Error feedback-based corrections function properly
3. After several correction iterations, the system converges to a state meeting specifications

This "convergence of the correction loop" is the essence of practical stability in AI coding.

This leads to thinking about modeling—what structure supports this correction loop?

---

# 3. The Structure Supporting the Correction Loop

## Can We Build Stable Systems on Unstable Foundations?

After realizing "convergence of the correction loop" in the previous section, a fundamental question remained:

Can we build stable systems (practical coding environments) on unstable foundations (non-deterministic AI)?

This seems contradictory.

## Applying TCP/IP Concepts

Intuitively, I thought of applying TCP/IP concepts.

From personal experience:

This idea probably stems from memories of the RS-232C communication era. Learning about TCP/IP's structure was mind-blowing back then.

The RS-232C world:

• Direct 1-to-1 connections
• Communication cuts when cables disconnect
• All error handling implemented application-side
• Applications directly dealt with unstable physical layers

The TCP/IP world:

• Clear responsibilities through layer separation
• Upper layers absorb lower layer instability
• Retransmission, sequencing, error correction provided as standard
• Applications only need to think about stable communication

This structure of "achieving stability through layer separation" seemed to overlap with the AI coding problem.

What TCP/IP teaches us:

TCP/IP, which underpins the Internet, achieves stable communication (TCP) atop unstable networks (IP).

【Lower Layer】
Packet loss        → Unstable
Variable delays    → Unstable
Uncertain routing  → Unstable

【Upper Layer】
TCP retransmission → Guarantees stable communication
Sequencing         → Guarantees data integrity
Error correction   → Guarantees reliability

Key insight:

• Lower layers (packets) can be unstable
• With error detection and feedback mechanisms
• Upper layers (applications) can provide stability

## Analogy to AI Coding

Applying this structure to AI coding:

【Lower Layer: AI Generation】
Initial generation       → Non-deterministic (unstable)
Diverse implementations  → Unstable
Trial and error          → Unstable

【Feedback Mechanism】
Type errors       → Detect what's wrong
Runtime errors    → Detect where failures occur
Test failures     → Detect gaps with specifications
etc.

【Upper Layer: Practical Development】
Correction loop convergence → Stable results
Meeting specifications      → Reliability
Explainability              → Auditability

## What This Analogy Means

Conditions for TCP/IP to function:

1. Can detect packet arrival/non-arrival
2. Has retransmission mechanism
3. Can guarantee order
4. Can correct errors

Conditions for AI coding to function:

1. Can detect errors → Type systems, tests
2. Has correction mechanism → AI feedback loop
3. Can guarantee consistency → Implementation-runtime alignment
4. Can confirm convergence → Specification verification through tests

## The Evaluation Perspective Emerges

In other words, what's required of language ecosystems:

Not "AI can generate perfectly"

But "providing mechanisms to detect errors, provide feedback, and converge corrections"

The TCP/IP equivalent of "retransmission and sequencing" is:

• Clear error messages
• Consistent type systems
• Runtime observability
• Specification clarification through tests

## The First Major Realization

This was the first major realization in building this framework.

Through the TCP/IP analogy, vague intuitions became clear structures:

Building stable systems on unstable foundations is possible.

But it requires appropriate feedback mechanisms.

This insight guided the entire process of discovering evaluation criteria that followed.

## The Next Question: Layering Language Ecosystems

After seeing this structure, the next consideration was layering language ecosystems.

Starting point of consideration:

• AI coding agents run validation loops
• Bridging AI models and languages
• But surveying information required for feedback mechanisms...

Realization:

Might language specifications alone be insufficient?

New questions:

• From an AI perspective, how are languages perceived?
• Where does information needed for feedback mechanisms come from in languages?
• How should we structure the entire language ecosystem to reveal evaluation criteria?

Therefore: I thought layering the language side would help grasp the full picture of information AI needs.

---

# 4. The Structure of the Correction Loop

I detailed the structure where AI generates code and runs validation loops. Clarifying under what conditions regeneration occurs at each stage of the correction loop.

## Seven Phases of the Validation Loop

Phases that emerged from observation:

Phase1 Static Knowledge (Prior Knowledge)
   - AI's training data (OSS, Q&A, official docs, blogs, etc.)
   - Language specifications, standard libraries
   - Common coding patterns

Phase2 Generation (Initial Generation)
   - AI's initial code generation
   - From prompts and context
   
Phase3 Static Semantic Validation
   - Type checking
   - Syntax validation
   - Build/compilation
   - Linter validation
   - → If failed, go to Phase7
   
Phase4 Startup Check
   - Dependency resolution
   - Basic startup confirmation
   - → If failed, go to Phase7
   
Phase5 Test Execution
   - Unit tests
   - Integration tests
   - Specification verification (application-dependent)
   
Phase6 Test Feedback (Runtime Feedback)
   - Test results
     - → If successful, loop ends
   - Error messages
   - Stack traces
   - Performance information
     - Observed data is `fact`
     - Requires `semantic interpretation`
   
Phase7 Regeneration (Corrective Generation)
   - Corrections based on feedback
   - → Return to Phase3 and loop

Important realization: I thought this information targets not just language specifications, but the entire language ecosystem.

---

# 5. The Full Picture of Language Ecosystems

## Reorganizing Information Sources

Analyzing the seven phases of the validation loop, organizing information sources:

Phase1 Static Knowledge Phase:

• AI's training data → OSS, Q&A, documentation
• Language specifications → Official specs
• Community practices → Blogs, tutorials

Phase3 Static Validation Phase:

• Type system → Core language features
• Linter → Development tools
• API information → Metadata
• Build tools → Ecosystem services

Phase4 Startup Check Phase:

• Dependencies → Package management
• Runtime environment dependencies

Phase5 Phase6 Execution/Feedback Phase:

• Runtime → Language implementation
• Test frameworks → External tools
• Observability → Profilers, debuggers

## Discovery of Layering

Surveying these reveals four layers:

1. Core Layer

• Language specifications
• Type system
• Basic syntax

2. Service Layer

• Compiler/Interpreter
• Toolchain
• Development support tools (LSP, debuggers, etc.)

Rationale for naming:

Named "Service" separately from "Core" to represent feedback from language to AI

3. Dependency Layer

• Dependency management
• Package ecosystem
• Runtime environment information

4. Community Layer

• Shared knowledge (OSS, Q&A)
• Best practices
• Community culture

## Scope of "Language" from AI's Perspective

Conventional definition of "language":

• Refers to Core + Service domain
• Language specifications and their implementations

But from AI's perspective:

Viewing the entire language ecosystem, including how it's used in the language community and community culture, as "language" seems more natural

Why I think this:

• AI learns from training data
• Training data includes not just language specs, but actual usage examples
• Community practices shape AI's understanding
• In other words, for AI, "language" = entire ecosystem

## Organizing Inter-Layer Dependencies

This realization raised the next question:

Should we further subdivide the four layers?

To consider this, I thought to organize inter-layer dependencies.

---

# 6. Dual Perspective: AI View / Human View

At that moment, intuitively, the concentric circles of Clean Architecture came to mind. Though displayed as text here, I tried mapping each layer onto concentric circles.

       ┌─────────────┐
       │  Community  │  ← Outer
       ├─────────────┤
       │ Dependency  │
       ├─────────────┤
       │   Service   │
       ├─────────────┤
       │    Core     │  ← Center
       └─────────────┘

I naturally placed language specifications at the center.

Human developer's perspective:

• Core (center) → Language specifications are foundational
• Dependencies spread outward
• Domain (Core) → Application (Service) → Infrastructure (Dependency)
• Community is reference information

## A Surprising Discovery: AI's Center is Reversed

But reconsidering from AI's perspective revealed a different structure...

Dependency relationships for AI:

• AI learns from training data
• Most training data comes from Community Layer
• This has the greatest influence on initial code generation

In other words:

       ┌─────────────┐
       │    Core     │  ← Outer (human's center)
       ├─────────────┤
       │   Service   │
       ├─────────────┤
       │ Dependency  │
       ├─────────────┤
       │  Community  │  ← Center (AI's center)
       └─────────────┘

## Key Realization: Humans and AI Perceive Languages in Opposite Directions

Human developers:

• Core (language specs) at center
• Extending outward
• Core → Service → Dependency → Community

AI's perspective:

• Community (training data) at center
• Understanding language from there
• Community → Dependency → Service → Core

Humans and AI have opposite dependency relationships to language ecosystems

What this discovery means:

• "Common usage" is paramount for AI
• Community practices over Core specifications
• Community Layer quality determines AI generation quality

Implications for evaluation:

• Simply "excellent language specifications" isn't enough
• "How it's used in the community" matters
• Quality and quantity of practical examples in training data

This insight provides the rationale for positioning Community Layer as the first evaluation criterion in building evaluation criteria that follows.

---

# 7. Strategy Pattern and Language Differences

## How to Understand Language-Specific Differences

After seeing the four-layer structure, the next question arose:

In which layer and which parts do differences between languages manifest?

Starting point of consideration:

• Want to evaluate C#, Python, TypeScript
• But each has different characteristics
• Can we compare them from a unified perspective?

## The Strategy Pattern Image

Here, intuitively, the Strategy pattern image emerged.

Advantages of this perspective:

• Interface is the same
• Strategy (implementation) differs
• Can explain concepts including not just language specification differences, but values and culture

## Yet, a New Question

Just as I was satisfied with this organization, a fundamental question emerged:

Can static and dynamic languages really be discussed with the same interface?

Specific problem:

• Static languages (C#, TypeScript) → Compile-time type checking
• Dynamic languages (Python) → Runtime type checking
• Can this be called "the same interface"?

## Seeking a Unified Perspective

Can we find a structure that unifies discussion of static and dynamic languages?

Conventional perspective:

【Static Languages】
Source Code → Compilation → Executable → Execution

【Dynamic Languages】
Source Code → (No Compilation) → Execution

In this structure, unified discussion is indeed challenging...

## Key Realization: What is the Essence of Compilation?

Here, I reconsidered from AI's perspective:

For AI, what does a compiler do?

Conventional understanding:

• Compiler = Converts source code into executables

But from AI's view:

• Compiler = Semantic validation engine
• Executable = Merely an "artifact" upon successful validation

## Paradigm Shift: Redefining the Value of Compilation

For running correction loops, what matters to AI: Not that executables are generated, but that semantic errors are detected.

From this perspective:

【Static Languages】
Source Code → Semantic Validation (Compilation) → Error or Artifact

【Dynamic Languages】
Source Code → Semantic Validation (Runtime) → Error or Success

Unified structure:

Source Code → Semantic Validation → Feedback
                ↓
        (Artifact on success)

## What This Unified Perspective Means

Static languages:

• Semantic validation = Compile-time type checking
• Timing = Implementation time
• Artifact = Executable file

Dynamic languages:

• Semantic validation = Runtime type checking
• Timing = Runtime
• Artifact = (None, or bytecode)

Key insight:

Both provide the same interface of "semantic validation"

But their strategy (timing) differs

## Implications for the Evaluation Model

This discovery enables:

1. Clarifies the meaning of Static Semantics

   • Static languages: Compile-time semantic validation
   • Dynamic languages: Partial validation via type hints, etc.

2. The importance of Runtime Continuity

   • Static languages: Implementation-runtime consistency
   • Dynamic languages: Runtime semantic validation, runtime error clarity

3. Fair cross-language comparison becomes possible

   • Evaluate with the same interface
   • Understand as strategy differences

This insight established a foundation for thinking about languages in a generalized state.

---

# 8. Port & Adapters: Syntax is the UI

## How to Implement Strategy Switching

In Section 7, we gained a perspective to handle static and dynamic languages uniformly. But a new question arose:

So, how do we actually switch implementations (Strategies)?

Specifically:

• When AI handles both C# and Python
• How does it process internally?
• How does it absorb language differences?

## The Port & Adapters Concept

Here, the Port & Adapters (Hexagonal Architecture) concept seemed to fit better.

Key insight:

• Domain logic doesn't change
• Adapters (DB and UI) are interchangeable
• Loosely coupled via Ports

## Reversal from AI's Perspective

AI perceives language ecosystem layers in the opposite direction from humans. At that point, the outermost layer corresponds to language specifications, specifically language syntax. In other words, I realized that the syntax of the generated language is the user interface (UI) between AI and human users.

## What This Perspective Means

Humans grasp the meaning of "conditional branching" from the syntax "if".

Human understanding process:

Syntax → Semantics
  "if"    → Conditional branching
  "for"   → Repetition

Conversely, AI generates language-appropriate syntax from the meaning of "conditional branching."

AI's generation process:

Semantics → Syntax
Conditional branching → "if"
Repetition → "for" 

Key insight:

For AI, syntax is merely an interchangeable "display format"

## Syntax as Adapter

Organizing from the Port & Adapters perspective, the semantic layer is the true essence This discovery led to an important conclusion:

For AI, language syntax is "interchangeable"

What matters is the semantics provided by the Service layer before that

AI's thought process:

1. Understand semantic requirements
2. Convert to target language syntax
3. Validate at Service layer (compiler, etc.)

## Implications for Evaluation

This insight enables:

1. Should evaluate semantics over syntax

   • Type system expressiveness
   • Semantic clarity of error messages
   • Semantic consistency

2. The importance of Service Layer

   • Quality of semantic validation
   • Feedback clarity
   • Implementation-runtime consistency

3. Redefining Runtime Continuity

   • Not syntax consistency
   • Semantic consistency

Impact on the evaluation model:

• Static Semantics → Quality of semantic validation
• Runtime Continuity → Semantic consistency
• Error Clarity → Semantic error messages

## Returning to the Initial Question

This discovery brought a new perspective to the initial question:

When considering "languages optimized for AI,"

Whether generated syntax is human-readable might not be a fundamental issue

Because:

• For AI, syntax is merely "UI"
• What matters is the semantic layer
• If semantics are sound, syntax is interchangeable
• Then human-readable is preferable
  • However, the language's semantic layer supporting correction loops should be externalized (Appendix A-3)
  • Then existing languages might work just as well

However, in this document: We won't delve deeper into this question.

Because this framework's purpose is:

• Evaluating existing languages, not
• Designing new languages

While this insight offers important implications for future language design, this framework focuses on evaluating existing languages.

---

# 9. Concretizing Evaluation Criteria: Mapping Validation Loop to Layers

## From Inspiration to Systematization

Insights so far:

• Convergence of correction loops is essential (Sections 2-3)
• Seven-phase validation structure (Section 4)
• Four-layer ecosystem structure (Section 5)
• Reversal from AI perspective (Section 6)
• Importance of semantics (Sections 7-8)

At this point, the model's complete form became visible.

At which points in the validation loop is which information needed?

Which layer of the ecosystem does that information come from?

Beyond this, no particular insights—just systematic organization work.

## Working Approach

Specifically:

1. Reconfirm each phase of the validation loop
2. Identify information needed at each phase
3. Classify which layer that information belongs to
4. Organize as evaluation criteria

Desktop work:

• Create correspondence table of 7 phases and 4 layers
• Fill each cell with "required information"
• Check for overlaps and gaps
• Group and consolidate into evaluation criteria

## Validation Loop and Layer Correspondence

Phase1 Static Knowledge Phase:

Required information: Common language usage
Corresponding layer: Community Layer
Evaluation criterion candidate: Public Knowledge

Phase3 Static Semantic Validation Phase:

Required information: Type system, syntax rules, API specs
Corresponding layer: Core Layer + Service Layer
Evaluation criterion candidates: 
  - Static Semantics (semantic validation)
  - Spec Conformance (specification conformance)
  - Tool Support (tool support)
  - Error Clarity (error clarity)

Phase4 Startup Check Phase:

Required information: Dependencies, environment configuration
Corresponding layer: Dependency Layer
Evaluation criterion candidates:
  - Dependency Management
  - Environment Semantics

Phase5 Phase6 Execution/Feedback Phase:

Required information: Runtime behavior, error information
Corresponding layer: Service Layer (runtime)
Evaluation criterion candidates:
  - Runtime Continuity
  - Runtime Semantics

## Consolidation into 9 Criteria

From this organization, 9 evaluation criteria emerged:

Community Layer → 1 criterion

1. Public Knowledge

Core + Service Layer → 5 criteria 2. Static Semantics 3. Spec Conformance 4. Tool Support 5. Error Clarity 6. Runtime Continuity

Dependency Layer → 2 criteria 7. Dependency Management 8. Environment Semantics

Runtime Layer → 1 criterion 9. Runtime Semantics

## Criterion Priority

Reflecting AI perspective (Section 6):

• Community Layer (1 criterion) highest priority
• Training data quality affects initial generation
• Next, semantic validation (criteria 2-6)
• Dependencies and runtime (criteria 7-9)

Evaluation flow:

Public Knowledge (initial generation quality)
    ↓
Static Semantics (implementation-time validation)
    ↓
Runtime Continuity (runtime consistency)
    ↓
Correction loop convergence

## Systematization Complete

At this point:

• Clarified relationship between 7 phases and 4 layers
• Completed 9-criterion evaluation framework
• Organized meaning and importance of each criterion

Remaining work:

• Detailed definition of each criterion (main text)
• Validation with C#, Python, TypeScript (main text)
• Organizing theoretical background (Appendices A-C)

This systematization completed the transition from thought process to practical evaluation model.

---

# Conclusion

This document records the thought process as of late December 2025.

Sections 0-2: Intuition and realization

• AI-specific languages unnecessary? (Section 0)
• Need evaluation criteria (Section 1)
• Fixability is essence (Section 2)

Sections 3-6: Discovering structure

• TCP/IP analogy (Section 3)
• Seven-phase structuring (Section 4)
• Four-layer discovery (Section 5)
• Dual perspective discovery (Section 6)

Sections 7-8: Gaining unified perspective

• Unifying semantic validation (Section 7)
• Syntax = UI (Section 8)

Section 9: Systematization

• Consolidation into 9 criteria (Section 9)

Characteristics of this process:

• Started with vague questions
• Past experiences like TCP/IP generated analogies
• The AI perspective clarified structures
• Ended with systematic organization work

I hope this record serves as a reference for those tackling similar problems.