This document defines the formal rules for Caveman Compression: a lossless semantic compression technique that removes linguistic overhead while preserving all information content.
Design Goal: Minimize token count while maintaining complete semantic fidelity.
Core Principle: Remove only what LLMs can deterministically reconstruct. LLMs excel at predicting grammar, connectives, and sentence structure. They cannot reliably predict facts, numbers, or constraints. We compress the former, preserve the latter.
Each sentence MUST express exactly one atomic thought.
Formal Definition: An atomic thought is a single fact, action, constraint, or logical step that cannot be meaningfully subdivided without creating sentence fragments.
❌ "Database needs index and query runs faster"
✅ "Database needs index. Query runs faster."
Rationale: Compound sentences introduce connectives ("and", "but", "or") that consume tokens.
Sentences SHOULD contain 2-5 words.
Exceptions allowed when:
- Preserving a constraint requires 6-7 words
- Breaking further would lose semantic clarity
- Technical terms cannot be shortened without ambiguity
✅ "Need fast queries." (3 words)
✅ "Hash map offers O(1) lookup." (5 words)
⚠️ "All cups must contain same number." (6 words - acceptable for constraint)
❌ "We need to implement a fast query system that uses indexes." (11 words - must break)
Remove ALL logical connectives unless omission creates ambiguity.
Prohibited words:
- Causal: because, since, due to, owing to, as a result
- Contrastive: however, nevertheless, although, despite, but
- Sequential: therefore, thus, consequently, hence, then
- Purpose: in order to, so that, for the purpose of
- Conditional: if, unless (unless essential to logic)
Replacement strategy: Express cause-effect through sequential sentences.
❌ "Use index because query too slow"
✅ "Query too slow. Use index."
❌ "Although expensive, index improves speed"
✅ "Index costs space. Index improves speed."
Use active voice and present tense unless temporal distinction is critical.
Active voice:
❌ "The value is calculated by the function"
✅ "Function calculates value"
Present tense:
❌ "Will need to check constraints"
✅ "Need check constraints"
✅ "Was O(n) before. Now O(log n)." (temporal distinction preserved)
Keep specific numbers and quantities. Don't replace with vague terms.
✅ "three variables" not "several variables"
✅ "15 engineers" not "a few engineers"
✅ "50 million requests" not "many requests"
Rationale: Numbers carry information. Vague terms lose it.
❌ "The system handles many daily requests"
✅ "System handles 50 million daily requests"
(only if original contained "50 million")
Keep adjectives/adverbs that describe requirements. Remove only intensifiers.
Intensifiers to remove: very, extremely, quite, rather, really, somewhat
❌ "very important constraint" → ✅ "important constraint"
❌ "extremely fast" → ✅ "fast"
❌ "quite difficult" → ✅ "difficult"
Keep meaningful descriptors:
✅ "quickly" (speed requirement)
✅ "critical" (priority level)
✅ "optional" (vs required)
✅ "same" (vs different)
Omit articles (a, an, the) when context provides sufficient specificity.
❌ "The database needs an index"
✅ "Database needs index"
❌ "Add the value to the array"
✅ "Add value to array"
Exception: Keep articles when omission creates ambiguity between generic and specific references.
Keep short pronouns (it, we, he, she) when unambiguous. Only replace when ambiguous.
✅ "Function returns value. Store it in variable. Use it later."
(it = value, clear from context)
❌ "Function returns value and timestamp. Store it in variable."
✅ "Function returns value and timestamp. Store value in variable."
(which "it"? ambiguous)
Every inference step MUST be explicit. No implicit logical leaps allowed.
Test: Can a reader reconstruct the full reasoning chain from the compressed text?
❌ "Need hash map. Use hash map."
✅ "Need fast lookup. Array too slow. Hash map offers O(1). Use hash map."
(Shows: need → problem → solution → decision)
❌ "Bug in code. Fix bug."
✅ "Bug in code. Bug causes crash. Find crash location. Fix null check. Bug fixed."
(Shows: problem → symptom → investigation → solution → verification)
Simple conditionals: Omit "if" when condition-action pair is clear.
❌ "If value greater than ten, return error"
✅ "Value greater than ten. Return error."
Complex conditionals: Keep "if" when multiple interleaved conditions exist.
✅ "If A and B then do X. If A and not B then do Y."
(Removing 'if' creates ambiguity)
Keep collective references when already concise.
✅ "Check all connections"
✅ "Verify each service"
✅ "Test three endpoints"
Only enumerate when specificity adds information:
❌ "Install dependencies A, B, and C"
✅ "Install dependencies: React, Node, PostgreSQL"
Preserve technical terms that have precise meanings. Do not "simplify" domain-specific vocabulary.
✅ "Use binary search" (not "use fast search")
✅ "Implement recursion" (not "function calls function")
✅ "Check Big-O" (not "check speed")
To verify Caveman Compression correctness:
- Extract facts: List all discrete facts from original text
- Extract facts from compressed: List all discrete facts from compressed text
- Compare sets: Facts must be identical
- Check logical flow: Verify reasoning chain has no gaps
- Measure compression: Calculate token reduction percentage
Acceptance criteria:
- Fact preservation: Complete
- Logical completeness: Complete
- Token reduction: ≥30%
❌ "Function error return null"
✅ "Function has error. Function returns null."
Problem: Word order unclear; relation between words ambiguous.
❌ "Try fix"
✅ "Try option A. Measure result. Pick best option. Fix applied."
Problem: Skips intermediate logical steps.
❌ Original: "Use hash map"
❌ Caveman: "Use hash map. Hash map is fast."
✅ Caveman: "Use hash map."
Problem: Added information not in original ("is fast").
Original (61 tokens):
Binary search works by repeatedly dividing the search space in half. First,
compare the target value to the middle element. If they match, return the
position. If target is less, search the left half. If target is greater,
search the right half.
Level 2 Caveman (49 tokens):
Binary search divides search space in half. Compare target to middle element.
Match found. Return position. Target less than middle. Search left half.
Target greater than middle. Search right half.
Reduction: 20%
Original (66 tokens):
When debugging the null pointer exception, start by checking the stack trace
to identify where the error occurred. Then examine the code at that line to
see which variable is null. Add a null check before the dereference to
prevent the error from happening again.
Level 2 Caveman (54 tokens):
Debugging null pointer exception. Check stack trace. Find error location.
Examine code at error line. Identify null variable. Variable needs value
check. Add null check before dereference. Prevents error next time.
Reduction: 18%
- v1.0 (2025-01): Initial specification release
- Token counting based on GPT tokenizer conventions
- Semantic preservation inspired by information theory (Shannon entropy)
- Influenced by controlled natural languages (Simplified English, ASD-STE100)