Assembly-Level Malware Detection

A heuristic-based ransomware detector written in 8086 Assembly Language for DOS/DOSBox. Uses opcode density analysis to detect polymorphic malware without relying on signature databases.

🎯 Project Overview

This project implements a static analysis engine that detects crypto-ransomware by analyzing the frequency of suspicious instruction patterns (XOR encryption, bit shifting) in executable files. Inspired by academic research on opcode n-gram analysis and the Microsoft Malware Classification Challenge (BIG 2015).

🚀 Features

1. Heuristic Malware Detector (detector.asm)

• Scans files for XOR and SHL/ROL opcodes (encryption indicators)
• Calculates "Threat Density Percentage"
• Visual risk level bar chart
• Flashing red screen alert for dangerous files
• Configurable sensitivity threshold (default: 5%)

2. Demo Ransomware (malware.asm)

• Encrypts files using XOR cipher
• Polymorphic behavior simulation
• Safe test case for detector validation

📁 Repository Structure

├── detector_v2.asm           # Main heuristic scanner & analyzer
├── detectorv2.com            # Compiled analyzer binary
├── malware.asm               # Sample ransomware simulator
├── malware.com               # Compiled malware binary
├── ember_data.asm            # EMBER statistical data source
├── database.txt              # Sample malware signature database
├── victim.txt                # Test file for encryption demo
├── victim_original.txt       # Backup for resetting demo
├── reset_victim.sh           # Cleanup script for demo
├── QUICK_START.md            # Fast-track execution guide
├── DEMO_INSTRUCTIONS.md      # Detailed walkthrough
└── README.md                 # Main project documentation

🛠️ Requirements

• NASM (Netwide Assembler) for compilation
• DOSBox for execution
• macOS, Linux, or Windows

📦 Installation & Usage

cd asm_projects
nasm -f bin detector_v2.asm -o detectorv2.com
nasm -f bin malware.asm -o malware.com

Run in DOSBox

1. Start DOSBox:

   dosbox

2. Mount the directory:

   mount c /Users/ashwinacharya/newadldel/det
   c:
   

3. Run the detector:

   detectorv2.com
   

   • Enter filename: malware.com
   • Watch it detect the ransomware!

4. Run the demo ransomware:

   type victim.txt        # View original file
   malware.com            # Encrypt the file
   type victim.txt        # See encrypted gibberish
   

🧪 Demo Workflow (Presentation Mode)

# Step 1: Show clean file
type victim.txt

# Step 2: Attack with malware
malware.com

# Step 3: Show encrypted file
type victim.txt

# Step 4: Scan with detector
detectorv2.com
# Enter: malware.com
# Threshold: 10

# Result: "RANSOMWARE DETECTED" with red flashing screen

🔬 Technical Details

Detection Algorithm

The detector uses opcode frequency analysis:

1. Scan Phase: Read file bytecode and count specific opcodes

   • XOR (0x30-0x37) - Encryption operations
   • SHL/ROL/ROR (0xC0-0xD3) - Bit manipulation

2. Analysis Phase: Calculate threat density

   Threat % = (Suspicious Opcodes / Total Bytes) × 100
   

3. Classification: Compare against threshold

   • < 5% → SAFE
   • ≥ 5% → RANSOMWARE DETECTED

Opcode Patterns Detected

Based on Ramnit malware family characteristics:

Pattern	Opcodes	Significance
LODSB → XOR → STOSB	0xAC 0x30 0xAA	String decryption loop
XOR → ROL → XOR	0x30 0xC0 0x32	Multi-layer encryption
MOV → XOR → LOOP	0x8A 0x30 0xE2	Decryption routine

📊 Research Foundation

This project validates findings from:

1. Moskovitch et al. (2008) - "Opcode Representation for Malware Detection"
2. Microsoft Malware Classification Challenge (2015) - Kaggle dataset with 20,000+ samples
3. Ensemble AI Approaches - Decision Trees + Neural Networks achieving 99.87% accuracy

Key Insight

"Normal software uses XOR occasionally. Ransomware uses XOR obsessively (for encryption loops). By counting instruction frequency instead of matching signatures, we can detect unknown variants of polymorphic malware."

🎓 Educational Value

Concepts Demonstrated

• Static analysis vs. dynamic analysis
• Heuristic detection vs. signature-based detection
• Polymorphic malware evasion techniques
• Low-level x86 instruction set architecture
• DOS interrupt programming (INT 21h, INT 10h)

Skills Practiced

• Assembly language programming
• Malware reverse engineering principles
• Algorithm design for pattern matching
• User interface design with limited resources (80x25 text mode)

⚠️ Disclaimer

This project is for educational purposes only.

• The malware simulator is harmless - it only affects test files in the same directory
• Do NOT run on production systems
• Do NOT modify to create actual malicious software
• Intended for cybersecurity education and research

🚧 Future Enhancements

• Implement full N-gram pattern analysis (2-grams, 3-grams)
• Add comparative analysis against known malware families (Jerusalem, Cascade, Brain)
• Integrate with malware signature database
• Support for PE/ELF file format parsing
• Machine learning integration (SVM/Neural Networks)

📚 References

• Moskovitch et al. (2008) - "Opcode Representation for Malware Detection"
• Microsoft Malware Classification Challenge Dataset
• Ramnit Malware Analysis (AV vendors)
• x86 Instruction Set Reference (Intel/AMD)

👨‍💻 Author

Created as part of a cybersecurity research project exploring static malware analysis techniques using low-level programming.

📄 License

MIT License - Feel free to use for educational purposes with proper attribution.

---

⚡ Quick Test:

dosbox -c "mount c ." -c "c:" -c "detectorv2.com"