PyClean_DSL_Language_Guide.txt

================================================================================
                    PYCLEAN DSL - LANGUAGE GUIDE
         Domain-Specific Language for Data Cleaning Operations
================================================================================

TABLE OF CONTENTS
-----------------
1. Introduction
2. Language Syntax & Keywords
3. Statement Types
4. Data Types
5. Complete Examples
6. Execution Flow
7. Error Handling
8. Best Practices

================================================================================
1. INTRODUCTION
================================================================================

PyClean DSL is a domain-specific language designed for data cleaning operations
on CSV files. It provides a simple, intuitive syntax for common data cleaning
tasks without requiring programming knowledge.

Purpose:
--------
- Clean and transform CSV data
- Fill missing values (nulls)
- Remove duplicates
- Validate data ranges
- Transform text (trim, case conversion)

Target Users:
-------------
- Data analysts
- Data scientists
- Business analysts
- Anyone working with messy CSV data

================================================================================
2. LANGUAGE SYNTAX & KEYWORDS
================================================================================

RESERVED KEYWORDS:
------------------
CLEAN              - Start a cleaning process
FILL_NULL          - Fill null/empty values
WITH               - Specify fill value
REMOVE_DUPLICATES  - Remove duplicate rows
COLUMN             - Target a specific column
TRIM               - Remove leading/trailing whitespace
UPPERCASE          - Convert text to uppercase
LOWERCASE          - Convert text to lowercase
VALIDATE           - Validate data against rules
range              - Range validation function

OPERATORS:
----------
;                  - Statement terminator (required)
,                  - Parameter separator
( )                - Function parameters
" "                - String literals

COMMENTS:
---------
//                 - Single-line comment
/* */              - Multi-line comment (if supported)

================================================================================
3. STATEMENT TYPES
================================================================================

3.1 CLEAN STATEMENT
-------------------
Purpose: Initialize the cleaning process for a dataset

Syntax:
  CLEAN dataset_name;

Example:
  CLEAN dataset;
  CLEAN mydata;

Notes:
- Must be the first statement in most scripts
- dataset_name is an identifier (usually "dataset")
- Does not perform any actual cleaning, just declares intent


3.2 FILL_NULL STATEMENT
------------------------
Purpose: Fill empty/null values in a specific column with a default value

Syntax:
  FILL_NULL column_name WITH value;

Examples:
  FILL_NULL age WITH 0;
  FILL_NULL name WITH "Unknown";
  FILL_NULL salary WITH 50000;
  FILL_NULL email WITH "no-email@example.com";

Supported Value Types:
- Numbers:   FILL_NULL age WITH 25;
- Strings:   FILL_NULL name WITH "Default Name";
- Decimals:  FILL_NULL price WITH 99.99;

Notes:
- Column name must exist in the CSV
- String values must be in double quotes
- Numbers don't need quotes
- Only fills cells that are empty or null


3.3 REMOVE_DUPLICATES STATEMENT
--------------------------------
Purpose: Remove duplicate rows from the dataset

Syntax:
  REMOVE_DUPLICATES;

Example:
  REMOVE_DUPLICATES;

Notes:
- Removes rows that are completely identical
- Keeps the first occurrence
- No parameters needed
- Works on entire rows, not specific columns


3.4 COLUMN TRANSFORM STATEMENTS
--------------------------------
Purpose: Apply transformations to a specific column

Syntax:
  COLUMN column_name operation;

Operations Available:
---------------------

a) TRIM - Remove leading and trailing whitespace
   COLUMN name TRIM;
   COLUMN email TRIM;

b) UPPERCASE - Convert all text to uppercase
   COLUMN name UPPERCASE;
   COLUMN department UPPERCASE;

c) LOWERCASE - Convert all text to lowercase
   COLUMN email LOWERCASE;
   COLUMN username LOWERCASE;

Examples:
  COLUMN name TRIM;
  COLUMN name UPPERCASE;
  COLUMN email LOWERCASE;
  COLUMN address TRIM;

Notes:
- Operations are applied to all rows in the column
- Column must exist in the dataset
- Converts non-string values to string first
- Multiple operations can be chained on same column


3.5 VALIDATE STATEMENT
-----------------------
Purpose: Validate data and remove invalid rows

Syntax:
  COLUMN column_name VALIDATE validation_type(parameters);

Validation Types:
-----------------

a) range - Validate numeric values are within a range
   Syntax: range(min, max)
   
   Examples:
   COLUMN age VALIDATE range(0, 120);
   COLUMN temperature VALIDATE range(-50, 50);
   COLUMN score VALIDATE range(0, 100);

Parameters:
- min: Minimum valid value (inclusive)
- max: Maximum valid value (inclusive)

Behavior:
- Removes rows where value < min OR value > max
- Converts column to numeric type
- Non-numeric values are treated as invalid

Notes:
- Currently only range validation is supported
- Future versions may support: regex, email, phone, etc.
- Invalid rows are completely removed from dataset

================================================================================
4. DATA TYPES
================================================================================

4.1 IDENTIFIERS
---------------
- Column names: age, name, email, salary
- Dataset names: dataset, mydata
- Must start with letter or underscore
- Can contain letters, numbers, underscores

4.2 STRING LITERALS
-------------------
- Enclosed in double quotes: "Hello"
- Examples: "Unknown", "no-email@example.com", "0"
- Cannot contain unescaped quotes

4.3 NUMERIC LITERALS
--------------------
- Integers: 0, 25, 100, -5
- Decimals: 99.99, 3.14, -2.5
- Used in FILL_NULL and VALIDATE statements

4.4 SPECIAL VALUES
------------------
- Empty/Null: Represented as empty cells in CSV
- NaN: Result of failed numeric conversions

================================================================================
5. COMPLETE EXAMPLES
================================================================================

EXAMPLE 1: Basic Data Cleaning
-------------------------------
// Clean employee data
CLEAN dataset;

// Fill missing values
FILL_NULL name WITH "Unknown Employee";
FILL_NULL age WITH 25;
FILL_NULL department WITH "Unassigned";

// Remove duplicates
REMOVE_DUPLICATES;

// Clean up text
COLUMN name TRIM;
COLUMN name UPPERCASE;
COLUMN email LOWERCASE;


EXAMPLE 2: Sales Data Processing
---------------------------------
CLEAN sales_data;

// Handle nulls
FILL_NULL product WITH "Unknown Product";
FILL_NULL quantity WITH 0;
FILL_NULL price WITH 0.0;

// Standardize text
COLUMN product TRIM;
COLUMN product UPPERCASE;
COLUMN customer_email LOWERCASE;
COLUMN customer_email TRIM;

// Remove invalid data
COLUMN quantity VALIDATE range(0, 10000);
COLUMN price VALIDATE range(0, 999999);


EXAMPLE 3: Student Records
---------------------------
CLEAN student_records;

// Fill missing grades
FILL_NULL grade WITH 0;
FILL_NULL attendance WITH 0;

// Clean names
COLUMN first_name TRIM;
COLUMN last_name TRIM;
COLUMN first_name UPPERCASE;
COLUMN last_name UPPERCASE;

// Email standardization
COLUMN email LOWERCASE;
COLUMN email TRIM;

// Validate ranges
COLUMN grade VALIDATE range(0, 100);
COLUMN attendance VALIDATE range(0, 100);
COLUMN age VALIDATE range(16, 25);

// Remove duplicates
REMOVE_DUPLICATES;


EXAMPLE 4: E-commerce Products
-------------------------------
CLEAN products;

// Default values
FILL_NULL product_name WITH "Untitled Product";
FILL_NULL price WITH 0.00;
FILL_NULL stock WITH 0;
FILL_NULL category WITH "Uncategorized";

// Text cleanup
COLUMN product_name TRIM;
COLUMN category TRIM;
COLUMN category UPPERCASE;
COLUMN description TRIM;

// Price validation
COLUMN price VALIDATE range(0, 100000);
COLUMN stock VALIDATE range(0, 999999);

REMOVE_DUPLICATES;

================================================================================
6. EXECUTION FLOW
================================================================================

Compilation Process:
--------------------
1. LEXICAL ANALYSIS (Tokenization)
   - Source code → Tokens
   - Identifies keywords, operators, literals
   - Removes whitespace and comments

2. SYNTAX ANALYSIS (Parsing)
   - Tokens → Abstract Syntax Tree (AST)
   - Validates grammar rules
   - Checks statement structure

3. CODE GENERATION
   - AST → Pandas Python code
   - Converts DSL to executable operations

4. EXECUTION
   - Runs Pandas operations on CSV data
   - Applies transformations in order
   - Returns cleaned CSV

Execution Order:
----------------
Statements are executed in the order written:

1. CLEAN statement (initialization)
2. FILL_NULL statements (fill missing data)
3. REMOVE_DUPLICATES (deduplicate)
4. COLUMN transforms (TRIM, UPPERCASE, LOWERCASE)
5. VALIDATE statements (remove invalid rows)

IMPORTANT: Order matters!
- Fill nulls BEFORE validation
- TRIM BEFORE case conversion
- Remove duplicates AFTER filling nulls

================================================================================
7. ERROR HANDLING
================================================================================

7.1 LEXICAL ERRORS
------------------
- Invalid characters
- Unterminated strings
- Unknown tokens

Example:
  FILL_NULL name WITH 'invalid';  // Error: Use double quotes
  COLUMN age @TRIM;                // Error: Invalid character @


7.2 SYNTAX ERRORS
-----------------
- Missing semicolons
- Wrong keyword order
- Invalid statement structure

Examples:
  FILL_NULL age WITH 0            // Error: Missing semicolon
  WITH 0 FILL_NULL age;           // Error: Wrong order
  COLUMN TRIM name;               // Error: Column name before operation


7.3 RUNTIME ERRORS
------------------
- Column not found
- Invalid data type
- Validation failures

Examples:
  FILL_NULL nonexistent WITH 0;   // Error: Column doesn't exist
  COLUMN age VALIDATE range(0);   // Error: Range needs 2 parameters


7.4 ERROR MESSAGES
------------------
Format: Line X:Y - Error message

Example:
  Line 5:10 - Column 'agee' not found. Available columns: [age, name, email]
  Line 8:15 - Expected SEMICOLON, got IDENTIFIER

================================================================================
8. BEST PRACTICES
================================================================================

8.1 ORDERING
------------
✓ DO: Start with CLEAN statement
✓ DO: Fill nulls before validation
✓ DO: TRIM before case conversion
✓ DO: Remove duplicates after filling nulls

✗ DON'T: Validate before filling nulls
✗ DON'T: Skip the CLEAN statement


8.2 NAMING
----------
✓ DO: Use descriptive column names
✓ DO: Match exact column names from CSV

✗ DON'T: Use spaces in column names
✗ DON'T: Use special characters


8.3 VALUES
----------
✓ DO: Use appropriate data types
✓ DO: Use quotes for strings
✓ DO: Use meaningful defaults

✗ DON'T: Fill numbers with strings
✗ DON'T: Use empty strings as defaults


8.4 VALIDATION
--------------
✓ DO: Set realistic ranges
✓ DO: Validate after filling nulls
✓ DO: Document why ranges are chosen

✗ DON'T: Use overly restrictive ranges
✗ DON'T: Forget to handle edge cases


8.5 COMMENTS
------------
✓ DO: Comment complex logic
✓ DO: Explain business rules
✓ DO: Document data assumptions

Example:
  // Age range based on company policy (18-65)
  COLUMN age VALIDATE range(18, 65);

================================================================================
9. GRAMMAR REFERENCE (BNF)
================================================================================

program          ::= statement+

statement        ::= clean_stmt
                   | fill_null_stmt
                   | remove_dup_stmt
                   | column_stmt
                   | validate_stmt

clean_stmt       ::= "CLEAN" IDENTIFIER ";"

fill_null_stmt   ::= "FILL_NULL" IDENTIFIER "WITH" value ";"

remove_dup_stmt  ::= "REMOVE_DUPLICATES" ";"

column_stmt      ::= "COLUMN" IDENTIFIER operation ";"

operation        ::= "TRIM" | "UPPERCASE" | "LOWERCASE"

validate_stmt    ::= "COLUMN" IDENTIFIER "VALIDATE" 
                     IDENTIFIER "(" parameter_list ")" ";"

parameter_list   ::= value ( "," value )*

value            ::= STRING | NUMBER

IDENTIFIER       ::= [a-zA-Z_][a-zA-Z0-9_]*
STRING           ::= '"' [^"]* '"'
NUMBER           ::= [0-9]+ ( "." [0-9]+ )?

================================================================================
10. QUICK REFERENCE CARD
================================================================================

CLEAN dataset;
FILL_NULL column WITH value;
REMOVE_DUPLICATES;
COLUMN column TRIM;
COLUMN column UPPERCASE;
COLUMN column LOWERCASE;
COLUMN column VALIDATE range(min, max);

// Comment
"string value"
123
45.67

; - Required at end of statement
, - Separates parameters

================================================================================
END OF GUIDE
================================================================================

For more information, examples, or support:
- Check the project README.md
- Review example scripts in the project
- Open the web interface at http://localhost:5173

Version: 1.0.0
Last Updated: October 2025
================================================================================