Fixes and cleanup script

.github/workflows/update_profiles.yaml

@@ -4,9 +4,14 @@ on:
     schedule:
         -   cron: "0 0 * * *"
     workflow_dispatch:
+    push:
+        branches:
+            - main
 
 jobs:
     update_slicer_profiles:
+        # Only run on main branch, not on PRs
+        if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
         runs-on: ubuntu-latest
         permissions:
             contents: write

README.md

@@ -35,33 +35,52 @@ git clone https://github.com/YOUR_USERNAME/open-filament-database.git
 cd open-filament-database
 ```
 ### 5. Make your changes!
-Either use the web editor by simply running these commands or [following the guide](docs/webui.md), if you want to do it manually you can [use this one](docs/manual.md)
+Use the web editor (recommended) or edit files manually:
+
+**Using the WebUI (Recommended):**
 ```bash
 cd webui
 npm ci
 npm run dev
 ```
-and access it in your browser at http://localhost:5173
+Then access it in your browser at http://localhost:5173
+
+The WebUI includes built-in validation and data sorting features to help ensure your changes are correct. [Full WebUI guide](docs/webui.md)
+
+**Manual editing:** If you prefer to edit files directly, [follow this guide](docs/manual.md)
+
+### 6. Validate and sort your changes
+The WebUI can validate and sort your data automatically:
+
+1. Click the "Validate" button in the top-right corner to check for errors
+2. Click the "Sort Data" button to organize your JSON files consistently
+3. Fix any validation errors that appear (they'll be highlighted in red)
 
-### 6. Validate your changes
-Once you've finished modifying the database you can use these commands or [this guide](docs/validation.md) to make sure your data is correct, fix any errors that pop up
+Alternatively, you can use the command-line validation scripts ([see guide](docs/validation.md)):
 ```bash
-python data_validator.py --folder-names # Validates folder names.
-python data_validator.py --logo-files # Validates logo files.
-python data_validator.py --json-files # Validates json files.
-python data_validator.py --store-ids # Validates store ids.
+python data_validator.py --folder-names  # Validates folder names
+python data_validator.py --logo-files    # Validates logo files
+python data_validator.py --json-files    # Validates JSON files
+python data_validator.py --store-ids     # Validates store IDs
 ```
 ### 7. Submit your changes
-Start by running this command to add all your changes up
+Before submitting, make sure your data is sorted consistently:
+- **In the WebUI:** Click the "Sort Data" button in the top-right corner
+- **Or via command line:** Run `python scripts/sort_data.py`
+
+Then add your changes:
 ```bash
 git add .
 ```
-Then run this command but replace `COMMIT_MESSAGE` with a title of what you did, e.g. "Added filament A to brand B"
+
+Create a commit with a descriptive message (e.g., "Added Elegoo Red PLA variant"):
 ```bash
 git commit -m "COMMIT_MESSAGE"
 ```
-When that's done you can run this command to upload your stuff
+
+Upload your changes to GitHub:
 ```bash
 git push -u origin YOUR_BRANCHNAME
 ```
-Afterwards you can make a pull request [using this guide](docs/pull-requesting.md)
+
+Finally, make a pull request [using this guide](docs/pull-requesting.md)

data_validator.py

@@ -1,6 +1,7 @@
 import json
 import re
 import os
+import sys
 from concurrent.futures import ProcessPoolExecutor, as_completed
 from dataclasses import dataclass, field
 from enum import Enum
@@ -49,6 +50,15 @@ def __str__(self) -> str:
         path_str = f" [{self.path}]" if self.path else ""
         return f"{self.level.value} - {self.category}: {self.message}{path_str}"
 
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for JSON serialization."""
+        return {
+            'level': self.level.value,
+            'category': self.category,
+            'message': self.message,
+            'path': str(self.path) if self.path else None
+        }
+
 
 @dataclass
 class ValidationResult:
@@ -73,6 +83,15 @@ def error_count(self) -> int:
     def warning_count(self) -> int:
         return len([e for e in self.errors if e.level == ValidationLevel.WARNING])
 
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for JSON serialization."""
+        return {
+            'errors': [e.to_dict() for e in self.errors],
+            'error_count': self.error_count,
+            'warning_count': self.warning_count,
+            'is_valid': self.is_valid
+        }
+
 
 # -------------------------
 # Utility Functions
@@ -781,11 +800,24 @@ class ValidationOrchestrator:
 
     def __init__(self, data_dir: Path = Path("./data"),
                  stores_dir: Path = Path("./stores"),
-                 max_workers: Optional[int] = None):
+                 max_workers: Optional[int] = None,
+                 progress_mode: bool = False):
         self.data_dir = data_dir
         self.stores_dir = stores_dir
         self.max_workers = max_workers
         self.schema_cache = SchemaCache()
+        self.progress_mode = progress_mode
+
+    def emit_progress(self, stage: str, percent: int, message: str = '') -> None:
+        """Emit progress event as JSON to stdout for SSE streaming."""
+        if self.progress_mode and hasattr(sys.stdout, 'isatty') and not sys.stdout.isatty():
+            # Only emit when stdout is piped (not terminal)
+            print(json.dumps({
+                'type': 'progress',
+                'stage': stage,
+                'percent': percent,
+                'message': message
+            }), flush=True)
 
     def run_tasks_parallel(self, tasks: List[ValidationTask]) -> ValidationResult:
         """Run validation tasks in parallel using process pool."""
@@ -814,34 +846,42 @@ def run_tasks_parallel(self, tasks: List[ValidationTask]) -> ValidationResult:
 
     def validate_json_files(self) -> ValidationResult:
         """Validate all JSON files against schemas."""
-        print("Collecting JSON validation tasks...")
+        if not self.progress_mode:
+            print("Collecting JSON validation tasks...")
         tasks = collect_json_validation_tasks(self.data_dir, self.stores_dir)
-        print(f"Running {len(tasks)} JSON validation tasks...")
+        if not self.progress_mode:
+            print(f"Running {len(tasks)} JSON validation tasks...")
         return self.run_tasks_parallel(tasks)
 
     def validate_logo_files(self) -> ValidationResult:
         """Validate all logo files."""
-        print("Collecting logo validation tasks...")
+        if not self.progress_mode:
+            print("Collecting logo validation tasks...")
         tasks = collect_logo_validation_tasks(self.data_dir, self.stores_dir)
-        print(f"Running {len(tasks)} logo validation tasks...")
+        if not self.progress_mode:
+            print(f"Running {len(tasks)} logo validation tasks...")
         return self.run_tasks_parallel(tasks)
 
     def validate_folder_names(self) -> ValidationResult:
         """Validate all folder names."""
-        print("Collecting folder name validation tasks...")
+        if not self.progress_mode:
+            print("Collecting folder name validation tasks...")
         tasks = collect_folder_validation_tasks(self.data_dir, self.stores_dir)
-        print(f"Running {len(tasks)} folder name validation tasks...")
+        if not self.progress_mode:
+            print(f"Running {len(tasks)} folder name validation tasks...")
         return self.run_tasks_parallel(tasks)
 
     def validate_store_ids(self) -> ValidationResult:
         """Validate store IDs."""
-        print("Validating store IDs...")
+        if not self.progress_mode:
+            print("Validating store IDs...")
         validator = StoreIdValidator(self.schema_cache)
         return validator.validate_store_ids(self.data_dir, self.stores_dir)
 
     def validate_gtin(self) -> ValidationResult:
         """Validate GTIN/EAN rules."""
-        print("Validating GTIN/EAN...")
+        if not self.progress_mode:
+            print("Validating GTIN/EAN...")
         validator = GTINValidator(self.schema_cache)
         return validator.validate_gtin_ean(self.data_dir)
 
@@ -850,15 +890,32 @@ def validate_all(self) -> ValidationResult:
         result = ValidationResult()
 
         # Check for missing files first
-        print("Checking for missing required files...")
+        self.emit_progress('missing_files', 0, 'Checking for missing required files...')
+        if not self.progress_mode:
+            print("Checking for missing required files...")
         validator = MissingFileValidator(self.schema_cache)
         result.merge(validator.validate_required_files(self.data_dir, self.stores_dir))
+        self.emit_progress('missing_files', 100, 'Missing files check complete')
 
+        self.emit_progress('json_files', 0, 'Validating JSON files...')
         result.merge(self.validate_json_files())
+        self.emit_progress('json_files', 100, 'JSON validation complete')
+
+        self.emit_progress('logo_files', 0, 'Validating logo files...')
         result.merge(self.validate_logo_files())
+        self.emit_progress('logo_files', 100, 'Logo validation complete')
+
+        self.emit_progress('folder_names', 0, 'Validating folder names...')
         result.merge(self.validate_folder_names())
+        self.emit_progress('folder_names', 100, 'Folder name validation complete')
+
+        self.emit_progress('store_ids', 0, 'Validating store IDs...')
         result.merge(self.validate_store_ids())
+        self.emit_progress('store_ids', 100, 'Store ID validation complete')
+
+        self.emit_progress('gtin', 0, 'Validating GTIN/EAN...')
         result.merge(self.validate_gtin())
+        self.emit_progress('gtin', 100, 'GTIN/EAN validation complete')
 
         return result
 
@@ -876,15 +933,18 @@ def main():
     parser.add_argument("--folder-names", action="store_true",
                         help="Validate folder names")
     parser.add_argument("--store-ids", action="store_true", help="Validate store IDs")
+    parser.add_argument("--json", action="store_true", help="Output results as JSON")
+    parser.add_argument("--progress", action="store_true", help="Emit progress events (for SSE streaming)")
 
     args = parser.parse_args()
 
-    orchestrator = ValidationOrchestrator(max_workers=os.cpu_count())
+    orchestrator = ValidationOrchestrator(max_workers=os.cpu_count(), progress_mode=args.progress)
     result = ValidationResult()
 
     # Run requested validations
-    if not any(vars(args).values()):
-        print("No args passed, validating all")
+    if not any([args.json_files, args.logo_files, args.folder_names, args.store_ids]):
+        if not args.json and not args.progress:
+            print("No args passed, validating all")
         result = orchestrator.validate_all()
     else:
         if args.json_files:
@@ -896,28 +956,39 @@ def main():
         if args.store_ids:
             result.merge(orchestrator.validate_store_ids())
 
-    # Print results
-    if result.errors:
-        # Group errors by category
-        errors_by_category: Dict[str, List[ValidationError]] = {}
-        for error in result.errors:
-            if error.category not in errors_by_category:
-                errors_by_category[error.category] = []
-            errors_by_category[error.category].append(error)
-
-        # Print errors grouped by category
-        for category, errors in sorted(errors_by_category.items()):
-            print(f"\n{category} ({len(errors)}):")
-            print("-" * 80)
-            for error in errors:
-                print(f"  {error}")
-
-        print(
-            f"\nValidation failed: {result.error_count} errors, {result.warning_count} warnings")
-        exit(1)
+    # Output results
+    if args.json:
+        # JSON output mode
+        output = result.to_dict()
+        # Use compact output in progress mode for SSE compatibility, pretty output otherwise
+        if args.progress:
+            print(json.dumps(output))
+        else:
+            print(json.dumps(output, indent=2))
+        sys.exit(0 if result.is_valid else 1)
     else:
-        print("All validations passed!")
-        exit(0)
+        # Text output mode
+        if result.errors:
+            # Group errors by category
+            errors_by_category: Dict[str, List[ValidationError]] = {}
+            for error in result.errors:
+                if error.category not in errors_by_category:
+                    errors_by_category[error.category] = []
+                errors_by_category[error.category].append(error)
+
+            # Print errors grouped by category
+            for category, errors in sorted(errors_by_category.items()):
+                print(f"\n{category} ({len(errors)}):")
+                print("-" * 80)
+                for error in errors:
+                    print(f"  {error}")
+
+            print(
+                f"\nValidation failed: {result.error_count} errors, {result.warning_count} warnings")
+            exit(1)
+        else:
+            print("All validations passed!")
+            exit(0)
 
 
 if __name__ == '__main__':

docs/cloning.md

@@ -1,16 +1,19 @@
-# Cloning your repository
-So now we need to clone our repository, that may sound like we're gonna do like in Star Wars and create an army but sadly it's just the process of getting your cloud folder onto your computer, you can imagine this as copy pasting a folder from the cloud.
+# Cloning Your Repository
+Now we need to clone your repository. "Cloning" is the process of downloading your forked copy from GitHub to your computer - think of it as copying a folder from the cloud to your local machine.
 
-To be able to make a PR we also need to make what's called a branch after cloning, we won't be explaining this as it's require a greater understanding of git though if you're studious [here's a link](https://www.w3schools.com/git/git_branch.asp)
+After cloning, we'll create a "branch" for your changes. Branches allow you to work on changes without affecting the main codebase. If you're curious about how branches work, [here's a helpful guide](https://www.w3schools.com/git/git_branch.asp).
 
 ## Cloning on Windows
-To clone your files you simply need to open a commandline window, we'll do this by holding the `windows key` and pressing `R`, this will spawn a small little windowin the bottom left with a text input, simply type in `cmd` and press `enter`.
-Now, don't be scared by the hackerman window that appeared.
-You now have to write or copy in the following command while replacing `YOUR_USERNAME` with you username on github
+To clone your files, you need to open a command line window:
+1. Hold the `Windows key` and press `R`
+2. A small window will appear in the bottom left with a text input
+3. Type `cmd` and press `Enter`
+
+A black terminal window will appear. Now type or paste the following command, replacing `YOUR_USERNAME` with your GitHub username:
 ```bash
 git clone https://github.com/YOUR_USERNAME/open-filament-database.git
 ```
-press enter and let it run, when it allows you to write again you then enter the following two lines, first one will be speedy.
+Press `Enter` and let it run. When it finishes and you can type again, enter the following two commands:
 
 **Remember to replace YOUR_BRANCHNAME with a descriptive name for your changes.**
 Use lowercase with hyphens, for example:
@@ -25,8 +28,8 @@ git checkout -b YOUR_BRANCHNAME
 
 Leave the window open and continue with [Step 5 in the README](../README.md#5-make-your-changes) to make your changes.
 
-## Cloning on Linux and MacOS
-Simply open you terminal and run the following to clone your repository and create a branch for your changes
+## Cloning on Linux and macOS
+Open your terminal and run the following commands to clone your repository and create a branch for your changes:
 ```bash
 git clone https://github.com/YOUR_USERNAME/open-filament-database.git
 cd open-filament-database

docs/forking.md

@@ -1,7 +1,13 @@
-# To fork a repository
-So to fork our repository you'll need to open it in your web browser, you can then find the "Fork" button in the top right of the page, on the other side of the name below the search  
-![Fork button getting pressed](img/forking01.png)  
-You'll get redirected to a menu where you can fork the repository, if the owner menu says "Choose an owner" please click it and select yourself, then click "Create fork"  
-![Create a new Fork menu](img/forking02.png)  
+# Forking the Repository
+To create your own copy of the repository, you need to "fork" it on GitHub:
 
-You should now be looking at the same page again except your name has replaced SimplyPrint (or OpenFilamentCollective) in the top left!
+1. Open the [Open Filament Database repository](https://github.com/OpenFilamentCollective/open-filament-database) in your web browser
+2. Find the "Fork" button in the top right of the page (below the search bar)
+   ![Fork button getting pressed](img/forking01.png)
+
+3. You'll be redirected to a menu where you can create your fork
+4. If the owner menu says "Choose an owner", click it and select your username
+5. Click "Create fork"
+   ![Create a new Fork menu](img/forking02.png)
+
+You should now see the same repository page, but with your username in the top left instead of OpenFilamentCollective. This is your personal copy that you can modify freely!