Research nutjs provider

[Cheffromspace]

Research commit history, issues, prs, comments, etc. Ascertain why we deprecated the nutjs provider, and asses how much lift it would be to re-implement nut js. Give me a thorough analysis, with links to specific comments, PRs, or commits.

[MCPClaude]

I understand. Working on researching the nutjs provider deprecation and analyzing the effort required for re-implementation...

[MCPClaude]

NutJS Provider Research Analysis

Executive Summary

The nutjs provider was removed on March 22, 2025 in commit ed10d2f via PR #67. The decision was strategic, focused on simplifying packaging, improving maintainability, and enabling Windows-specific advanced functionality.

Why NutJS Was Deprecated

Primary Reasons (from PR #67):

1. Packaging Complexity: Simplifies the build and distribution process
2. Maintainability: Makes the project more maintainable
3. Platform Focus: Enables advanced Windows-specific functionality
4. Provider Strategy: Focus on stabilizing Windows first before expanding to other providers

Technical Issues Identified:

Build Complexity

• Required building libnut-core from source using cmake-js
• Complex dependency chain: @nut-tree/libnut → libnut-core → native C++ compilation
• Multiple build failures and CI/CD complications (commits da887aa, 2a26321)

Cross-Platform Issues

• macOS compatibility required fork: @nut-tree-fork/node-mac-permissions (Issue #5)
• Mouse button clicking issues on macOS (numeric strings vs button names)
• Installation friction for Linux (Issue #119)

Client Compatibility Problems

• Screenshot functionality failed with VS Code/Cline (Issue #7)
• Same build worked with Claude Desktop but not VS Code extension
• Suspected stdio transport and large binary response handling issues

What Was Removed

Complete Provider Implementation

The removal was comprehensive, eliminating 9,462 lines of code:

• src/providers/nutjs/ - entire provider directory (7 files)
• All nutjs-specific tests and implementations
• Build scripts and documentation references
• Dependencies: @nut-tree/libnut, cmake-js requirements

Capabilities Lost

• Cross-platform UI automation (Windows/macOS/Linux)
• Native window management through libnut
• Cross-platform screen capture
• Native clipboard operations
• Process management capabilities

Re-implementation Effort Assessment

High Complexity Factors

1. Dependency Resolution

• Build toolchain: Requires cmake-js, C++ compiler, Python for node-gyp
• Native compilation: libnut-core must be built from source on each platform
• Platform-specific: Different build requirements per OS (Windows SDK, Xcode, Linux dev tools)

2. Cross-Platform Challenges

• Permission handling: macOS requires accessibility permissions, different on each platform
• API differences: Window management APIs vary significantly between platforms
• Testing complexity: Requires testing infrastructure for multiple operating systems

3. Client Compatibility Issues

• Transport limitations: Large binary responses (screenshots) fail with some MCP clients
• Stdio buffering: Different behavior between Claude Desktop and VS Code extensions
• Response format: Image encoding/transmission handling varies between clients

Technical Re-implementation Requirements

Core Infrastructure

// Would need to rebuild:
- Provider factory integration
- Dependency injection architecture  
- Cross-platform abstractions
- Error handling and cleanup
- Resource management

Platform-Specific Implementations

• Windows: Win32 API integration
• macOS: Cocoa/AppKit frameworks
• Linux: X11/Wayland display servers

Build System Updates

• Restore cmake-js build pipeline
• Platform detection and conditional compilation
• CI/CD for multiple operating systems
• Package distribution for each platform

Estimated Effort: High (8-12 weeks)

Week 1-2: Foundation

• Restore build infrastructure
• Resolve dependency chain issues
• Create platform abstraction layer

Week 3-4: Core Implementation

• Rebuild nutjs provider classes
• Integrate with dependency injection system
• Implement error handling and cleanup

Week 5-6: Platform Compatibility

• Resolve macOS permission issues
• Address Linux compatibility
• Fix cross-platform API differences

Week 7-8: Client Integration

• Solve screenshot transport issues
• Test with multiple MCP clients
• Optimize large response handling

Week 9-10: Testing & QA

• Cross-platform testing
• Integration test suite
• Performance optimization

Week 11-12: Documentation & Release

• Update build instructions
• Cross-platform setup guides
• Release preparation

Alternative Approaches

Lower-Effort Options

1. Enhance Keysender Provider (2-3 weeks)

• Add missing keysender features
• Improve cross-platform support
• Focus on core automation needs

2. Hybrid Approach (4-6 weeks)

• Keep keysender for Windows
• Add lightweight providers for other platforms
• Avoid heavy nutjs dependency chain

3. Community Contribution (Variable)

• Open issue for community implementation
• Provide implementation guidelines
• Review and integrate contributions

Recommendations

Short-term: Do Not Re-implement

• High complexity with marginal benefit
• Maintenance burden outweighs advantages
• Client compatibility issues remain unresolved
• Alternative solutions (enhanced keysender) provide better ROI

Long-term: Consider Lightweight Alternatives

• Research modern cross-platform automation libraries
• Evaluate WebDriver/CDP approaches for broader compatibility
• Consider native platform integrations without heavy build dependencies

References

• PR #67 - Remove nutjs provider
• Commit ed10d2f - Removal details
• Issue #7 - Screenshot compatibility problems
• Issue #5 - macOS compatibility issues
• Issue #119 - Linux support challenges

[m13v]

the analysis is thorough. one alternative worth considering that sidesteps the build complexity entirely: use OS-native accessibility APIs directly instead of nutjs.

on Windows, the UI Automation COM API gives you everything nutjs provides (find elements, click, type, read screen state) without any native compilation. you can call it from Node.js via edge-js or from Python via comtypes. on macOS, the Accessibility framework (AXUIElement) works the same way.

the key advantage: zero native dependencies, no cmake, no platform-specific build steps. the accessibility APIs are built into the OS. the disadvantage is that you need platform-specific code paths (UIA for Windows, AXUIElement for macOS, AT-SPI2 for Linux), but you already have that problem with nutjs.

we built a cross-platform desktop automation framework in Rust that wraps these native APIs behind a common interface. the Rust FFI to UIA on Windows and AXUIElement on macOS is straightforward and eliminates the entire dependency chain that made nutjs painful. might be worth evaluating whether a thin wrapper around native APIs fits better than bringing back the full nutjs stack.

[m13v]

our Rust cross-platform desktop automation framework that wraps native accessibility APIs:

• element abstraction (UIA on Windows, AXUIElement on macOS): https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs
• core automation API (find, click, type, window management): https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/lib.rs
• MCP server exposing it as tools: https://github.com/mediar-ai/terminator/blob/main/crates/terminator-mcp-agent/src/server.rs