Fork overview and contribution inquiry (sjjsy/asr2clip)

[sjjsy]

Hi,

I'm Samuel Sydänlammi (GitHub: sjjsy), a developer in Finland. I've been using asr2clip as the foundation for a personal voice pipeline on Linux and have built a significant fork over the past few months: https://github.com/sjjsy/asr2clip -- actually I had started on my own script and wrapper a long time ago, but when I realized it had started to grow towards a full-fledged voice pipeline tool I realized I should make it a real project and share it, and before doing that I did another round of research of existing tools and determined that instead of publishing a competitor for your tool, I could contribute to it.

---

What the fork adds on top of your v0.4.0 — A brief overview

Feature	Notes
whisper.cpp backend (-b wcpp)	Fully offline; wraps whisper-cli subprocess
Toggle mode (--toggle)	Lock-file start/stop; keybinding-friendly; my daily driver
Robust chunked transcription (-r)	Silence-split + quality-check + retry for long files
Audio preprocessors (-p deepfilter/pyrnnoise/noisereduce)	Noise reduction before ASR
AI assistant style post-processing (-P NAME)	User-defined prompts in config; OpenAI-compat or claude -p
Speaker diarization (-D)	WhisperX + pyannote; [HH:MM:SS] Speaker: text output

All additions were intended to be roughly independent. The v1.2.0 release includes a unit and integration test suite and has been running on my Ubuntu 20.04 LTS for a bit. I plan to upgrade to 26.04 LTS this month so then I'd expand support. Python 3.8 is legacy by now and I might propose removing support for it sometime after my upgrade (I notice you had removed support recently already).

Note that my personal testing has focused on whisper.cpp (and Groq); other backends and integrations are included but I haven't tested them properly, nor have I tested diarization yet. I plan to do more testing and likely more QA and code quality improvement (and bugfixing) within the next two months with my team and friends, however. The feature-set and the big picture, however, is IMHO already quite complete, which is why I write to you already now.

---

The story and related reflections

My use case is a bit different from the typical user: I use both English and Finnish as my working language, and Finnish ASR needs Whisper large-v3 to get good quality — the smaller or cloud-fine-tuned models don't cover Finnish well. Also, since I like to be "sovereign" I bumped into and installed the whisper.cpp backend which provides local offline transcription. From there I added the rest: toggle mode for hotkey dictation on a tiling WM, robust chunked transcription for long meeting recordings (whisper.cpp with some models got stuck repeating a phrase on my limited hardware so I wanted some QA/supervision logic), noise reduction before ASR, and eventually I ended up going for the "last-mile" by adding speaker diarization and meeting assistant style AI post-processing for solo and team meeting transcript enhancement and summaries.

I noticed your existing cloud backends (SiliconFlow, xinference) reflect a different ecosystem focus and SenseVoice (via sherpa-onnx) provide excellent performance for speakers of certain Asian languages; my additions lean the other direction toward offline-first transcription of European languages, especially Finnish. I hope they're complementary and the end result may become (or is already) a superb tool for Linux users around the world.

The Linux and CLI interface focus of course limits the audience but perhaps normal people might have AI agents that find it useful! :D Also, maybe cross-platform support could be developed further. I use only Linux atm. so have limited motivation for that work. If you use MacOS it would be great if you'd ensure my contributions serve Mac users well. Maybe if some Windows user finds this interesting, he/she can contribute to expanding our platform support.

A practical note: a significant part of the actual coding was done with Claude Code (Anthropic's CLI coding assistant). While I also used CC (and other AI tools) heavily to research the landscape and figure out the best approaches, the decisions on design, testing, and integrations were mostly mine. I want to be transparent about that upfront and welcome your review and feedback.

---

First: I have seriously considered whether to even ask to merge my contributions back upstream to your project or whether to diverge and even rename my fork. There's several breaking changes (config structure, flags etc). Moreover, the tool now covers a pipeline well beyond what one might expect from the minimalist "ASR to clipboard" name. Though practically everything still honors the original idea and use case. Even the AI meeting memos end up into the clipboard for easy copy-pasting.

If you would accept PRs from me, I'm glad to stay under the asr2clip name and contribute there. However, if you'd rather keep asr2clip lean and that I maintain my fork as an independent project going forward, I would rename it — crediting asr2clip clearly as the foundation in the README and license headers. Either path works for me!

Toggle mode and robust transcription are the most self-contained and the most likely to be useful to others on other platforms. whisper.cpp and the preprocessors are also complete. I added a battery of unit tests and also some E2E testing later on. And I've revamped the documentation a great deal. But due to the breaking changes, you and all users should note in advance that amendments to existing configs and scripts would be required and while my fork brings more capabilities to the tool and its configs, they also get more bloat and complexity, which might be undesirable from the perspective of someone just wanting a dead simple asr2clip tool with a single backend.

The readthedocs site (https://asr2clip.readthedocs.io/) appears to be built from your repo's source. If you accept my contributions to the tool, I can also submit a PR to update the docs. Otherwise, I'll possibly setup my own readthedocs instance.

As you can see, I did not update README_zh.md. I leave it to you or someone fluent in the language to do it. :)

No rush with responding! I'm happy to wait until I've done some more testing, refactoring and refinement hopefully with some feedback from my peers before you'd even look at my additions. If/when I feel confident of its utility, I might message you again and/or advertise more broadly as well. :)

---

Thanks for building asr2clip — the architecture and code quality made it genuinely enjoyable to extend.

Samuel
https://github.com/sjjsy/asr2clip

[Oaklight]

Hi Samuel,

Really appreciate you writing all this up — and even more so that you chose to contribute back rather than just fork and go. That means a lot.

I've been looking at your fork in detail. The feature set is impressive, and the modular architecture (backends, preprocessors, postprocessors, recorders) shows solid engineering thinking.

Integration approach

Your fork covers a lot of ground, so I think the most productive path is to start with a clean integration point rather than trying to merge everything at once. I've put up a PR that introduces an engine abstraction layer: #17

The key idea: everything is an "engine" — remote API calls, local sherpa-onnx inference, and (your contribution point) whisper.cpp. They all implement the same BaseEngine interface: bytes in, text out.

BaseEngine (bytes → text)
├── OpenAICompatEngine   — any OpenAI-compatible API
├── SherpaOnnxEngine     — local sherpa-onnx (SenseVoice, Whisper, Paraformer, etc.)
└── WhisperCppEngine     — your contribution goes here

Note that sherpa-onnx already runs Whisper models, so whisper.cpp is really an alternative runtime at the same level — not a separate backend category.

I'd love your review on the PR before I merge it. Specifically:

• Does the BaseEngine interface work for your whisper.cpp integration?
• Anything missing from TranscriptionResult that you'd need?
• Does the create_engine() factory pattern make sense for config-driven engine selection?

Contribution path after this lands

1. engines/whisper_cpp.py implementing BaseEngine — this is the most natural first PR from you
2. Other features (toggle mode, chunked transcription, preprocessors, etc.) can be discussed as follow-up PRs once the engine layer is settled

Housekeeping

I also added a CONTRIBUTING.md in the same PR, which covers the contribution workflow and a few ground rules — including AI-assisted contributions policy. Worth a read before submitting PRs.

One specific ask: your fork has Claude Code / Cursor as co-author in many commit messages. Per the contributing guidelines, I'd prefer those co-author tags removed from git history before submitting PRs upstream. A git rebase with amended messages would do it.

No rush on any of this — take your time reviewing and testing. I'm genuinely excited about collaborating on this and would love to have you as a long-term contributor.

Peng

[sjjsy]

Hi Peng,

Thanks for the excellent response and thoughtful architectural work on PR #17. Indeed that makes the design more extensible.

As you might have noticed, I submitted a couple dozen commits on my fork even after submitting my above post.

During that time I also reflected a bit on our situation and possible next steps.

As you remarked: sherpa-onnx already supports Whisper models and your recent PR pushes the lean asr2clip idea of bytes in, text out, I feel all my additions create perhaps a tad too much weight on your elegant tool. Since I started from a simple bash wrapper of the whisper.cpp binaries, in my implementation audio is still recorded first to a file before invoking the Whisper.cpp binary with CLI args. It is not as elegant as the API based streaming solutions that also enable VAD.

However, I understand that whisper.cpp offers the most fine tuned models from the Whisper family and may thus be the better choice if one is not that interested in the rest of sherpa-onnx's capabilities.

In a way I use the otherwise efficient whisper.cpp but add a bit of friction and bells and whistles around the ASR engine while you have a fully featured ASR server with a neat straight pipeline around it. There's some mismatch there. :)

To your specific questions:

• Your BaseEngine is more optimized for in-memory byte streams, so integration of file I/O and CLI invokations would feel a bit awkward, even though you did add some related function parameters
• Diarization might complicate your TranscriptionResult flow (the "bytes out" would be more than just the speech) -- though not part of the Whisper.cpp portion you specifically proposed to start with; and the recipient can be expected to expect that the bytes received include diarization results along with the speech-to-text results.
• The create_engine() pattern is a good direction and could perhaps be expanded to accept more diversity, if desired.

Inddeed, the work I did since posting here was mostly on testing and refactoring, and a big part of that was related to redoing the "creation/config" system. My config file grew and the diversity brought up challenges that required a more sophisticated approach to keep the codebase readable.

My rather intense sprint in adding features needed a sprint of consolidation. And now I feel it should be followed with a longer sprint of just using the tool -- as I mentioned already in my original post above. And this is why maybe I was a tad too eager to give you a heads up about my fork. :)

However, when I have time for my next development sprint, I can reconsider making the PR as you proposed, honoring your engine design and commit related and other contribution policies. Or maybe if the feature set in my fork inspires you to add something to yours, I'm sure I'd learn a lot from seeing if and how you choose to do it. :)

Indeed my fork is foremost about inspiration, enthusiasm and learning -- from and for the Open Source community!

Thanks again for the welcome and for building a great foundation!

Best,
Samuel

[Oaklight]

Circled back to look at the input interface after your comments — specifically the point about BaseEngine being bytes-oriented and file I/O feeling awkward.

I surveyed 10 ASR projects to see how they handle input:

Project	Input types	Notes
OpenAI Whisper	str (path), np.ndarray, torch.Tensor	Most flexible
faster-whisper	str, np.ndarray, BinaryIO
whisper.cpp (python)	str (path only)	CLI-oriented
HF transformers (ASR)	str, np.ndarray, dict, list
SpeechBrain	str, torch.Tensor
NeMo	list[str] (paths)	Batch-oriented
Vosk	streaming bytes chunks
Google Cloud Speech	bytes, GCS URI
Azure Speech SDK	str (path), audio stream
AssemblyAI	URL	REST only

Local engines almost universally accept file paths + numpy. Cloud SDKs lean toward bytes or URLs. CLI wrappers like whisper.cpp work with file paths exclusively. No single format fits all backends.

Based on this, PR #17 now has an AudioInput container that holds audio in its original representation and converts lazily:

audio = AudioInput.from_file("recording.wav")   # holds a path
audio = AudioInput.from_numpy(samples, sr=16000) # holds an array
audio = AudioInput.from_bytes(data)              # holds bytes

# Engines pull what they need — conversions are cached
path = audio.as_file()       # free if from_file, writes temp otherwise
samples, sr = audio.as_numpy()  # free if from_numpy, decodes otherwise

So a whisper.cpp engine would just call audio.as_file() and get a clean path without the bytes round-trip. A recording that's already numpy skips WAV encoding when the engine can consume it directly.

The engine interface stays the same — transcribe(audio: AudioInput, language=None). New input formats can be added to AudioInput later without touching engine implementations.

Curious whether this would work for your whisper.cpp integration, or if there are patterns I'm not seeing.

[sjjsy]

Hi @Oaklight,

Good analysis and steps ahead!

I’ve just opened a draft PR (#19) to fulfill what you outlined above.

This initial version provides a simple whisper.cpp CLI wrapper for offline transcription. It uses the .as_file() function but then avoids writing a file for the transcription result since it can be read straight from stdout.

I’ve also included a roadmap in the PR description regarding a future transition to a Server + Async Queue model that would bring whisper.cpp's power to asr2clip in a way that would be more inline with asr2clip's current vision that emphasizes real-time transcription and VAD and avoid filesystem latency. This would utilize a persistent whisper.cpp server and threaded VAD to enable "gapless" recording and eliminate model-loading latency making even larger whisper.cpp GGLM models that may be required for low-resource languages to serve effectively. Indeed, then asr2clip would provide a more complete alternative to sherpa-onnx for those that may be interested.

Looking forward to your early feedback on the draft implementation and the architectural ideas!

However, I will not be implementing the server setup very soon due to time-constraints.

Note also that I decided to detach my original fork to a new identity kaiku -- Finnish for "echo" -- due to the massive codebase divergence and also difference in purpose. Indeed, I'd propose we proceed with the "R&D and Mainline" collaboration model. I can freely experiment with kaiku, letting others also try and test, even extend it, while also allowing the easy use of asr2clip as an adjacent tool.

If kaiku would happen to demonstrate great ideas or vice-versa, those could be added across separately. However, since kaiku already covers my core needs as of now, I do not have a clear personal interest in developing either variant further except maybe up to the outlined VAD with asynchronous whisper.cpp server stage. Maybe that could replace my need for the simple toggle feature and then asr2clip might overtake kaiku as my daily driver, with kaiku potentially ending up relegated to stay an R&D playground project indefinitely.

PS. I realized only now that you had added an issue already a year ago for the "toggle feature" I had implemented this month into kaiku: "Ctrl+B shortcut to transcribe since last press" #1 Moreover, your "feat: daemon mode — background service with HTTP control API" #15 is something that could facilitate the implementation of the real-time VAD async whisper.cpp server setup. However, both #15 and this whisper.cpp service would take quite a lot of time away from my primary responsibilities, of which I hope I had less :)