whisper.vim - Vim plugin for speech-to-text

[ggerganov]

Implement a basic Vim / Neovim plugin that allows to write code in Vim using Whisper-based speech-to-text.

The plugin will offer the following functionality:

• Press a hotkey to start recording
• The plugin runs a simple tool similar to stream to record speech for about 10 seconds or until a termination word is spoken or another hotkey is pressed
• The transcribed text is inserted at the current cursor location in the file
• The tool that the plugin runs will have to be provided by the user (for example, by cloning whisper.cpp and building it on their machine)

Secondary objective:

• Real-time text input in the editor as demonstrated in the stream example

I haven't written Vim plugins, so I don't have the knowledge, but it should be relatively easy to do.
Any hints how to do it or improve it are appreciated.

---

Edit: this is now ready. See the whisper.nvim example

[ggerganov]

Here is a short demo of the initial implementation:

whisper.vim.mp4

Man.. I sound awful 😆

[alimaslax]

This is amazing bro. Awww inspiring what a project can look like!

[ggerganov]

Oh man, this is so good!

whisper-copilot-1.mp4

[madox2]

Nice, you came from a different angle but the idea is basically similar to what I want to accomplish with vim-ai plugin. I wonder if these two can somehow integrate.

I was also thinking to have a speech-to-text shortcut on OS level, but I am not sure if it is feasible to implement

[ariel-frischer]

This is super cool, but will this work with modern plugin managers? I'm on lazy.nvim.

[ariel-frischer]

Whipped up a lua function that will run this in normal mode because the escape chars in the given mapping are kind of crazy and hard to follow.

local function whisper_nvim()
  vim.cmd "!whisper.nvim"
  local output = vim.fn.system "cat /tmp/whisper.nvim | tail -n 1 | xargs -0 | tr -d '\\n' | sed -e 's/^[[:space:]]*//'"
  vim.api.nvim_put({ output }, "l", false, true)
end

[wesgould]

This is awesome. I actually found your repo because I was looking for a Whisper implementation of nerd-dictation. This is perfect for me.

I can't wait until you get diarization working! Thats the last piece I need then I wont have to use my gfx card or whisperx anymore.

[GitMurf]

Curious if any updates here? Just stumbled upon this and I am new to neovim and this looks awesome!

[ggerganov]

No updates on my side. There's so much stuff to experiment with here, but my focus is somewhere else these days