MahmoudAshraf97 /whisper-diarization

Automatic Speech Recognition with Speaker Diarization based on OpenAI Whisper

★ 5,676⑂ 503👁 50⑃ 52● 272▤ 0
Jupyter NotebookBSD-2-ClauseCreated 2023-01-25
+1
Daily
5,676
stars
503
forks
50
watchers
52
prs
272
issues
0
releases

README

Speaker Diarization Using OpenAI Whisper




  
    
  
  
    
  
  
        
  
  
        
  
  
    
   
  
  
    
  


  
    
  



 
Speaker Diarization pipeline based on OpenAI Whisper

 Please, star the project on github (see top-right corner) if you appreciate my contribution to the community!

 What is it
This repository combines Whisper ASR capabilities with Voice Activity Detection (VAD) and Speaker Embedding to identify the speaker for each sentence in the transcription generated by Whisper. First, the vocals are extracted from the audio to increase the speaker embedding accuracy, then the transcription is generated using Whisper, then the timestamps are corrected and aligned using ctc-forced-aligner to help minimize diarization error due to time shift. The audio is then passed into MarbleNet for VAD and segmentation to exclude silences, TitaNet is then used to extract speaker embeddings to identify the speaker for each segment, the result is then associated with the timestamps generated by ctc-forced-aligner to detect the speaker for each word based on timestamps and then realigned using punctuation models to compensate for minor time shifts.


Whisper and NeMo parameters are coded into diarize.py and helpers.py, I will add the CLI arguments to change them later
 Installation
Python = 3.10 is needed, 3.9 will work but you'll need to manually install the requirements one by one.

FFMPEG and Cython are needed as prerequisites to install the requirements
shell
pip install cython

or
shell
sudo apt update && sudo apt install cython3

shell
 on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg

 on Arch Linux
sudo pacman -S ffmpeg

 on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg

 on Windows using Chocolatey (https://chocolatey.org/)
choco install ffmpeg

 on Windows using Scoop (https://scoop.sh/)
scoop install ffmpeg

 on Windows using WinGet (https://github.com/microsoft/winget-cli)
winget install ffmpeg

shell
pip install -c constraints.txt -r requirements.txt

 Usage 

shell
python diarize.py -a AUDIO_FILE_NAME


If your system has enough VRAM (=10GB), you can use diarize_parallel.py instead, the difference is that it runs NeMo in parallel with Whisper, this can be beneficial in some cases and the result is the same since the two models are nondependent on each other. This is still experimental, so expect errors and sharp edges. Your feedback is welcome.

 Command Line Options

- -a AUDIO_FILE_NAME: The name of the audio file to be processed
- --no-stem: Disables source separation
- --whisper-model: The model to be used for ASR, default is medium.en
- --suppress_numerals: Transcribes numbers in their pronounced letters instead of digits, improves alignment accuracy
- --device: Choose which device to use, defaults to "cuda" if available
- --language: Manually select language, useful if language detection failed
- --batch-size: Batch size for batched inference, reduce if you run out of memory, set to 0 for 

Top issues

Discussions

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON