[Notes] OpenAi Whisper for Video Subtitles

Links/notes for my own use to find again.

• https://drive.google.com/drive/my-drive New > More > Google Colaboratory
  • or https://colab.research.google.com/#create=true
• Rename Untitled.ipynb to TranscribeAudio.ipynb
• Runtime > Change runtime type > GPU
• Install Whisper AI:

!pip install git+https://github.com/openai/whisper.git
!sudo apt update && sudo apt install ffmpeg

• Upload audio (drag n drop)
• Run Whisper AI: in Medium or Large. (testing the file, medium took about 30 seconds for my 1 hr podcast, and large took 13minutes and 9 seconds, scroll down to view the accuracy differences to see whether it’s worth the extra time).

!whisper "FileName.mp3" --model medium

• Download text or subtitles (files will appear to the left when it’s finished transcribing)
• Additional Whisper AI arguments to learn about: (useful for people who want to transcribe the video into different languages!!)

!whisper -help

My everyday “go-to” code command that I will use to create English text files:

!whisper "Marvin.mp3" --model medium.en --output_format txt --task transcribe --beam_size 5 --patience 13 --threads 4 --length_penalty 0

Testing Accuracy using different settings & models

Two speakers with accents, 60 minute mp3

Results of test with two speakers (with accents) from this 1-hr podcast episode (Marvin vs Virology: COVID Taken To Court). Female NZ speaker (Dr. Sam Bailey) and male German speaker (Marvin Haberland).

Medium Setting (took 7 minutes to transcribe) example:

His case should require the virologists to provide evidence that they followed the scientific method when they claimed that SARS-CoV-2 exists. … so you studied engineering in Germany and you got a Fulbright scholarship. Is that right to bootclears? Yeah, exactly. So I did my undergraduate studies in Hamburg, Germany, …

Large Setting (took 13 minutes to transcribe) example:

His case should require the virologists to provide evidence that they followed the scientific method when they claimed that SARS-CoV-2 exists. … so you studied engineering in Germany and you got a Fulbright scholarship. Is that right? To Berkeley? Yeah, exactly. So I did my undergraduate studies in Hamburg, Germany, …

Considering both speakers have an accent, it picked up their conversation extremely well, with only tiny mistakes, so I think the medium setting will be fine for most English-speaking audio (even if the speakers have accents), and just a “read through” to pickup on small mistakes will be the best time-saving setting to stick to.

I just did another test using the fastest setting, and specific parameters, and it completed the 60min file in 2mins 21 secs, and it seems that specifying “–length_penalty 0” did the whole 60min podcast (whereas it seemed to have only outputted only the first 29mins in the previous tests)

!whisper "Marvin.mp3" --model tiny --output_format txt --task transcribe --language en --length_penalty 0

Quickly scrolling through, it seems the only errors made, even on the quickest setting, was Dority Institute instead of Doherty Institute. It spelt his name Harbourland instead of Haberland. It made the same error “boot clears” instead of Berkeley.

Testing the fastest model “Tiny.en”:

!whisper "Marvin.mp3" --model tiny.en --output_format txt --task transcribe --length_penalty 0

This took 2m 25s to transcribe the one hour audio file

Errors:

• …since Corona started the Corona Fucking German telegram channel also published
  • (woah, oops! lol)
• …since Corona started, the Corona Fakten, a German telegram channel, also published

Error comparison using different models:

• Tiny:
  • and the Dority Institute because this isn’t interesting Australian paper
• Tiny.en with above code:
  • the dirty institute because this is an interesting Australian paper (lol.. that’s a good name for them!)
• Medium:
  • Dougherty Institute because this is an interesting Australian paper
• Large:
  • and the Doherty Institute, because this is an interesting Australian paper.

Using the “Tiny” setting will create errors that might not be worth the extra time saved.

Using small video clip to test accuracy differences in models

example using a tiny young Beyoncé video…

“I know we’re gonna stay humble. If you ever meet me, I have a little attitude, just slap me. Just slap me right back into shape.“

1. (Tiny): I know we’re gonna stay home if you ever meet me. I have no abs to just slap me just slap me right back in the shape.
2. (Tiny.en): I know we’re gonna stay humble if you ever meet me. I have a little attitude. Just slap me. Just slap me right back into shape.
3. (Small.en): I know we’re gonna stay humble if you ever meet me. I have a little attitude. Just slap me. Just slap me right back into shape.
4. (Small): I know we’re gonna stay humble if you ever meet me. I have a little attitude. Just slap me Just slap me right back into shape.
5. (Medium.en): I know we’re gonna stay humble if you ever meet me. I have a little attitude. Just slap me. Just slap me right back into shape.
6. (Large): I know we’re gonna stay humble. If you ever meet me, I have a little attitude, just slap me. Just slap me right back in the shade.
7. (Large-v2): I know we’re gonna stay humble. If you ever meet me, I have a little attitude, just slap me. Just slap me right back in the shade.

• Python: Long but detailed walk-through of entire setup (YouTube)
• Alternative walk-through that looks easier to follow that I wish I’d found first (YouTube) *video above

(My notes below are for the first link, but I recommend following the video above instead, it’s way clearer and you’ll be up and going much faster)

• Using: https://www.python.org/downloads/release/python-399/
  • for my pc: Windows installer (64-bit)
  • And using: https://github.com/openai/whisper
    • pip install -U openai-whisper
      • (paste into a command prompt)
  • And using: https://ffmpeg.org/download.html
  • Created new folder c:\Vid-Transcribe and placed a video in the folder test.mp4
    1. In command prompt, cd c:\Vid-Transcribe
    2. whisper “c:\Vid-Transcribe\test.mp4” –model small.en –output_format txt –task transcribe –beam_size 5 –patience 13 –threads 4 –length_penalty 0
       • here it will automatically download the small.en model
  • To use GPU: you have to uninstall torch and install the latest torch
    1. pip3 uninstall torch
    2. get code from https://pytorch.org/
    3. For my PC, code is:
       • pip3 install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cu117
       • (Wow, it’s 3GB!!)
    4. Once you have the above, can use this code to use GPU:
       • whisper “c:\Vid-Transcribe\test.mp4” –model small.en –device cuda

Additional Notes I learnt from 2nd link:

1. Navigate to folder that has your audio/video files
   • Type in cmd in that folder, which will open up command prompt using that directory
   • whisper “file name.mp4”
2. To translate a file from german to english
   • whisper “file name.mp4” –language German –task translate

Notes while troubleshooting:

• Memory errors.
  1. Break up files to be <25mb if getting memory errors
  2. Try –patience 2
  3. Change mode:
     • small model needs 2GB Ram, base or tiny (1GB)
       • Settings > Display > Advanced > Display adapter properties
       • (shows you how much dedicated video memory you have) (Video)
  4. Convert file into mp3 rather than video
     • ffmpeg -i “testvideo.mp4” -q:a 0 -map a testvideo.mp3
       • this also works:
         • ffmpeg -i example.mp4 example.mp3
  5. Set NVIDIA as default video driver (Video)
     1. Check if you have an NVIDIA Card (Device Manager > Display Adapters > NVIDIA)
     2. If yes, close out of that, right-click desktop > NVIDIA control panel
        • If NVIDIA control panel is not listed via desktop (Video)
          • Control Panel > Search NVIDIA
        • To fix Error: NVIDIA Display settings are not available. You are not currently using a display attached to an NVIDIA GPU
          1. Device Manager > Display Adapters > NVIDIA GeForce > Properties
          2. Driver tab > Disable device > Yes to warning > Ok out of it
          3. Device Manager > Display Adapters > NVIDIA GeForce > Properties
          4. Driver tab > Enable device > Restart PC
     3. NIVIDIA control panel > Manage 3D settings > Global Settings > Preferred Graphics Processer > NVIDIA > Apply
     4. NIVIDIA control panel > PhysX > Select PhysX processor > GeForce GTX > Apply