Porting to Mac and improve performance

[oldsongsz]

This is a really good project. I'm learning from it.

I just did some dirty hard-code and make it work on my Macbook Air (Intel model). It runs pretty slow.

Could you share any solution to optimize? (Yes, I"m asking Google and ChatGPT at the same time)

Thanks!

p.s. I'm trying to on an ARM Linux SBC as well, hope it works.

[SevaSk]

Thanks! If the transcription is slow, most likely culprit would be if this logging message:
print(f"[INFO] Whisper using GPU: " + str(torch.cuda.is_available()))

is false. Then Whisper cannot transcript with the GPU and will use the CPU instead, which is way slower.

Outside of using whisper, none of the code should run noticeably slow on any OS or computer.

[oldsongsz]

Agree.

torch.cuda.is_available() is only true with Nvida CUDA system, any AMD, Intel or other embedded GPU would not by default support it.

Would you consider integrate Whisper online API to not rely on local resource? Consider LangChain?

[SevaSk]

It would be nice to have an option to use a API for whisper transcriptions. Should be fairly straight forwards since we can use this
https://platform.openai.com/docs/guides/speech-to-text/speech-to-text-beta. I will create a specific issue for this.

[Klaudioz]

I've been using whisper with a M2 with good results using this: openai/whisper#382

[SevaSk]

You can check out this branch https://github.com/SevaSk/ecoute/tree/29-add-option-to-use-speech-to-text-api-rather-than-transcribing-locally

use the command

python main.py --api

it honestly way faster and better then the local model

edit:
merged that branch to main now.

[oldsongsz]

Thanks for sharing and the work, on my way to try.