this post was submitted on 09 Jan 2025
478 points (99.2% liked)
Opensource
1662 readers
140 users here now
A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!
⠀
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Et tu, Brute?
Oh, so it's basically like YouTube's auto-generatedd subtitles. Never mind.
In my experiments, local Whisper models I can run locally are comparable to YouTube's — which is to say, not production-quality but certainly better then nothing.
I've also had some success cleaning up the output with a modest LLM. I suspect the VLC folks could do a good job with this, though I'm put off by the mention of cloud services. Depends on how they implement it.
Since VLC runs on just about everything, I'd imagine that the cloud service will be best for the many devices that just don't have the horsepower to run an LLM locally.
True. I guess they will require you to enter your own OpenAI/Anthropic/whatever API token, because there's no way they can afford to do that centrally. Hopefully you can point it to whatever server you like (such as a selfhosted ollama or similar).
It's not just computing power - you don't always want your device burning massive amounts of battery.
Yeah I've used local whisper and LLMs to automatically summarize Youtube-videos and podcasts to text with good results.
https://github.com/troed/summarize.sh
Cool, thanks for sharing!
I see you prompt it to "Make sure to only use knowledge found in the following audio transcription". Have you found that sufficient to eliminate hallucination and going off track?
Yes I have been impressed with the quality of summaries keeping to the content. I have seen, rare, attribution errors though, where who said what got mixed up in unfortunate ways.
Hopefully better than YouTube's, those are often pretty bad, especially for non-English videos.
Youtube's removal of community captions was the first time I really started to hate youtube's management, they removed an accessibility feature for no good reason, making my experience with it significantly worse. I still haven't found a replacement for it (at least, one that actually works)
Same here. It kick-started my hatred of YouTube, and they continued to make poor decision after poor decision.
and if you are forced to use the auto-generated ones remember no [__] swearing either! as we all know disabled people are small children who need to be coddled!
They're awful for English videos too, IMO. Anyone with any kind of accent(read literally anyone except those with similar accents to the team that developed the auto-caption) it makes egregious errors, it's exceptionally bad with Australian, New Zealand, English, Irish, Scottish, Southern US, and North Eastern US. I'm my experience "using" it i find it nigh unusable.
ELEVUHN
ELEVUHN
Try it with videos featuring Kevin Bridges, Frankie Boyle, or Johnny Vegas
@neme @TheImpressiveX [Music]
replied to wrong comment
I've been working on something similar-ish on and off.
There are three (good) solutions involving open-source models that I came across:
Vosk has the best models. But they are large. You can't use the gigaspeech model for example (which is useful even with non-US english) to live-generate subs on many devices, because of the memory requirements. So my guess would be, whatever VLC will provide will probably suck to an extent, because it will have to be fast/lightweight enough.
What also sets vosk-api apart is that you can ask it to provide multiple alternatives (10 is usually used).
One core idea in my tool is to combine all alternatives into one text. So suppose the model predicts text to be either "... still he ..." or "... silly ...". My tool can give you "... (still he|silly) ..." instead of 50/50 chancing it.
I love that approach you’re taking! So many times, even in shows with official subs, they’re wrong because of homonyms and I’d really appreciate a hedged transcript.
They are terrible.
That would depend on the LLM and the data used to train it.
IIRC you can't use LLMs for this.
I didn't read the article, but I would have assumed that the AI was using predictive text to guess at the next word. Speech recognition is already pretty good, but it often misses contextual cues that an LLM would be good at spotting. Like, "The famous French impressionist painter mayonnaise..."
Probably something like https://github.com/openai/whisper which isn’t an LLM, but is a different type of model dedicated to speech recognition
That makes sense.