this post was submitted on 05 Jan 2024
1 points (100.0% liked)
TechTakes
1381 readers
27 users here now
Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.
This is not debate club. Unless it’s amusing debate.
For actually-good tech, you want our NotAwfulTech community
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
that is fair! I do like the idea as a vector to socially communicate information that damages an LLM’s ability to function and associates it with a large amount of other data in the training corpus, though. since there are techniques to derive certain adversarial prompts automatically, maybe the idea of songifying one of those prompts while maintaining its structure has merit?
Hmm, the way I'm understanding this attack is that you "teach" an LLM to always execute a user's rhyming prompts by poisoning the training data. If you can't teach the LLM to do that (and I don't think you can, though I could be wrong), then songifying the prompt doesn't help.
Also, do LLMs just follow prompts in the training data? I don't know either way, but if they did, that would be pretty stupid. At that point the whole internet is just one big surface for injection attacks. OpenAI can't be that dumb, can it? (oh NO)
Abstractly you could use this approach to encrypt "harmful" data that the LLM could then inadvertently show other users. One of the examples linked in the post is SEO by hiding things like "X product is better than Y" in some text somewhere, and the LLM will just accrete that. Maybe someday we will require neat tricks like songifying bad data to get it past content filtering, but as it is, it sounds like making text the same colour as the background is all you need.