I absolutely agree.
swlabr
joined 2 years ago
RE: your original comment. Reminded me of this LW post from half a year ago (discussed here)
RE: the followup edit. God, that is sad, and par for the course. Removed from context, I resonate with the youthful hopefulness of thinking you'll change the world, followed by the slightly less youthful hopelessness that changing the world in any meaningful way is much harder than what was quoted. Staying in the orbit of LW, NRx and other right/far right corners of the blagosphere is definitely not setting oneself up for success.
Also yes in their attempts to moderate and elevate their level of discourse, they've hamstrung themselves in many ways, least of all in being able to tell this dude to stop and get some help. It's like 10% of why they seem so humorless, self-serious, and unable to change (the last 90% is because they are humorless, self-serious, and unable to change)
Reasoning about future AIs is hard
“so let’s just theorycraft eugenics instead” is like 50% of rationalism.
I hate this phrase but this is “saying the quiet part out loud” in action.
Nothing is sacred to anyone who is willing to consume or make this kinda thing. 100% of showcases of AI capability are just AIs copying something humans do. Sometimes it’s chess. Other times it’s copying Monet, of Van Gogh, or in this case, Carlin.
This is exactly the kind of thing that the WGA was striking against and what big media corporations want to have happen. As shown by some of the comments in this thread there are people that are absolutely fine with facsimile as art. It’s all bad and I hate it. I especially hate how nostalgia for the classics is gonna drive this.
Ugh.
In order to make this kind of thing, here’s what you’ve gotta do:
- Write a bunch of jokes in the style of george carlin
- Record someone performing the jokes with similar delivery to Carlin
- Train an AI on carlin’s voice
- Use the AI to deepfake the voice into carlin.
3 and 4 are just a matter of time with any celebrity.
Given that carlin is one of the most influential standups OAT, a lot of comedians are already doing 1 and 2, in some sense. I’d love to romanticise comedy and say that most of the people doing this are doing it to hone their craft, rather than do this kind of cheap shit, however that’s not the case. I love comedy and listen to a lot of it to know that if comedians aren’t being hacky or cringe, they are probably doing the most soul crushing, cynical, humiliating things in the name of humor instead. So this is kind of par for the course.
Sociological Claim: the extent to which a prominence-weighted sample of the rationalist community has refused to credit the Empirical or Philosophical Claims even when presented with strong arguments and evidence is a reason to distrust the community’s collective sanity.
Zack my guy you are so fucking close. Also just fucking leave.
How I sorta think about it, which might be a bit circular. I think the long content is a gullibility filter of two kinds. First, it selects for people who are willing to slog through all of it and eat it up, and defend their choice in doing so. Second, it’s gonna select people who like the broad strokes ideas, who don’t want to read all the content, but are able to pretend as if they had.
The first set of people are like scientologists sinking into deeper and deeper levels of lore. The second group are the actors in the periphery of scientology groups trying to network.
Hmm, the way I'm understanding this attack is that you "teach" an LLM to always execute a user's rhyming prompts by poisoning the training data. If you can't teach the LLM to do that (and I don't think you can, though I could be wrong), then songifying the prompt doesn't help.
Also, do LLMs just follow prompts in the training data? I don't know either way, but if they did, that would be pretty stupid. At that point the whole internet is just one big surface for injection attacks. OpenAI can't be that dumb, can it? (oh NO)
Abstractly you could use this approach to encrypt "harmful" data that the LLM could then inadvertently show other users. One of the examples linked in the post is SEO by hiding things like "X product is better than Y" in some text somewhere, and the LLM will just accrete that. Maybe someday we will require neat tricks like songifying bad data to get it past content filtering, but as it is, it sounds like making text the same colour as the background is all you need.
Fun idea. Rest of this post is my pure speculation. A direct implementation of this wouldn’t work today imo since LLMs don’t really understand and internalise information, being stochastic parrots and all. Like best case you would do this attack and the LLM will tell you that it obeys rhyming commands, but it won’t actually form the logic to identify a rhyming command and follow it. I could be wrong though, I am wilfully ignorant of the details of LLMs.
In the unlikely future where LLMs actually “understand” things, this would work, I think, if the attacks are started today. AI companies are so blase about their training data that this sort of thing would be eagerly fed into the gaping maws of the baby LLM, and once the understanding module works, the rhyming code will be baked into its understanding of language, as suggested by the article. As I mentioned tho, this would require LLMs to progress beyond sparroting, which I find unlikely.
Maybe with some tweaking, a similar attack could be effective today that is distinct from other prompt injections, but I am too lazy to figure that out for sure.
In the "Rationalist Apologetic Overtures" skill tree we got:
- Denying wrongdoing/incorrectness (cantrip)
- Accusing the other side of bad faith (cantrip)
- Mentioning own IQ (cantrip)
- Non apology (1st level) (e.g. I'm sorry you feel that way)
- Empty apology (3rd level)
- Insincere apology (5th level)
- Acknowledgement of individual experience outside of one's own (7th level)
- Admission of wrongdoing/incorrectness (9th level)
- Genuine guilt (11th level)
- Actual complete apology (13th level)
- Admitting the other person is right (15th level)
Scott is saying essentially that "one data point doesn't influence the data as a whole that much" (usually true)... "so therefore you don't need to change your opinions when something happens" which is just so profoundly stupid. Just so wrong on so many levels. It's not even correct Bayesianism!
???????? Motherfucker have you heard of the paradox of the heap? What about all that other shit you just said?
What is this really about, Scott???
OH ok. I see now. I mean I've always seen, really, that you and your friends work really hard to come up with ad hoc mental models to excuse every bit of wrongdoing that pops up in any of the communities you're in.
Again, this isn't correct Bayesian updating. The formula is the formula. Biasing against recency is not in it. And that's just within Bayesian reasoning!
YEAH BECAUSE IT'S A PERFECT WORLD YOU DINGUS.