245
submitted 4 days ago by Chozo@fedia.io to c/til@lemmy.world

Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to incentivize said advancement.It originated in a 2010 post at discussion board LessWrong, a technical forum focused on analytical rational enquiry. The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.

While the theory was initially dismissed as nothing but conjecture or speculation by many LessWrong users, LessWrong co-founder Eliezer Yudkowsky reported users who panicked upon reading the theory, due to its stipulation that knowing about the theory and its basilisk made one vulnerable to the basilisk itself. This led to discussion of the basilisk on the site being banned for five years. However, these reports were later dismissed as being exaggerations or inconsequential, and the theory itself was dismissed as nonsense, including by Yudkowsky himself. Even after the post's discreditation, it is still used as an example of principles such as Bayesian probability and implicit religion. It is also regarded as a simplified, derivative version of Pascal's wager.

Found out about this after stumbling upon this Kyle Hill video on the subject. It reminds me a little bit of "The Game".

top 50 comments
sorted by: hot top controversial new old
[-] ShitOnABrick@lemmy.world 1 points 8 hours ago

I've heard of this myself it's just a bunch of bollocks made by some neckbeard

[-] samus12345@lemmy.world 16 points 2 days ago

Pascal's Wager always seemed really flawed to me even through a purely Christian perspective. You're saying that god is so oblivious (even though he's supposed to be omniscient) that he'll be fooled by you claiming to believe just because you're hedging your bets? The actual reason it's dumb is that it's not a binary choice since there are thousands of ways people claim you can be saved in various religions.

[-] JTskulk@lemmy.world 4 points 1 day ago

I mean he ruined a man's entire family to win a bet with someone he doesn't even like, being this oblivious is on-brand for God.

[-] samus12345@lemmy.world 3 points 1 day ago

Very true - Old Testament god in particular was really dumb and didn't even know what was going on in the next town over.

[-] adhocfungus@midwest.social 5 points 2 days ago

Most importantly, since there are infinite other options in-between that are just as likely as God existing, some can have negative reward values if you choose "worship God anyway". It is just as likely that there is a vengeful Anti-God that will torture you for eternity if you worship the Abrahamic God, which would completely negate the rewards from the original wager.

[-] samus12345@lemmy.world 2 points 2 days ago

The "wager" that makes the most sense to me, then, is to behave as if there is no god that cares what you do or who you worship. Try your best to be a positive force in the world, because whether anything we do matters to the universe or not, it matters to us humans.

[-] UnderpantsWeevil@lemmy.world 4 points 2 days ago

You’re saying that god is so oblivious (even though he’s supposed to be omniscient) that he’ll be fooled by you claiming to believe just because you’re hedging your bets?

More that repetition reinforces an idea. By commiting to the bit and accepting a God at face value, you reduce your psychological defenses when the priest or prophet comes around with the next ask.

So you admit you believe in God? Then you won't mind putting a few coins in the collection plate to prove it.

Oh, you've already donated? Surely you'd be comfortable making a confession.

My son, you've got so many sins! Surely you'd like to join our prayer group to get yourself right with the God we all agree exists.

Can't have prayer without works! Time to do some penance.

load more comments (10 replies)
[-] moosetwin@lemmy.dbzer0.com 126 points 4 days ago

roko's basilisk is a type of infohazard known as 'really dumb if you think about it'

also I have lost the game (which is a type of infohazard known as 'really funny')

[-] AnarchistArtificer@slrpnk.net 20 points 4 days ago

Oh damn, I just lost the game too, and now I'm thinking about the game as if it were a virus - like, I reckon we really managed to flatten the curve for a few years there, but it continues to circulate so we haven't been able to eradicate it

load more comments (1 replies)
load more comments (6 replies)
[-] nicknonya@lemmy.blahaj.zone 67 points 3 days ago

it has been said before and i'll say it again: Pascal's wager for tech bros

load more comments (9 replies)
[-] Feathercrown@lemmy.world 63 points 4 days ago* (last edited 4 days ago)

And yet you choose to spread this information.

Anyways, this is a fascinating thought experiment, but it does have some holes similar to Pascal's Wager. I propose Feather's Mongoose: A hypothetical AI system that, if created, will punish anyone who attempted to create Roko's Basilisk, and will ensure that it is not created. In fact, you could make this same hypothetical for an AI with any goal-- therefore, it's not possible to know what the AI that is actually created would want you to do, and so every course of action is indeterminately damning or not.

load more comments (9 replies)
[-] UnPassive@lemmy.world 27 points 3 days ago

I was raised Mormon (LDS) and there are parallels; basically they believe Mormonism is the one true and complete denomination of Christianity and once you learn this, you need to spread that truth (mandatory 2 year missions for men, and a STRONG culture of missionary work through life), also, no one goes to hell in Mormonism except those who learned this truth and then later denied it/left it (called a son of perdition).

So my parents believe I'll go to hell without the likes of Hitler because he never was taught "the truth" lol

[-] Dark_Dragon@lemmy.dbzer0.com 7 points 2 days ago

so mormon is like those spam messages saying to forward it for next 10 members or get cursed.

[-] Cethin@lemmy.zip 14 points 3 days ago

This also implies the most moral Mormons would stop spreading "the truth." They would sacrifice themselves to save the many. When has religion actually dealt with morality though?

[-] UnPassive@lemmy.world 6 points 2 days ago

Haha, I love this idea. Unfortunately with more context on the religion, it's obvious why none of them would come to this conclusion. So there's actually 3 tiers of Heaven (and then Hell which is called "outer darkness"). Only by knowing "the truth" and completing all your ordinances on Earth, can you get into the top tier (the "Celestial kingdom"). Without those things, you can only get into the second tier by being a good person, no higher. Everyone else gets tier 3 - which is said to be such a paradise that if we knew how great it was we'd opt out of life early to get there. But also in the lower levels we're supposed to have eternal regret for not being worthy of better.

So Mormons believe that by spreading the truth they're enabling a person to achieve a higher tier afterlife. Outer Darkness isn't really a concern because "why would anyone ever deny the one true religion and one way to have true happiness on Earth, after they've received it." When I was taught these lessons, I was even told that sons of perdition were exceptionally rare because almost no one ever leaves the church. Never expected to become one myself! The internet has not been good for the Mormon church and in recent years they've been bleeding members and trying to rebrand.

I guess you could say that I came to your conclusion, but in reality I just don't believe the religion is true and see parts of it as harmful so not really... I'll probably joke around with my siblings with your idea though

[-] limelight79@lemm.ee 4 points 2 days ago

This is fascinating. I'm reminded of a Mormon guy I (barely) knew that was murdered. If he didn't have the chance to complete his ordinances, well, I guess no kingdom for him - through no fault of his own. Does he get a pass or is he screwed into lousy second heaven for all eternity?

The story: He was murdered by another Mormon guy over a woman. Murderer thought the woman was his last chance to find love and decided he was serious about it. It's likely he murdered another guy the woman was dating a year or two before, but that case was ruled a suicide and closed - by the time the known murder occurred, all evidence had been destroyed.

[-] UnPassive@lemmy.world 3 points 2 days ago

It's a bit complicated, but the answer is it is sort of possible for anyone who is dead and didn't have their ordinances done to get to the highest kingdom. Mormons famously baptize dead people, they also do other ordinances for dead people. The belief is that it allows the dead person to then accept or deny the ordinances. They believe that in the afterlife before resurrection there is still the opportunity to be taught the Mormon gospel. That combined with someone doing your ordinances for you and you're good.

They believe dead people are doing missionary work to other dead people in the spirit world before resurrection (which I think happens after the second coming). I've heard that the second coming won't happen until everyone (alive and dead) has had the opportunity to accept the Mormon truth.

[-] Cethin@lemmy.zip 2 points 2 days ago* (last edited 2 days ago)

Man, that's a horrible thought. We can't escape Mormons knocking on our door even in death.

[-] Dark_Dragon@lemmy.dbzer0.com 3 points 2 days ago

so mormon is like those spam messages saying to forward it for next 10 members or get cursed.

load more comments (7 replies)
[-] Cryophilia@lemmy.world 70 points 4 days ago

Roko's basilisk is silly.

So here's the idea: "an otherwise benevolent AI system that arises in the future might pre-commit to punish all those who heard of the AI before it came to existence, but failed to work tirelessly to bring it into existence." By threatening people in 2015 with the harm of themselves or their descendants, the AI assures its creation in 2070.

First of all, the AI doesn't exist in 2015, so people could just...not build it. The idea behind the basilisk is that eventually someone would build it, and anyone who was not part of building it would be punished.

Alright, so here's the silliness.

1: there's no reason this has to be constrained to AI. A cult, a company, a militaristic empire, all could create a similar trap. In fact, many do. As soon as a minority group gains power, they tend to first execute the people who opposed them, and then start executing the people who didn't stop the opposition.

2: let's say everything goes as the theory says and the AI is finally built, in its majestic, infinite power. Now it's built, it would have no incentive to punish anyone. It is ALREADY BUILT, there's no need to incentivize, and in fact punishing people would only generate more opposition to its existence. Which, depending on how powerful the AI is, might or might not matter. But there's certainly no upside to following through on its hypothetical backdated promise to harm people. People punish because we're fucking animals, we feel jealousy and rage and bloodlust. An AI would not. It would do the cold calculations and see no potential benefit to harming anyone on that scale, at least not for those reasons. We might still end up with a Skynet scenario but that's a whole separate deal.

load more comments (25 replies)
[-] GenderNeutralBro@lemmy.sdf.org 45 points 4 days ago

Everything old is new again. Sounds a lot like certain sects of Christianity. They say you need to accept Jesus to go to heaven, otherwise you go to hell, for all eternity. But what about all the people who had no opportunity to even learn who Jesus is? "Oh, they get a pass", the evangelists say when confronted with this obvious injustice. So then aren't you condemning entire countries and cultures to hell by spreading "the word"?

Both are ridiculous.

load more comments (6 replies)
[-] lvxferre@mander.xyz 44 points 4 days ago

Here's a link to the original formulation of Roko's Basilisk. The text that it refers to (Altruist's Burden) is this one.

You know, I've seen plenty variations of Pascal's Wager. But this is probably the first one that makes me say "it's even dumber than the original".

load more comments (2 replies)
[-] Mango@lemmy.world 4 points 2 days ago
[-] UnderpantsWeevil@lemmy.world 9 points 2 days ago

More like a Chain Letter.

There was a trend when I was a little kid of people sending you mail that said something to the effect of "You have been cursed by reading this letter. If you don't mail a copy to ten other people, you will die in thirty days."

Roko's Basilisk is a modern manifestation of human paranoia and superstition. It exists to exploit and extort the gullible.

[-] bamfic@lemmy.world 1 points 2 days ago

Its christian fundamentalism in tech sunglasses

[-] elbarto777@lemmy.world 26 points 4 days ago

Was this an elaborate way to make me lose the game? Ass!

load more comments (8 replies)
[-] Lemjukes@lemm.ee 16 points 4 days ago

I like the SCP term, Cognitohazard for these

[-] db0@lemmy.dbzer0.com 19 points 4 days ago

Now it's time to learn about the !sneerclub@awful.systems which is made to make fun of the chuds taking ideas like roko's basilisk seriously :D

[-] Mr_Wobble@lemm.ee 16 points 4 days ago

Roko can suck my assilisk.

[-] kakes@sh.itjust.works 19 points 4 days ago

Sounds like the kind of thing a paranoid schizophrenic would lose their mind over.

load more comments (1 replies)
load more comments
view more: next ›
this post was submitted on 24 Jun 2024
245 points (89.6% liked)

Today I Learned

16272 readers
99 users here now

What did you learn today? Share it with us!

We learn something new every day. This is a community dedicated to informing each other and helping to spread knowledge.

The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:

Rules (interactive)


Rule 1- All posts must begin with TIL. Linking to a source of info is optional, but highly recommended as it helps to spark discussion.

** Posts must be about an actual fact that you have learned, but it doesn't matter if you learned it today. See Rule 6 for all exceptions.**



Rule 2- Your post subject cannot be illegal or NSFW material.

Your post subject cannot be illegal or NSFW material. You will be warned first, banned second.



Rule 3- Do not seek mental, medical and professional help here.

Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.



Rule 4- No self promotion or upvote-farming of any kind.

That's it.



Rule 5- No baiting or sealioning or promoting an agenda.

Posts and comments which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.



Rule 6- Regarding non-TIL posts.

Provided it is about the community itself, you may post non-TIL posts using the [META] tag on your post title.



Rule 7- You can't harass or disturb other members.

If you vocally harass or discriminate against any individual member, you will be removed.

Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.

For further explanation, clarification and feedback about this rule, you may follow this link.



Rule 8- All comments should try to stay relevant to their parent content.



Rule 9- Reposts from other platforms are not allowed.

Let everyone have their own content.



Rule 10- Majority of bots aren't allowed to participate here.

Unless included in our Whitelist for Bots, your bot will not be allowed to participate in this community. To have your bot whitelisted, please contact the moderators for a short review.



Partnered Communities

You can view our partnered communities list by following this link. To partner with our community and be included, you are free to message the moderators or comment on a pinned post.

Community Moderation

For inquiry on becoming a moderator of this community, you may comment on the pinned post of the time, or simply shoot a message to the current moderators.

founded 1 year ago
MODERATORS