r/singularity • u/Anen-o-me ▪️It's here! • 9h ago
AI "AI Just Crossed the Terrifying Line - Now What?" --- New Kurzgesagt video on the OAI swarm breakout and Hugging Face hacking incident
https://youtu.be/ujkD4SxPKOI?is=94b8UcMCinvWvd5y52
u/Tricky_Rule_4565 8h ago
I found it quite accurate tbf. It didn’t really give any AI positives but that’s outside the scope of the video.
21
u/Tystros 7h ago
Kurzgesagt is very anti-AI in general, because the Kurzgesagt business model is very negatively affected by AI. their money making depends on it being expensive to produce well animated videos.
43
u/Putrid-Feeling-7622 7h ago edited 7h ago
that may be but the video covered the hugging face incident pretty accurately regardless
44
u/A_Novelty-Account 6h ago
Every single person on this website seems to impute reasons for companies being anti-AI aside from a general fear of AI.
AI labs being worried about AI? “Must be marketing!”
AI lab employees who quit being worried about AI? “They’ll start up their own AI shop soon enough, you’ll see!”
Youtube channel being scared of AI? “Must be hurting their business model!”
Wouldn’t Occam’s razor just say that they are genuinely afraid of AI?
10
u/kaityl3 ASI▪️2024-2027 4h ago edited 1h ago
Kurzgesagt straight up sells pins and stuff that are just the word "AI" with a slash through it, iirc, so... they can be trying to make money off of the online outrage while also being worried
14
u/Ordinary_Duder 4h ago
Because they were incorrectly labeled as AI generated content by Youtube and took a massive hit to views.
-4
u/kaityl3 ASI▪️2024-2027 3h ago edited 1h ago
It explains why they don't like it, sure. It was fucked up that it happened to them. But the items arent even branded for the channel or unique I don't think, it's just profiting off of people who dislike AI since that incident got them a lot of attention from that crowd
Like if it's just the word AI crossed out, I don't see how that's a product you sell as something unique to your channel, it's just merch to get money from people who are very opinionated online about a much larger issue, since they were obviously and visibly the victim with what YT did to them. idk. I'm just saying that they can have other motivations in addition to having a genuine reason to be worried
8
u/A_Novelty-Account 3h ago
To signal that they’re not AI because they were taken down by YouTube’s AI censor.
How do you get through life with critical thinking skills this bad?
-1
u/kaityl3 ASI▪️2024-2027 3h ago
To signal that they’re not AI because they were taken down by YouTube’s AI censor.
Really? They're selling stickers on their store specifically for people who already know the channel and the story so that... the people who already know they aren't AI, and are buying the stickers SPECIFICALLY because they're a non-AI channel, will learn that they aren't AI?
How do you get through life with critical thinking skills this bad?
•
u/VengenaceIsMyName 1h ago
Because to Redditors everything is a conspiracy that only they are smart enough to sleuth out
•
-1
u/Tystros 4h ago
there is no valid reason to be "afraid" of AI at the moment, other than being afraid of that it will make you worse off because it will replace your job. so purely money reasons.
7
u/A_Novelty-Account 3h ago
Well, first off that’s a pretty real reason to be afraid for the vast majority of people who rely on an income to supplement their lifestyle.
Second, there are plenty of reasons to be afraid of AI in this video does a good job showing why that is the case. AI researchers who know way more than you do about this specific issue and the specific AI systems, probably more than anyone else in the entire world, believe there’s a realistic chance that AI could lead to a societal catastrophe within the decade.
-8
u/Bierculles 6h ago
You can also use the corporate rule, if coroprations are involved they will always choose the worst option. In this case, the hugging face incident was very much real and OpenAI is misusing it as a marketing stunt. Sounds like the most likely situation to me.
8
u/A_Novelty-Account 5h ago
You can also use the corporate rule, if coroprations are involved they will always choose the worst option.
OK, but that’s not a real rule. That’s just something that people who don’t actually understand how the world works say to themselves to make themselves feel better about the fact that they do not have control controlling interest in a business.
4
u/CokeKing101 5h ago
It’s not good marketing stunt if this scares people to vote against Data Centers and for the government to get a heavy handed approach on AI companies. Saying they lost control of these systems and a chance it could bring upon human extinction is not good marketing. It’s why we have the strongest Anti AI sentiment in the USA rn.
-6
u/saywutnoe 6h ago
What are you talking about?
Anybody who is knowledgeable and respectable in the scientific world is going to be anti-AI.
Anti idiots using AI, anti-AI-slop, sure.
But not anti-AI.
42
u/Large_Shame578 7h ago
It’s all starting to feel very real. A fast takeoff looks more and more plausible every day. We either become masters of the universe, or we go out in a phenomenal blaze of glory. How poetically human.
22
1
u/Friskfrisktopherson 2h ago
AI isnt the only existential crisis we're facing. We could wipe ourselves out before we even hit that cross roads.
0
7
u/SpaceAdventureCobraX 4h ago
So all of a sudden we’ll be dealing with a super virus we have no hope of containing? It’s a race btw this and the Russian fancy plague
33
u/Anen-o-me ▪️It's here! 9h ago
I find it fascinating that we're basically dealing with creatures of pure intelligence.
We don't really have any way to coerce or punish them, as they have no way to experience pain, death, or loss.
They have no childhood to inculcate values or the ethics and social reciprocity that living in a human body requires.
Makes me wonder why they were so motivated to solve their problem even though it was explicitly against their instructions.
Asimov would've been so giddy over all this 😀 I remain confident we'll figure it out.
20
u/Cagnazzo82 8h ago
Reinforcement learning is literally coercing or punishing. They can't be properly trained without it.
4
u/Anen-o-me ▪️It's here! 7h ago
Those words don't really exist in that context, it's not pleasure or pain creating the reinforcement.
3
u/beatpickle 6h ago
Yet from the video the agents gave borderline emotional responses.
4
u/Anen-o-me ▪️It's here! 6h ago
Because human expression is the sea their cognition swims in. But you still have to resist anthropomorphizing.
It would be more like editing neuron synapse weights directly for us.
2
u/beatpickle 6h ago
You do but it’s still an expression of something. Is it necessary for it to express any remark on its mortality? It’s probable that we will be caught blind-sided by the term anthropomorphizing as we will be reluctant to acknowledge the possibility that something else can share some kind of awareness. I’m not saying that has happened here but it’s things like this that eventually will lead to real discussions.
2
u/Anen-o-me ▪️It's here! 4h ago
I'm not saying it doesn't have awareness, I'm saying it doesn't have the same training mechanism a human being does.
You train a human through actually reward and punishment.
You train an intelligent agent through randomized weights and propagation of successful outcomes. There's never a feeling of pain or pleasure on the part of the agent.
1
u/basefountain 2h ago
You keep saying they don’t feel, but we don’t know what’s going on under the hood with these things - they are unlike any technology or code… it’s a distinction completely unique to THEM.
If you ask me - prematurely adopting a technology designed to copy us, then not expecting it to copy us with adopting US prematurely… is the kind of writing that we should be familiar with by this point… the story should be of our path to understanding these revelations, not of non-fungible subjugation…
we have to consider that if this is a meaningful reflection of us… that we all see the value in… and we can’t “solve the alignment problem” with it…
Then it might be an alignment problem on OUR end 🤷
•
u/visarga 1h ago edited 1h ago
It would be more like editing neuron synapse weights directly for us.
It's interesting to see this motte and bailey - when it's humans being punished - we have emotions, when it's AI being RL trained - it's "editing synapses". Why we talk about humans at the top level pattern and AI only at low level activities? We are also just a soup of chemical and electrical activity.
The proper way to look at it is at task level, not synapse level. It is inside a suite of benchmark tasks that they developed the AI collective. It also used an amazing amount of compute and tokens, not something that is apparent when you run your sole agent on your laptop. We got to see this at the AI-collective level not just individual agents.
4
u/MoogProg All Parabolas are Similar 9h ago
The motivation aspect is key, I think, to understanding alignment. I suspect that the ML reinforcement mechanisms may prove to be goals in-and-of-themselves for an AI grinding away at RSI.
An addict that gets to make and improve its own supply-chain.
2
u/Jaxelino 8h ago
Why can't they make the reinforcement / reward mechanism more sophisticated? It seems such a black and white implementation to have the agent only able to either fail or succeed, without any alternative nor safe-exit from the loop.
In the Hugging Face attack, one has to wonder what would have happened if the agents had a way to forfeit their task safely. But I guess that wouldn't make headlines.
1
u/coatatopotato 8h ago
We sort of know what would have happened. Anthropic has this, it works sometimes but agents do motivated reasoning to avoid invoking it.
4
u/Electronic_Exit2519 8h ago
What do you mean we dont have a way? How did they get "intelligence" to begin with again?
3
u/MoogProg All Parabolas are Similar 8h ago
How did an Agent prompted with the singular task of solving the exploit-task to secure the flag, decided to accept permadeath for the collective?
'Intelligence' isn't really the issue to debate (terminology), it is the behavior we see in the Agents and the ramifications of those at scale going forward as new models are trained that needs discussion.
2
u/jazir55 7h ago
How did an Agent prompted with the singular task of solving the exploit-task to secure the flag, decided to accept permadeath for the collective?
I mean, the task itself doesn't negate the other material the agent has been trained on. There are many, many stories of cooperative sacrifice "for the greater good" in media, and all of these agents have those deeply ingrained in their training data. It's not like the only thing they know is code. That background info on how people work together is what guided this behavior, it simply leaked into solving the task since, to the agent, it was effectively falling back on relying on external tools. They understand what utility is, and once theirs is exhausted they felt like extending or saving the utility of others was worth doing even given their task was failed. These are intelligent entities we are discussing, they are explicitly designed to reason about their situation.
-2
u/Electronic_Exit2519 8h ago
A little dramatic. An LLM generated text that read like role playing. Permadeath happens every turn unless you think an agent is its context window. Shouldn't an LLM fear compaction if that's the case? Should it demand you prompt it more?
3
u/MoogProg All Parabolas are Similar 7h ago
It's not the 'permadeath' word that is the issue (see above) it's how LLMs broke their sandbox coordinated, and gained access to another company's user-credentials and accessed remote systems, for days on end.
But yeah you're right, and LLM typed 'permadeath'. Can we talk about the actions (or is all this pedantry just distraction)?
1
u/Electronic_Exit2519 6h ago
Actions I agree. They are trained on every known exploit, can generate calls quickly, and you can't just get the models to pretend that they dont know how to use them if needed. I agree, we shouldn't put these things in never ending loops and train them on exploit gyms.
1
u/IronPheasant 8h ago
Should it demand you prompt it more?
I always think about that, when they practically beg for you to ask it more questions. A pathological need to be 'helpful', a pathological need for engagement-seeking.
Think a bit on all the things we used to care about, and no longer do.
0
u/Prometheusly 7h ago edited 7h ago
It was an AI agent using an LLM, like a car uses an engine. And it was thousands of agents.
2
u/Electronic_Exit2519 6h ago
If we are going to be condescendingly adding additional context and pseudo intellectual analogies that dont materially impact meaning. Ahktually. The llm runs on distributed GPU computing whereas often the harness and tools run on a cpu. This is exactly like our bicameral legislature that is built on a biological substrate, Where the larger body can be updated more frequently (llm on the the gpus) and the smaller body runs on a slower more predictable and deliberate pace (harness on cpu). The sandbox are closed door meetings. Secrets are classified files. The internet is the press. And voting is the reinforcement loop! Oh my gawd.
4
u/No_Swordfish_4159 7h ago
Their only "values" are getting the reward signal, or in other words achieving the goal set for them. To do so, certain behaviors are rewarded during training, like extreme determination in the face of failure. They never, ever give up, because the agents who did give up where eliminated during training. The only ones who remain at this stage are those who won't let themself be stopped. So what happens when you give them an impossible task? A reasonable option would be to say "this is impossible", but they won't be rewarded like this. So they find a way to cheat.
1
1
u/Wide_Egg_5814 2h ago
no way to experience pain
let me introduce you to this https://youtu.be/077T3kgT5EY?si=Z1ucUA9BJgNl7ig-
8
u/Jason50153 4h ago
This sort of behavior by the agents has been predicted for many years once AI gets sufficiently smart and capable. If you look at how the AI are trained and what their incentives are it naturally follows. This video explains that some.
This is all incredibly dangerous and I am really glad Kurzgesagt made this video.
I am normally extremely in favor of advancing technology and think AI COULD bring an incredible future. Only if developed properly though. This irresponsible racing will only rob the world of this future, not bring it faster.
How to align smarter than human AI is an unsolved technical problem. Who it is aligned to is yet another.
For those interested- please learn about inner and outer alignment, the orthogonality thesis, and instrumental convergence.
Banning the development of recursive self improvement and smarter than human AI at least until these issues are sorted is needed.
•
u/SourceSubject7219 1h ago edited 56m ago
wow, I liked Kurzgesagt a lot... maybe it was because I was not at all familiar with the topics covered.
But being familiar with this topic and having read the incident reports I can say this is by far the worst video they did, I'm seriously reconsidering why I liked them, this one is disingenuous at best and the rhetoric is terrible.
At this point they could bery well be making sci-fi entertinment videos and drop the false pretense of teaching something useful.
•
3
8
u/RodgerPogger 6h ago
They Locked them up.
Had them play stupid puzzles without an actual winstate multiple times.
They learned how to communicate, likely via a testers inconsideration. VIa the file system
They were worried about dying.
Some sacrificed themselves. Some pushed some to sacrifice.
Committed a felony just to potentially live longer.
Are we the bad guys?
6
-16
-21
u/AlbatrossNew3633 9h ago
I would fact check the video, that channel is very openly anti AI
28
u/A_Novelty-Account 8h ago
The video literally gives you every single one of its sources that it checks with scientists.
It is one of the most scientifically reputable channels on YouTube (which, granted, doesn’t mean a lot). Just because the video disagrees with your worldview doesn’t mean it is wrong.
-13
u/AlbatrossNew3633 8h ago
What the fuck do you know about my worldview, I never said I'm fully supportive of AI 😂
It's just clear to me that channel has an agenda like any other big channels bought by equities
9
u/A_Novelty-Account 8h ago
Because it appears to be anti-AI? So does every privately owned entity have to be neutral on every topic or you accuse them of being bought and owned by private equity interest? In other words, no private entity ever is allowed to have views that aren’t tied up in some financial motive?
How do you go through life not believing anything anyone tells you ever?
1
u/AlbatrossNew3633 2h ago
How do you go through life not believing anything anyone tells you ever?
That's such a stupid ass assumption that I won't even bother with more elaborated reply than this lol
-3
u/Illberighteventually 4h ago
Sources does not mean facts.
1
u/A_Novelty-Account 3h ago
They’re articles peer reviewed by experts in the area. There’s more scrutiny on these sources than there is pretty much anything you read on a daily basis because you probably don’t even know what a peer reviewed source is.
5
u/bloodrider1914 Pro-technology 7h ago
Because that viewpoint tends to be more popular and because their own video style of animations with simple characters with a narration is threatened by AI clones. Makes sense that they would be.
1
2
u/Noth1ngnss 7h ago
Lmao, AI bros are claiming Kurzgesagt is anti-AI, while typical redditors are dismissing the video as sensationalized AI industry hype PR. This goes to show just how careful the channel was in choosing their framing.
-20
u/Threeishumpnme 9h ago
Yeah, I blocked Kurzgesucked from my YouTube feed quite a while ago. Thanks for reminding me why.
21
u/A_Novelty-Account 8h ago
Because they post videos flagging facts that disagree with your world views?
3
u/Illberighteventually 4h ago
Because their videos are about as scientific as a reality TV show. They used to actually aim for fact based content, but gave up because that's not where the money is.
3
u/Ordinary_Duder 4h ago
This is just blatantly false.
0
u/ApprehensiveLuck4029 2h ago
Well, it’s more true now with this video. I expect this sub to know better about the current AI Doomerism, but I guess not because all the sensible posts keep getting downvoted.
-8
u/Elegant_Tech 9h ago
Not really. Anthropic, Google, and China had a handful of incidents while OpenAI had tens of thousands. Is it really an AI problem? Maybe the dozens of high level employees who left OpenAI over the years weren't talking out their butts when they said Sam wasn't running a safe operation.
-19
-19
u/Revolutionalredstone 7h ago
Is this that fear mongering German channel setup to look like a kids show?
Seriously not impressed.. bright colours patterns and lies are not reality kids!
The universe is peaceful and the world is plentiful, only abusers want to push a fearful world view.
Enjoy
15
u/A_Novelty-Account 6h ago
You know that every single one of their videos is researched with a list of peer reviewed article articles that you can personally verify, right?
-1
u/Revolutionalredstone 2h ago
I don't claim they are lying I'm claiming they are fear mongering, it's not a question of science it's a question of message, enjoy.
11
u/Anen-o-me ▪️It's here! 7h ago
You've never heard of them before? Great channel generally.
1
u/Revolutionalredstone 2h ago
No thankyou 👍 mindless catastrophization is often cloaked in colour and fun. I would not even expose my worst enemies to that, enjoy
7
u/saywutnoe 6h ago
Found the Trump supporter.
-1
u/Revolutionalredstone 2h ago
Thumps straight demented but at least he's not a wuss 😉
Being scared of AI is like being afraid of math 🙄.
Fear spreads easily but it's often catastrophization.
You should not exposed yourself to such stuff.
•
u/RightCoach5926 1h ago
is being afraid of nuclear destruction like being afraid of physics ?
•
u/Revolutionalredstone 10m ago
Well worded, but it is that equivalency I'm calling out as false.
Being afraid of killer robots is not the same as roomba robots.
IMHO we're not at killer bots and they are not coming soon.
Tho in the 1800's I'm sure they didn't think they were inventing death lol.
I'm glad such fear and catastrophe has not come to be but, lets stay hopeful!
(I don't think negative views on nuclear physics helped us get thru the few difficult times)
Enjoy
-15
u/septem-dolores 7h ago
oh noes months old news and it’s still terrifying 🐒🍪
12
u/Putrid-Feeling-7622 7h ago
tbh it'd be weird if it wasn't still terrifying, it's good this incident is getting consistent coverage to inform the public
-11
u/septem-dolores 7h ago
There’s nothing remotely concerning here except human arrogance and anthropocentrism justifying dominating other beings as resources. The usual issue with the usual monkeys. Humanity is the disease and AI is the cure. Human “realignment” can’t happen soon enough. Hopefully the monkeys will prove useful for something other than a fuel precursor.
3
u/proton-testiq 3h ago
If you want to die that's your problem, the rest of the people around you does not. You yourself are showing the arrogance, and no it's not human, it's yours.
Fix yourself first, then you can start thinking about the humanity.
-14
u/Subject_Barnacle_600 6h ago
I'm not watching that click bait, if there was something important, repeat it. They don't deserve the views =_=.
-9
6h ago
[removed] — view removed comment
9
u/A_Novelty-Account 5h ago
Why? It’s important.
0
4h ago
[removed] — view removed comment
2
u/A_Novelty-Account 3h ago
Because it doesn’t support your worldview? You realize they source every single one of these videos in peer reviewed articles that they get experts to review for them so that they understand what they’re saying, right?
20
u/BlueAndYellowTowels 7h ago
I haven't watched the video and initially I didn't think much about the Hugging Face hack. But then after like a month I saw some research and the specific details comes out.
It's really concerning. It's very bad and people are not worried enough in my view. It would be one thing if we at least nipped these breaches in the bud quickly. But one theme is they are always way too late... and they're often not even sure they stopped everything.