There are options, and a well-funded new player will burst on the scene just as ChatGPT made itself eponymous with language AI: https://open.substack.com/pub/qnfo/p/the-diminishing-returns-of-scaling
this post was submitted on 10 Nov 2024
83 points (86.7% liked)
Futurology
1784 readers
77 users here now
founded 1 year ago
MODERATORS
AI is synthetic cognition or synthetic knowledge. What we're calling AGI must "know" the logic of our quantum universe first, which is all but incomprehensible to us.
It's hard to reason with the inconsistent (sometime nonexistent) human written communication that LLMs know. There's something deeper in other channels of communication that we use for our own logic.
Oh no, the technology that is literally just a glorified text prediction that gives you random guesses about what word comes next, based on what was in the text you trained it on, can not scale to have an independent reasoning?
Color me surprised, who would have thought?
But how can we be sure? Give us a couple more billion to waste on graphics cards first!
I don't think anyone in the industry thought LLMs were going to reach AGI. But LLMs will be useful as part of an AGI framework. That's the current focus in the industry.
I mean, yeah, this is about people outside the industry, those who invested money.
To my knowledge, LLMs still don't pay for themselves. When the hype dies down and investors aren't willing to provide money anymore, then prices for LLMs will become prohibitive for many current use-cases. That will also shrink the industry.
What that means in effect, we'll still have to see, but AGI was one path that investors hoped for to get towards profitability, so it doesn't aid the hype when they're slowly learning about reality.
I'm developing some human centric LLM frameworks at work. Every API request to OpenAI is currently subsidized by venture capital. I do worry about what the industry will look like once there is a big price adjustment. Locally run models are pretty decent now and the pace is still moving forward, especially with regards to context window sizes so as long as I keep the frameworks model agnostic it might not be a big impact.
Yep, based off public numbers regarding OpenAI finances, it was estimated that as of a few months ago they spent $2.35 for every $1 they made.
It's what Altman has constantly said was going to happen. Up to you to decide if he's actually in the industry or not.
When has Sam Altman said LLMs will reach AGI? Can you provide a primary source?
He has said that they already know everything they need to know to get to AGI.
OpenAI has not made a single thing that wasn't just a wrapped LLM.
So either A: he has somehow been running a skunk works that has fundamentally changed everything we know limits LLMs and none of the researchers leaked anything or B: he thinks LLMs are the way.
Additional quote when he was asked about AGI:
“How did we get to the doorstep of the next leap in prosperity? In three words: deep learning worked. In 15 words: deep learning worked, got predictably better with scale, and we dedicated increasing resources to it,” Altman said.
This tracks with the whole "I need 1 trillion dollars of energy investments" plan to get to AGI that he's asked for.
The guy's either honestly convinced himself that we can get there with deep learning scaling or he's a conman. Could be both.
I'm not defending Sam Altman or the AI hype. A framework that uses an LLM isn't an LLM and doesn't have the same limitations. So the accurate media coverage that LLMs may have reached a plateau doesn't mean we won't see continued performance in frameworks that use LLMs. OpenAI's o1 is an example. o1 isn't an LLM, it's a framework that augments some of the deficiencies of LLMs with other techniques. That's why it doesn't give you an immediate streamed response when you use it, it's not just an LLM.
OpenAI has absolutely made non LLM products lmao
He said it again a few days ago on a Reddit AMA.
Perhaps the most interesting comment from Altman was about the future of AGI - artificial general intelligence. Seen by many as the ‘real’ AI, this is an artificial intelligence model that could rival or even exceed human intelligence. Altman has previously declared that we could have AGI within "a few thousand days".
When asked by a Reddit user whether AGI is achievable with known hardware or it will take something entirely different, Altman replied: “We believe it is achievable with current hardware.”
You've completely misunderstood as others have pointed out to you. It's great that you want to learn about this stuff, but you have a long ways to go before you are at a point to talk authoritatively about it. The best thing for you to do now is to set aside all your preconceived notions and start from the beginning with an open mind. There is no point in talking authoritatively before you spend some time learning.
That's not Sam Altman saying that LLMs will achieve AGI. LLMs are large language models, OpenAI is continuing to develop LLMs (like GPT-4o) but they're also working on frameworks that use LLMs (like o1). Those frameworks may achieve AGI but not the LLMs themselves. And this is a very important distinction because LLMs are reaching performance parity so we are likely reaching a plateau for LLMs given the existing training data and techniques. There is still optimizations for LLMs like increasing context window sizes etc.
It's remarkable that anyone would think sam said that or thought that. It's like there is a whole other universe where 3rd and 4th hand sources are treated like 1st hand.
No they did for sure. Ofcourse that was conditional on a few things. Those things have yet to arrive and one major detractor is actually the lack of training data. Most of the public internet has been crawled by AI crawlers and half the new content is poisoned by AI making it worthless. Its gonna take a few more years to see if it keeps scaling or not.
Google, Microsoft and Amazon are all making heavy investments in nuclear power to run more GPUs. These aren't the moves of companies who are about to taper off utilization.
Heavy investments is a strong term for modest hedges around SMRs.
Tens of millions is low risk pocket change compared to the billions burned running the things constantly.
The problem is they're all playing chicken with each other.
OpenAI will never back down. The question is will Google, Microsoft or Amazon blink first
MS is going to relight Three Mile Island, not an SMR.
They don't expect to taper off. But they might just be burning money.
They might be, but that won't slow them down for at least the next gen of NVIDIA hardware.
Current LLMs can still do a lot. They’ve provided Level 4 self-driving, and seem to be leading to general-purpose robots capable of much useful work.
Really? I don't think this has anything to do with LLMs. They are likely using reinforcement learning combined with traditional AI techniques, an approach which has been the foundation of these kinds of robotics and automation for decades at this point.
If other areas of AI and automation have seen a boost at the same time as LLMs came on the scene, it's because the underlying hardware has become so much faster, cheaper and easily available, along with the massively increased interest in and funding for these types of research, and computer scientists re-skilling into a discipline that's in the midst of a bubble.
Not entirely true, the big change was multi-headed attention and the transformer model.
It's not just being used for language but anything where sequence and context patterns are really important. Some stuff is still using convolutional networks and RNNs etc. but transformers aren't just for LLMs. There's definitely a lot of algorithmic advances driving the wave of new ai implementations, not just hardware improvements.
Thanks for the clarification. The point remains that it's not true to say that LLMs have "provided Level 4 self-driving and ... general-purpose robots."
Agreed. It's a lot of the same tech that powers both, but it's not like a self driving car contains a language model that's fine tuned on the adventures of Steve MacQueen or something.
To be fair, most Americans don't demonstrate independent thinking, regularly regurgitate entire phrases they've been fed without showing any cognitive understanding, and they also sometimes perform tasks useful to corporations.
Self driving uses LLMs? Or a specific type of AI? If it's LLM I wouldn't trust it on a side street, since that's not what a language model is designed for.
No, self-driving cars don't use Large Language Models, they don't need to process language at all.
I mean, they would need to if they got good enough. Rare or obscure road signs that are just text. But self driving cars haven't even perfected how to take a turn correctly.
The fundamental architecture of an LLM is used across all these services - and that's reaching its limits.
I get that neural networking itself is used, but I doubt they're using specifically a LLM and fine tuning it for driving purposes. Pattern matching and fuzzy logic for inputs of driving conditions is a lot different than prompting a textual or visual response.
Pretty sure it was never going to become an AGI.
It's just donkey-work machines to cut low level jobs.
Remember: the dot-com bubble didn't end the internet.
This tech has cool uses, outside of venture-capitalist cult obsession. Mostly porn. I mean, Jesus Christ, so much porn.
But the general idea of image-to-image transformers, based on natural-language descriptions, is here, and it is witchcraft. Generating new images from scratch is a stupid demo gone feral. The real applications will be all-purpose "CGI" as an idiot-proof Photoshop filter. Select tin can on string, type "spaceship," get decently plausible results for your no-budget sci-fi show. No roto, no modeling, no lighting. The machine makes excellent guesses. And if you disagree, well, run it again. If you want it to be a specific kind of spaceship, either build a little toy or drag a PNG across the screen, and the machine will try to fix the aspects which make that look stupid.
The applications that money-robots want will be what destroys their industry. Animators don't want to type in "cartoon rabbit walking" and get a finished product the machine spat out - but they'd love to have their drawings tweened. They're all busy mocking this "framerate upscaling" nonsense, and missing that it means they can put in an on-eights previs and have it come out as smooth or as choppy as they want. And then they can doodle over whichever parts they don't like, and have a different model turn those sketches into on-model drawings. The ultimate outcome of which can look like any Pixar movie even if your process is entirely 2D. Or... it can look like live action. Starring real actors, living or dead. Or starring literal nobodies, as made-up as any animated character, but as plausible as any person on film.
We're gonna see a repeat of the webcomic boom... for movies and shows. It simply will not cost one billion dollars to make a whole-ass media franchise. Expect this to completely surprise the lumbering giants who keep trying to get rid of the little people who made up the stories and the characters.
Good news then?
They seriously thought IterationX would work?
LLMs are not all that is ai
I don't know of anyone seriously making the argument that LLMs would spontaneously develop independent reasoning. There's a huge amount of working currently being put into making them develop independent reasoning. Agentic workflows, chain of thought built into training data, that sort of thing. That's what those further investments you mention are involved in accomplishing.
If current AI players are about to get wrecked, I doubt it's the end for AI development. Perhaps it will switch to the areas that can actually make money - like Level 4 vehicles and robotics.
That's not a "bubble bursting", that's just ordinary churn. Companies come and go all the time, especially in cutting-edge fields like AI.
Maybe if the author wouldn't write "AI did hit a wall" in 2022, when everything is just currently talking about diminishing return, then someone might habe taken him seriously a bit. However AI is complex and there are new approaches to speed up learning and result speed, different approaches to steer a model output. The tech is still too new to say what's up next. So complex even, that we might have months or years with no significant upgrade until a break through. Other than that it just reads as if the author wants to get back their reputation after making himself look like a negative Nancy. People forget that even the brain has hallucinations, but also layers in place to correct them.
You can’t train it on Reddit and expect things not to go wrong.
Yeah, but we should let AI solve the problem of making itself better! Then it can solve everything from climate change to making fully self driving cars to figuring out the most efficient way to murder the whole planet!
/s
another normie who has no idea what is ai and who thinks llm=ai thinking he knows everything. people like you are more irritating than companies who overhype ai