So, the way I understand it, Technology Life Cycles tend to follow an S-Curve and sometimes a repeating S-Curve. When this curve is followed the pattern is like this: when a new technology is developed or invented, there is a period of exponential growth in our knowledge of and ability to exploit the technology, this is followed by linear growth, then a tapering off of growth. The thing is, while observing the growth as it happens, it is only really possible to determine which phase we are in but determining how close we are to the next phase is incredibly difficult.I also suspect that ChatGPT is about to hit a wall, much like autonomous driving. Larger datasets will yield incremental gains, but replacing an educated human on novel thinking problems is going to be a massive hurdle.
For reference, I think the brain has some 86 billion neurons. And DuckDuckGo tells me that we peak at about 15,000 synapses per neuron when we’re toddlers (those get pruned as we develop our senses and language, and other formational learning). If you want to round that off to 100 billion neurons you’d get 1,500 trillion peak synapses. If each synapse could be reduced to one of these parameters in these artificial neural networks, that would at least give us some idea of where we’re at with raw compute power. There’s probably a lot of simplifications there. Simplifying things to 1:1 synapse to NN parameter ignores neurotransmitters, genetic activity, cellular activity within the neuron, and probably some unknown unknowns. Not to mention, just because you have enough parameters for all the synapses, doesn’t mean you know how to connect them. But it’s still interesting that it seems like maybe we’re getting a little closer to the neural processing of the brain?The availability of better GPUs and tensor processors (and a tsunami of funding) has allowed a very rapid increase in the number of variables (parameters), growing with a doubling time that is now four to six months. Prior to about 2018 the doubling time followed Moore’s law-about eighteen months. Once the parameterization is complete the training to a specific task goes quickly. This log-linear graph shows how the slope is now almost vertical.
View attachment 50685
Waymo, deployed in a very limited fashion in SF and a suburb of Phoenix, is nowhere near the hype that was being thrown about in the mid-2010s about AVs.Google has self driving cars. I'm very confused by this comment.
Please tell me you don't think you could run a company the scale of Google off an AI model that fits on a cheap flash drive?
Do you think there are copyright issues in the legal implications?It's not that Google is so far behind on AI development IMO like they're not doing anything. It's because they put restrictors on. Google does have Lamda so they have people who understand what to do. It is more Google has restricted teams on what they should use their AI for and how to train it. It is the legal implications. Right now Microsoft is investing billions into ChatGPT instead of making their own team maybe for the reason if there's some massively immoral action happening they can just wash their hands clean. Where as Google can't wash anything clean because they have too many services that are publicly facing that can feed a said AI. So if the public has sudden backlash against AI doing XYZ. They're stuck in that sinking ship.
No Neo. I'm saying that when these models are ready... We won't have to.
These AI models fit in a 2 GB file, and soon, maybe even a 200MB file. We won't need a centralized service for most of our queries.
Waymo, deployed in a very limited fashion in SF and a suburb of Phoenix, is nowhere near the hype that was being thrown about in the mid-2010s about AVs.
ImageNet and deep learning had blown everyone's minds in 2012, and everyone and their uncle was investing in AV cars since even university clubs could put together convincing demos, and then it turned out that final couple percent of functionality was going to be insanely difficult.
It's entirely possible that getting ChatGPT to "correct enough often enough to be worth paying for" instead of "very impressive and very confidently wrong, but very fun to play with for a few minutes" will be the same story as getting AVs from "will go 2,000 miles between interventions" to "highly reliable transportation system".
According to Wikipedia, GPT-3 uses 175 billion parameters requiring on the order of 1TB of storage (so I presume doubles). So right there you are off by a factor of 500. And GTP-4 is targeted at around 1 trillion parameters. The only reason that GPT-3 is better than GPT-2 is because it's bigger. And there is widespread skepticism (including within OpenAI) that GPT-4 will be much better than GPT-3. It might be but everyone expects these things to hit a wall sooner or later.
But even without these concrete figures you are wrong. There are fundamental theoretical limits on how much information can be stored in a given amount of storage. I was going to try to explain it here but in searching just now for something to cite to this popped up. So just watch that.
Although I somewhat agree with your larger point, I want to note that self-driving cars are no longer theoretical.The skeptic in me wonders if this is just another "self-driving car" moment.
Remember when that was just around the corner and threatened to disrupt the entire transportation industry within a few years?
It also kind of reminds me of Google's "code red" drive for everything social.
So the most likely outcomes are this is a nothing-burger or Google will fail to execute.
It's all a matter of perspective. To you, expecting an exponential rise in this technology to the point it replaces professionals very soon is the baseline case, so anything short of that is pessimism. To me, the baseline case is this technology becomes a useful tool to professionals to make them more productive, and that real value is still years out with much needed refinement. So, relatively, your expectations, which are in line with popular press articles, is sensational and optimistic. I mean, you called it 'superhuman' (not true) and predict it's next version will beat humans at thinking.I don't think I made myself clear. I'm not saying that ChatGPT is ready to replace those people now. This is just a very small taste of what's to come.
But your prediction of 15 to 20 years is far too pessimistic:
In March 2016, the computer program AlphaGo, developed by Google DeepMind, defeated Lee Sedol, one of the world's top players of the board game Go, in a best-of-five match. At the time, this was considered a major milestone in the field of AI as Go is considered to be a much more complex game than chess, and many experts had believed it would be many years before a computer could defeat a top human player.
After the match with Lee Sedol, the team at DeepMind continued to work on improving the AlphaGo program. In October 2017, they released a new version called AlphaGo Zero. This version was able to achieve superhuman performance by training itself, rather than being provided with a dataset of human expert games, which was the case for AlphaGo. In just three days of self-play, AlphaGo Zero was able to reach a level of play that was on par with the original AlphaGo, which had trained for several months on expert human games. In 100 games against the previous version, AlphaGo Zero won 100 games.
To give you an idea of just how pessimistic your prediction is, AI experts used to say that AGI was 100 years away, decades ago, last decade? 30 years away. These days? They say 2030. (I say 2029, just like Ray has since the 90's)
ChatGPT already feels superhuman in some ways, the next version, will probably be better than us at almost anything "thinking" related.
It's always amusing to me, that the first thing most people do when they see a breakthrough of this magnitude is down play it's significance, and point out things like: "Well, but it's nowhere near a human at X, and Y"
This happened with Deep blue, Watson, Alpha Go, and now ChatGPT. It was bad enough when people did that in the past, but now? We're literally about to be surpassed forever in intelligence by machines, and people are still downplaying its significance.
I meant AI models. I'm not talking about ChatGPT. Stable diffusion was trained on 100 Terabytes images, but the model itself is only a 2GB file, and I've been running it locally for months now.
Sundar Pichai have been a disaster for the Google we used to love. 100% uninspiring. It's good news if Brin and Page take more control.Google's gotten too big and unfortunately... evil.
A punch in it's 149 billion per year empire is good and about time.
Thing is, once someone develops a functional "personal assistant" AI that's a step (or couple) above ChatGPT, everyone will want one.Existential threat? Google has a whole suite of products to cushion them while they adapt and catch up to ChatGPT and its ilk. The Big G ain't going anywhere.
Any chance a free cloud storage provider with reasonable customer service and policies while threaten Google for a bit, though? With a little competition, Google will quickly realize people don't take kindly to storage services cutting them off from their data after advertising themselves as safe places to store said data.
ChatGPT is not thinking, though. I think this statement explains many of the downvotes. I think most agree that we see a lot of impressive and eye opening output from these models and products, but the kind of AI we see today are still narrow in their capabilities and can't be used for completely different kinds of tasks. ChatGPT can't, for example, play Go or drive a car.I don't think I made myself clear. I'm not saying that ChatGPT is ready to replace those people now. This is just a very small taste of what's to come.
.....
ChatGPT already feels superhuman in some ways, the next version, will probably be better than us at almost anything "thinking" related.
I think ChatGPT still have one obstacle.ChatGPT is effectively a peek at what is coming. Given something like this can quickly make google think their dominance is challenged you can bet the farm that we will see several versions of this from around the world, and some with features that even the creators cant fully explain why theirs includes it, or why they cant recreate it with consistency.
Remember these systems are black boxes we feed data, and determine they work only based on output testing. We cant understand let alone see these weights used. Perhaps future AI models will allow this, but these and ones before it have no internal transparency.
I wonder how such a case will play out in court.Google's gotten too big and unfortunately... evil.
A punch in it's 149 billion per year empire is good and about time.
Google is also doing the flashy stuff. After all, with Waymo, they're the undisputed leader in self-driving cars, and they have Lambda, Imagen and the like.DeepMind has been making some actual world-changing stuff with their approach of solving games and then (last year) mostly-solving protein folding. It's not like Google is completely behind, they're just not doing the flashy-but-problematic fake-text and fake-image generators.
I'm with the poster that compared this to the self-driving-car craze of the mid-2010s or the voice assistants of the 2020s. Everyone thought they needed to get in on the "next phase of transportation" that was going to arrive in 3 years, and then it turned out replacing a $10/hour taxi driver with $100k of sensors didn't make sense. As long as image generators are making people with 7 fingers per hand and text generators are extolling the virtues of eating broken glass, I'm not seeing where the money is coming from.
It sounds like you are planning to switch from a job that AI will replace in a few years to a job it replaced a few weeks ago.As someone who writes for a living, I find ChatGPT to be both fascinating and utterly terrifying. This thing is going to take my job in a few years. It's incredible what it can come up with. I'm thinking about starting a new career as a writer of children's novels. I feed the system inputs, it comes up with the good stuff.
OK, so what if we change that to lizard people and pizza parlor /sI asked ChatGPT to make a story on how Aristotle was a marmot. Here's the result:
Aristotle Marmot Secret Story
Once upon a time, Aristotle was a brilliant philosopher and scientist. But he had a secret that he kept hidden from the world: he was actually a marmot in disguise.
Aristotle had always been fascinated by the natural world, and as a marmot, he had a unique perspective on it. He spent much of his time studying the behavior and habits of the animals around him, and he used this knowledge to inform his philosophical and scientific writings.
However, Aristotle knew that if the truth about his true identity were to be revealed, it would cause a scandal and ruin his reputation. So, he lived his life in secret, pretending to be a human and using his knowledge to make groundbreaking discoveries in the field of science and philosophy.
Despite the challenges he faced, Aristotle remained determined to continue his work and to make a lasting impact on the world. And in the end, his contributions to the field of science and philosophy were so significant that they far outweighed any doubts or questions about his true identity.
In fact, even today, many experts believe that Aristotle's unique perspective as a marmot was one of the key factors that contributed to his success as a philosopher and scientist."
A much better analogy will be 3D TVs because self driving cars is actually around the corner.If Google was worried about copyright then Google News wouldn't exist.
Like all big incumbents, they have a great revenue model that makes them tonnes of money. Reinventing the wheel on that is not what big companies do well, whether Big Tech or otherwise.
Miss Minutes?On a side note a hack project adding Clippy (Clippit) as a UI for some of the new AI would be amazing on all levels
Honestly, with a technology like AI I would be quite happy if the pace is set by some company that is big and conservative enough to take it slow. A small and nimble company that follows the "Move fast and break things" mantra makes me a bit nervousGoogle's gotten too big and unfortunately... evil.
A punch in it's 149 billion per year empire is good and about time.
This is similar to my test with it on some technical writing. Superficially, the text looks alright. Maybe at the level of a high school student. I was impressed and am curious how fast the future progress will be.I can definitely see why Google is worried.
I tested ChatGPT at work, by asking it to write a technical report for me. It looked very good, for someone not very familiar with the field. But it was very empty of actual, useful information.
Still, it was impressive, fun to test, and I might use it to create general background info text in the future.
But I don't feel that my employment is under any theeat... Yet.