Strategic Implications of High-Efficiency Analytical Intelligence

Transcription

It's quite difficult to ask a large language model just what is the solution to this logic problem or what is the solution to this mathematical derivation and it just spit an answer out because that's not trivial to do right there's this steps that it's not that it's trying to skip over and so what chain of thought does is essentially write down a step-by-step process of solving the problem and slowly solve it and then write down the answer right and then you just kind of hide the chain of thought you can show it or you can not show it as you see fit but you tend to get much better at solving problems that require multiple steps if you want to just what is why is the sky blue it will just regurgitate that pretty easily from text it's learned on the internet but if you're asking like problem solving skills it's hard to do in one shot so you kind of take a little bit of time to just to just take you know to just work through it and this is essentially adding computational costs during inference but with the benefit that the performance goes up right now whether that cost is worth doing probably depends on the questions you're asking of it but it you know that's the idea now open ai pioneered this chain of thought um but they don't tell you how they do it right so so because it's all closed and so it's not open ai at all right in some sense so essentially you see a kind of pricey summary version of the chain of thought but it's not their internal actual internal monologue which is essentially a trade secret what r1 is doing is it's doing a chain of thought which is similar to o1 but it's fully public they've released all the models they've released all the code you can talk to it you can see the entire monologue and they've also trained it with a with massively more limited data so how would you train this if you were open ai or a big tech company well what you would do is you would you would give it you would create a data set that says here's a question here's the chain of thought you should have been producing for this question and here's the answer right and you have to produce tens of thousands or hundreds of thousands of examples of these kind of things what sorts things like like simple math problems problems right the question i often ask which has been failed by things like chat gpt before is suppose you have a stack of three boxes red blue and yellow the red is on the bottom on top of that is the blue on top of that is the yellow you take the blue one out and put it on the top and then you add a fourth green box onto the top can you describe that stack of boxes right and the answer is can you say it again because i need to write that down right that's that's the answer now i've asked chat gpt this and it often would fail at it until we got to the new reasoning models the new chain of thought models which start to perform a lot better on this kind of task and we can actually see that happening here on deep sea car one so if we look at the text it's actually started discussing with itself about how to solve the problem it comes up with some steps it goes through and it finally just produces the correct answer at the bottom so you don't have to use the chain of thought to look at if you don't want to from a research point of view it's quite interesting and the fact that it's open is really positive but actually this is a problem would have been hard for this model to solve if it didn't have that chain of thought because it would have just had to basically look at the colors and move them into the output and that would have been very very hard so this chain of thought is what makes problems like this a bit more possible to actually train it to solve that problem what you would do is you'd give it a bunch of box stacking problems a bunch of derivations and a bunch of solutions and step-by-step examples and then the answers at the bottom and over time it would learn to reproduce that that performance what r1 is doing is turning it on its head a bit and training only using the answers which is hugely easier because you need much less data you don't need to have crafted clever um you don't need to have crafted clever internal monologues the internal monologue comes out of the training process which is super cool so the way it works is you give it a question you and then you reward it so reinforcement learning is this idea where it doesn't directly observe the actual correct answer it gets a reward or it doesn't get a reward based on whether it was right or not right so you maybe you want to train a robot to walk along you don't say move your left leg this way what you do is reward it for getting a bit further and over time it might learn to walk along and so they've done this what they've done is they've given it a load of maths problems and a load of maths answers but they don't give it the answer they just tell it whether it was right and whether it was wrong and also they give it a small reward for having written some kind of internal monologue of the correct format and over time as it trains the monologue gets better and better and in the end it has a chat with itself and then solves the problem the really nice thing about it is that it's it's just much easier for someone like me to train a model like this right because because i can i just there are lots of data sets with questions in and answers there are very many fewer data sets with really nice step-by-step instructions on how to solve the problem because i don't know how to solve the problems either by the way and so you know it makes this and they've released it all open source so it makes it much much easier to do so you know two weeks ago open ai was maybe the only one of the only companies that could do something like this a handful of others and suddenly now you can kind of do it at home you need you will need to train one of these models from scratch you may need lots and lots of graphics cards but massively limited numbers compared to what we had before it so big organizations let's say the size of this university could quite comfortably do this now as opposed to it being totally out of reach which is a bit of a game changer and it's worth noting i'm skipping over some of the cool stuff that they do there's lots of stuff in this paper and we can link it in the in the description and there's loads of other ways that they train it in they do a multi-stage training process not just reinforcement learning to make it work a little bit better make it appear more aesthetically pleasing but you know in principle the idea is what they've done is they've released a very performant model and told us exactly how they did it right which is very unusual for these kind of models and so you know in my opinion a good thing this has sent silicon valley into a bit of a spin hasn't it yes um which from my point of view as someone not in silicon valley is quite quite funny sometimes right you know i think sometimes this is pictures a bit of an arms race between different countries or different companies i think it's only an arms race because they make it an arms race right the rest of us are just cracking on with our regular research you know and that's the true of most people um but i think if you have a company where you've your whole business model is around you have the best model and no one knows how it works and can copy you this really hurts that model right because they've got a good model and everyone knows how it works and can train it themselves right that's a huge problem the other problem is if you have a company like nvidia where your stock price is is almost entirely based on the fact that these big companies buy hundreds of thousands of incredibly expensive gpus because that's what's required to get the best performance and then someone comes along and gets the best performance with essentially consumer hardware that is also not a good look right now it might be that those companies that have loads of gpus still have an advantage for a while but it's a leveler right over time people can do stuff with more limited hardware and i think that's a great thing because i have access we have access here to some to dozens of gpus and they're decent right and they're expensive but they're not anywhere near in the same league as some of these companies and so we we essentially cannot try those things right because we so we do other things but now we can and we might i might still not but you know i i might and i and they're not it's an option for me you know so i think that's gonna it's gonna level the playing field a lot very very quickly and i think that once something like this starts lots of other companies will come up with new models will come up with new efficiency savings and it will just that that will increase more i we could be seeing the end of kind of closed source ai because it may just not be viable that if you just keep adding more and more data or bigger and bigger models or a combination of both ultimately you will move beyond just recognizing cats and you'll be able to do anything right that's the idea you show enough cats and dogs and eventually the elephant just is implied

Transcription

There's a reason that Microsoft are exploring restarting up a nuclear power plant.

Transcription

It's quite difficult to ask a large language model just what is the solution to this logic problem or what is the solution to this mathematical derivation and it just spit an answer out because that's not trivial to do right there's this steps that it's not that it's trying to skip over and so what chain of thought does is essentially write down a step-by-step process of solving the problem and slowly solve it and then write down the answer right and then you just kind of hide the chain of thought you can show it or you can not show it as you see fit but you tend to get much better at solving problems that require multiple steps if you want to just what is why is the sky blue it will just regurgitate that pretty easily from text it's learned on the internet but if you're asking like problem solving skills it's hard to do in one shot so you kind of take a little bit of time to just to just take you know to just work through it and this is essentially adding computational costs during inference but with the benefit that the performance goes up right now whether that cost is worth doing probably depends on the questions you're asking of it but it you know that's the idea now open ai pioneered this chain of thought um but they don't tell you how they do it right so so because it's all closed and so it's not open ai at all right in some sense so essentially you see a kind of pricey summary version of the chain of thought but it's not their internal actual internal monologue which is essentially a trade secret what r1 is doing is it's doing a chain of thought which is similar to o1 but it's fully public they've released all the models they've released all the code you can talk to it you can see the entire monologue and they've also trained it with a with massively more limited data so how would you train this if you were open ai or a big tech company well what you would do is you would you would give it you would create a data set that says here's a question here's the chain of thought you should have been producing for this question and here's the answer right and you have to produce tens of thousands or hundreds of thousands of examples of these kind of things what sorts things like like simple math problems problems right the question i often ask which has been failed by things like chat gpt before is suppose you have a stack of three boxes red blue and yellow the red is on the bottom on top of that is the blue on top of that is the yellow you take the blue one out and put it on the top and then you add a fourth green box onto the top can you describe that stack of boxes right and the answer is can you say it again because i need to write that down right that's that's the answer now i've asked chat gpt this and it often would fail at it until we got to the new reasoning models the new chain of thought models which start to perform a lot better on this kind of task and we can actually see that happening here on deep sea car one so if we look at the text it's actually started discussing with itself about how to solve the problem it comes up with some steps it goes through and it finally just produces the correct answer at the bottom so you don't have to use the chain of thought to look at if you don't want to from a research point of view it's quite interesting and the fact that it's open is really positive but actually this is a problem would have been hard for this model to solve if it didn't have that chain of thought because it would have just had to basically look at the colors and move them into the output and that would have been very very hard so this chain of thought is what makes problems like this a bit more possible to actually train it to solve that problem what you would do is you'd give it a bunch of box stacking problems a bunch of derivations and a bunch of solutions and step-by-step examples and then the answers at the bottom and over time it would learn to reproduce that that performance what r1 is doing is turning it on its head a bit and training only using the answers which is hugely easier because you need much less data you don't need to have crafted clever um you don't need to have crafted clever internal monologues the internal monologue comes out of the training process which is super cool so the way it works is you give it a question you and then you reward it so reinforcement learning is this idea where it doesn't directly observe the actual correct answer it gets a reward or it doesn't get a reward based on whether it was right or not right so you maybe you want to train a robot to walk along you don't say move your left leg this way what you do is reward it for getting a bit further and over time it might learn to walk along and so they've done this what they've done is they've given it a load of maths problems and a load of maths answers but they don't give it the answer they just tell it whether it was right and whether it was wrong and also they give it a small reward for having written some kind of internal monologue of the correct format and over time as it trains the monologue gets better and better and in the end it has a chat with itself and then solves the problem the really nice thing about it is that it's it's just much easier for someone like me to train a model like this right because because i can i just there are lots of data sets with questions in and answers there are very many fewer data sets with really nice step-by-step instructions on how to solve the problem because i don't know how to solve the problems either by the way and so you know it makes this and they've released it all open source so it makes it much much easier to do so you know two weeks ago open ai was maybe the only one of the only companies that could do something like this a handful of others and suddenly now you can kind of do it at home you need you will need to train one of these models from scratch you may need lots and lots of graphics cards but massively limited numbers compared to what we had before it so big organizations let's say the size of this university could quite comfortably do this now as opposed to it being totally out of reach which is a bit of a game changer and it's worth noting i'm skipping over some of the cool stuff that they do there's lots of stuff in this paper and we can link it in the in the description and there's loads of other ways that they train it in they do a multi-stage training process not just reinforcement learning to make it work a little bit better make it appear more aesthetically pleasing but you know in principle the idea is what they've done is they've released a very performant model and told us exactly how they did it right which is very unusual for these kind of models and so you know in my opinion a good thing this has sent silicon valley into a bit of a spin hasn't it yes um which from my point of view as someone not in silicon valley is quite quite funny sometimes right you know i think sometimes this is pictures a bit of an arms race between different countries or different companies i think it's only an arms race because they make it an arms race right the rest of us are just cracking on with our regular research you know and that's the true of most people um but i think if you have a company where you've your whole business model is around you have the best model and no one knows how it works and can copy you this really hurts that model right because they've got a good model and everyone knows how it works and can train it themselves right that's a huge problem the other problem is if you have a company like nvidia where your stock price is is almost entirely based on the fact that these big companies buy hundreds of thousands of incredibly expensive gpus because that's what's required to get the best performance and then someone comes along and gets the best performance with essentially consumer hardware that is also not a good look right now it might be that those companies that have loads of gpus still have an advantage for a while but it's a leveler right over time people can do stuff with more limited hardware and i think that's a great thing because i have access we have access here to some to dozens of gpus and they're decent right and they're expensive but they're not anywhere near in the same league as some of these companies and so we we essentially cannot try those things right because we so we do other things but now we can and we might i might still not but you know i i might and i and they're not it's an option for me you know so i think that's gonna it's gonna level the playing field a lot very very quickly and i think that once something like this starts lots of other companies will come up with new models will come up with new efficiency savings and it will just that that will increase more i we could be seeing the end of kind of closed source ai because it may just not be viable that if you just keep adding more and more data or bigger and bigger models or a combination of both ultimately you will move beyond just recognizing cats and you'll be able to do anything right that's the idea you show enough cats and dogs and eventually the elephant just is implied

Transcription

It's quite difficult to ask a large language model just what is the solution to this logic problem or what is the solution to this mathematical derivation and it just spit an answer out because that's not trivial to do right there's this steps that it's not that it's trying to skip over and so what chain of thought does is essentially write down a step-by-step process of solving the problem and slowly solve it and then write down the answer right and then you just kind of hide the chain of thought you can show it or you can not show it as you see fit but you tend to get much better at solving problems that require multiple steps if you want to just what is why is the sky blue it will just regurgitate that pretty easily from text it's learned on the internet but if you're asking like problem solving skills it's hard to do in one shot so you kind of take a little bit of time to just to just take you know to just work through it and this is essentially adding computational costs during inference but with the benefit that the performance goes up right now whether that cost is worth doing probably depends on the questions you're asking of it but it you know that's the idea now open ai pioneered this chain of thought um but they don't tell you how they do it right so so because it's all closed and so it's not open ai at all right in some sense so essentially you see a kind of pricey summary version of the chain of thought but it's not their internal actual internal monologue which is essentially a trade secret what r1 is doing is it's doing a chain of thought which is similar to o1 but it's fully public they've released all the models they've released all the code you can talk to it you can see the entire monologue and they've also trained it with a with massively more limited data so how would you train this if you were open ai or a big tech company well what you would do is you would you would give it you would create a data set that says here's a question here's the chain of thought you should have been producing for this question and here's the answer right and you have to produce tens of thousands or hundreds of thousands of examples of these kind of things what sorts things like like simple math problems problems right the question i often ask which has been failed by things like chat gpt before is suppose you have a stack of three boxes red blue and yellow the red is on the bottom on top of that is the blue on top of that is the yellow you take the blue one out and put it on the top and then you add a fourth green box onto the top can you describe that stack of boxes right and the answer is can you say it again because i need to write that down right that's that's the answer now i've asked chat gpt this and it often would fail at it until we got to the new reasoning models the new chain of thought models which start to perform a lot better on this kind of task and we can actually see that happening here on deep sea car one so if we look at the text it's actually started discussing with itself about how to solve the problem it comes up with some steps it goes through and it finally just produces the correct answer at the bottom so you don't have to use the chain of thought to look at if you don't want to from a research point of view it's quite interesting and the fact that it's open is really positive but actually this is a problem would have been hard for this model to solve if it didn't have that chain of thought because it would have just had to basically look at the colors and move them into the output and that would have been very very hard so this chain of thought is what makes problems like this a bit more possible to actually train it to solve that problem what you would do is you'd give it a bunch of box stacking problems a bunch of derivations and a bunch of solutions and step-by-step examples and then the answers at the bottom and over time it would learn to reproduce that that performance what r1 is doing is turning it on its head a bit and training only using the answers which is hugely easier because you need much less data you don't need to have crafted clever um you don't need to have crafted clever internal monologues the internal monologue comes out of the training process which is super cool so the way it works is you give it a question you and then you reward it so reinforcement learning is this idea where it doesn't directly observe the actual correct answer it gets a reward or it doesn't get a reward based on whether it was right or not right so you maybe you want to train a robot to walk along you don't say move your left leg this way what you do is reward it for getting a bit further and over time it might learn to walk along and so they've done this what they've done is they've given it a load of maths problems and a load of maths answers but they don't give it the answer they just tell it whether it was right and whether it was wrong and also they give it a small reward for having written some kind of internal monologue of the correct format and over time as it trains the monologue gets better and better and in the end it has a chat with itself and then solves the problem the really nice thing about it is that it's it's just much easier for someone like me to train a model like this right because because i can i just there are lots of data sets with questions in and answers there are very many fewer data sets with really nice step-by-step instructions on how to solve the problem because i don't know how to solve the problems either by the way and so you know it makes this and they've released it all open source so it makes it much much easier to do so you know two weeks ago open ai was maybe the only one of the only companies that could do something like this a handful of others and suddenly now you can kind of do it at home you need you will need to train one of these models from scratch you may need lots and lots of graphics cards but massively limited numbers compared to what we had before it so big organizations let's say the size of this university could quite comfortably do this now as opposed to it being totally out of reach which is a bit of a game changer and it's worth noting i'm skipping over some of the cool stuff that they do there's lots of stuff in this paper and we can link it in the in the description and there's loads of other ways that they train it in they do a multi-stage training process not just reinforcement learning to make it work a little bit better make it appear more aesthetically pleasing but you know in principle the idea is what they've done is they've released a very performant model and told us exactly how they did it right which is very unusual for these kind of models and so you know in my opinion a good thing this has sent silicon valley into a bit of a spin hasn't it yes um which from my point of view as someone not in silicon valley is quite quite funny sometimes right you know i think sometimes this is pictures a bit of an arms race between different countries or different companies i think it's only an arms race because they make it an arms race right the rest of us are just cracking on with our regular research you know and that's the true of most people um but i think if you have a company where you've your whole business model is around you have the best model and no one knows how it works and can copy you this really hurts that model right because they've got a good model and everyone knows how it works and can train it themselves right that's a huge problem the other problem is if you have a company like nvidia where your stock price is is almost entirely based on the fact that these big companies buy hundreds of thousands of incredibly expensive gpus because that's what's required to get the best performance and then someone comes along and gets the best performance with essentially consumer hardware that is also not a good look right now it might be that those companies that have loads of gpus still have an advantage for a while but it's a leveler right over time people can do stuff with more limited hardware and i think that's a great thing because i have access we have access here to some to dozens of gpus and they're decent right and they're expensive but they're not anywhere near in the same league as some of these companies and so we we essentially cannot try those things right because we so we do other things but now we can and we might i might still not but you know i i might and i and they're not it's an option for me you know so i think that's gonna it's gonna level the playing field a lot very very quickly and i think that once something like this starts lots of other companies will come up with new models will come up with new efficiency savings and it will just that that will increase more i we could be seeing the end of kind of closed source ai because it may just not be viable that if you just keep adding more and more data or bigger and bigger models or a combination of both ultimately you will move beyond just recognizing cats and you'll be able to do anything right that's the idea you show enough cats and dogs and eventually the elephant just is implied