Ilya Sutskever 说的“模型就是想学习”到底是什么意思？

It was just before OpenAI started that I met Ilya, who you interviewed. One of the first things he said to me was — “Look. The models, they just want to learn. You have to understand this. The models, they just want to learn.” And it was a bit like a Zen Koan. I listened to this and I became enlightened.

And over the years, I would be the one who would formalize a lot of these things and kind of put them together, but what that told me was that the phenomenon that I'd seen wasn't just some random thing. It was broad. It was more general. The models just want to learn. You get the obstacles out of their way. You give them good data, you give them enough space to operate in, you don't do something stupid like condition them badly numerically, and they want to learn. They'll do it.

I'm actually not entirely satisfied with Dario's elaboration! If Ilya meant precisely "given sufficient resources, do your job, and the models will do theirs [during training]", I think he would have phrased his advice slightly differently. What does it mean to "want" to learn? Why personify the model in that way, and not the optimizer or gradient descent itself or something? And I don't get why the koan would necessarily imply that scaling would have worked so well.

I realize that Ilya likes to cultivate an air of prophecy about himself (and I'd be lying if I said it didn't work on me). But it's just such a strange way to communicate the message, that I can't help but think the message must have been about something broader.

Possibly much more mundane in context, this must have been 10 years ago. Idk I wonder what it must have been like to be on the frontier at that time, maybe I take such things for granted now.

submitted by /u/XTXinverseXTY
[link] [留言]