Highlights from #GenNYC: PyTorch Co-Founder on The Evolution of PyTorch, celebrity attention for OpenAI, and the Future of the AI Race

Soumith Chintala lives and breathes generative AI — both inside and outside his role as VP/Fellow of Engineering at Meta, leading PyTorch and other Infra initiatives. Most notably, he’s the co-founder of PyTorch, one of the world’s most popular machine learning libraries. PyTorch started as a passion project, then took on a life of its own. Today, it powers most of the world’s AI research and products, including elements of the technology used by ChatGPT, Tesla, drug discovery companies, cancer research and NASA. 

At Lightspeed’s most recent #GenNYC meetup, in front of a packed house during New York Tech Week, Lightspeed Partner Michael Mignano sat down with Soumith to discuss PyTorch’s evolution, the future of the AI race, Scarlett Johanson’s saga with OpenAI and more. 

Here are some of the highlights from their conversation. Quotes have been edited for clarity. 

Soumith never planned for PyTorch to take off the way it did.

“We didn’t have a grand vision for what PyTorch could be in the world,” Soumith shared. “We built it for ourselves and it ended up being something that was useful for other people. At the same time, the addressable market has expanded a lot and that made it a bigger deal than I ever thought.”

In many ways, PyTorch’s early days followed the basic startup playbook. The team built a feedback loop to understand who was using the product and what support they needed. If users showcased their work with PyTorch online, the team amplified it. And over time, they built a community of people who were connected by their identities as PyTorch users. 

PyTorch evolved from being a great tool to an ecosystem in itself, backed by a network of support. Today, if you Google nearly any PyTorch problem you were grappling with, someone else has been there and shared the answer online. 

Soumith’s philosophy on AI at Meta: “If it helps the world, it probably helps us.”

Michael and Soumith discussed the role PyTorch played in the rollout and strategy of Llama — Meta’s family of open-source large language models that were released in early 2023. It turns out, Llama came extremely naturally to Meta. When it came time to release it as open source, it wasn’t a hard thought-out decision. 

“Meta’s FAIR research lab was started with the mission of advancing AI, with the thesis that advancing AI generally also helps Meta build better products and safety features. With open-sourcing Llama, the thinking is the same – we need to help advance Generative AI, getting researchers to understand these models better, people to build new experiences, and understand the limits of GenAI, and one of the great ways to help accelerate that is to open-source Llama”

The future of the AI race is a bit unknown right now.

“We’re in a state where we’re just starting to see that you can build more and more intelligent models by throwing more and more data at it — and [specifically] certain kinds of data,” Soumith shared. “We haven’t really reached the upper level of intelligence.” 

We also still need to discover how much intelligence is useful from a product perspective. He gave the hypothetical example of a 200b model vs 70b model — where the 200b model is 30 percent smarter but costs 10 times as much to serve. We need to consider that building the most intelligent model might provide a technological advantage, but it might not be viable from a cost perspective. 

Something else on Soumith’s mind when it comes to the AI race: the need for generative AI models to go through a “personalization” revolution. He shared how people are just starting to figure out that having a single prompt to serve all users across the world isn’t ideal. For example, ChatGPT has no understanding of local sensitivities. 

Legal disputes around Generative AI are a good thing. 

In May of this year, OpenAI started making headlines for an unusual reason: the company held a live demo of its new ChatGPT voice assistant. The voice, named “Sky,” sounded a lot like Scarlett Johansson. The actresses’ lawyers sent letters to OpenAI asking the company to disclose how the voice was developed. Ultimately, OpenAI removed the voice — but not before raising a lot of questions with the general public around what’s fair game for AI. 

Soumith’s take: “People in current society have rights based on various aspects: property, rights, copyrights, likeness rights. Some of those rights are being questioned by AI models. Various creative artists have been getting unhappy. I think we need moments like these to understand when AI is not expressing something that works according to current social norms. These moments are going to allow people to understand each other, organize with each other, and question the direction we want to go around preserving or not preserving these rights with generative AI.” 

Ultimately, he shared, it’s one of many examples that will test how society responds to technology. 

As Michael added, it could take a decade for this to play out in the courts. In the meantime, it will play out in the market. 

Will the future be closed source or open source? It’s hard to say.

Given Meta’s focus, you might expect Soumith to be all-in on open source. But even he can’t say for certain what the future holds. Ultimately, each has its advantages and disadvantages. 

“Open source AI is probably not going to be as well integrated and smooth as a vertically integrated product that uses OpenAI,” he shared. “But the advantages are that it’s way more democratized. Anyone can build a product around it without having to go through a bunch of agreements with a separate provider and doing things only in a certain way.”

In the open source category, he referenced the earlier example of AI accounting for local sensitivities. An AI provider based in California might have a certain set of biases that don’t work for a large part of the world. Open source AI can apply personalization to various markets much faster than closed source. 

Right now AI is a capital game. But it might not be that way forever.

Towards the end of their conversation, Michael asked Soumith: “Can anyone compete with Meta on the open source side?” After all, this is a capital game. To train each model requires hundreds of millions of dollars. And Meta has a lot of capital. 

Soumith’s take? “The current hypothesis around AI is that if you want to train a smarter model, you have to throw more data and compute at the model for it to get smarter. It’s not clear when it’s going to over-saturate. But it’s going to happen. I don’t know if it’s going to be a capital game until we capture all the energy in the universe.”

If AI stops being about the capital game and starts being about something else (like human feedback), it’s very possible that other open source companies will start catching up. 

Interested in attending a Generative Live event? Sign up here to receive notifications about future meetups, and be sure to listen to the Generative Now podcast, where AI builders talk about how they’re creating the future.