Building the Metaverse with Jon Radoff

Perplexity allows you to ask questions and receive answers supported by citations from online sources: Wikipedia-meets-AI. I spoke with Aravind Srinivas, founder and CEO of Perplexity.ai, the company behind it. They use a combination of GPT, good UI/UX and a lot of behind the-scenes magic to equip you with an artificial intelligence that can make you more informed, smarter--and maybe even more compassionate.

Ali El Rhermoul and Jon Radoff's team at the a16z Tech Week AI/Virtual Worlds Hackathon won grand prize: it's a multiplayer online game featuring a persistent world, generative AI for storytelling (Anthropic Claude) and immersive environments (Blockade Labs, Scenario) and a client/server architecture using Beamable + Unity to create a persistent world using a new technique called "semantic programming," which uses an XML-based format to provide consistency between generative and game components.

I spoke with Adam B. Levine, cofounder of Blockade Labs, the company responsible for the generative skybox AI that can create 3D skyboxes from text prompts or sketches. One of the best applications of this technology is created environments for use in game engines like Unity and Unreal. We explored some demos of what this can do covered, and discussed the future of games, creativity and technology. Beyond the technology and its applications, we discussed the merits of decentralized AI vs. decentralized approaches ] (i.e., open-source models like Stable Diffusion compared to API-centric models like OpenAI).

Voice Synthesis is about to transform game development, virtual worlds and simulations of all kinds. I spoke with Zach Johnson, cofounder of LMNT, an AI voice synthesis company that uses the magic of prosody imitation to make voices that sound strikingly like the original speaker. We covered how these technologies work; demonstrated cases with David Attenborough and a flight simulator; discussed the new type of creator economy this will form; technologies to enable scalability and reliability (such as memory safe languages, e.g., Rust) as well as other things Zach learned from his time working on Google Glass.

One of the fastest growing areas of creative expression in games has been the emotes and animations you can apply to your character in games like Fortnite. Yassine Tahi saw this realm of self-expression coming, and created a platform that democratizes the creation of character animations. With Kinetix, individual creators can capture movement and make their avatar animations available through a marketplace--and end-users can invoke them through text-to-animation generative prompting. This requires a huge number of challenging technology problems to be solved: motion capture with a normal camera that maps to the character rig; adjusting animation to the geometry and physics of an environment; labeling the data in a way that generative AI technology can make use of it; and then making animations interoperable across different games, virtual worlds and metaverse platforms. Some of the games and platforms we discuss include Zepeto, Inworld, The Sandbox Fortnite and NextDancer.

Lovelace Studios is building Nyric, a world-generation platform that starts with a text prompt and brings entire worlds to life. I spoke with Kayla Comalli (co-founder and CEO) about how they're doing this. This is a great episode for anyone who is interested in learning how you can combine generative AI technologies like ChatGPT, Polyhive, Sloyd and Convai -- along with Unreal's Nanite and Lumen capabilities to speak entire worlds into existence. (Note: this is an episode you may want to watch on Metavert.tv because the clips of the technologies used are really interesting).

Linus Ekenstam is a long-time product leader, AI educator, and the creator of Bedtimestory.ai. Parents use Bedtimestory to create personalized stories for their children. We discussed how language models enable new types of products that deliver higher levels of personalization and new forms of narrative structure; his learnings from product leadership at companies such as Typeform. We also spent time on the big picture: what it means to transition from the Age of Information to the Age of Intelligence; whether Prompt Engineering is an actual job; and how AI might help restore nuance to the currently-polarized conversations that pervade the internet.

Generative AI is accelerating creativity: complicated creative tasks that once took hours can now take only a minute. My conversation with @bilawal spans the transformative impact of creative tools like Midjourney, Neural Radiance fields, ControlNet, Ebsynth, phone-based photogrammetry, and the potential to "reskin reality" as generative technologies fuse with augmented reality. We also talk about his journey and success as a creatorpreneur, with words of wisdom for anyone hoping to combine their creative passion with entrepreneurship...and finish up with some of the risks and opportunities around generative artificial intelligence, as well as our different viewpoints around centralized vs. decentralized technologies.

Cameron Wills used ChatGPT to make a ChatGPT plugin for Unity, which became #1 in AI, and #2 overall on the Unity Asset Store. We talked about his experience as an indie game developer, his Unity asset for making games with ChatGPT, and the how generative AI is changing the craft of game-making for Indies.

Kris Kashtanova is an AI artist who published the world's first copyrighted AI-assisted comic book. Her journey with the US Copyright Office has been documented in the news. I spoke to her about her work as an artist and photographer, what she learned about copyrighting AI art, the safe community she is building--and some of her newer work around ControlNet and training her own models.