Over the past year, we’ve made substantial progress in the realm of artificial intelligence research and development at Somnium Space. This groundbreaking work has led to the implementation of an array of innovative features in the V3.0 update, paving the way for transformative user experiences and valuable scientific contributions.

A standout feature is the new Somnium Space AI & Data Systems. These revolutionary tools empower users to collect data and integrate machine learning and AI models into their VR experiences, serving a multitude of purposes — from content creation and training digital twins, to contributing to scientific projects and more.

In order to facilitate data sharing and analysis across various disciplines, we’re proud to introduce the Experience Data Format (EDF). This open standard has been designed to capture multimodal data about lived experiences in VR, creating a common language that unifies the realms of AI, science, art, and engineering.

The initial release supports recording of comprehensive data including full body motion, finger tracking, eye and face tracking, speech, hearing, vision, avatar and animation data, and even basic physiological data. This can be collected for any number of users interacting within our VR environment.

A crucial aspect of our AI and data system’s design is its commitment to user privacy and data ownership. At Somnium Space, we firmly believe in your right to control your own data. As such, WE DO NOT COLLECT OR STORE any user data on our servers. Instead, all data gathered from your VR experiences are stored locally on your own PC in the universal and sharable Experience Data Format (EDF).

This user-centric approach not only empowers you with complete control over your personal data but also opens up possibilities for decentralized and transparent data sharing or trading. Researchers seeking to gather data for various purposes can directly connect with users within the Somnium Space platform. Users can then choose to share or trade their locally stored data in a manner that is both consensual and transparent.

In essence, the Somnium Space Platform V3.0 offers a secure and decentralized solution for data collection, storage, sharing & trading, designed with your privacy and data ownership rights in mind. We’re excited to see the new avenues of exploration and discovery this will open for both users and researchers alike.

To maximize accessibility, we’ve developed EDF read and write tools in both Python and C#, with plans to create further tools to integrate with common workflows in machine learning. Our aim is to advance research into highly multimodal machine learning and virtual humans.

The flexibility and robustness of EDF comes from its format design, which consists of Apache Avro files and optional referenced media files, akin to the design of GLTF and other multi-modal files. The Avro files adhere to the EDF metaschema, which defines the structure and semantics of EDF records. We chose Apache Avro for its robustness, efficiency, compression capabilities, inter-process communication support, and schema evolution features.

The EDF metaschema is the backbone of the format, providing a detailed ontology of all possible data categories, including bio data, peripherals data, objects data, physical environment data, virtual environment data, and activity data. It also supports data recorded at different frequencies, organizing data into Avro files based on the frequency of change. This separation allows for more efficient storage by avoiding unnecessary repetition of static data.

To complement the new data system, we are adding an AI system to allow you to interact with AI characters and integrate AI models into your VR experiences. These characters can be customized using by allowing users to easily fine tune the models with your own data. With this tight and easy-to-use AI integration we hope more users will explore and use the latest AI advances as part of their own experiences!

Lastly, we’re thrilled to announce that these advancements are paving the way for the launch of the highly anticipated “Live Forever Mode Alpha” later this year. This mode will be a testament to our commitment to creating a platform that is as dynamic, engaging, and interactive as the users it serves.