Invented by Benjamin GROSSMANN, Robert LEGATO, Disney Enterprises Inc

enhanced user experience. In recent years, there has been a significant rise in the demand for immersive technologies that offer users a more interactive and engaging experience. One such technology that has gained traction is the Collaborative Multi-Modal Mixed-Reality System, which combines virtual reality (VR), augmented reality (AR), and tactile user interfaces to create a truly immersive and interactive environment. The market for Collaborative Multi-Modal Mixed-Reality System and Methods leveraging reconfigurable tactile user interfaces is expected to witness substantial growth in the coming years. This can be attributed to the increasing adoption of immersive technologies across various industries, including gaming, entertainment, education, healthcare, and manufacturing. One of the key factors driving the market growth is the enhanced user experience offered by these systems. By leveraging reconfigurable tactile user interfaces, users can not only see and hear the virtual environment but also feel and interact with it. This adds a new dimension to the overall experience, making it more realistic and engaging. Moreover, the collaborative nature of these systems allows multiple users to interact with each other in the same virtual environment. This opens up new possibilities for team collaboration, training simulations, and remote communication. For example, in the healthcare industry, surgeons can collaborate and practice complex procedures in a virtual environment, reducing the risk of errors during real surgeries. The gaming and entertainment industry is also expected to drive the demand for Collaborative Multi-Modal Mixed-Reality Systems. With the increasing popularity of multiplayer games, these systems offer a unique and immersive gaming experience. Players can interact with virtual objects, feel the impact of their actions, and communicate with other players in real-time. Furthermore, the education sector is another potential market for these systems. By leveraging tactile user interfaces, educators can create interactive and engaging learning experiences for students. For instance, students can explore historical sites, perform virtual science experiments, or practice language skills in a realistic virtual environment. In terms of technology, the market for Collaborative Multi-Modal Mixed-Reality Systems is witnessing advancements in reconfigurable tactile user interfaces. These interfaces are designed to provide haptic feedback, allowing users to feel the texture, shape, and weight of virtual objects. This technology is crucial for creating a truly immersive and realistic experience. However, there are certain challenges that need to be addressed for the widespread adoption of these systems. One of the major challenges is the cost associated with the hardware required for these systems. High-end VR and AR devices, along with reconfigurable tactile user interfaces, can be expensive, limiting their accessibility to a wider audience. Additionally, there is a need for standardization and interoperability among different platforms and devices. Currently, there are several competing technologies and platforms in the market, which can create compatibility issues and hinder collaboration between different systems. In conclusion, the market for Collaborative Multi-Modal Mixed-Reality System and Methods leveraging reconfigurable tactile user interfaces is poised for significant growth in the coming years. The enhanced user experience, coupled with the collaborative nature of these systems, makes them ideal for various industries, including gaming, entertainment, education, healthcare, and manufacturing. However, addressing the cost and compatibility challenges will be crucial for the widespread adoption of these systems.

The Disney Enterprises Inc invention works as follows

The disclosure is directed at collaborative multi-modal mixed reality systems and methods that leverage reconfigurable tangible interfaces to produce immersive, cinematic and interactive content. Content can be produced using a combination of computer generated and live action media in context, or entirely computer-generated content. The technology described in this document may use physical and software-based interfaces to merge the real world with the virtual one. This allows users to create new forms of content that mimic traditional processes.

Background for Collaborative Multi-Modal Mixed-Reality System and Methods leveraging reconfigurable tactile user interfaces for immersive, cinematic and interactive content production

The disclosure is directed at collaborative multi-modal mixed reality systems and methods that leverage reconfigurable tangible interfaces to produce immersive, cinematic and interactive content.

The method in one embodiment includes initializing a VR client device instance, loading scene files, and dependencies; rendering a world based on loaded scene files, and dependencies; and making changes to scene files by the VR device using a media production tool that is rendered within the virtual world, or a media production device communicatively connected to the VR device.

In some implementations, receiving data corresponding user input activating a control on the physical media production device and registering that data with the virtual environment includes tracking movement of the physical medium production tool using a motion sensor mounted on the physical medium production tool. In some implementations receiving data corresponding user input actuating control of the media production device and registering data with the world includes tracking movement of the media production using a sensor mounted on the media production; encoding tracked movement and transmitting encoded tracked movements to the VR devices for registration with virtual world.

In some implementations, a VR client device may include a VR head mounted display (HMD), and the virtual world will be displayed on the HMD. The VR client device can then make changes to scene files by using a controller that is communicatively connected to the HMD. In these implementations receiving data corresponding user input actuating the control of the physical production tool, and registering data with the virtual environment may include mounting the handheld on a mount that is coupled with the production tool. Tracking movement of the production tool with a motion detector of the mounted handheld controller.

In some implementations, physical media production tools include a videocamera, a dolly or camera crane or tripod. The virtual media production tools include a virtual videocamera, a virtual dolly or camera crane or tripod. In some implementations, a physical media tool is a camera-dolly. Receiving data corresponding user input, actuating the control of the physical tool, includes moving the dolly along the dolly track. Registering the data with the Virtual World includes registering movement of the dolly at a virtual world attachment point.

In some implementations, registering data with the virtual environment includes: passing data corresponding user input activating a control on the physical media production tool to a node-graph; and evaluating node-graph using the input.

In some implementations, a method may also include: displaying in the rendered virtual environment a user interface for selecting virtual media tools; receiving user input via the controller of the VR Headset to select the virtual production tool; and executing and rendering the virtual production tool. The method can also include receiving data from a user who activates a virtual control to change the state of a virtual media production device, and making the corresponding changes to a physical production tool communicatively connected to the VR HMD.

In some implementations, this method also includes: The client device transmitting changes to scene files to the server; the server broadcasting changes to scene files to multiple VR client devices in order to synchronize the virtual world rendered on the client device with that rendered by other VR clients devices.

The system can render a virtual environment for the VR Headset based on loaded scene files, and dependencies.

In one embodiment, the method comprises: consolidating scene and dependencies on a master media creation device; transmitting these scene and dependencies, from the master to each of a plurality VR client production devices (which includes a plurality VR HMD); transmitting this scene and dependency information from the master to a server. Each of the multiple VR client production devices loads the scene and dependencies in order to render a virtual environment, with each VR HMD displaying a view of that rendered virtual environment.

In some implementations, changes can be synchronized by the server in real time. In some implementations the master media creation device can transmit an instruction to all the VR client media creation devices to start recording changes to the scene files.

The term “virtual reality” is used in this document. “As used herein, the term?virtual reality? “VR” is a term that refers to the simulation of an individual’s presence within a real or imagined environment.

The terms “augmented reality” as used in this document are not to be confused with the term “augmented reality”. ?AR,? ?mixed reality,? “Mixed Reality” (MR) and “MR” refer to a view of a physical, real-world environment that is augmented or supplemented by computer-generated or digital information such as video, sound, and graphics. The terms?MR? and?MR? refer to a view or a real-world physical environment that has been enhanced or complemented by digital or computer-generated information, such as sound, graphics, or video. Digital information can be registered directly in the physical environment of the user, allowing the user to interact with it in real time. Digital information can be in the form of audio, images, haptic feedbacks, videos, texts, etc. “For example, three-dimensional digital objects can be superimposed in real time over the user’s view on the real world environment.

The following detailed description will reveal other features and aspects, when taken together with the accompanying illustrations, which show, as an example, features according to embodiments of this disclosure. The summary does not limit the scope or the disclosure claimed, which are defined by the attached claims.

Creating content which is a mix of live action media, (e.g. photographs or footage filmed by a camera of the real world) and computer-generated imagery has always been challenging because it’s difficult to see these images combined together into a final? Composite images can be created in real-time. It is difficult to see subjects “in context” when using this feature. Or ‘as a whole picture? In the past, it wasn’t possible to display the visual effects, which were computer-generated images that would be added later, in conjunction with live action footage. Previously, it was not possible to show the visual effects (computer-generated images to be added later) in conjunction with live action footage. This forced people to use imagination to imagine things that weren’t there. Everyone imagines different things, so their rational minds have to put together the final images conceptually. This hinders their ability to use subconscious instincts, resulting in unnatural or lower-quality results. The communication and coordination required by large groups to achieve a consistent visual over time was also excessive.

Furthermore creating computer-generated material has been difficult because the interface to create that content is typically only within a computer. People can only use computer monitors to see the virtual world and keyboards and mouse as substitutes for physical presence and hands. In order to perform similar tasks, the computer user must translate their knowledge of the tools they use in the real world to the tools that are available on the computer.

Embodiments” described herein refer to systems and methods which enable users (e.g. immersive, interactive and/or cinematic videos) to create content, either by combining live action media with computer generated media in context, or entirely using computer-generated media. The technology described in this document may use physical and software-based interfaces to merge the real world with the virtual one. This allows users to create new forms of content that mimic traditional processes. The system can allow users to import, modify and manipulate objects in three-dimensional spaces, change their appearance, function and/or record all changes over time.

The systems described herein enable both the physical and virtual worlds to use the exact same tools. Users can view both worlds simultaneously fused together, without having to look through a monitor. The users can enter the virtual world with their skills, knowledge, and tools from the physical one. They can bring out the data needed to create virtual worlds that are accessible in other systems.

Embodiments” described herein can be implemented by software running on one or more connected personal computers. Users can interact with the system via a variety of interfaces, from “outside” to “inside”. Users can interact with the system through a range of interfaces, from?outside? Through a combination of hardware and software, the virtual world can be accessed. The system can also include hardware input devices, such as traditional film production equipment and user input devices modified to transmit functionality and position. The system can also output a number of views that are scalable to be displayed on devices such as virtual reality or augmented-reality head-mounted displays, traditional television monitors, or computer displays.

The outputs of this system may include video/audio recordings compatible with standard video/audio hardware, data recordings (including sensor/input/positional data) that can be subsequently imported into 3D digital content creation programs for use or modification, and metadata, which provides information on the data created and the way it was modified by the system.

FIG. FIG. 1 shows an example of a virtual media production system that may be used to implement embodiments described in this document. The virtual media production system is shown to include a master or central system 110, an asset database 120, a server for synchronization 130, as well as a number of client devices 140 150 160 and 170 which may communicate over a communication network 190 with the master 110 or other client devices or server 130.

Click here to view the patent on Google Patents.