Invented by Benjamin GROSSMANN, Robert LEGATO, Disney Enterprises Inc
The Disney Enterprises Inc invention works as followsThe disclosure is directed at collaborative multi-modal mixed reality systems and methods that leverage reconfigurable tangible interfaces to produce immersive, cinematic and interactive content. Content can be produced using a combination of computer generated and live action media in context, or entirely computer-generated content. The technology described in this document may use physical and software-based interfaces to merge the real world with the virtual one. This allows users to create new forms of content that mimic traditional processes.
Background for Collaborative Multi-Modal Mixed-Reality System and Methods leveraging reconfigurable tactile user interfaces for immersive, cinematic and interactive content production
The disclosure is directed at collaborative multi-modal mixed reality systems and methods that leverage reconfigurable tangible interfaces to produce immersive, cinematic and interactive content.
The method in one embodiment includes initializing a VR client device instance, loading scene files, and dependencies; rendering a world based on loaded scene files, and dependencies; and making changes to scene files by the VR device using a media production tool that is rendered within the virtual world, or a media production device communicatively connected to the VR device.
In some implementations, receiving data corresponding user input activating a control on the physical media production device and registering that data with the virtual environment includes tracking movement of the physical medium production tool using a motion sensor mounted on the physical medium production tool. In some implementations receiving data corresponding user input actuating control of the media production device and registering data with the world includes tracking movement of the media production using a sensor mounted on the media production; encoding tracked movement and transmitting encoded tracked movements to the VR devices for registration with virtual world.
In some implementations, a VR client device may include a VR head mounted display (HMD), and the virtual world will be displayed on the HMD. The VR client device can then make changes to scene files by using a controller that is communicatively connected to the HMD. In these implementations receiving data corresponding user input actuating the control of the physical production tool, and registering data with the virtual environment may include mounting the handheld on a mount that is coupled with the production tool. Tracking movement of the production tool with a motion detector of the mounted handheld controller.
In some implementations, physical media production tools include a videocamera, a dolly or camera crane or tripod. The virtual media production tools include a virtual videocamera, a virtual dolly or camera crane or tripod. In some implementations, a physical media tool is a camera-dolly. Receiving data corresponding user input, actuating the control of the physical tool, includes moving the dolly along the dolly track. Registering the data with the Virtual World includes registering movement of the dolly at a virtual world attachment point.
In some implementations, registering data with the virtual environment includes: passing data corresponding user input activating a control on the physical media production tool to a node-graph; and evaluating node-graph using the input.
In some implementations, a method may also include: displaying in the rendered virtual environment a user interface for selecting virtual media tools; receiving user input via the controller of the VR Headset to select the virtual production tool; and executing and rendering the virtual production tool. The method can also include receiving data from a user who activates a virtual control to change the state of a virtual media production device, and making the corresponding changes to a physical production tool communicatively connected to the VR HMD.
In some implementations, this method also includes: The client device transmitting changes to scene files to the server; the server broadcasting changes to scene files to multiple VR client devices in order to synchronize the virtual world rendered on the client device with that rendered by other VR clients devices.
The system can render a virtual environment for the VR Headset based on loaded scene files, and dependencies.
In one embodiment, the method comprises: consolidating scene and dependencies on a master media creation device; transmitting these scene and dependencies, from the master to each of a plurality VR client production devices (which includes a plurality VR HMD); transmitting this scene and dependency information from the master to a server. Each of the multiple VR client production devices loads the scene and dependencies in order to render a virtual environment, with each VR HMD displaying a view of that rendered virtual environment.
In some implementations, changes can be synchronized by the server in real time. In some implementations the master media creation device can transmit an instruction to all the VR client media creation devices to start recording changes to the scene files.
The term “virtual reality” is used in this document. “As used herein, the term?virtual reality? “VR” is a term that refers to the simulation of an individual’s presence within a real or imagined environment.
The terms “augmented reality” as used in this document are not to be confused with the term “augmented reality”. ?AR,? ?mixed reality,? “Mixed Reality” (MR) and “MR” refer to a view of a physical, real-world environment that is augmented or supplemented by computer-generated or digital information such as video, sound, and graphics. The terms?MR? and?MR? refer to a view or a real-world physical environment that has been enhanced or complemented by digital or computer-generated information, such as sound, graphics, or video. Digital information can be registered directly in the physical environment of the user, allowing the user to interact with it in real time. Digital information can be in the form of audio, images, haptic feedbacks, videos, texts, etc. “For example, three-dimensional digital objects can be superimposed in real time over the user’s view on the real world environment.
The following detailed description will reveal other features and aspects, when taken together with the accompanying illustrations, which show, as an example, features according to embodiments of this disclosure. The summary does not limit the scope or the disclosure claimed, which are defined by the attached claims.
Creating content which is a mix of live action media, (e.g. photographs or footage filmed by a camera of the real world) and computer-generated imagery has always been challenging because it’s difficult to see these images combined together into a final? Composite images can be created in real-time. It is difficult to see subjects “in context” when using this feature. Or ‘as a whole picture? In the past, it wasn’t possible to display the visual effects, which were computer-generated images that would be added later, in conjunction with live action footage. Previously, it was not possible to show the visual effects (computer-generated images to be added later) in conjunction with live action footage. This forced people to use imagination to imagine things that weren’t there. Everyone imagines different things, so their rational minds have to put together the final images conceptually. This hinders their ability to use subconscious instincts, resulting in unnatural or lower-quality results. The communication and coordination required by large groups to achieve a consistent visual over time was also excessive.
Furthermore creating computer-generated material has been difficult because the interface to create that content is typically only within a computer. People can only use computer monitors to see the virtual world and keyboards and mouse as substitutes for physical presence and hands. In order to perform similar tasks, the computer user must translate their knowledge of the tools they use in the real world to the tools that are available on the computer.
Embodiments” described herein refer to systems and methods which enable users (e.g. immersive, interactive and/or cinematic videos) to create content, either by combining live action media with computer generated media in context, or entirely using computer-generated media. The technology described in this document may use physical and software-based interfaces to merge the real world with the virtual one. This allows users to create new forms of content that mimic traditional processes. The system can allow users to import, modify and manipulate objects in three-dimensional spaces, change their appearance, function and/or record all changes over time.
The systems described herein enable both the physical and virtual worlds to use the exact same tools. Users can view both worlds simultaneously fused together, without having to look through a monitor. The users can enter the virtual world with their skills, knowledge, and tools from the physical one. They can bring out the data needed to create virtual worlds that are accessible in other systems.
Embodiments” described herein can be implemented by software running on one or more connected personal computers. Users can interact with the system via a variety of interfaces, from “outside” to “inside”. Users can interact with the system through a range of interfaces, from?outside? Through a combination of hardware and software, the virtual world can be accessed. The system can also include hardware input devices, such as traditional film production equipment and user input devices modified to transmit functionality and position. The system can also output a number of views that are scalable to be displayed on devices such as virtual reality or augmented-reality head-mounted displays, traditional television monitors, or computer displays.
The outputs of this system may include video/audio recordings compatible with standard video/audio hardware, data recordings (including sensor/input/positional data) that can be subsequently imported into 3D digital content creation programs for use or modification, and metadata, which provides information on the data created and the way it was modified by the system.
FIG. FIG. 1 shows an example of a virtual media production system that may be used to implement embodiments described in this document. The virtual media production system is shown to include a master or central system 110, an asset database 120, a server for synchronization 130, as well as a number of client devices 140 150 160 and 170 which may communicate over a communication network 190 with the master 110 or other client devices or server 130.
Click here to view the patent on Google Patents.