Invented by Samuel A. Miller, Rony Abovitz, Magic Leap Inc
The Magic Leap Inc invention works as followsOne embodiment relates to a system that allows two or more users interact in a virtual environment comprising virtual data. The system comprises a computer network that includes one or multiple computing devices. Each computing device has memory, processing circuitry and software that is stored in the memory at least partially and is executable by the processor circuitry in order to process a portion of virtual data. At least a part of the data is derived from a local first user’s virtual world.
Background for Recognizing Objects in a Passable World Model in Augmented or Virtual Reality Systems
Computers generate virtual and augmented environments using data that describe the environment. These data can describe objects that a user could sense or interact with. These objects can include rendered objects, audio played for the user’s hearing, or tactile feedback (or haptic feedback) for the user. Users can sense and interact with virtual and augmented realities through visual, audio and tactical means.
Embodiments” of the present invention relate to devices, methods and systems that facilitate virtual reality or augmented reality interactions for one or several users.
The user display device includes a housing frame that can be mounted on a person’s head, a pair of cameras to track eye movements and estimate the depth of focus using the tracked eye movement, a projector module with a light-generating mechanism for modifying projected light to make it appear in focus by adjusting the light generated based upon the depth of focal point, a lens attached to the housing, and a processor communicating with the projector module to send data related to the display image. The lens can include at least one transparent reflector positioned in front the user’s eye to bounce the projected lighting into the user?s eyes. The transparent mirror can selectively transmit light from the surrounding environment.
The user display device can also include a second set of cameras that are mounted on the housing frame and capture the field-of view image of each eye. The processor can calculate the head pose of the person based on these field-of view images.
The projection module can include a scanned-laser arrangement that modifies the projected light beam in relation to the display object according to the estimated depth. The projected light beam diameter may be less that 0.7 mm.
In one embodiment, a first pair of cameras can be infrared camera paired with an infrared source to track the movement of each user’s eye. The user display device can further include a sensor assembly that includes at least one sensor for sensing a user’s movement, location, direction, and orientation. At least one sensor can be an accelerometer or a digital compass. The processor can estimate the head pose of a user by using at least one of movement, location, direction, and orientation. The user display device can include a GPS system. The user display device can also include a haptic device that is communicatively connected to the projection module for tactile feedback. 20. The user display device can also include an environment sensing sensor to digitally reconstruct the environment of the user.
The processor can be connected to a computer system to send at least part of a virtual data world and to receive another part of it.
The user display device can include an audio speaker mounted on the head frame for sound output. The user display device can further include a microphone mounted on the housing frame for capturing sounds close to the user.
The projection module can modify another projected light that is associated with an object other than the display object so that it appears blurred. The processor can render frames at a rate at least 60 frames per seconds.
The display object can be at least one out of a virtual item, a rendered object, an image, and a video.
In another embodiment, the method involves tracking the eye movements of a user, estimating the depth of focus based on that tracked movement, modifying the light beam of an object displayed on a screen based on this estimated depth of focal point so that the object appears in sharp focus, and then projecting the modified beam into the eyes of the user. The projected light beam to the user’s eye may have a diameter less than 0.7mm.
The method can further include selectively allowing light to be transmitted from a user’s local environment based on the visualization mode of an object displayed. The visualization mode can be an augmented reality, virtual reality, or a combination thereof.
The method can also include capturing an image of the field of view of each user’s eye. The field-of-view image captured may be used to estimate the head pose of a user. The captured field of view image can be used to convert a physical object into a physically rendered object and display it to the user.
The method can further include extracting a group of points from the captured field of view image and creating a fiducial based on that extracted group of points for at least one object. The method can also include transmitting at least one extracted set and created fiducial, as well as tagging at least one extracted set and created fiducial with a type object. The method can further include recognizing that a physical object belonging to a type of item is based on the at least one tagged set associated with the object type and the tagged fiducial associated.
The method can also include sensing at the least one of an accelerometer, a compass, or a gyroscope. Sensors include an accelerometer, GPS, and gyroscope.
The method may also include processing virtual world information associated with a display object on a cloud network and transmitting a portion or all of the virtual data associated to the display to a user at a different location so that the user can experience at least part of the virtual data at the other location.
The method can also include sensing an object and altering, based upon a relationship predetermined with the object sensed, at least part of the virtual data associated with the object displayed. The method also includes presenting the virtual world data modified to the second user.
The method can also include modifying a light associated with an object other than the display object so that it appears blurred.
The method can further include receiving user input via a user interface and modifying the displayed object based on that input. The user interface can be at least one haptic device, keyboard, mouse, joystick, motion capture controllers, optical tracking devices, and audio input devices. Display objects may include a virtual object or rendered physical object as well as an image or video.
The method of interacting with virtual data in a virtual environment through a head mounted user display device includes rendering a display image associated at least with a portion to the user based on the estimated depth of focus for the user’s eye, creating additional virtual data from the interaction between the head mounted user device and the virtual data and the interaction with the physical environment of a user and transmitting that additional virtual data to a network. The virtual world can be displayed in either a two-dimensional or three-dimensional format.
The method can further include transmitting the additional virtual data for presentation to a user at a different location so that the user may experience the virtual data from the other location. The additional virtual data can be linked to a field of view image captured by the user display device mounted on a head. The additional virtual data can be linked to at least one of a sensed user movement, a Sensed user location, a Sensed user direction, and a Sensed user orientation. The additional virtual data can be associated with an object that is sensed by the user display head. The additional virtual data can be associated with a display object that has a predetermined relation to the sensed physical objects.
Click here to view the patent on Google Patents.