Invented by Samuel A. Miller, Rony Abovitz, Magic Leap Inc

The market for recognizing objects in a passable world model in augmented or virtual reality systems is experiencing significant growth as these technologies continue to advance. Augmented reality (AR) and virtual reality (VR) have become increasingly popular in recent years, with applications ranging from gaming and entertainment to education and training. One of the key challenges in AR and VR systems is the ability to accurately recognize and interact with objects in the user’s environment. This is where the concept of a passable world model comes into play. A passable world model refers to a digital representation of the real world that can be traversed and interacted with in AR or VR. It allows users to seamlessly blend the virtual and real worlds, creating a more immersive and realistic experience. Recognizing objects within this passable world model is crucial for a variety of reasons. Firstly, it enables users to interact with virtual objects in a more natural and intuitive manner. For example, in a gaming scenario, users can physically pick up and manipulate virtual objects as if they were real. This enhances the overall user experience and makes interactions feel more realistic. Secondly, object recognition in a passable world model opens up new possibilities for applications in fields such as architecture, interior design, and industrial training. Professionals in these industries can use AR or VR systems to visualize and manipulate virtual objects within a real-world context. This can aid in the design process, allowing for better decision-making and more efficient workflows. The market for recognizing objects in a passable world model is driven by several factors. Firstly, the increasing adoption of AR and VR technologies across various industries is fueling the demand for advanced object recognition capabilities. As more businesses and individuals recognize the potential of these technologies, the need for accurate and reliable object recognition becomes paramount. Additionally, advancements in computer vision and machine learning algorithms are contributing to the growth of this market. These technologies enable AR and VR systems to analyze and interpret visual data in real-time, allowing for more accurate object recognition. As these algorithms continue to improve, the capabilities of object recognition in passable world models will also advance. Moreover, the COVID-19 pandemic has accelerated the adoption of AR and VR technologies in various sectors. With social distancing measures in place, businesses are increasingly turning to virtual solutions for training, collaboration, and customer engagement. Object recognition in passable world models can enhance these virtual experiences, making them more interactive and engaging. In terms of market players, several companies are at the forefront of developing object recognition technologies for AR and VR systems. These include tech giants like Google, Microsoft, and Facebook, as well as specialized startups such as Magic Leap and Unity Technologies. These companies are investing heavily in research and development to improve object recognition capabilities and create more immersive AR and VR experiences. In conclusion, the market for recognizing objects in a passable world model in augmented or virtual reality systems is witnessing significant growth. The demand for advanced object recognition capabilities is driven by the increasing adoption of AR and VR technologies across various industries. Advancements in computer vision and machine learning algorithms, as well as the impact of the COVID-19 pandemic, are further propelling the market forward. As these technologies continue to evolve, the possibilities for object recognition in passable world models are bound to expand, opening up new opportunities for innovation and growth.

The Magic Leap Inc invention works as follows

One embodiment relates to a system that allows two or more users interact in a virtual environment comprising virtual data. The system comprises a computer network that includes one or multiple computing devices. Each computing device has memory, processing circuitry and software that is stored in the memory at least partially and is executable by the processor circuitry in order to process a portion of virtual data. At least a part of the data is derived from a local first user’s virtual world.

Background for Recognizing Objects in a Passable World Model in Augmented or Virtual Reality Systems

Computers generate virtual and augmented environments using data that describe the environment. These data can describe objects that a user could sense or interact with. These objects can include rendered objects, audio played for the user’s hearing, or tactile feedback (or haptic feedback) for the user. Users can sense and interact with virtual and augmented realities through visual, audio and tactical means.

Embodiments” of the present invention relate to devices, methods and systems that facilitate virtual reality or augmented reality interactions for one or several users.

The user display device includes a housing frame that can be mounted on a person’s head, a pair of cameras to track eye movements and estimate the depth of focus using the tracked eye movement, a projector module with a light-generating mechanism for modifying projected light to make it appear in focus by adjusting the light generated based upon the depth of focal point, a lens attached to the housing, and a processor communicating with the projector module to send data related to the display image. The lens can include at least one transparent reflector positioned in front the user’s eye to bounce the projected lighting into the user?s eyes. The transparent mirror can selectively transmit light from the surrounding environment.

The user display device can also include a second set of cameras that are mounted on the housing frame and capture the field-of view image of each eye. The processor can calculate the head pose of the person based on these field-of view images.

The projection module can include a scanned-laser arrangement that modifies the projected light beam in relation to the display object according to the estimated depth. The projected light beam diameter may be less that 0.7 mm.

In one embodiment, a first pair of cameras can be infrared camera paired with an infrared source to track the movement of each user’s eye. The user display device can further include a sensor assembly that includes at least one sensor for sensing a user’s movement, location, direction, and orientation. At least one sensor can be an accelerometer or a digital compass. The processor can estimate the head pose of a user by using at least one of movement, location, direction, and orientation. The user display device can include a GPS system. The user display device can also include a haptic device that is communicatively connected to the projection module for tactile feedback. 20. The user display device can also include an environment sensing sensor to digitally reconstruct the environment of the user.

The processor can be connected to a computer system to send at least part of a virtual data world and to receive another part of it.

The user display device can include an audio speaker mounted on the head frame for sound output. The user display device can further include a microphone mounted on the housing frame for capturing sounds close to the user.

The projection module can modify another projected light that is associated with an object other than the display object so that it appears blurred. The processor can render frames at a rate at least 60 frames per seconds.

The display object can be at least one out of a virtual item, a rendered object, an image, and a video.

In another embodiment, the method involves tracking the eye movements of a user, estimating the depth of focus based on that tracked movement, modifying the light beam of an object displayed on a screen based on this estimated depth of focal point so that the object appears in sharp focus, and then projecting the modified beam into the eyes of the user. The projected light beam to the user’s eye may have a diameter less than 0.7mm.

The method can further include selectively allowing light to be transmitted from a user’s local environment based on the visualization mode of an object displayed. The visualization mode can be an augmented reality, virtual reality, or a combination thereof.

The method can also include capturing an image of the field of view of each user’s eye. The field-of-view image captured may be used to estimate the head pose of a user. The captured field of view image can be used to convert a physical object into a physically rendered object and display it to the user.

The method can further include extracting a group of points from the captured field of view image and creating a fiducial based on that extracted group of points for at least one object. The method can also include transmitting at least one extracted set and created fiducial, as well as tagging at least one extracted set and created fiducial with a type object. The method can further include recognizing that a physical object belonging to a type of item is based on the at least one tagged set associated with the object type and the tagged fiducial associated.

The method can also include sensing at the least one of an accelerometer, a compass, or a gyroscope. Sensors include an accelerometer, GPS, and gyroscope.

The method may also include processing virtual world information associated with a display object on a cloud network and transmitting a portion or all of the virtual data associated to the display to a user at a different location so that the user can experience at least part of the virtual data at the other location.

The method can also include sensing an object and altering, based upon a relationship predetermined with the object sensed, at least part of the virtual data associated with the object displayed. The method also includes presenting the virtual world data modified to the second user.

The method can also include modifying a light associated with an object other than the display object so that it appears blurred.

The method can further include receiving user input via a user interface and modifying the displayed object based on that input. The user interface can be at least one haptic device, keyboard, mouse, joystick, motion capture controllers, optical tracking devices, and audio input devices. Display objects may include a virtual object or rendered physical object as well as an image or video.

The method of interacting with virtual data in a virtual environment through a head mounted user display device includes rendering a display image associated at least with a portion to the user based on the estimated depth of focus for the user’s eye, creating additional virtual data from the interaction between the head mounted user device and the virtual data and the interaction with the physical environment of a user and transmitting that additional virtual data to a network. The virtual world can be displayed in either a two-dimensional or three-dimensional format.

The method can further include transmitting the additional virtual data for presentation to a user at a different location so that the user may experience the virtual data from the other location. The additional virtual data can be linked to a field of view image captured by the user display device mounted on a head. The additional virtual data can be linked to at least one of a sensed user movement, a Sensed user location, a Sensed user direction, and a Sensed user orientation. The additional virtual data can be associated with an object that is sensed by the user display head. The additional virtual data can be associated with a display object that has a predetermined relation to the sensed physical objects.

Click here to view the patent on Google Patents.