The US Patent and Trademark Office today published a patent application from Apple that sheds some light on the matter a unique lens system design well suited for head-worn augmented reality devices that incorporates a new lens system with ‘lenslets’.‘ This could potentially revolutionize lens systems for AR glasses.
Apple states in its patent background that augmented reality (AR) or mixed reality (MR) systems merge virtual images with a view of the real world. In AR head-mounted devices (HMDs), virtual content is presented using an image projection system that is close to the eye. Real-world content can either be viewed directly with an “optical see-through” lens design, or rendered digitally with scene cameras for “see-through video.”
Current see-through video HMDs suffer from a shifted perspective of the real world because the scene cameras are placed in front of (in front of), above (above), and/or to the side (to the side) of the user’s eyes.
Scene cameras placed laterally to the eyes create an interpupillary distance (IPD) mismatch compared to the user’s eyes. As a result, the user may experience double vision, blurred vision, dizziness, headaches, nausea, and fatigue. This mismatch can also lead to misperceived distance and scaling of objects. Scene cameras placed above or in front of the eyes can create perspective that, with exaggeration, translates to head movement or false motion parallax. These effects can also cause discomfort to the user, and that’s what the patent granted by Apple aims to solve.
The Apple patent application describes a novel lens system having an entrance pupil at a location that matches or closely matches the entrance pupil of the eye’s optical system.
Apple’s lens system is significantly thinner than a traditional lens system, making it well-suited for head-mounted device (HMD) applications. The slimness of the system is achieved through the use of a number of camera elements.
Each camera element includes camera optics called “lenslets” and a sensor (e.g., the sensor could be an image sensor or a depth sensor), where a lenslet may include one or more lenses (e.g., a compound lens).
Individual camera elements can be arranged so that their optical axes intersect at or near the location of the eye’s entrance pupil. Also, the field of view (FOV) of each camera element corresponds to a small sector of a large FOV. The FOVs of adjacent camera elements may not or slightly overlap, so that a wide-angle image can be formed by concatenating the individual camera element images.
Apple’s patent FIG. 2 below shows a camera system whose individual elements are arranged along a spherical surface.
Apple’s patent FIG. 6A above illustrates an augmented reality (AR) HMD system #600 that includes multiple camera elements (represented by shaped elements) for each eye (e.g., 605L and 605R) arranged along a spherical surface, according to one or more embodiments are arranged, one or more display elements (No. 610L and No. 610R), one or more modules (No. 615L and No. 615R) and structure No. 620 to which the other components can be attached and which are on the head of a user can be attached. While only five camera elements are shown for each eye, more or less may be used in any given implementation depending on the intended use of the device and the target FOV.
Apple’s patent FIG. 6B above is another embodiment of AR-HMD system #625 that includes optical and electronic systems (6051, 6101, and modules 615) attached to single-eye structure (#630). In the illustrated system, electronics modules #6151 and #615R are shown, although neither may be required.
Interestingly, although not mentioned in the body of the patent, Apple’s claims state that “the computer-readable code identifies one or more objects in the scene includes computer-readable code for performing facial recognition to identify a person in the scene. This could be a key feature of Apple’s future glasses, and it’s unfortunate that Apple’s patent doesn’t elaborate on this. If we’re lucky, Apple may include more details of this feature in a secondary patent for Lenslets in the future.
See Apple’s patent application number 20220146820 for more details. As this is a patent application, the launch date of such a product is currently unknown.
Noah Bedard: prototyping engineer. Previously, he worked at Ricoh on prototypes of novel imaging systems.
Branko Petlyansky: Senior Engineer – Camera Electronics (Apple, Munich Germany). He previously worked at Panavision.
Catherine Berkner: Senior Engineering Manager – Camera Incubation. Incubates user experiences, imaging architectures and technologies to pave the future of camera applications.
Richard Motta: Distinguished Engineer 6 years: Creates and leads Camera Incubation, a team focused on developing new camera technology. Motta now works at Google.