. Scientific Frontline: NUS scientists develop a novel light-field sensor for 3D scene construction with unprecedented angular resolution

Thursday, May 11, 2023

NUS scientists develop a novel light-field sensor for 3D scene construction with unprecedented angular resolution

Prof Liu Xiaogang (right) and Dr Yi Luying from the NUS Department of Chemistry capturing a 3D image of a model using the light-field sensor.
Photo Credit: Courtesy of National University of Singapore

Color-encoding technique for light-field imaging has potential applications in fields such as autonomous driving, virtual reality and biological imaging

A research team from the National University of Singapore (NUS) Faculty of Science, led by Professor Liu Xiaogang from the Department of Chemistry, has developed a 3D imaging sensor that has an extremely high angular resolution, which is the capacity of an optical instrument to distinguish points of an object separated by a small angular distance, of 0.0018o. This innovative sensor operates on a unique angle-to-color conversion principle, allowing it to detect 3D light fields across the X-ray to visible light spectrum.  

A light field encompasses the combined intensity and direction of light rays, which the human eyes can process to precisely detect the spatial relationship between objects. Traditional light sensing technologies, however, are less effective. Most cameras, for instance, can only produce two-dimensional images, which is adequate for regular photography but insufficient for more advanced applications, including virtual reality, self-driving cars, and biological imaging. These applications require precise 3D scene construction of a particular space.

For example, self-driving cars could use light-field sensing to view streets and more accurately assess road hazards so as to adjust their speed accordingly. Light-field sensing could also enable surgeons to accurately image a patient’s anatomy at varying depths, allowing them to make more precise incisions and better assess a patient’s risk of injury. 

“Currently, light-field detectors use an array of lenses or photonic crystals to obtain multiple images of the same space from many different angles. However, integrating these elements into semiconductors for practical use is complicated and costly,” explained Prof Liu.  “Conventional technologies can detect light fields only in the ultraviolet to visible light wavelength range, leading to limited applicability in X-ray sensing.”

In addition, compared with other light-field sensors such as microlens arrays, the NUS team’s light field sensor has a larger angular measurement range of more than 80 degrees, high angular resolution which can potentially be less than 0.015 degrees for smaller sensors, and a wider spectral response range of between 0.002 nm and 550 nm. These specifications make the novel sensor able to capture 3D images at higher depth resolution. 

The breakthrough was published in the prestigious journal Nature on 10 May 2023.

A large scale angle-sensing structure comprising nanocrystal phosphors, a key component of the sensor, illuminated under ultraviolet light. Three light emitting phosphors that produce red, green and blue light are arranged in a pattern to capture detailed angular information which is then used for 3D image construction. The team is looking into using other materials for the structure too.
Photo Credit: Courtesy of National University of Singapore

Made possible by perovskite nanocrystals

At the core of the novel light-field sensor are inorganic perovskite nanocrystals – compounds that have excellent optoelectronic properties. Due to their controllable nanostructures, perovskite nanocrystals are efficient light emitters, with an excitation spectrum that spans X-rays to visible light. The interactions between perovskite nanocrystals and light rays can also be tuned by carefully altering their chemical properties or by introducing small amounts of impurity atoms.

NUS researchers have patterned perovskite crystals onto a transparent thin-film substrate and integrated them into a color charge-coupled device (CCD), which converts incoming light signals into a color-coded output. This crystal-converter system comprises a basic functional unit of the light-field sensor.

When incident light hits the sensor, the nanocrystals become excited. In turn, the perovskite units emit their own light in varying colors depending on the angle at which the incoming light ray strikes. The CCD captures the emitted color, which can then be used for 3D image reconstruction.

“A single angle value, however, is not enough to determine the absolute position of the object in a three-dimensional space,” shared by Dr Yi Luying, Research Fellow at the NUS Department of Chemistry and the first author of the paper. “We discovered that adding another basic crystal converter unit perpendicular to the first detector and combining it with a designed optical system could provide even more spatial information regarding the object in question.”

They then tested their light-field sensor in proof-of-concept experiments and found that their approach can indeed capture 3D images — with accurate reconstructions of depth and dimension — of objects placed 1.5 meters away.

Their experiments also demonstrated the capacity of the novel light-field sensor to resolve even very fine details. For example, a precise image of a computer keyboard was created that even captured the shallow protrusions of individual keys.

Figure showing the design (left) and output (right) of 3D light-field sensor. The designed device (left) encodes the light field as color output. Patterned perovskite nanocrystals arrays convert different directions of light into different colors, which can be detected by a color charge-coupled device camera. The right image shows a reconstructed 3D depth image of a Merlion model produced by the camera.
Illustration Credit: Yi Luying

Future research

Prof Liu and his team are looking into methods to improve the spatial accuracy and resolution of their light-field sensor, such as using higher-end color detectors. The team has also applied for an international patent for the technology.

“We will also explore more advanced technologies to pattern perovskite crystals more densely onto the transparent substrate, which could lead to better spatial resolution. Using materials other than perovskite may also expand the detection spectrum of the light-field sensor,” said Prof Liu.

Published in journal: Nature 

Research MaterialX-ray-to-visible light-field detection through pixelated color conversion

Source/CreditNational University of Singapore

Reference Number: eng051123_01

Privacy Policy | Terms of Service | Contact Us

Featured Article

Hypoxia is widespread and increasing in the ocean off the Pacific Northwest coast

In late August, OSU's Jack Barth and his colleagues deployed a glider that traversed Oregon’s near-shore waters from Astoria to Coos Bay...

Top Viewed Articles