Description: In this project, Applied Research LLC (ARLLC), the University of Tennessee, Knoxville (UTK), and the Arizona State University (ASU) propose high performance image processing algorithms that will support current and future Mastcam imagers. The algorithms fuse the acquired Mastcam stereo images at different wavelengths to generate multispectral image cubes, which can then be used for high quality virtual reality (immersive) visualization in 3D, data clustering, anomaly detection, and rough composition estimation from relatively long distance when compared to LIBS instrument. One major challenge in constructing a multispectral image cube from the two Mastcams is the alignment of the images in the stereo image pair which needs to have registration errors in the subpixel level. To address the challenge in the stereo image alignment, we propose a two-step image registration framework. In this framework, we also provide a set of image processing techniques, including pansharpening, debayering, data clustering, and anomaly detection. In Phase II, we will further validate the above algorithms using Mastcam-Z data. Moreover, we will develop new data products for generating 12-band image cubes, high resolution stereo images, and new layers for Java Mission-planning and Analysis for Remote Sensing (JMARS).

Benefits: Our system can be used in NASA's existing and future planetary rover missions. The new Mars rover that is expected to be sent to Mars in 2020 will also contain stereo Mastcam instrument that has Mastcam-Z. In addition, the two-step registration approach can be used in remote sensing applications for environmental monitoring and for damage assessment after a natural disaster where precise registration is critical. Moreover, the pansharpening algorithms will be useful for fusing Landsat and MODIS images, and THEMIS and TES images. We will create new layers (stereo and 3D images, 12-band high resolution Mastcam images) in JMARS so that scientists can interact with JMARS to have better user experiences. Finally, we will also create a data product related to the high resolution 12-band Mastcam image in PDS.

Our technology will be useful for border monitoring and security monitoring using high resolution images. It can also be useful in the biometrics (face detection and recognition) using stereo and/or multimodal stereo images. It can be also used by military for surveillance and reconnaissance that utilize multiple cameras with different views to the same scene. Another important field that our technology can have an impact is the biomedical field. In some biomedical applications, sequential imaging techniques are commonly used to detect changes in the spatial distribution of various molecules and biological materials. As an example, multispectral imaging is used to detect hemoglobin, melanin; narrowband imaging is used for cancer detection; multispectral fluorescence imaging is used to indicate molecular targeting in flexible endoscopy. All these techniques need to acquire multiple images of a sample at different wavelengths and/or polarization states in order to construct a complete spectrum for each pixel.