Yesterday in Hawaii at the Snapdragon Technology Summit, the long-anticipated Qualcomm Snapdragon 855 was launched. While the new chip brings various substantial changes (from significantly more accurate GPS to the first new 2.4 GHz WiFi version in a decade, to 5G cellular connectivity), the discussion keeps coming back to imaging and the massive improvements that the Qualcomm Spectra 380 brings.

Traditionally the camera's ISP would handle transforming the raw color outputs of the camera into something that the rest of the phone can use for machine vision, but as AI processing has become more complex, integration with specialized hardware has become critical. Qualcomm took the first steps towards this with their Qualcomm Spectra 280, baking support for multi-frame noise reduction and depth sensing right into the ISP, but there was still a long way to go. This year Qualcomm takes that leap, building hardware computer vision acceleration right into the Qualcomm Spectra 380 ISP, and creating what they're calling the world's first CV-ISP.

The Qualcomm Spectra 380 will remove bottlenecks in the image processing pipeline and result in up to 4x power savings over previous generations by performing machine vision processing right on the ISP itself, removing the need to wake up the CPU, GPU, and HSP for complex photography AI tasks, and greatly improving the processor's AR capabilities.

This means that hardware depth sensing, object classification, and object segmentation all can happen in real-time for up to 4k HDR 60Hz on the Qualcomm Snapdragon 855. This will give us live object or background replacement in real-time video, opening up new possibilities for extended reality (XR) applications with the Qualcomm Snapdragon 855 chipset both in phones and in AR headsets.

The improvements for the Qualcomm Spectra 380 don't stop at AI and AR though. Qualcomm put a lot of work into color space and brightness improvements, adding support for full HDR10+ video recording, making it possible to record videos with greater dynamic range and wider color spaces, to take full advantage of the capabilities of modern phones and TVs.

The new ISP also brings improved hardware acceleration for VP9 video and for the HEVC based HEIF image format which can be 50% smaller than older image formats and brings expanded functionality (although we are highly anticipating the addition of hardware acceleration for the AV1 based AVIF, which will hopefully be available in future years)

The Qualcomm Spectra 380 will also be able to handle higher resolution imaging than previous years, with its dual 14-bit CV-ISP pipeline supporting 22MP 30Hz concurrent capture on dual cameras, and 48MP 30Hz capture with a single image sensor. The faster sensor read pipeline doesn't translate over to video in this case, however, where the Qualcomm Spectra 380 supports the same native 720p 480Hz (without using on-sensor cache) that the Spectra 280 did.

Best of all, the Qualcomm Spectra 380 may continue to improve in the future, as its new modular ISP design brings a newfound flexibility to the imaging pipeline, opening up the potential for different processing methods for both RAW and YUV pixel domains.

Are you excited for expanded AI functionality in cameras? What do you hope to be able to use machine vision for? Tell us in the comments below!