IoT smart devices are increasingly moving toward deep integration of audio, video, and AI technologies. Products such as smart speakers, video intercom systems, IPC cameras, Bluetooth audio terminals, and human-machine interaction panels require multiple capabilities, including audio playback, video capture, and intelligent voice algorithms. For a long time, developers had to switch between multiple independent frameworks, resulting in inconsistent interfaces, difficult code reuse, and high learning costs, which limited product iteration efficiency.
Espressif has released ESP-GMF (General Multimedia Framework) v1.0! This is the first officially released API-stable version of Espressif’s general multimedia framework. It integrates audio, video, and AI multimedia capabilities into a unified framework, providing an all-in-one multimedia development foundation for ESP32 series chips. It officially replaces ESP-ADF and marks the beginning of a new era for Espressif IoT multimedia development.
What is ESP-GMF?
ESP-GMF is a unified multimedia development framework designed for Espressif SoCs. Through a unified data stream model, component system, and development architecture, it integrates audio, video, and AI multimedia capabilities into a single platform, providing developers with a more efficient, flexible, and scalable development experience.
Compared with the previous ESP-ADF (Audio Development Framework), which mainly focused on audio applications, ESP-GMF has been comprehensively upgraded in terms of architecture design, hardware support range, and code reuse capabilities. It can support a wide range of application scenarios, including audio playback, audio recording, AI voice applications, video calls, Bluetooth audio, image rendering, and more.
Building a Unified Multimedia Software Platform
With the continuous integration of audio, video, and AI applications, multimedia development is becoming increasingly complex and diversified.
Through a unified software architecture, ESP-GMF integrates Espressif’s previously distributed multimedia capabilities and establishes a unified Data Stream model, modular component system, and scalable Pipeline architecture. This provides a consistent development approach for audio, video, and AI multimedia applications.
Developers no longer need to switch between multiple frameworks and can complete the development of different types of multimedia applications within the same development system. This significantly reduces learning costs and improves code reuse efficiency.
At the same time, Espressif has developed ESP Multimedia Core as the underlying capability foundation of ESP-GMF. ESP Multimedia Core consolidates fundamental capabilities such as media protocols, audio/video codecs, audio algorithms, and image processing algorithms into underlying libraries. It provides a stable and long-term software foundation for ESP-GMF and future multimedia products, while continuously evolving along with Espressif’s multimedia ecosystem.
ESP-GMF v1.0: The First API Stable Version
ESP-GMF v1.0 is the first officially released API Stable version of the framework. This release unifies multiple previous development versions and upgrades all official components to 1.0.x, meaning future versions will continue to evolve while maintaining API stability, making the framework more suitable for product development and long-term maintenance.
This release mainly introduces the following updates:
All Official Components Upgraded to 1.0.x
All official components have completed their transition to formal releases, and public APIs have reached a stable state, providing long-term compatibility assurance for product development.
Four New Core Modules Added
ESP-GMF v1.0 introduces several important capability modules, including:
esp_player
A new embedded multimedia player supporting demultiplexing, decoding, audio/video rendering, and Seek operations.
esp_asrc
An audio sample rate conversion module capable of converting sample rates, bit depth, and audio channels, with support for software and hardware collaborative acceleration.
esp_video_render
A video and UI rendering module supporting multiple display backends, dual video streams, Overlay, and Widget systems.
gmf_fft
A fixed-point FFT/IFFT computation module supporting hardware acceleration on chips such as ESP32-S31, ESP32-P4, and ESP32-S3.
Further Enhancement of Multimedia Capabilities
With the introduction of new components, ESP-GMF v1.0 establishes a complete multimedia capability system covering audio, video, multimodal AI, and Bluetooth multimedia. It provides unified functional interfaces across content capture, processing, playback, rendering, and transmission, delivering developers a more comprehensive and efficient multimedia development experience.
ESP Capture provides a unified audio and video capture entry point, integrating device access, format conversion, encoding, image processing, Overlay, synchronization, and multiplexing capabilities. It supports automatic negotiation, parallel stream transmission, and local storage, making it widely applicable to scenarios such as multimodal AI, WebRTC, live streaming, and remote monitoring.
ESP Player provides an all-in-one playback solution from demultiplexing and decoding to audio/video rendering. It supports multiple media sources, including local files, HTTP(S), and HLS, while integrating audio-video synchronization, playback control, Seek, and variable-speed playback functions, helping developers quickly build stable and smooth multimedia playback applications.
ESP Audio Simple Player provides a lightweight pure audio playback solution. It supports mainstream formats such as MP3, AAC, FLAC, WAV, and Opus, as well as multiple audio sources. It can automatically complete decoding and audio format conversion, enabling applications to achieve efficient audio playback with lower resource consumption.
ESP Video Render provides high-performance video composition and display capabilities. It supports multiple video streams, Overlay UI, integrated rendering, and display backends such as LCD and LVGL, meeting the requirements of applications including video players, smart displays, robots, and camera previews.
ESP Audio Render is designed for multi-channel audio integration and high-quality output. It supports PCM mixing, audio effects processing, and post-processing capabilities, enabling flexible implementation of typical scenarios such as background music and TTS mixing, notification sound overlay, and voice interaction.
ESP BT Audio unifies Classic Bluetooth and LE Audio capabilities. It supports protocols including A2DP, HFP, AVRCP, BAP, and TMAP, and can be seamlessly integrated into the ESP-GMF pipeline. Developers can also combine the basic Bluetooth audio examples provided by Espressif to quickly experience typical applications such as LE Audio and Auracast, accelerating Bluetooth audio product development.
Continuous Optimization for Multiple Espressif SoCs
ESP-GMF v1.0 continues to be optimized for Espressif SoCs such as ESP32-S31, ESP32-P4, and ESP32-S3.
Among them, ESP32-S31 introduces new V4L2 camera support and supports MJPEG-to-RGB decoding, multi-region video Overlay, audio/video rendering examples, as well as FFT and PPA hardware acceleration capabilities. These enhancements provide more comprehensive development support for applications such as AI cameras, smart displays, and human-machine interaction terminals.
In addition, the new version further optimizes the development toolchain. esp_board_manager has been separated into an independent component, and example projects now uniformly use the esp-bmgr-assist Python package to manage development board resources. This makes configuration methods more consistent across different projects and further improves development efficiency.
Typical Product Application Scenarios
Smart Voice Terminals:
Offline wake-word smart speakers, voice control panels, Bluetooth voice microphones
Visual Smart Devices:
Screen-equipped voice speakers, indoor video intercom systems, access control cameras
IoT Cameras (IPC):
Video capture + audio pickup, voice alerts, two-way intercom
Bluetooth Audio Products:
LE Audio broadcasting, Auracast public audio terminals
Industrial Human-Machine Interaction:
Equipment voice announcements, local audio alerts, image capture terminals
Conclusion
ESP-GMF v1.0 has been officially open-sourced. Developers can access the source code through the ESP-GMF GitHub repository and combine it with official documentation, complete release notes, and the Espressif General Multimedia Framework Guide to quickly experience the audio, video, and AI multimedia development capabilities enabled by the unified multimedia framework.
The release of ESP-GMF v1.0 marks a strategic upgrade of Espressif’s multimedia software ecosystem. A single framework now supports the integration of audio, video, and AI requirements, connecting the entire development chain from hardware acquisition and media processing to intelligent algorithms.
