In our last blog post from August 2024 we identified the optimal setup for a live video streaming embedded platform. Since then, we have conducted lots of tests with various Embedded Linux hardware platforms and can present you some results.
Hardware Test Setup
The basic test-setup for the next following benchmarks is a single automotive-grade ethernet camera (ArkCam Basic Plus Mini, resolution 1280 x 960) with an Toradex Apalis i.Mx8 QuadMax. This CPU contains 6 cores, four of them are Arm Cortex-A53, two are Arm Cortex-A72. The Arm Cortex-A72 cores have a maximum clock speed of 1.6 GHz, the Arm Cortex-A53 cores have 1.26 GHz. There is also 8 GB of RAM and 32 GB of flash memory available for the embedded Linux root filesystem. The GPU installed on the board contains two Vivante GC7000 XSVX units and has hardware video encoder and decoder for MJPEG, H264 and various other popular formats.
Network load H264 vs MJPEG
When comparing the video codecs H.264 and MJPEG, it can be expected that the data rate a video stream with the MJPEG codec is a lot higher because the codec MJPEG sends all images in full. As shown the image below, the H.264 codec requires around a third of the bandwidth compared to the Codec MJPEG (orange area). This is clearly due to the fact that the H.264 codec is more efficient and can transmit a compareable video quality with a smaller data rate. Also, the H.264 codec data rate is more inconsistent because the size depends on how much change there is to the previous video image frame. The data rate of the H.264 codec is around 1.1 MB/s. The MJPEG codec has a stable data rate of about 4.4 MB/s.
Frames per Second (FPS) H264 vs MJPEG with hardware decoder support
As can be seen in the graph below, there is quite a difference between the FPS performance of H264 and MJPEG. The MJPEG codec offers significantly more stable performance with deviations of a maximum of 2 frames per second from the full 50 FPS of the source. This results in an average frame rate of 49.92 frames per second during a 26 minutes long-term test. On the other hand, the H.264 codec often has larger deviations, sometimes up to 9 images per second. The average value here is at 46.82 FPS throughout the longterm test. In both tests the hardware decoder unit (VPU) of the i.Mx8 Vivante GPU was used.
Software Decoding: FPS with H264 and MJPEG
As can be seen in the graphics below, the i.Mx8 is capable to decode an MJPEG stream via software encoder at full frame rate. The 1280×960 pixels are rendered as fluent as with the hardware decoder, the FPS measurements are at the maxium with stable 50 frames per second. On the other hand, the H264 codec was not able to be rendered at 50FPS. The measurement shows a performance of about 30 frames with some drops below 10 frames. Due to the more complex nature of the codec H264, it obviously is necessary to have hardware decoder support to be able to handle a HD-video-stream in real-time with a stable 30 frames per second.
Latency Measurements
Ethernet cameras like the ArkCam Basic+ from Ark Vision, Orlaco Emos or Mekra Lang are designed to work in rough environments and perform critical monitoring tasks. Inherently, a low latency is crucial for such a product to be used in a safe way. Moreover, in the automotive sector the ISO 16505 lists some minimum system requirements for camera monitoring systems such as digital mirrors or rear view camera systems: Latency from glass to glass is required to be <200ms. In order to test such a system, it is crucial to test the overall system latency which means that the latency has to be measured within the final product under a realistic system load. This is the reason why it is not sufficient to just open a gstreamer-window and measure latency. No, you have to measure the latency in the final application (probably Qt/QML application) with the video component integration you have chosen to use in your system. As there are quite some different possiblities how the video stream gets rendered, it is interesting, which video stream component configuration is optimal to be used in terms of low latency. The following possible configurations were tested:
- Qt5 QtQuick Multimedia RTP H264
- Qt5 QtQuick Multimedia RTP MJPEG
- Qt5 QtQuick Multimedia RTSP H264
- Qt5 QtQuick Multimedia RTSP MJPEG
- Qt5 QtQuick qmlglsink RTP MJPEG
- Qt5 QtQuick qmlglsink RTP MJPEG
- Qt5 Widgets RTSP H264
- Qt5 Widgets RTSP MJPEG
- Qt6 QtQuick Multimedia RTSP H264, no hardware decoder
- Qt6 QtQuick Multimedia RTSP MJPEG, no hardware decoder
- Qt6 Widgets H264, no hardware decoder
- Qt6 Widgets MJPEG, no hardware decoder
In the first measurement setup, we focused on Qt5 based configurations. In terms of the integration of a video stream within QML, you basically have the choice of using the MediaPlayer Element in QML (Qt Quick Multimedia), or to use the qmlglsink C component from gstreamer. Whereas pipeline configuration with Qt Quick Multimedia can be done via a string property in the component, using the qmlglsink component needs to be written fully in C. All Qt5 latency measurements were done with support from the hardware decoder of the i.MX8 GPU.
Overall, the Qt5 measurements show, that MJPEG is much better in terms of latency compared to H264. All other different configuration factors play a much smaller role. In this measurement setup, using Qt Multimedia delivered a slightly better latency result over using the qmlglsink component. But as we know from our ongoing projects, we also already had embedded-linux-systems on our desks, where qmlsink was better than Qt Multimedia. I would call this a quite balanced result.
Again, the Qt6 measurements show, that the choice of the used codec mostly defines latency performance. But there is one big drawback our developer stumbled over when performing the tests: Due to the change of the video processing architecture in Qt6, it is no longer possible to configure a gstreamer-pipeline in the QML Element. In favor to also support ffmpeg in Qt6, all gstreamer-specific interfaces are no longer available in the MediaPlayer component. The logical next possibility then would be to use the qmlsink-gstreamer component in C++. But here the developer recognized, that this was still not compatible with the i.Mx8-Yocto-Linux (Toradex yocto Multimedia reference image, June 2024) he used to test the system. As a result one could state, that in this test setup Qt6 is not a good choice to support low-latency live video streaming. Maybe this statement changes, once you can use GStreamer Version 1.22 or newer, because there the qml6glsink is available.
Reliable, low-latency video streaming integrated in Qt Quick applications somewhat is a challenge: All the system components need to be optimzed to get the best results: From the choice of the exact i.Mx8 model up to the setup and configuration of the embedded linux system, the selection of Qt and gstreamer version and the definition of the software system component architecture. In some use cases, other video streaming interfaces also might look interesting (e.g. GSML or Mipi-CSI-2), but the task to optimally integrate the video stream (with gstreamer) into your Qt user space application remains.
If you have any further questions, feel free to send me an email unter: . Also see our last blog post to this topic here: https://www.sequality.at/blog/2024/live-video-streaming-with-qt/
