Embedded Realtime Linux: An Evaluation

Embedded systems increasingly take on complex and time-critical tasks. Linux is an attractive foundation because of its extensive software ecosystem, broad hardware support, and open-source nature. Many embedded applications, however, must meet timing constraints in addition to their functional requirements. This makes more than average response time relevant. The crucial question is whether the system can adhere to the timing constraints under heavy system load.

The PREEMPT_RT patch introduces mechanisms intended to improve Linux’s real-time capabilities. A Bachelor’s thesis at the University of Applied Sciences Upper Austria, conducted in collaboration with Sequality, examined how these changes affect the system latency of an embedded Linux platform. The focus on the measurements was not to measure the average response time but instead to evaluate the worst-case latency and the predictability of the system under load.

Kernel Preemption

A general-purpose operating system (GPOS) such as Linux is primarily designed for throughput, fair resource sharing between tasks, and versatility. Under load, kernel sections, interrupt handling, or contention for locks can delay a higher-priority task pending for execution. Such difficult-to-predict delays reduce determinism and make it near impossible to guarantee real-time behavior.

Linux offers several kernel preemption models. They define when a running task may be preempted by a higher-priority task. The PREEMPT_RT patch introduced an additional fourth mode to the existing three preemption models:

PREEMPT_NONE
PREEMPT_VOLUNTARY
PREEMPT
PREEMPT_RT

PREEMPT_RT goes further than the conventional models: interrupt handlers are, to a large extent, executed as schedulable threads, and locking mechanisms are adapted to limit long non-preemptible sections and handle the problem of priority inversion. The goal is not necessarily higher throughput, but more predictable timing behavior.

Test Setup

At a high level, the setup consists of two components: a microcontroller-based latency measurement device and the system under test (SUT).

The measurement device sends CAN frames to the SUT and measures the time until a response frame is received. The SUT is the embedded Linux platform whose timing behaviour is being evaluated. It receives the trigger frame, processes it, and sends a response over the CAN bus. The development machine is used to configure and orchestrate the measurement, but is not part of the measured response path.

The SUT was a Toradex Verdin iMX8MP module mounted on a Yavia carrier board and running a Yocto-based Linux image (Linux Kernel 6.12). Its application represents the type of Qt-based software that could be used in an industrial embedded project: it runs as a user-space application, uses Qt’s event-driven programming model, and communicates with the hardware through Qt-provided abstractions. A user-space application built with Qt was deliberately chosen to include the scheduling, event-handling, and communication latencies that can occur in realistic industrial applications. For CAN communication, the application uses Qt Serial Bus which provides a common high-level abstraction for Qt application to interact with their platform specific CAN implementation.

The measured round-trip time therefore represents the system latency from an external event to an observable response. It includes the CAN communication, the Linux scheduling path, and the processing performed by the application. This high-level topology makes it possible to evaluate the complete application path from outside the system rather than measuring an isolated function call.

Measurements

The evaluation compared PREEMPT_RT, PREEMPT, PREEMPT_VOLUNTARY, and PREEMPT_NONE on Linux 6.12. The observed system latencies were plotted as a box plot to visualize the distribution and identify outliers.

Idle System: The first measurement was performed with the SUT in an idle state and without additional synthetic load.

The average and median latencies were close across all preemption models. However, PREEMPT_RT had the highest average latency at 0.2967 ms, compared with 0.2850 ms for PREEMPT, 0.2839 ms for PREEMPT_VOLUNTARY, and 0.2865 ms for PREEMPT_NONE.

This small increase is consistent with the additional mechanisms introduced by PREEMPT_RT. Real-time preemption improves predictability, but it can also add some scheduling and synchronization overhead. When the system is idle, there is little contention for the kernel, so the determinism improvements do not yet translate into a clear latency advantage.

Kernel Mode	Min	Q1	Median	Q3	Max	Average	Samples
RT_PREEMPT	0.2728	0.2929	0.2969	0.3009	0.4208	0.2967	100.000
DESKTOP	0.2649	0.2809	0.2849	0.2888	0.4409	0.2850	100.000
VOLUNTARY	0.2649	0.2809	0.2848	0.2849	0.5048	0.2839	100.000
NONE	0.2649	0.2848	0.2849	0.2889	0.4809	0.2865	100.000

System under heavy load: The second measurement subjected the system to a target CPU load of 100 percent as well as synthetic kernel-related stressors, including timer events, context switches, file I/O, and futex operations.

The average and median latencies remained close across all preemption models. The relevant difference is visible at the upper end of the distribution: PREEMPT_RT reached a maximum latency of 0.6449 ms, whereas PREEMPT and PREEMPT_VOLUNTARY exhibited outliers of 3.6727 ms and 4.0608 ms, respectively. Under this combined user-space and kernel-related load, PREEMPT_RT therefore primarily improves timing predictability by limiting latency outliers, rather than by reducing the average latency. In fact, the average latency of PREEMPT_RT is larger than that of the other preemption models. This is due to the overhead introduced by the real-time preemption mechanisms added to the kernel.

Kernel Mode	Min	Q1	Median	Q3	Max	Average	Samples
RT_PREEMPT	0.3049	0.3649	0.3729	0.3889	0.6449	0.3805	100.000
DESKTOP	0.2849	0.3568	0.3609	0.3689	3.6727	0.3659	100.000
VOLUNTARY	0.2769	0.3529	0.3608	0.3689	4.0608	0.3701	100.000
NONE	0.2809	0.3529	0.3569	0.3689	10.9885	0.3718	100.000

Conclusion

PREEMPT_RT does not automatically turn Linux into a hard real-time operating system. The measurements do show, however, that it can substantially improve determinism under heavy kernel-related load. This makes it an attractive foundation for many soft real-time applications.

The series of measurements and experiments conducted do not serve a formal proof of real-time behavior, but they do provide valuable insights into the impact of different preemption models on system latency under load.

—

Do you want to know more details and get access to the full technical report (bachelor thesis PDF, 65 pages)? Give us a notice at office@sequality.at!