The "Holy Bible" for embedded engineers
Deterministic Timing in Linux Systems
Understanding PREEMPT_RT, Xenomai, and real-time extensions for embedded applications
Real-time computing involves systems that must respond to events within strict timing constraints. In embedded systems, real-time capabilities are crucial for applications like industrial control, automotive systems, and medical devices.
Real-time System Characteristics:
General-purpose Systems:
Real-time Systems:
┌─────────────────────────────────────┐
│ Real-time Requirements │
├─────────────────────────────────────┤
│ Hard Real-time: │
│ • Must meet deadlines │
│ • System failure if missed │
│ • Example: Airbag deployment │
├─────────────────────────────────────┤
│ Soft Real-time: │
│ • Should meet deadlines │
│ • Degraded performance if missed │
│ • Example: Video streaming │
├─────────────────────────────────────┤
│ Firm Real-time: │
│ • Must meet deadlines │
│ • Acceptable to miss occasionally │
│ • Example: Data logging │
└─────────────────────────────────────┘
Standard Linux was not designed for real-time applications. Various extensions and patches have been developed to add real-time capabilities while maintaining Linux’s general-purpose functionality.
Real-time Linux extensions follow the minimal modification principle—add real-time capabilities with minimal changes to the existing Linux kernel, ensuring compatibility and maintainability.
Extension Design Goals:
1. PREEMPT_RT Patch:
2. Xenomai Framework:
3. RTAI (Real-Time Application Interface):
The PREEMPT_RT patch transforms Linux into a fully preemptible kernel, allowing high-priority real-time tasks to interrupt kernel operations.
PREEMPT_RT follows the kernel preemption principle—make the Linux kernel fully preemptible to reduce worst-case latency and improve real-time responsiveness.
PREEMPT_RT Goals:
Kernel Configuration:
# Enable PREEMPT_RT in kernel config
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_DEBUG_PREEMPT=y
Lock Conversion:
// Standard Linux: spinlock
spinlock_t lock;
spin_lock(&lock);
// Critical section
spin_unlock(&lock);
// PREEMPT_RT: mutex
struct rt_mutex lock;
rt_mutex_lock(&lock);
// Critical section
rt_mutex_unlock(&lock);
Interrupt Threading:
// Standard Linux: interrupt handler
irqreturn_t irq_handler(int irq, void *dev_id) {
// Handle interrupt immediately
return IRQ_HANDLED;
}
// PREEMPT_RT: threaded interrupt
irqreturn_t irq_handler(int irq, void *dev_id) {
// Schedule work for later
return IRQ_WAKE_THREAD;
}
irqreturn_t irq_thread(int irq, void *dev_id) {
// Handle interrupt in thread context
return IRQ_HANDLED;
}
Xenomai provides a co-kernel approach where a real-time kernel runs alongside Linux, offering excellent real-time performance while maintaining Linux compatibility.
Xenomai follows the dual-kernel principle—separate real-time and general-purpose kernels to achieve optimal performance for both domains.
Xenomai Design Goals:
┌─────────────────────────────────────┐
│ User Applications │
├─────────────────────────────────────┤
│ Real-time Applications │
│ (Xenomai APIs) │
├─────────────────────────────────────┤
│ Linux Applications │
│ (POSIX, System Calls) │
├─────────────────────────────────────┤
│ Xenomai Co-kernel │
│ (Real-time Scheduler) │
├─────────────────────────────────────┤
│ Linux Kernel │
│ (General-purpose) │
├─────────────────────────────────────┤
│ Hardware Layer │
└─────────────────────────────────────┘
Xenomai APIs:
1. Native API:
#include <native/task.h>
#include <native/timer.h>
RT_TASK task_desc;
RT_TIMER timer_desc;
void real_time_task(void *arg) {
// Real-time task code
rt_task_sleep(1000000); // 1ms sleep
}
int main() {
// Create real-time task
rt_task_create(&task_desc, "rt_task", 0, 99, T_CPU(0));
rt_task_start(&task_desc, &real_time_task, NULL);
// Start Xenomai
rt_task_shutdown();
return 0;
}
2. POSIX API:
#include <pthread.h>
#include <sched.h>
#include <time.h>
void *real_time_thread(void *arg) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
// Real-time thread code
ts.tv_nsec += 1000000; // 1ms
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL);
return NULL;
}
int main() {
pthread_t thread;
struct sched_param param;
param.sched_priority = 99;
pthread_create(&thread, NULL, real_time_thread, NULL);
pthread_setschedparam(thread, SCHED_FIFO, ¶m);
pthread_join(thread, NULL);
return 0;
}
Real-time programming requires careful attention to timing, resource management, and system behavior. Understanding real-time programming principles is essential for building reliable real-time applications.
Real-time programming follows the deterministic execution principle—ensure predictable timing behavior through careful design, resource management, and system understanding.
Real-time Programming Goals:
Task Structure:
#include <native/task.h>
#include <native/timer.h>
#include <native/mutex.h>
RT_TASK task_desc;
RT_MUTEX mutex_desc;
RT_TIMER timer_desc;
// Real-time task function
void real_time_task(void *arg) {
RTIME start_time, current_time;
int iteration = 0;
// Set task to periodic mode
rt_task_set_periodic(NULL, TM_NOW, 1000000); // 1ms period
while (1) {
start_time = rt_timer_read();
// Acquire mutex
rt_mutex_acquire(&mutex_desc, TM_INFINITE);
// Critical section
// Perform real-time operations
// Release mutex
rt_mutex_release(&mutex_desc);
// Wait for next period
rt_task_wait_period(NULL);
// Check timing
current_time = rt_timer_read();
if (current_time - start_time > 500000) { // 500μs
rt_printf("Timing violation: %lld ns\n",
current_time - start_time);
}
iteration++;
}
}
// Main function
int main() {
// Initialize Xenomai
rt_print_auto_init(1);
// Create mutex
rt_mutex_create(&mutex_desc, "rt_mutex");
// Create real-time task
rt_task_create(&task_desc, "rt_task", 0, 99, T_CPU(0));
rt_task_start(&task_desc, &real_time_task, NULL);
// Wait for user input
printf("Press Enter to exit\n");
getchar();
// Cleanup
rt_task_delete(&task_desc);
rt_mutex_delete(&mutex_desc);
return 0;
}
Priority Management:
#include <native/task.h>
#include <native/sem.h>
RT_TASK high_priority_task;
RT_TASK low_priority_task;
RT_SEM semaphore;
// High priority task
void high_priority_handler(void *arg) {
while (1) {
// Wait for semaphore
rt_sem_p(&semaphore, TM_INFINITE);
// Handle high priority event
rt_printf("High priority task executing\n");
// Simulate work
rt_task_sleep(100000); // 100μs
rt_task_wait_period(NULL);
}
}
// Low priority task
void low_priority_handler(void *arg) {
while (1) {
// Perform background work
rt_printf("Low priority task executing\n");
// Simulate work
rt_task_sleep(1000000); // 1ms
rt_task_wait_period(NULL);
}
}
int main() {
// Create semaphore
rt_sem_create(&semaphore, "rt_sem", 0, S_FIFO);
// Create high priority task
rt_task_create(&high_priority_task, "high_task", 0, 99, T_CPU(0));
rt_task_set_periodic(&high_priority_task, TM_NOW, 1000000);
rt_task_start(&high_priority_task, &high_priority_handler, NULL);
// Create low priority task
rt_task_create(&low_priority_task, "low_task", 0, 50, T_CPU(0));
rt_task_set_periodic(&low_priority_task, TM_NOW, 5000000);
rt_task_start(&low_priority_task, &low_priority_handler, NULL);
// Signal high priority task periodically
while (1) {
rt_sem_v(&semaphore);
rt_task_sleep(2000000); // 2ms
}
return 0;
}
Real-time performance is measured by latency, jitter, and throughput. Understanding how to measure and optimize these metrics is crucial for building high-performance real-time systems.
Latency Metrics:
Jitter Metrics:
Interrupt Latency Measurement:
#include <native/task.h>
#include <native/timer.h>
#include <native/irq.h>
RT_TASK measurement_task;
RT_TIMER measurement_timer;
volatile RTIME interrupt_time, task_time;
// Interrupt handler
void irq_handler(int irq, void *dev_id) {
interrupt_time = rt_timer_read();
}
// Measurement task
void measurement_handler(void *arg) {
RTIME latency;
while (1) {
// Wait for timer interrupt
rt_task_sleep(1000000); // 1ms
// Calculate latency
latency = task_time - interrupt_time;
rt_printf("Interrupt latency: %lld ns\n", latency);
rt_task_wait_period(NULL);
}
}
// Timer callback
void timer_callback(void *arg) {
task_time = rt_timer_read();
}
int main() {
// Create measurement task
rt_task_create(&measurement_task, "measure", 0, 99, T_CPU(0));
rt_task_set_periodic(&measurement_task, TM_NOW, 1000000);
rt_task_start(&measurement_task, &measurement_handler, NULL);
// Create timer
rt_timer_create(&measurement_timer, "measure_timer",
TM_NOW, 1000000, TM_PERIODIC, &timer_callback, NULL);
// Wait for user input
printf("Press Enter to exit\n");
getchar();
// Cleanup
rt_task_delete(&measurement_task);
rt_timer_delete(&measurement_timer);
return 0;
}
Real-time systems require careful design and implementation to ensure reliable operation. Following best practices is essential for building robust real-time applications.
1. Minimize Interrupt Processing:
// Good: Minimal interrupt handler
irqreturn_t irq_handler(int irq, void *dev_id) {
// Only essential operations
schedule_work(&deferred_work);
return IRQ_HANDLED;
}
// Bad: Complex interrupt handler
irqreturn_t irq_handler(int irq, void *dev_id) {
// Complex processing in interrupt context
process_data();
update_display();
send_network_packet();
return IRQ_HANDLED;
}
2. Use Appropriate Scheduling:
// Good: Real-time scheduling
struct sched_param param;
param.sched_priority = 99;
pthread_setschedparam(thread, SCHED_FIFO, ¶m);
// Bad: Default scheduling
// Uses SCHED_OTHER (time-sharing)
3. Resource Management:
// Good: Pre-allocate resources
static char buffer[1024];
static RT_MUTEX buffer_mutex;
// Bad: Dynamic allocation in real-time context
char *buffer = malloc(1024); // Could block
Latency Testing:
#include <native/task.h>
#include <native/timer.h>
#include <native/mutex.h>
RT_TASK test_task;
RT_MUTEX test_mutex;
RT_TIMER test_timer;
volatile RTIME max_latency = 0;
volatile RTIME min_latency = 1000000000;
volatile int test_count = 0;
void latency_test(void *arg) {
RTIME start_time, end_time, latency;
while (test_count < 10000) {
start_time = rt_timer_read();
// Acquire mutex
rt_mutex_acquire(&test_mutex, TM_INFINITE);
// Simulate work
rt_task_sleep(1000); // 1μs
// Release mutex
rt_mutex_release(&test_mutex);
end_time = rt_timer_read();
latency = end_time - start_time;
// Update statistics
if (latency > max_latency) max_latency = latency;
if (latency < min_latency) min_latency = latency;
test_count++;
rt_task_wait_period(NULL);
}
rt_printf("Latency test completed\n");
rt_printf("Min latency: %lld ns\n", min_latency);
rt_printf("Max latency: %lld ns\n", max_latency);
}
int main() {
// Create test task
rt_task_create(&test_task, "test", 0, 99, T_CPU(0));
rt_task_set_periodic(&test_task, TM_NOW, 1000000);
rt_task_start(&test_task, &latency_test, NULL);
// Wait for completion
while (test_count < 10000) {
rt_task_sleep(100000); // 100ms
}
rt_task_delete(&test_task);
return 0;
}
Real-time Linux provides powerful capabilities for building deterministic embedded systems. Understanding PREEMPT_RT, Xenomai, and real-time programming principles is essential for creating reliable real-time applications.
Key Takeaways:
The Path Forward:
As embedded systems become more complex and real-time requirements become stricter, the importance of real-time Linux skills will only increase. Modern systems continue to evolve, providing new real-time capabilities and optimization techniques.
Remember: Real-time Linux is not just about running Linux in real-time—it’s about understanding how to design, implement, and validate systems that meet strict timing requirements. The skills you develop here will enable you to create robust, reliable, and high-performance real-time embedded systems.