ARTiS, an Asymmetric Real-Time Scheduler on Multi-Processor

							 Real-Time




                    ARTiS, an Asymmetric Real-Time
Systems for
      Multi-




                      Scheduler on Multi-Processor
 Processor
Architectures




State of the
Art
RTOS
                             Architectures
GPOS
Linux based
approaches
Proposition      Éric Piel, Philippe Marquet, Julien Soula, Jean-Luc Dekeyser
ARTiS
Partitioning
Automatic                   Laboratoire d'informatique fondamentale de Lille
migration
Specic                     Université des sciences et technologies de Lille
load-balancing
                                                France
Performance
measure-
ments
Interrupt
latency
Execution time
variation                       WPDRTS'06, Rhodes, Greece
Load-balancing
                                     April 26, 2006
Conclusion
                 Outline
 Real-Time
Systems for
      Multi-
 Processor
                 State of the Art
Architectures



                 ARTiS
State of the
Art
RTOS
GPOS
Linux based
                 Performance measurements
approaches
Proposition
ARTiS
Partitioning
                 Conclusion
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
 Real-Time
Systems for
      Multi-
 Processor
                 State of the Art
Architectures       RTOS
                    GPOS
State of the
Art
                    Linux based approaches
RTOS                Proposition
GPOS
Linux based
approaches
Proposition
ARTiS
                 ARTiS
Partitioning
Automatic
migration
Specic          Performance measurements
load-balancing
Performance
measure-
ments
                 Conclusion
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Real-Time Dedicated OS
 Real-Time
Systems for         Focused on RT from the beginning
                     +
      Multi-
 Processor                excellent hard real-time performance
Architectures
                     +    readily available and extensively tested
                         multi-processing is an extension (not so good at number
State of the              crunching)
                      
Art
RTOS                      mostly targeted for embedded applications (no IA-64
GPOS
Linux based               support)
approaches
Proposition         ex: RTEMS
ARTiS
                          limited MP support (no task migration)
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 GPOS Extended to Real-Time
 Real-Time
Systems for         Originally focused on performance computing
                     +
      Multi-
 Processor                MP really enables High Performance Computing
Architectures
                    Developer friendly
                     +    known environment
                     +
State of the
Art                       plethora of tools available
RTOS
GPOS                RT is an extension
Linux based
approaches
Proposition
                         stability less sure

ARTiS
                         soft or rm real-time
Partitioning
Automatic           ex: REACT/Pro (by SGI), several Linux modications
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Co-kernel Approach
 Real-Time
Systems for         Small hard real-time micro-kernel
      Multi-
 Processor
                         real-time tasks are ran in the micro-kernel
Architectures            Linux is the lowest priority task
                    Pros and Cons
State of the         +   hard real-time performance
Art
RTOS                    real-time tasks have no Linux services direct access
GPOS
Linux based
                        poor SMP support
approaches
Proposition         ex: RTLinux, RTAI                                          Linux user processes



ARTiS
Partitioning
Automatic                                                RT FIFO
migration                           RT process                               Linux kernel

Specic
load-balancing
Performance                                               Virtual
                                                          Interrupt
measure-
                                                          Controller
ments
Interrupt
latency                                    Real−Time Micro−Kernel

Execution time
variation
Load-balancing                                           Hardware Platform

Conclusion
                 Patched Linux Kernel
 Real-Time
Systems for         Modied version of the kernel
      Multi-
 Processor               preemption more often possible
Architectures
                    Based on Linux
                     +   every Linux application is supported
                     +
State of the
Art                      very good HPC performance
RTOS
GPOS
                        modication technically hard to implement
Linux based
approaches              very dicult to verify real-time properties
Proposition
ARTiS
                    ex: Preemptible kernel option, preempt-rt project
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Asymmetric Multi-Processing
 Real-Time
Systems for         Temporal reservation fails in GPOS because of latencies
      Multi-
 Processor
Architectures
                    Spatial reservation
                          reserved CPUs for the RT tasks
                          AMP: Asymmetric Multi-Processing
State of the
Art                       (opposed to SMP, Symmetric Multi-Processing)
RTOS
GPOS                Resource wasting
Linux based
approaches
Proposition
                         non-RT task waiting while RT CPU idle

ARTiS               ex: CCC RedHawk
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Asymmetric Multi-Processing
 Real-Time
Systems for
      Multi-               NRT CPU   RT CPU   RT CPU   RT CPU
 Processor
Architectures


                                                                Linux
                 cluster
State of the
Art
RTOS
GPOS
Linux based
approaches
Proposition
ARTiS
Partitioning
Automatic
migration                                                       RT
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 ARTiS: an evolution of AMP
 Real-Time
Systems for         Asymmetric Real Time Scheduler
      Multi-
 Processor
Architectures
                    Dynamic partitioning
                         two sets of processors
                         all tasks can run on all CPUs
State of the
Art                      non-preemtable code not executed on RT CPUs
RTOS
GPOS                Based on Linux
Linux based
approaches
Proposition
                     +   POSIX compatible, Linux ABI compatible

ARTiS
                     +   high performance available
Partitioning         +   multi-architecture support (x86 and IA-64)
Automatic
migration            +   rm RT possible
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
 Real-Time
Systems for
      Multi-
 Processor
                 State of the Art
Architectures


                 ARTiS
State of the
Art
                    Partitioning
RTOS
GPOS
                    Automatic migration
Linux based
approaches          Specic load-balancing
Proposition
ARTiS
Partitioning
Automatic
                 Performance measurements
migration
Specic
load-balancing
Performance      Conclusion
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 System Partitioning
 Real-Time
Systems for         Processor partitioning
      Multi-
 Processor              RT CPU vs Non-RT CPU
Architectures
                        both type can execute all tasks
                        non-RT CPUs handle all the IRQs
State of the                 (excepted the one for RT)
Art
RTOS
GPOS
                    Types of tasks
Linux based             RT0, xed to one RT CPU, hard real-time
approaches
Proposition             RT1+, soft real-time
ARTiS
                        Linux task
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Principle of Automatic Migration
 Real-Time
Systems for         How to always re-schedule in time?
      Multi-
 Processor              never block interrupts,   local_irq_disable()
Architectures
                        never forbid preemption,    preempt_disable()
                        functions available only via syscalls (= in the kernel)
State of the
Art
                    Migration of the task from RT CPU to NRT CPU
RTOS                    kernel automatically migrates RT1+ and Linux tasks
GPOS
Linux based             a RT CPU can not share a lock with a NRT CPU
approaches
Proposition             lock-free (& wait-free) queue: RT-FIFO
ARTiS
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 A Particular Migration Mechanism
 Real-Time
Systems for
      Multi-               NRT CPU         RT CPU       RT CPU   RT CPU
 Processor
Architectures


                                                                          Linux
                 cluster
State of the
Art
                                            ARTiS
RTOS                                        migration
GPOS
Linux based
approaches                                                                RT1+
Proposition
                                     ARTiS
ARTiS                                migration
Partitioning
Automatic
migration                                                                 RT0
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 An Ecient Load-Balancing
 Real-Time
Systems for         Usual Linux load-balancing
      Multi-
 Processor              never idle CPU while a ready task on another CPU
Architectures
                        under-loaded CPUs grab tasks from busy CPUs

                    Asymmetry related enhancements
State of the
Art                     RT1+ tasks come back to a RT CPU asap
RTOS                    tasks which deactivate preemption often should stay on
GPOS
Linux based             NRT CPUs
approaches
Proposition
                    Real-time related enhancements
ARTiS
Partitioning            lock-free inter-CPU transfer of tasks
Automatic
migration               correctly estimate the usage of RT tasks
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 An Ecient Load-Balancing
 Real-Time
Systems for
      Multi-               NRT CPU         RT CPU        RT CPU        RT CPU
 Processor                                                        load−
Architectures
                                       load−
                                       balancing                  balancing

                                                                                Linux
                 cluster
State of the
Art
                                             ARTiS
RTOS                                         migration
GPOS
Linux based
approaches                                                                      RT1+
Proposition
                                     ARTiS
ARTiS                                migration
Partitioning
Automatic
migration                                                                       RT0
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
 Real-Time
Systems for
      Multi-
 Processor
                 State of the Art
Architectures


                 ARTiS
State of the
Art
RTOS
GPOS             Performance measurements
Linux based
approaches
Proposition
                    Interrupt latency
ARTiS
                    Execution time variation
Partitioning
Automatic
                    Load-balancing
migration
Specic
load-balancing
Performance      Conclusion
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Implementation Evaluation
 Real-Time
Systems for         Hardware
      Multi-
 Processor              4x Itanium2 at 1.3GHz
Architectures
                        Linux kernel 2.6.11

                    Loads
State of the
Art                     intensive use of the network card
RTOS
GPOS
                        call of   ioctl()   (implies a big kernel lock)
Linux based             intensive use of the hard-disk
approaches
Proposition             innite loop in user mode only on each CPU
ARTiS
                        CPU cache usage (implies cache line reads)
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Interrupt Latency
 Real-Time
Systems for          Measurements
      Multi-
 Processor                measurement task is highest priority RT
Architectures
                          same coding than an usual real-time task
                          run of 8 hours (300 millions interrupts)
State of the              2 types of latencies: driver (kernel) / task reschedule (user)
Art
RTOS                      loaded system
GPOS
Linux based
approaches
Proposition
ARTiS            Table: Maximum Kernel/User latencies of the dierent congurations
Partitioning
Automatic
migration
Specic
                            Congurations               Kernel        User
load-balancing
                            standard Linux                 63ms        49ms
                            Linux with preemption          60ms      1155ms
Performance
measure-

                            ARTiS                          43ms       104ms
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Execution Time Variation
 Real-Time
Systems for          Measurements
      Multi-
 Processor                same routine executed one million time
Architectures
                          maximum execution time should be close from minimum
                          loaded system
State of the                                             m
                          minimum execution time is 9269 s
Art
RTOS
GPOS
Linux based
approaches
Proposition      Table: Execution time dierence between minimum obtained and
ARTiS            conguration maximums
Partitioning
Automatic
migration               kernel            Normal priority RT0 priority
Specic
load-balancing          standard Linux        516.1ms              20.61ms
Performance
measure-
                        ARTiS                 659.7ms              21.61ms
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Load-Balancing Observation
 Real-Time
Systems for         Measurements
                                                                   m
      Multi-
 Processor               special tool to observe load-balancing (lb )
Architectures
                         1 NRT CPU, 3 RT CPUs
                         13 NRT tasks, 3 RT tasks taking 90% of the CPU
State of the             fairness should lead to 10 NRT tasks on the NRT CPU
Art
RTOS
GPOS
                    Table: NRT task repartition on the CPUs (at the beginning)
Linux based
approaches
Proposition                kernel              NRT RT RT RT
ARTiS                      standard Linux        4       3     3        3
Partitioning
Automatic
migration
                           ARTiS                10       1     1        1
Specic
load-balancing
Performance
                                 Table: Task completion time (s)
measure-
ments                      kernel              Min Average Max
Interrupt
latency
Execution time
                           standard Linux      190       315           447
variation
Load-balancing             ARTiS               377       381           485
Conclusion
 Real-Time
Systems for
      Multi-
 Processor
                 State of the Art
Architectures


                 ARTiS
State of the
Art
RTOS
GPOS             Performance measurements
Linux based
approaches
Proposition
ARTiS            Conclusion
Partitioning
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion
                 Conclusion
 Real-Time
Systems for
      Multi-
                 Summary
 Processor
Architectures
                    ARTiS = AMP with dynamic partitioning
                        automatic migration of tasks endangering RT
State of the
Art
                        adapted load-balancing with respect to asymmetry
RTOS
GPOS                Comfort of Linux
Linux based
approaches
Proposition         Firm real-time and HPC
ARTiS
Partitioning
Automatic
migration
                 Future work
Specic
load-balancing
                    Real-time scheduling
Performance
measure-                EDF...
ments
Interrupt
latency
                    Implementation on MPSoC
Execution time
variation               within the GASPARD design ow (Intensive Signal
Load-balancing
                        Processing)
Conclusion
 Real-Time
Systems for
      Multi-
 Processor
Architectures




State of the
Art
RTOS
                 www.lifl.fr/west/artis
GPOS
Linux based
approaches
Proposition
ARTiS
Partitioning     eric.piel@lifl.fr
Automatic
migration
Specic
load-balancing
Performance
measure-
ments
Interrupt
latency
Execution time
variation
Load-balancing
Conclusion