Material TDDB72: Real-Time Operating Systems Most slides by courtesy of Jörgen Hansson Real-Time Systems Laboratory Department of Computer Science Linköping University, Sweden Articles: Stepner, Rajan, and Hui, ”Embedded Application Design Using a Real-Time Operating System” Li, Potkonjak, and Wolf, ”Real-Time Operating Systems for Embedded Computing”, 1997 Stewart, Schmitz, and Khosla, ”The Chimera II Real-Time Operating System for Advanced Sensor-Based Control Applications”, 1992 Friedrich et al., ”A Survey of Configurable Component-Based Operating Systems for Embedded Applications” (Configurability) Books: J. Liu, ”Real-Time Systems”, Prentice Hall, 2000. (Basic theory in Chapters 1-11, Chapter 12 on RTOS, NT/Linux) W. Wolf, ”Computers as Components”, see chapter 6 on ”Processes and Operating Systems”, Morgan Kaufmann Publishers, 2000. Chapter 19 of [SGG7] Silberschatz, Galvin, Gagne 7th edition, 2005 (very short overview; RTLinux, VxWorks) Revised 2005 by Christoph Kessler, IDA A. Bednarski, IDA, 2006 TDDI12 - Real-Time Operating Systems Lecture Outline Definitions Desirable features of OSs targeted for real-time and embedded applications Foundations of real-time systems Real-time scheduling algorithms RTOS Requirements Case study: Commonalities of commercial RTOS Real-time Linux and Windows NT VxWorks Benchmarking of RTOS/RTS Research trends Oxford Dictionary of Computing: ”Any system in which the time at which output is produced is significant. This is usually because the input corresponds to some movement in the physical world, and the output has to relate to that same movement. The lag from input time to output time must be sufficiently small for acceptable timeliness.” Randell (1985) ”A real-time system is a system that is required to react to stimuli from the environment (including the passage of physical time) within time intervals dictated by the environment.” TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems What is a Real-time System? Interface A real-time system is any information processing system which has to respond to externally generated input stimuli within a finite and specified period. Correctness depends not only on the logical result but also the time it was delivered! Failure to respond is as bad as the wrong response! The computer is a component in a larger engineering system => EMBEDDED COMPUTER SYSTEM. TDDI12 - Real-Time Operating Systems Input flow reading Pipe Flow meter Processing Output valve angle Time Valve Computer TDDI12 - Real-Time Operating Systems 1 Misconceptions about Real-Time Systems MISCONCEPTION #1: ”Real-time computing is equivalent to fast computing!” CORRECT: Real-time computing aims at predictability foremost, and secondly, efficiency. Predictability: Time-cognizant behavior of the system, i.e., the system is designed and developed to enforce temporal correctness, preferably by direct awareness and explicit notion of time. Misconceptions about Real-Time Systems Misconception #3: ”There is no science in real-time system design!” ”Real-time research is about performance tuning!” CORRECT: Methods for systematic and explicit treatment of temporal behavior to enforce overall system correctness MISCONCEPTION #2: ”Advances in hardware will solve real-time problems” CORRECT: Improving hardware performance is always desirable, but does not imply (without support of analysis) that predictability has been maintained/achieved (e.g., release times of tasks). TDDI12 - Real-Time Operating Systems Hard real-time – systems where it is absolutely imperative that responses occur within the required deadline Soft real-time – systems where deadlines are important but which will still function correctly if deadlines are occasionally missed. Firm real-time – systems which are soft real-time but in which there is no benefit from late delivery of service A single system may have all hard, soft and firm real-time subsystems. (Value of result relative to time) Hard real-time system: utility vv(t) i (t ) ri Release time di Deadline TDDI12 - Real-Time Operating Systems Soft Real-Time time TDDI12 - Real-Time Operating Systems Soft Real-Time (cont.) Multiple requirements Deadlines can be missed occasionally, typically with an upper limit of misses within a defined interval, e.g., a constraint on the maximum number of consecutive deadline misses Service can occasionally be delivered late, with an upper bound on tardiness, i.e., a.k.a. deadline tolerance. Utility functions for soft real-time systems: utility Decay rate Lin ear Ex pon enti al ri utility loss Utility Function benefit Terminology TDDI12 - Real-Time Operating Systems penalty di di + time Hard real-time is hard, but soft real-time is harder! -p - infinity TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems 2 Characteristics of a RTS Where are OSs found? General purpose systems Embedded systems Microprocessor market shares in 1999 99% Large and complex OR small and embedded vary from a few hundred lines of assembler or C to 20 million lines of Ada estimated for the Space Station Freedom Concurrent control of separate system components devices operate in parallel in the real-world, hence, better to model this parallelism by concurrent entities in the program Facilities to interact with special purpose hardware need to be able to program devices in a reliable and abstract way 1% TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Characteristics of a RTS Extreme reliability and safety Embedded systems typically control the environment in which they operate. Failure to control can result in loss of life, damage to environment or economic loss. Algorithms for Digital Control Real-Time Clock We need to be able to predict with confidence the worst case response times for systems; Efficiency is important but predictability is essential • In RTS, performance guarantees are: • Task- and/or class centric • Often ensured a priori • In conventional systems, performance is: • system oriented and often throughput oriented • post-processing (… wait and see…) Data Logging Remote Monitoring System Data Retrieval and Display Display Devices Database Operator’s Console Operator Interface TDDI12 - Real-Time Operating Systems Real-Time Programming Languages Engineering System Guaranteed response times Interface Assembly languages Sequential systems implementation languages — e.g. RTL/2, Coral 66, Jovial, C. Both normally require OS support. High-level concurrent languages. Impetus from the software crisis. e.g. Ada, Chill, Mesa, Java, Erlang… No operating system support! Real-Time Languages and OSs User Programs Operating Hardware System Typical OS Configuration TDDI12 - Real-Time Operating Systems Real-Time Computer TDDI12 - Real-Time Operating Systems User Program Including Operating Hardware System Components Typical Embedded Configuration TDDI12 - Real-Time Operating Systems 3 Observation... Time to think… There is currently about 100-140 different types of OS for embedded and/or real-time applications available... (number depends on how generous your definition of realtime and embedded is...) Why are conventional operating systems not used as a platform for most (hard) real-time systems? However, a recent survey indicates that 66% of embedded systems developed in Japan avoid having an OS, instead implementing the necessary functionality directly... Are there any - Design mismatches? - Sources of unpredictability? More tailorability of OSs for embedded and real-time applications is required... What is likely to happen if one uses a conv. OS? TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Questions to be Answered Before Design Starts (and choosing an RTOS!) GPOS RTOS Req’s What are the worst case performance requirements for each activity? What are the number and complexity of activities to be performed? How should these activities be distributed among the software tasks so that the processor load is balanced (i.e., optimize cost/performance)? What is the degree of coupling of these tasks (criticality, type of data flow, precedence constraints, event interdependencies)? How much RAM and ROM does the hardware design provide? How much RAM and ROM will be consumed for the specific set of tasks, ISRs, queues, and so on? How should we avoid priority inversion or deadlock for resources? Real-Time Operating Systems TDDI12 - Real-Time Operating Systems RTOS criteria All of this is possible using Linux or even Windows 95 So why is there a multitude of RTOSes? TDDI12 - Real-Time Operating Systems RTOS Req’s RTOS Req’s What OS would you select to build an UMTS base station? IP router? Tele text module? Dialysis machine? Avionics GPS receiver? Speech coder for GSM terminals? Application-Specific OS Embedded applications High degree of configurability and extensibility required SPIN 2K VEST S.H.a.R.K. Etc... TDDI12 - Real-Time Operating Systems RTOS criteria Why RTOS? ASOS General Purpose OS Too costly for embedded applications Increased demand on RT functionality Windows NT, 2K, XP… Solaris, HP-UX Linux Etc... Common technical design criteria when selecting RTOS CPU and memory requirements Kernel model Process model Scheduling model Memory management model Interrupt model Inter process communication (IPC) Fault management model Device driver model (DD) Program loading and continuous operation TDDI12 - Real-Time Operating Systems 4 RTOS criteria Common technical design RTOS criteria Non-technical design criteria when selecting RTOS criteria when selecting RTOS Add-on products (protocols, memory protection, file systems...) Integrations with 3rd party products Performance of all this Safety provisions Security provisions Development environment, tools RTOS Req’s RTOS Req’s Nice sales people Price, pricing model Support (responsiveness, knowledge, language, time zone, ...) Classes (level, where, when) Consultants, staff with knowledge about the RTOS Perceived quality Source code availability (and responsibility) Delivery time Stable (=large) supplier Exciting demos … TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems ASOS RTOS criteria 1/6 Choosing CPU RTOS Req’s RTOS Req’s Some choose CPU first, RTOS later. Some vice versa. Price per unit Computation power Interrupt latency Available compilers Integrated functions on die (built-in Ethernet controller, …) Physical size Power consumption, heat dissipation requirements Instruction size (memory consumption) Interference resistance (RFI) Factory mounting handling Task assignment and scheduling (CPU management) Resource management (memory, disk, I/O, etc) Communication protocols Failure management and recovery Clock synchronization Configurability * Modularity * Extensibility - kernel optional components (SSP) System Service Providers. * Particularly true for ASOS, less so for GPOS TDDI12 - Real-Time Operating Systems Terminology Scheduling: order tasks such that a metric is maximized/minimized Real-time: guarantee hard deadlines, minimize the number of missed deadlines, minimize lateness Dispatching: carry out the execution according to the schedule Preemption, context switching, monitoring etc. TDDI12 - Real-Time Operating Systems Preliminaries GPOS Basic RTOS Requirements Admission control: Filter tasks coming into the system and thereby make sure the admitted workload is manageable activation Scheduling is the issue of ordering the use of system resources A means of predicting the worst-case behavior of the system scheduling dispatching ….. termination Execution preemption Allocation: designate tasks to CPUs and (possibly) nodes. Precedes scheduling. TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems 5 Non-Real-Time Scheduling (as covered earlier) Primary Goal: maximize performance Secondary Goal: ensure fairness Typical metrics: • minimize response time • maximize throughput • e.g., FCFS (First-Come-First-Served), RR (Round-Robin) Example: Workload Characteristics Tasks are preemptable, independent with arbitrary arrival times Tasks have deadlines (D) and known computation times (C) Tasks execute on a uni-processor system Example setup: C1 T1 D1 C2 T2 D2 C3 T3 D3 C4 D4 T4 TDDI12 - Real-Time Operating Systems Example: Non-preemptive FCFS Scheduling TDDI12 - Real-Time Operating Systems Example: Round-Robin Scheduling Missed deadline!! Missed deadline!! T1 T1 T2 T2 T3 T3 T4 T4 TDDI12 - Real-Time Operating Systems Real-Time Scheduling TDDI12 - Real-Time Operating Systems Scheduling: Problem Space Primary goal: ensure predictability Secondary goal: ensure predictability Typical metrics: guarantee miss ratio = 0 (hard real-time) guarantee Prob(missed deadline) < X % (firm real-time) maximize completion ratio / minimize miss ratio (firm real-time) minimize overall tardiness; maximize overall usefulness (soft real-time) e.g., EDF (Earliest Deadline First), LS (Least Slack), RMS (Rate-Monotonic Scheduling) Recall: Real-time is about enforcing predictability, and does not equal fast computing! TDDI12 - Real-Time Operating Systems Easier -> more difficult Possible -> desirable Heuristic -> Approximation (provably within X % of opt.) -> Provably optimal Uni-processor / multiprocessor / distributed system Periodic / sporadic / aperiodic tasks Independent / interdependent tasks Preemptive / non-preemptive Static (at system build time) / dynamic (at run time) Off-line (all tasks initially given) / on-line (don’t know future) Handle transient overloads Support fault tolerance TDDI12 - Real-Time Operating Systems 6 Task Assignment and Scheduling Cyclic executive scheduling (-> later) Fixed set of processes (tasks) Cooperative scheduling The scheduler relies on the current process to give up the CPU before it can start the execution of another process. Processes are periodic, with known periods Processes are independent of each other A static priority-driven scheduler can preempt the current process to start a new process. Priorities are set assigned pre-execution. E.g. Rate-monotonic scheduling (RMS/RMA) System overheads, context switches etc., are ignored (zero cost) Processes have a deadline equal to their period i.e., each process must complete before its next release Processes have fixed worst-case execution time A dynamic priority-driven scheduler can assign, and possibly also redefine, process priorities at run-time. E.g., Earliest Deadline First (EDF) TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Terminology: Temporal Scope of a Task C - Worst-case-execution time of the task D r - Deadline of task, i.e., the latest time by which the task should be complete. - Release time N, n - Number of tasks in the system π P J Performance Metrics Completion ratio / miss ratio Maximize total usefulness value (weighted sum) Maximize value of task Minimize lateness Minimize error (imprecise tasks) - Priority of the task - Minimum inter-arrival time (period) of the task periodic - inter-arrival time is fixed sporadic - minimum inter-arrival time aperiodic - random (Poisson distribution) • => bursty (no worst case) - Release jitter of a process TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Scheduling Approaches (Hard RTS) Cyclic Executive Approach Off-line scheduling /analysis (static analysis + static scheduling) All tasks, times and priorities given a priori (before system startup) Time-driven; schedule computed and hardcoded before system startup E.g., Cyclic executives Inflexible May be combined with static or dynamic scheduling approaches Fixed priority scheduling (static analysis + dynamic scheduling) All tasks, times and priorities given a priori (before system startup) Priority-driven, dynamic(!) scheduling: The schedule is (re-)constructed by the OS scheduler at run time For hard / safety-critical systems E.g., RMA/RMS - Rate Monotonic Analysis/Scheduling, … Dynamic priority scheduling / On-line scheduling Scheduler does not know about future task arrivals and their priorities Priority-driven For hard / best-effort systems E.g., EDF - Earliest Deadline First, (Least Slack) … TDDI12 - Real-Time Operating Systems Clock-driven (time-driven) scheduling algorithm Off-line algorithm Minor Cycle (e.g. 25ms) Major Cycle (e.g. 100ms) - gcd of all periods - lcm of all periods Construction of a cyclic executive is equivalent to bin packing Cyclic Exec. RTOS Req’s Simple Process Model Process A B C D E Period 25 25 50 50 100 Comp. Time 10 8 5 4 2 TDDI12 - Real-Time Operating Systems 7 Cyclic Executive (cont.) Observations Wait_For_Interrupt; Procedure_For_A; Procedure_For_B; Procedure_For_C; Wait_For_Interrupt; Procedure_For_A; Procedure_For_B; Procedure_For_D; Procedure_For_E; Wait_For_Interrupt; Procedure_For_A; Procedure_For_B; Procedure_For_D; end loop; Cyclic Exec. loop Wait_For_Interrupt; Procedure_For_A; Procedure_For_B; Procedure_For_C; Cyclic Exec. Cyclic Executive (cont.) No actual processes exist at run-time Each minor cycle is just a sequence of procedure calls The procedures share a common address space and can thus pass data between themselves. This data does not need to be protected (via a semaphore, for example) because concurrent access is not possible All `process' periods must be a multiple of the minor cycle time. TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Cyclic Executive (cont.) Disadvantages Dynamic Scheduling: States of Process/Task With the approach it is difficult to: incorporate sporadic processes; incorporate processes with long periods; major cycle time is the maximum period that can be accommodated without secondary schedules scheduling Run preemption construct the cyclic executive; and signal from resource handle processes with sizeable computation times. any `process' with a sizeable computation time will need to be split into a fixed number of fixed sized procedures wait on busy resource Waiting TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Schedulability Test RMA Assumptions Test to determine whether a feasible schedule exists Exact Test The task set is schedulable if and only if it passes the test. Sufficient Test + if test is passed, then tasks are definitely schedulable – if test is not passed, tasks may be schedulable, but not necessarily termination Ready Necessary Test + if test is passed, tasks may be schedulable, but not necessarily – if test is not passed, tasks are definitely not schedulable TDDI12 - Real-Time Operating Systems A1: Tasks are periodic (activated at a constant rate). Period Pi = interval between two consecutive activations of task τi A2: All instances of a periodic task τi have the same computation time Ci A3: All instances of a periodic task τi have the same relative deadline, which is equal to the period (Di = Pi). A4: All tasks are independent (i.e., no precedence constraints and no resource constraints). RMA Cyclic Exec. task activation Implicit assumptions: A5: Tasks are preemptable A6: No task can suspend itself A7: All tasks are released as soon as they arrive A8: All overhead in the kernel is assumed to be zero (or part of Ci). TDDI12 - Real-Time Operating Systems 8 Example: Rate Monotonic Scheduling Rate Monotonic Scheduling: Principle Principle: Each process is assigned a (unique) priority based on its period; the shorter the period, the higher the priority Pi < Pj ⇒ πi > πj ). (1= low priority) W.l.o.g. number the tasks in reverse order of priority Period 25 60 42 105 75 Priority 5 3 4 1 2 T4 Name T1 T3 T2 T5 T4 T3 RMA RMA Process A B C D E Tasks and Task instances (jobs), Release times / Periods, Computation times T2 T1 TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Example: Rate Monotonic Scheduling Utilization Gantt-Chart for RM schedule Ui = Ci / Pi P4 P4 T4 T4 T3 T3 RMA RMA utilization of task τi T2 T2 T1 T1 TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems RMA: Schedulability test #1 Example with RMA Theorem (Utilization-based Schedulability Test): A periodic task set τ1, τ2, ...τn, with Di = Pi , 1 ≤ i ≤ n, is schedulable by the rate monotonic scheduling algorithm if n Ci ≤ n(21/ n −1), n = 1,2,..... ∑ i =1 Pi RMA n This schedulability test is ”sufficient”! 1 2 3 5 10 n →∞ 100.0% 82.8% 78.0% 74.3% 71.8% 69.3% (ln 2) For harmonic periods (Tj evenly divides Ti) , the utilization bound is 100% For the average workload the utilization bound is ~ 88-92%. TDDI12 - Real-Time Operating Systems τi = < Pi , Ci > Τ = {τ1 = < 10 , 5 >, τ2 = < 20 , 5 >, τ3 = < 30 , 5 >} Given Utilization Bound (in %) n Ci ∑ P ≤ n(2 i =1 i 1/ n − 1), n = 1,2,..... n We get ∑ (5 / 10 + 5 / 20 + 5 / 30) = 0.916 which is > n( 21/ n − 1) = 0.78 i =1 RMA U1 + ....+ U n ≤U n* Here, the utilization-based schedulability test fails. A ”false negative” case ?! TDDI12 - Real-Time Operating Systems 9 Phase RMA: Towards Schedulability test #2 Phase Ii: release time of (the first job of) a periodic task τi Lemma: The longest response time for any job of τ1, τ2, ...τn occurs for the first job of τi when I1=I2= ... = In = 0. □ τn The case when I1=I2= ... = In = 0 is called a critical instant, because it results in the longest response time for the first job of each task. Consequently, this creates the worst case task set phasing and leads to a criterion for the schedulability of a task set. τi t=0 Ii Two tasks τi , τj are in phase if Ii=Ij τn RMA RMA W.l.o.g. set the time origin (t=0) to the minimum phase in the task set τi Theorem: (Schedulability test #2) A periodic task set can be scheduled by a fixed priority scheduling algorithm if the deadline of the first job of each task is met when using the scheduling algorithm starting from a critical instant. □ Proof: Liu and Layland, 1973 Approach: draw Gantt-Charts for testing schedulability! TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Proof: Lemma for Schedulability test #2 Proof: Schedulability test #2 - Figure Prove that critical instant is the worst case . τn Let Τ={τ1, τ2, ...τn} be the set of periodic tasks ordered by increasing periods (i.e., τn has the longest period, and thus, according to RMS, τn has the lowest priority). τi The response time of task τn is delayed by the interference of task τi with higher priority. τi Advancing the release time of τi may increase the completion time of τn (but will never decrease it). As a consequence, the response time of τn is largest when it is released simultaneously with τi . τn RMA RMA Response time of τn is delayed due to interference of a task τi with higher priority: τn τi The interference may increase when advancing the release time of τi . TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems RMA: Optimality Example with RMA (as above) Theorem (Optimality of RM scheduling): The rate monotonic scheduling algorithm is optimal among all fixed priority scheduling algorithms for scheduling periodic task sets where the deadline of tasks equals the end of their period (Di = Pi). τi = < Pi , Ci > Τ = {τ1 = < 10 , 5 >, τ2 = < 20 , 5 >, τ3 = < 30 , 5 >} Given n i =1 n We get Ci ∑ P ≤ n(2 i 1/ n − 1), n = 1,2,..... ∑ (5 / 10 + 5 / 20 + 5 / 30) = 0.916 which is > n( 21/ n − 1) = 0.78 i =1 RMA RMA But, there may be dynamic-priority algorithms that do better… (recall the example P1<50,25>, P2<80,35>) Image source: Silberschatz, Galvin, Gagne 7th ed. © Wiley 2005 TDDI12 - Real-Time Operating Systems Here, the utilization-based schedulability test fails. Next test, draw a Gantt-chart and check schedulability TDDI12 - Real-Time Operating Systems 10 Example with RMA continued Optimizations Reduce worst case execution times Optimization/Redesign of program Alternative/contingency actions τ2 Relax periods τ3 Note: RMA is optimal even when worst case execution times are not known. However, priority is then increasingly important. Thus, important processes must have high priorities even though they may have long periods. Solution: Transformation of periods! τ1 10 30 20 RMA RMA 0 Task τ3 meets its deadline and, thus, τ1 and τ2 also meet their deadlines. Hence, workload is schedulable!! TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems RMA Implementation Fixed priorities -> use pre-sorted array of PCB references … PCB1 PCB2 PCB3 PCB4 RMA τ1 τcurrent PCBn τi τn On release of new task τi : state(τi ) = READY; if (i < current) { preempt τcurrent; state(τcurrent) = READY; state(τi) = EXECUTING; dispatch τi to CPU; } else keep τcurrent on CPU and set state(τi ) = READY; small n. A static binary tree solution On termination of task τi: would allow for O(log n) insert/find state(τi) = NOT_READY; Find first READY task (e.g., by linear search). If there is one, dispatch it (τcurrent ), otherwise idle. Remark: Find in O(n) time is ok for EDF EDF Assumptions A1: Tasks are periodic or aperiodic. They are activated at a constant rate. The interval P between two consecutive activations is the period of the task. A2: All instances of a periodic task τi have the same computation time Ci A3: All instances of a periodic task τi have the same relative deadline, which is equal to the period (Di = Pi). A4: All tasks are independent (i.e., no precedence constraints and no resource constraints). Implicit assumptions: A5: Tasks are preemptable A6: No task can suspend itself A7: All tasks are released as soon as they arrive A8: All overhead in the kernel is assumed to be zero (or part of Ci). TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Example: Non-preemptive EDF Scheduling is Not Optimal EDF Scheduling: Principle Principle: Preemptive priority-based dynamic scheduling (as before) Each process is assigned a (current) priority based on how close its deadline is: the shorter the deadline, the higher the priority: Di < Dj ⇒ πi > πj (1= low priority) Missed deadline!! τ1 τ2 τ3 Example [SGG7]: P1 <50, 25>, P2 <80, 35> EDF EDF τ4 Image source: Silberschatz, Galvin, Gagne 7th ed. © Wiley 2005 TDDI12 - Real-Time Operating Systems Note: EDF is not optimal for non-preemptive scheduling problems. This picture demonstrates, however, the problem of non-preemptive tasks. TDDI12 - Real-Time Operating Systems 11 Example: Preemptive EDF Scheduling EDF Optimality Theorem (Optimality of EDF wrt. schedulability): Given a set of n independent tasks with arbitrary arrival times, any algorithm that, at any instant, executes the task with the earliest absolute deadline among all the ready tasks is optimal with respect to feasibility (i.e., schedulability). τ1 τ2 Proof: Next slide. τ3 Arrival time Feasible schedule!! Deadline EDF EDF τ4 Theorem (Optimality minimizing lateness) Given a set of n independent tasks with arbitrary arrival times, any algorithm that, at any instant, executes the task with the earliest absolute deadline among all the ready tasks is optimal with respect to minimizing the maximum lateness. Proof: [Dertouzas 1974] TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems EDF Example: Domino Effect EDF: Schedulability test Theorem (Utilization-based Schedulability Test): A task set τ1, τ2, ...τn, with Di = Pi , 1 ≤ i ≤ n, is schedulable by the earliest deadline first (EDF) scheduling algorithm if τnew Domino effect!!! τ1 U1 + .... + U n ≤ 1 τ2 Ci ≤ 1, n = 1,2,..... ∑ i =1 Pi n τ3 Schedulability test is ”exact”! Maximimizes CPU utilization EDF EDF τ4 Dynamic workloads may cause transient overloads scenario, i.e., we need to protect the system of being presented with unschedulable workloads!!! Dertouzos (1974): EDF minimizes lateness of the ”most tardy” task TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Dynamic Scheduling: Scheme of Guarantee Mechanism EDF Guarantee Test scheduling activation acceptance test termination Ready Run preemption Waiting wait on busy resource EDF signal from resource EDF NO YES Perform guarantee at run-time when tasks have dynamic activations the arrival times are not known a priori execute test every time there is a new task coming into the system Acceptance test is also known as performing ”admission control”! TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems 12 Methodology Let Ψ represent the set of already admitted tasks, i.e., tasks that are already guaranteed by schedulability tests. Let τnew be the new task arriving to the system We need to prove that Ψ’ =Ψ U {τnew} is schedulable! Assume that tasks in Ψ’ are ordered by increasing deadlines (without loss of generality) Let ci be the remaining execution time of τi. ci initially has the value of C i, and can be updated whenever τi is preempted). EDF Guarantee Test: Methodology Methodology (cont.) Hence, at time t, the worst-case finishing time (fi) of task τi can be computed as i fi = ∑ ck (t ) k =1 Schedulability can be guaranteed by i ∀i = 1,...,n fi = ∑ ck (t ) ≤ di EDF EDF EDF Guarantee Test: Methodology k =1 TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Algorithm EDF_guarantee (Ψ , τnew) { Ψ’ = Ψ U {τnew} (* ordered by deadline *) t = current_time(); f0 = 0; for (each τi in Ψ’) { fi = fi-1 + ci(t); if (fi > Di ) return (INFEASIBLE); } return (FEASIBLE); } Algorithm completes in time O(n). EDF and periodic tasks EDF has dynamic priority assignment and is intrinsically preemptive. EDF does not make any assumption about arrival rate, thus it can be used for periodic tasks. Under assumptions A1-A4 for RMA, the schedulability of an aperiodic task set handled by EDF can be verified through the processor utilization test for EDF. Thus, a set of periodic tasks is schedulable with EDF if and only if n Ci ∑ P ≤ 1, n = 1,2,..... EDF EDF EDF Guarantee Test: Algorithm i =1 i Note that the actual execution order will be different with EDF than with RMA. TDDI12 - Real-Time Operating Systems EDF Implementation Dynamic vs. Fixed Priority Scheduling EDF has higher run-time overhead than RMA: Need to keep PCB’s / TCB’s sorted by next deadline Operations: • deletemin(), to dequeue the task with closest deadline • insert(τi, di), to enqueue a task with its next deadline (when the current instance has terminated) TDDI12 - Real-Time Operating Systems i.e., a Priority Queue There exist data structures for ADT Priority Queue with deletemin() in (worst-case) time O(log n) insert() in (worst-case) time O(log n) E.g., a balanced search tree (2-4, red-black, …) with doubly linked list of leaves containing PCB’s, ordered by deadline, plus a pointer to the leaf with minimum deadline TDDI12 - Real-Time Operating Systems Dynamic Priority Scheduling (e.g., EDF) Dynamic scheduling With guarantee test: Tasks may be added dynamically on-line no a priori knowledge + flexibility + adaptability + few assumptions about workload characteristics - prone to overloads - non-negligible run-time overhead, more critical at high loads - only statistical guarantees More suitable for soft/firm RTS Fixed Priority Scheduling (e.g., RMA) Dynamic scheduling, but static analysis (RMA) or off-line scheduling assumes clairvoyance schedules planned for peak load + task workloads are periodic => no overloads + low (RMA) or no (off-line) overhead at run time + simple to implement - Conservative -> lower utilization Most suitable for hard RTS TDDI12 - Real-Time Operating Systems 13 What if the schedulability test failed? Real-Time OS What to do if the schedulability test failed for my system? Aggressive code optimization (compiler issue) (may help more or less) Re-negotiate the specification (i.e., deadlines) Micro-kernel System calls • Traps: case of…. create_thread, timer_sleep, open, read, … Re-design the tasks to take less execution time Nano-kernel Time and scheduling services, and clock interrupts (so called ”ticks”) RTOS Req’s Get a faster CPU, or replicate hardware (e.g., multiple CPUs) hardware re-design and/or larger/more expensive chip higher power consumption may also require recoding the program Time and scheduling Services Hardware and Software Exceptions External interrupts • Immediate Interrupt Service • Scheduling In short: scheduling, synchronization and interrupt handling TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Real-Time OS cont’d Nanokernel functionality Additional services are provided in kernel optional components, often called System Service Providers (SSPs) E.g., file system support, network support Normally, all tasks/threads scheduled with the same policy Task creation Task management - scheduling and dispatching Timing and interrupt management Time critical tasks are activated by the fast service interrupt routines (thus bypassing the kernel) Other tasks are controlled by the kernel RTOS Req’s RTOS Req’s TDDI12 - Real-Time Operating Systems Microkernel structure External Interrupts HW / SW exceptions TDDI12 - Real-Time Operating Systems Microkernel Functionality (1) [Liu’2000] 12.1.2 System calls Clock interrupts Functionality of nanokernel + … Trap Immediate Interrupt Service Case of… Time services and scheduling RTOS Req’s Create_thread Suspend_thread … Create_timer Timer_sleep Timer_notify … Open Read … Scheduling Use of BSP = board support package Custom and standard hardware design Simple development when interfacing software for new hardware designs Board specific functions, including general initialization, RTOS initialization and interrupt configuration Device-specific driver software supplied in template form (configured by the programmer) Support for development of special-purpose BSP functions Detailed low-level code used by the device drivers Return from exception TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems 14 Microkernel Functionality (2) Process (task) scheduling and control Declare, start, stop, destroy a task Set task priorities Lockout (make it non-preemptive), unlock, delay, resume a task Control the real-time clock (tick, relative time, and absolute time functions) Control use of interrupts RTOS Req’s System setup and special functions Initialize the OS (if not part of the BSP) Set up all special (non-kernel) interrupt functions Start execution of the application programs RTOS Req’s Microkernel Functionality (3) Mutual exclusion Gain control using semaphores (entry to critical region) Release control using semaphores (exit from critical region) Gain/release control using monitors Wait in monitors Synchronization functions (no data transfer) Initialize, send, wait, or wait for a signal flag Data transfer (without synchronization) Initialize a channel/pool Send to, receive from, or check state of a channel Write to, read from a pool TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Microkernel Functionality (4) Synchronization with data transfer Set up, post to, pend to, and check on a mailbox Dynamic Memory allocation Allocate and deallocate a block of memory ”General-purpose” Embedded RTOS Functionality of microkernel + RTOS Req’s RTOS Req’s TDDI12 - Real-Time Operating Systems Threads and Tasks (TCB) TDDI12 - Real-Time Operating Systems RTOS criteria 2/6 Memory management model TCB/PCB Thread/Process Control Block TDDI12 - Real-Time Operating Systems Memory management is central in embedded systems Virtual memory Different models for different OS’s Paging / Swapping Often requirement not to have this • SoU: Source of Unpredictability RTOS Req’s RTOS Req’s In addition to normal PCBs: Scheduling information Synchronization information Time usage information Timer information Task Parameters Phase, Period, Number of instances, Event lists, Relative Deadline Industrial networking, e.g., CAN, FieldBus etc General-purpose networking, e.g., Ethernet, ATM many embedded systems also have the ability/need to act as mini-web-servers Graphics-based user interfaces Long-term persistent storage disks, battery-backed RAM Fragmenting The most difficult problem (aka real-time garbage collection) to solve. Very important to solve well. (SoU) Multiple pools Different parts of the system have separate memory pools, so one part of the system may run out of memory, not affecting any other. TDDI12 - Real-Time Operating Systems 15 RTOS criteria Memory management model 2/6 Memory Management Fragmentation Virtual memory mapping (opt) Enables dynamic applications (e.g., task migration) Cost: Fragmentation[%] = 1 – (largest free block) / (free memory) 100 % • maintenance of address translation table • Complicates DMA-controlled I/O Alternative: Real addressing mode (P=L) Paging (opt) Memory locking for controlled paging needed Necessary for non-real-time applications needed during development to execute on target Memory Protection (opt) Enables reduced retesting when system is changed 50 % RTOS Req’s RTOS Req’s 0% External vs. internal fragmentation TDDI12 - Real-Time Operating Systems RTOS criteria Memory management model 2/6 t TDDI12 - Real-Time Operating Systems RTOS criteria 3/6 Interrupt model Fragmentation Largest free block 50kB Free memory 5MB Fragmentation[%] = 1 – (largest free block) / (free memory) ⇒ 99% fragmentation Interrupts are central in embedded systems Interrupts have higher priority than other processes Interrupts are prioritized among each other Interrupts with higher priority preempt lower priority interrupts Interrupts may be periodic or sporadic 100 % 50 % t test e.g. 2 hours t crash t RTOS Req’s RTOS Req’s 0% The time from an interrupt assertion until processing starts is called interrupt latency. e.g. 2 weeks TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Time Services: Clock Interrupt Basic steps HW and SW Interrupt Priorities: Hierarchy RTOS Req’s RTOS Req’s System shutdown priority Power down priority Clock interrupt priority Highest interrupt priority : Lower interrupt priority Dispatcher/scheduler priority Highest thread priority : Lowest thread priority TDDI12 - Real-Time Operating Systems The kernel performs the following at each ”tick”: Processes timer events A clock device has a timer queue Checks queue to determine if some timer events have occured Pending expiration times of all timers bound to the clock are stored in time order in the timer queue of the clock device Carries out the actions before returning control to user Updates execution budget (= time quantum in priority-RR) Decrements the budget of a thread/process of a tick size (e.g., time slice) • If budget expires, kernel preempts the thread Updates the ready queue and returns control Release of threads/processes that become ready (periodic) Note: Commercial OSs do not support periodic threads/processes TDDI12 - Real-Time Operating Systems 16 Time Services: Clock Interrupt External Interrupts Additional step: Update of monitoring information Interval timers, e.g., total CPU consumption RTOS solution: two steps in handling interrupts 1. Immediate Interrupt Service 2. Scheduled Interrupt Service Tick Hence, systems do not rely on tick scheduling only! Normally, a combination of tick-driven and event-driven scheduling Kernel invokes scheduler to update ready queue in case of thread creation, wakeup, release etc. RTOS Req’s RTOS Req’s Small tick size Æ fine granularity but high overhead Big tick size Æ coarse granularity but reasonable overhead Service time of an interrupt varies based on source of interrupt, e.g., DMA, network interface etc. Î unnecessary long blocking times This is also called Split Interrupting Handling TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems Immediate Interrupt Service Last step (see previous slide) instead invokes another ISR to complete interrupt handling + ISR itself becomes preemptable + Scheduled at software priority in a priority-driven system Executed as kernel threads (since they may modify kernel data) RTOS Req’s RTOS Req’s Interrupt handling executed at an interrupt priority level: Push necessary TCB/PCB data on interrupt stack Branch to starting address of kernel’s interrupt handling code Disable interrupts temporarily Modify data structures that might also be modified by interrupt service routines Save processor state on interrupt stack Enable interrupts Call interrupt service routine of the interrupting device = scheduled interrupt service Scheduled Interrupt Service TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems RTOS criteria 3/6 Interrupt latency Interrupt Latency Image source: Silberschatz, Galvin, Gagne 7th ed. © Wiley 2005 TDDI12 - Real-Time Operating Systems Max is the worst case latency The difference between Max and Min gives the length of the longest interrupt-disabled region Average gives interrupt throughput Latency is typically between a few hundred us (M68k) and a few ten ns (Texas Instruments DSP TMS320C62) RTOS Req’s RTOS Req’s Interrupt latency = Sum of the times it takes to + save the context of interrupted thread, identify the interrupting device and get the starting address of the interrupt service routine + complete current instructions, jump to the trap handler and interrupt dispatcher part of the kernel + disable interrupts + complete immediate interrupt service routines + start the scheduled interrupt service routine of the interrupting device TDDI12 - Real-Time Operating Systems 17 RTOS criteria 3/6 Interrupt latency Dispatch time Hardware interrupt assertion 40 ns at best RTOS ISR (Interrupt service routine) with no access to RTOS system calls RTOS Max exists? Interrupt process or ISR (Interrupt service routine) Desktop OS RTOS Req’s RTOS Req’s 2 us max at best TDDI12 - Real-Time Operating Systems RTOS criteria IPC 4/6 Inter process communication RTOS criteria Fault management model 5/6 Recovery strategy Forward recovery Traditional programming, find out what went wrong, fix it, try again. E.g. if parameter value exceeds max allowed, set to max allowed, continue. Backward recovery Important to really remove cause of error ⇒ backward recovery Backward recovery ⇒ kill processes abruptly Processes can die any time ⇒ RTOS Req’s RTOS Req’s IPC is extremely central in embedded systems Pure semaphores + shared memory Mail boxes Messages / Signals (not UNIX signals!) Pipes Sockets Many RTOSes have all of the above, with varying success. TDDI12 - Real-Time Operating Systems • Semaphores and mail boxes cannot be used easily • Clean-up by resource owner (server), not self (client) • Process supervision required TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems ASOS RTOS criteria 6/6 Program loading / unloading / updating In long running systems, such as base stations, switches, routers it is vital to be able to replace parts of the hard- and software in runtime In telecom, ”5 nines” is often mentioned, meaning system availability 99,999% of any given period of time. For example, 5 minutes planned and unplanned downtime/year Just as important to unload programs as to load Replacement of hardware sometimes called hot swapping, “plug & play” (or “plug & pray”) TDDI12 - Real-Time Operating Systems Case Study RTOS Req’s Commonalities of Commercial RTOS GPOS Compliant or partially compliant with POSIX 1003.1b (RT) (standardized 1993/95/96) [Liu’2000] Appendix Own set of API functions – portability at source code level Preemptive fixed-priority scheduling (>32 priority levels) [SGG7] 19.5.4 Synchronization, RT signals, timers Asynchronous/prioritized/synchronous I/O, memory locking May support only threads or processes (not necessarily both) Modularity and scalability Small (fixed) kernel, operating system configurable Speed Microkernel systems most common Low context switch time, interrupt latency, semaphore latency... System calls Nonpreemptable parts necessary for mutual exclusions are short. TDDI12 - Real-Time Operating Systems 18 ASOS ASOS GPOS Commonalities of Commercial RTOS cont’d GPOS Commonalities of Commercial RTOS cont’d Split Interrupt Handling Scheduling FIFO or Round-robin for equal-priority threads Dynamic alteration of priority possible (not POSIX-RT) No support for efficient and adequate support for True for the following operating systems: LynxOS pSOS QNX VRTX VxWorks • EDF scheduling • Hook for user-level bandwidth servers Priority Inversion Control Support for priority inheritance (enable/disable) Possibly ceiling-priority protocol Memory Management May provide virtual memory, no paging Sometimes support for memory protection Case Study Case Study TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems GPOS Windows NT (2K, XP…) and Real-Time? Real-Time Linux Timer and clock resolutions are sufficiently fine for all but the most time-stringent applicatons. Weak support for real-time scheduling and resource-access control no support for priority inheritance Unpredictable interrupt handling and IPC mechanisms Support for immediate interrupt with good interrupt latency SoU: DPCs are executed in FIFO order at a priority lower than HW interrupts but higher than scheduler/dispatcher. Hence, high-priority threads can be blocked by DPC (second step of ISR) caused by lower-priority threads, and blocking time is unbounded. :-O Embedded Linux projects MiniRTL Extension to RTLinux ELKS project HA-Linux (High Availability) Lineo Embeddic VME Linux Project Case Study Large memory footprint (rules NT & Co. out when size is an issue) Case Study Real-Time Linux projects: UTIME RT-Linux platform KURT platform RED Linux RTAI Linux/RK (from Mach/RK) TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems GPOS Linux Features/Shortcomings • Soft real-time scheduling (POSIX.1b) Other tasks: Time-sharing, dynamic priorities Clock and timer resolutions: 10 milliseconds (”jiffy”) Timing error can be large and unpredictable because of delays introduced by (i) kernel and (ii) large execution times of the timer functions TDDI12 - Real-Time Operating Systems Signal handling not RT POSIX Compliant No signal for application-defined use • The signals SIGUSR1 and SIGUSR2 are used by Linux Signals are not queued Signals may not be delivered in priority order Case Study Case Study Disabling of interrupts by Linux subsystems when they are in critical sections (disk system may disable for > 100ms), blocking clock interrupts Solution #1: Rewrite all offending drivers to make their nonpreemptable sections as short as possible Solution #2: Try to live with it Solution #3: Avoid it. Tasks = both processes and threads Clone: share memory with parent Scheduling: (first 100 priority levels) Fully preemptive (since kernel 2.6) Real-time tasks have static priorities; FIFO or RR Linux Features/Shortcomings cont’d TDDI12 - Real-Time Operating Systems 19 GPOS KURT Kansas University Real-Time Systems Linux UTIME Case Study Case Study Use both hardware clock and the Pentium time stamp counter Not tick driven Next timer interrupt occurs at the earliest of all future timer expiration times Calibration of timer is done when system is booted Timer interrupts periodically Compute length of jiffy in term of number of time stamp cycles per jiffy Compute number of time stamp cycles per second After system has booted, clock is reprogrammed to ”one-shot” mode Execution time of ISR is several times larger than in standard Linux. Uses UTIME Only for Firm RTS OS Modes: Focussed: only RT processes Mixed: RT and non-RT processes Normal: only non-RT processes Change performed when all processes in the real-time applications have registered to KURT (invocation times, sched policy etc) Real-time -> to non-real-time: unregister process Mode switch controlled by executive process, which computes a schedule of all events based on invocation times TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems GPOS GPOS KURT cont’d RTLinux KURT components Core, i.e., kernel with time-driven scheduler RTMods = real-time modules Provides functionality and is loadable Standard Linux kernel modules running in the same address space as Linux • driver may block timer interrupts significantly • Incorrect prioritization of file system processing Case Study Case Study Source of unpredictability (=not for hard RTS) large schedule files (large) only partially fits in mainmemory, requiring disc access Standard disk device driver, hence, Simulates interrupt disable/enable feature (flag) Interrupts are caught by real-time kernel (hardware interrupts are replaced by software emulated interrupts) Interrupts are relayed to Linux kernel when no real-time processes are scheduled for execution Interrupts are available to RTLinux, but disabled for Linux Linux kernel becomes completely preemptable Real-time kernel not preemptible, but optimized for not causing significant delays Preemptive fixed priority scheduler EDF and Rate Monotonic packages exist Runs on several hardware platforms Direct communication between processes on different kernels not possible TDDI12 - Real-Time Operating Systems TDDI12 - Real-Time Operating Systems GPOS RTLinux Architecture VxWorks 5.x Linux tasks System calls Device drivers RT tasks Linux Kernel Interrupts RTLinux Kernel Interrupts Hardware platform TDDI12 - Real-Time Operating Systems Case Study Case Study I/O TDDI12 - Real-Time Operating Systems 20 VxWorks 5.x [SGG7] 19.6 Case Study Popular RTOS supporting hard real-time systems Widely used in automobiles, consumer and industrial devices, network switches, routers etc. The 2 Mars rovers 2004 By WindRiverSystems Wind Microkernel: tasks = both processes and threads (with Pthreads API) Scheduling: Preemptive round-robin and FCFS Priority-Driven Scheduling with 256 priority levels Bounded interrupt and dispatch latency IPC: shared memory and message passing; • Semaphores and mutex locks • with priority inheritance protocol • pipes (file-buffered message passing) Information Sources RTS conferences RTSS, RTAS, RTCSA, ECRTS, Workshop on RT-Linux OS conferences and journals ACM Operating Systems Review ACM SOSP (Symposium on OS Principles) ACM ASPLOS (Arch. Support for Progr. Lang. And Syst.) Readings in Operating Systems IEEE Transactions on Parallel and Distributed Systems IEEE Transactions on Computers Magazines Dedicated Systems Magazine IEEE Computer, Communications of the ACM… TDDI12 - Real-Time Operating Systems URLs 2K Chimera Choice Coyote & Iguana Echidna eCos EROS ExoKernel HART ITRON KURT TDDI12 - Real-Time Operating Systems URLs cont’d http://devius.cs.uiuc.edu/2k/ http://www.ece.umd.edu/serts/bib/manuals/Chimera.shtml http://devius.cs.uiuc.edu/choices/ http://www.dsg.cs.tcd.ie/%7Ecoyote/ http://www.ece.umd.edu/serts/research/echidna/index.shtml http://sources.redhat.com/ecos/ http://www.eros-os.org/ http://www.pdos.lcs.mit.edu/exo.html http://hartik.sssup.it/ http://www.sakamura-lab.org/ http://www.ittc.ukans.edu/kurt/ TDDI12 - Real-Time Operating Systems Maruti Real-Time Linux Real-time POSIX RED-Linux RTAI RTLinux RT-Mach S.H.a.R.K SPIN Spring Timesys Linux/RK VEST VINO http://www.cs.umd.edu/projects/maruti/ http://www.realtimelinux.org/ http://www.osf.org http://linux.ece.uci.edu/RED-Linux/ http://www.aero.polimi.it/projects/rtai/ http://luz.cs.nmt.edu/~rtlinux/ http://www.cs.cmu.edu/afs/cs.cmu.edu/project/art6/www/rtmach.html http://shark.sssup.it/ http://www.cs.washington.edu:80/research/projects/spin/www/ http://none.cs.umass.edu/rts/spring.html http://www.timesys.com http://www.cs.virginia.edu/~stankovic/vest.html http://www.eecs.harvard.edu/~vino/ TDDI12 - Real-Time Operating Systems 21