Research Report 301 Dynamic Scheduling in the Presence of Faults: Specification and Verification Tomasz Janowski, Mathai Joseph RR3O1 a limited set of hardware resources and is hardware failures. Static analysis of the anticipated required to satisfy timing constraints, despite paper shows how to formally reason This infeasible. programs is often timing properties of such made on-line and take into account are decisions aboui these programs when scheduling process description language, CCS as a Timed We use failures. deadlines, toaA and hardware version of trr-calculus to specify and a and apply faults anticipated define a language to describe to be the outcome of an property of schedulability the pioperties. This allows verify timing is fault-monotonic: rf the logis reasoning, conventional problem. And unlike equaiion-solving of these faults is for any subset correctness faults, of proved for a number correctness is guaranteed. A distributed real-time program is usually executed on Department of Computer Science University of Warwick Coventry CY47AL United Kingdom March 1996 Dynamic Scheduling in the Presence of Faults: Specification and Verification * Tomasz Janorvskil and Mathai Josephz t 'Ihe tlnit"d Nations University International Institute for Softrvare Technology P.O. Box 3058. Macau 2 Department of Cornputer Science IJniversit)' of Warrvick, Coventry CV4 7AL, UK distribut,ed real-time program is usually executed on a resources and is required to satisfy timing constraints. despite anticipated liardware failures. Static analysis ofthe tirning properties of such programs is often infeasible. This paper shows how to formally reason about these programs when scheduling decisions are made on-line and take into account deadlines, load and hardrvare failures. We use Tirned CCS as a process description language, define a language to describe anticipated faults and appl.'- a version of a pcalculus to specify and verifl timing properties. This allows the propert,)' of scheclulability to be the outcome of an equation-solving problern. Ancl nnlike convent,ional rea-soning, the logic is fa'ult-monotonic: if correctuess is proved for a number offaults. correctness for anv subset ofthese fa.ults is gua.ranteecl. Abstract, A limited set of hardrvare Introduction Consider a real-time systerr which consists of a fixed number of tasks, each rvith a possibly unbounded numbel of invocations. Some tasks are periodic a:nd will be invoked at regular intervals by timers; the others are sporaclic tasks ancl are invoked by some other task or by the environment. Let the tasks be statically partitioned betrveen the nodes of the systern, all connected by a muitipleaccess network and each providing resources like clocks, memories and processors. Clocks are used to implement tir-ners, and asynchronous comlnunicatiot.t takes place using memory to implement Protected Shared Objects (PSO's) [5]. There will usually be more tasks than processors, so at each node the aliocatiort of local resources is controiled by a reai-time scheduler; tliere is also a protocol for scheduling the netlvork traffic. Tlie hardware of tlie system may be unreliable: for exarnple! processors may fail, memory lnay be corruptecl and cottttrtutrication may be delayed. A real-t,irne system operates under both Lesource ancl timing cotrstraiuts, for tliat a task produces 'correct' output rvithin a specified title. But such example Suppolt,e<l in part b1' EPSRC research grant GIt/H39499. to one task and rve let a lranscLctiott relate the timing of actions in one or r.nore ta.sks. Assume that tasks couttlunicate through PSO's alrcl tlial norrrally they are statically scheduled using the ceiling priority' protocol [5]. \\rhen hardrvare faults occur', ltorvevel, we let dytranlic scheduling be usecl for a more flexible reassignment of resources. Verification of timing properties rvill recluire solne a.ssuntptions: (1) about, the speecl and the number of processor.s, (2) about anficipated liardrvare failures and (3) about the ntittir.t'ttttrr irrter.-arrival tinre betweet] external invocations for each sporadic task. ,finecl process algebr.as (e.g. [20, i1,26] ) provide an obvious formal frarrt: work for tIis analysis. But basecl on t]re marirnal parallelisn assurnption [2'1]. 6ost a.r'e unable to represent clela.ys clue to resource contetrtion or to uroclel sclieciuling clecisions clilectly. illiis gave rise to ClCSR [9] n'hich lrrovides s1'nchrolols tilted actions ancl asynchrotrous instautaueous evettts, the fortrler resolving cornpetition for resources and the latter for syuclironization. BTit CCSR (ancl most other such formaiistns) assulrte the use of fixed priorities ancl are tfius ulsuit,able for rnoclelling dynarlic sclieduling decisious, e.g. to recover frotu faults. Furtlier, bisimulation-based reasoning is usually insufficient to verify faulttolerance: it ma,y be possible to provably tolerate a llumber of faults, yet be unable t,o provably tolerate only some of thern [12]. \Vitir unpredictable faults, such fatth-ntonolorticily is imltortant but is harcl to esta,blish in tnost branching-t'irne theories. Finally, fault,-toierant schecluling lias recently received sorne attelltiort [3,22] bLrt in a sernantic frarnervork lvhich does not give sufficient insight into horv proofs of feasibility of other scheduliug problerns can be olttaiuecl, antl rvith synchlonization restrictecl t'o sitlple prececletrcc lletrvectr tasks. This paper shorvs hou' to realistically analyse the tiruing ploperties of cotltttttt nicating systems in tlie fratnework of timed process algebras. We use a vcrsiou of Tirnecl LfCIS [26] ancl use timed plocesses to represeut tasks, to uoclel harclu'are a.ncl to clescribe scheclulers. D1'nan'ric scireduling is rrsecl (unlike [8, 9,6]) and priorities are assigrred to tasks. and not to individual actions (unlike [8.9] This rnakes it possible to use a single framervork for reasoniug, abstraction and autonatic verification [10] aud to relate scheclulability to equation-solving semautics and pro[21, 17].'lhe language is equipped with the usual transitional using additional semantically, of faults: vides r11ea1s of representing the effects by 'faulty'declarations for process cour-labelled transitious, and syntactically, pr-calculus which follows timed [26] version of the stants [12]. For verification, a logic is used; this is also faultof Hennessy-Milner and modal [14] extensions transactions aud euvideadlines, of expression monotonic [12]. The logic allows verify fa.ult-tolerauce. a,ncl deadlocks to detect is able rolment assun-rptions. ancl We start by assuming a,n unlimited nurnber (maxirnal pa,rallelism) of faultfree ltaldrvarc resollrces (Section 2). Assuming fault-free hardn'are, Section 3 provicles an alchitect ural model for describing atid reasoning about systems rvith liltiterl resourcos. Sect,ion 4 retains tlie assumption of unlimitecl resotlrces atrd introduces reasoning about tht effects of hardrvare failures. For this, the process languagt is assigned cliffelent fault-affected settrantics, an(l the logic refine<l ilto its fallt-rnonotonic version. 'lhese intpt'ovemetrts a,re cotttbineci in Sectiorl deacilines neecl not be restricted ) 2 5, wfiere it is shorvn holv to reason about timing under resource limitations, in tire presence of anticipated hardrvare faults. Section 6 provides a cliscussiotl. 2 Resource-basedSystems \Ve first briefly describe a timecl process la.nguage (based on 'l'CCIS) aud a logic (based on a rnodal p-calculus) to express and velify timing properties, assuutiug unlin-rited, fault-free hardware resources. 2.L The Language Let A be a set of actions consisting of untimecl a.ctiolts (f,) and tin'iecl actions e(l), o1e for each non-negative real I ancl represeuting a delay of I rrnits of time. \4/e shall norrnally exclude I = 0 and rvrite r instead of e (0), aucl represent sy.rrr'lrronization b; .orrrplernerrlary rrrrlirnecl acl.ionsn arrd A (u = u or,d e (1) :a".1 e(r)). Let Lc f., f :A-A(.f(e(t)) = E(l) and /(a) =/(a)) alid I €X,a set of process coustattts. Also, Iet a € ,C ancl a € A. There are three syntactic categories: process expressiotts Pe, declaraliotis 4 aucl processes P' Pe::=0 lXlo'PelPe*Pe l:::tllz\tr=P"l lao.1lzloc P ::=0 | pr.zl I o.P I P + P I PIP lP\, (l) I P[/] Inforlrally.0 represents cleadlock, a.P is process P rvit'h plefix o aucl process P + Q reprcst'nts altematioti. PIQ is usecl for coucurrent coltposition. P \ t for restrictio". P[/] for renarning and pI.:1 for the solution I of tlre recttt'sive equatioDs 21. The cieclara.tions include tire c.nrpty cleclaration [], /1[x'Pe]. to {eclare X as Pe and other constants as in :1, cr O C to prefix n to the right sicle of all cleclarat'ions in C. and zl tl V to sunl tlte right' sicles of the corresponding declarations in C and V. It is assumecl that in pX.A' X ancl all constants occulring in C are also declared in z\. \4re abbreviate [[X = Pe]l]''Qe] as i Pe [X ' Pe, 1' ] Qel and will often write [I ' Pe I p] for all declarat,ions I pa,rtial function Zl is a of the semantics p Formally, predicate holds. such that [Zl] defining the a,ssignmeut of process expressiotrs to process coustants, as itt Table 1, and rvith dont(A) for the constauts declarecl ir.r A. The semantics of processes is opera,tional, ancl is defined in Table 2 by structural induction. closely following [26] . The first rou' of rules applies to all unt imed act,ions plus r: \ € LL) {z}. Row-s two and three defiDe the passing of time and apply t,o l, u ) 0. Thc rules let o.P idie indefinitely until the environment is reacly to sl,nchronize. There is no rvaiting once syttchronization is possible (nlaxirnal progress). i.e. no delay for r.P rvhiie PIQ will idle unless P ancl Q can sylc|ronize: Sr(P) includes all actions a of P n,hich a.re possiltlt'rvithin I tirne ruyrits. For exanple, Sr(o.P) : {.},.S1(r.P) = 0, Sr+,(e(l).P) = S.,(P) and .S,(PlQ) = .Sr(P)U.Sr(Q). Time is corrtinuous, and delays do not ca,rtse loss of actiols (persistency) or result in reaching clifferent sta.tes (cleterrttinac)'). Ro_rv fortr applies to Lrotli timed and untimed actions. In Particular [ll](r){1r}..1/}-} is a process obtairled by simulta[eous substitution of all constants Y in /([z\](X)) by tlreir corresponding fixed point,s pr'.a. In t,he sequel rve shall appll' obvi ous extelsiols of the la.nguage to describe value-passing attcl a.ssttttte the rtsrtal lranslal,iorr itrto llt. basic latrguage [19]. 2.2 The Logic The logic is a versior.r of the ruoclal p-calculus rvhich follows the tiruecl exteusious of HN{ logic [26] (nreaning, antong othel things, tha.t rve need infinite conjunction) and rvhic| for sirnplicit.l', likc tlie process la.uguage, does not allorv for nestirlg of the fixecl point operator [13]. Let e be a.n etl.rl;tv seqllcllce ancl let 3 clenote s € A* rvith all r's retnoved (ancl deltrys sunltlled). a=e ri':-nf 7s-=i e(t)i a(lf[r)s:5(l f u)i e[)t-r.s = e(l)ni e([s: T[e forrlulas,| /.'\ ( j' are built using constants l1 ancl iclentifiers Z, t'regatiotr, disjunc- tiorr rvhich is possibly infinite, the existential modality. and tlie greatest fixecl point, rvith the other operators derivecl as usual. As in the process language. the syltax consist-q of formula expressious fe, declaratiols V and forlnrrlas l,. Fe V F For silrplicity rve rvlite ::: tt I Z l-fl. I V,er fe | (6)fe ::= lZ' Fe)llz ? FelY ::-tIlvZ.Y l-F I V,err I (a)f Y(Z) for the folmula e-rpt'cssiou rvhich is assignecl to Z by'V alcl as usual assunte that, iclentifiers in V( Z) occtr "vithitr an even ttttllrbt'r' of legations, each also cleclared in V.'llhen the setuautics of F (Ie) is clefirrecl relative to au a-qsigutlettt 6 of iclentifiers Z to process-sets and is the set [l:],r of processes that satisfy' M (we write P I ,11 rvhcnever P € [;1/]). Pointrvise inclusion ancl sutliuation is used to define [zZ.V]o and we assignnrent of a\I Z € dorn(V) to [V(Z)],r' [t3]. let [V]o' be the' 6(2) [-M]o =a"1 P - [.lf], |,.r[,'1/1], [r?.Ylo =a".r U{6' | 6' g [Vnd'](Z) [A,., 4 P' AT = 6) + P' e [Mno] n1a1,l,l1 =a"y {PlY r,-,,(P fftt\t --a,.r P [Z\r, =a"1 Llo]o =a..1 Table 1. Denotational senlantics of dom,(l)):a., [a i! "4](I) :a., [1[r'= Pc]11-r1:n", [l 0 if if if t firntrt if f [lni-\) (r vn(-{) :d"/ { [l](-\ )+[v](.\) ifif [ [v](r) a.[,r](x) !,, cleclarations. I € dom(A) -{: )'' Il). -{e dom(A) dorn.(V) donr(l)-{ € -I € dom(l)n dom(V) -Y E rlonr(V) * clom.(A\ Table 2. Operational sernautics of processes pt4p, e\e, pt4p, e+e' p\p'e5e' p p+aEV V+aEa' ptaT pla FtaE ptq -Vat Ptd ;.FT ;i-}; ;r''+ *P .11il7g.1"F .(,,)rg; p'('t p' p'"1 P' q !:! g' , g!J2:l'. s,(p)ns,(e) :0 I vrr' -r;ag+ t',.q, --p[lg pld '' P3+ P' P + P' [a](r){1i---r 111:+ e' (I 1a'o / t't F p,lf) P-i157'v 41J.-+. "t; pgF ' e rlonr(J)) l,-r.J Though the logic allorvs verification of timed processes, in general a.n rinlinrited numbeL of processols is assurnecl to be available to execute coucrirrent, tasks. C)onsider, for instance. n independent sporadic ta.sks P1, each irrvoked by an action a; and responding rvith Dl after 11 units of time, perhaps represerlting t'he spetd of tlre unclerlying processor; Pt :def /rX.[-X ] ai.e(ti).b,..\]. Let F =de.r Ao..o,lo)Z(i) Afa;lZ'(,rJ)) l\'i'_ruZ(i).lZ(i) ? : (bi)tt Albi)z(i) v \,/., (rr)rrn lz'(i,t) A,4a,[c)Z'(i,l) A A,,5,,- rle QL)]z'(i,t * Ir)] that if tasks are takerr toget,her then each is either reatly fol itrvocation rrrtlcss processiug takcs uo tirne, eaclt task must be executccl on its ou'n plocessor. .F states ol is able to complete n'ithin 1,. Since li"=rP, l.F. 3 Resource-LimitedSystems In orcler to reason about the timing properties of comtnut.ticating systents, it is essential to cousicler the limitations of the underlying liardware. Olle approach is to constrain the seura.ntics of pa.rallel cornposition, so that PIQ 4 P'lQ "not a,lways" follows P J+ P', and then verify the properties as usual; auother is have the usual sernantics ancl to verify the properties "rela,tive" to the euviroutttetrt constraints [15]. \\re take yet auother approach n'hich leaves the sema,titics and the logic unchanged but represents resources syntactically. The rnapping between Tasks and r?e.sorrrces is the goa,l of lhe Schetluler. Given a set .L of scheduling events and the T'im.ing properties to be estabiished, finding a fea.sible schecluler (if any) can be represettecl as the equation-solving probleru [21, 17]: (T u s k sl Re s otn' ce sl S ch erlui e r ) \tr | T i rn i tt, g This makes it possible to represent linrita.tions in the nurnbet'aucl also the speecl of processors. so that tasks need not represeut dela.ys explicitly. We first show horv a scheclLriel maps tasks to sharecl resources in a cetttralizecl systenr. \Ve sliow later horv t,o use a clistributecl rnoclel ancl hon'to specify':rncl verify transactiotrs. 3.1 Tasks and Resources Cgrrsicler a set of tasks /ask,;, sotne of thcttr periodic (i € [1,per] ) ancl invokt'd by tinrers, and thc others sporaclic (i € [per'*i.per*.spo])' atrd invoked bv the envirolurelt or by sorne other task. Let ?nsfr1 be a simple sporadic t,ask wliich is ilvokecl by action ?nr fronr the environment aucl rvhich returns a resuit by-oul1. To represent resource-limited executiotls, let, Td.sfri request a processor (reg,) irtu'r'recliately after it is invoked ancl release the processor (r'eli) rvheu retrtrniug the r.esrrlt. To t,ake account of the execut,ion speecl, assulle that after being allocated. Trrslr cart only proceecl if proviclecl rvith the actions lzicft;. Ta sk; - 4, f in,iQ:).rer16. p X (r).(t icfu rlr J) []'( y) ] re li.out i('1y). i ni Q).t' ern.f (r )] \\,'e rrse lricl.l to lepresent tlie basic rnachinecycle of the undcrll''iug plocessor aucl giverr Pr.ocessor'p (* € [1,pro]), speedp. is the minimunt tiure that, must elapse betrvecu tn'o ticks. Which task is currently executed b1'Processorl depends on thc value receivecl by the last action prl1, (for pt'e-etnpt). The actioti is available at, any tinre ancl can pre-empt executiott of the current task. l't'oet ssor'1.' a,.1 prl e(i\. sX(i). [X(t)' pri1, (7).I (i ) -t t(speerh ).r lc*;.I(i)] 'flie iclentity' of the executed 1,ask is available to Processor'1 but the cottvcrse is pot trrre; a task llaJ'be allocatecl to cliffelent, processols clrrling otte ittvocatiott. r\s contrrrorr iu scheduling theorl'. we assulne that ttrslis cauuot voluntarily' that all pt'ocessot's sltare a colttlttoll basic tnacirinc cy'cle. Wit,hout cotrttttrttricat,iorr. the for indepenclent tasks TosAt call only'take ttlo fortrls: suspencl themselves. One more assurrrptiou is instnrction set, each taking cleclarations t,it:k; r1,l ij a. -I(")'licAl.X'(f(r)) I(r) t ticki.if p(;r:) t,hen I'(r) else I"(r) ./(rr) is assumed to be a function evaluation and p(r) a test ott tlte argutttent value r, each taking one machine cycle. Any nlore complex courputa,tion I is assumed to be ma.cle of basic tnachine opelations like /(r) and p(r). So far lve have only considered one form of invocation. by'action rrln front t'he environrnent. Tasks ca.n also invoke each other (intry), ofteu as the last actioti of invocation, and be invoked by tin.rers. A tinler (Tinter;) is alrvays readl' to a.ccept a uerv time period (t,inte;) afl er n'hich it rvill tirlreout (lzirrieo'trl; ). Tinter;' :lirrre;(l).I'(l)] p-I.[X [-{'(l) t s(l).I'/ ! t irne;(u).I'( u)] [-\" : 1??]?€o?di.I I tintet(rr).I'(u)] Finally. Iet trl corrsist of the actions irr,u; for the invoca,tion of tasks and lirrring actions lirne1 antl timeouli. Theu T'asfts =a,, f (l|1--r+"1"'T,tsy l lfi"ff inrer'; )\/,1 nrrcl,Rr sorr 1 c( s : dc.f lli"=o, P roce s sorr. 3.2 Scheduling The schedulel' maps tasks to resources. Define tlte following sets of act,ions: lls for conrrnuuication bet,u,een tasks a,ncl the scheduler (reqi and rel;). 1-lr for actions between tasks and resources (licA;)and ,Lsr for actions bet'rveeu tlie schcduler and resources (prlt). Lel L =a,.f LtsU Ltr U Lsr. Thcn using a scheduler which accepts requests (req;) for processors, allocates tasks to processol's (prlp)' ancl keeps a,n updated knowledge of available resources (reli), tht' nrapping results irr tlie plocess (TaslslResourceslSchedzler) \2. For example, let /: [l,.s1.io*perl - lI,pro]U {f,T} I'ecorcl the status of t) : I then Tas[1 is rvaiting for an invocatiou; it f (i) = T then it t,asks: if "f( is act,ive but rrot. being executecl; ancl if /(t) € ll,pro) then it is trrtclt'r' e-\ecutiorr ori Processor'.11,). If /(t) = T then we say' that Tos[1 is suspenclecl and if k 4 rng{f) then Processorr. is idle. Let initially "fo(i) = I. Tlte lelative 'itttportance'oftasks is represeuted b1'their priorities r: [1, spotper) - f. trsing priolities, 7'asA1 n'ill be allocatecl a processor only if lno'Iaski of highel prioriti.' pX(/o).[f (/) = "'] s'here preclicate mcr:r(i.f):a,.t .f0): TA(/U) = T* ?r(i) > zr(j)) and r(/) = I,ro,=, reqi.x(flT lil)+ L t rote t L,r j reli' X ( f[L I i])+ Inro.r(i..f ) L o g,n 1 p;t k()' x ( fLk / i)) Once allocatecl, thc prioritl'-based scliedulel nili let a task run until its completiori. A 7,trc-etn1tlitte schecluler. itr coutrast, uay't't'place tlie task (7asl'r)rvitlr thc lou'est priolity anrong all executing tasks, rrrirr(j,./), lry the tasli (7'rr.'-{';) rvith the highest prioritl- alnong suspeudecl tasks, rnar:(r. /). Then prerlicat,e ntin(j..f) :a"f rns(f) - ll,prol A (/(*) € [1,pro] + n(-r) < r(t')) atrcl srtch a schecluler is pf (/o).[f (/) : ..] n'here neecls one. a.s represente'cl by the scheduler o1.r r(/) = Ir.r=, recl;.x(flT lil)+ 1s1r'r] reh' x (f lL I il)+ I''nr1 i, l ) D (,,, (.f ) pFt r ( i)' x ( f lk I il)l+ I.iii r, g Dnrin(j,J ) n''t 1 1t rtt1' X ( f lf 0 ) I i'T I i)) For pre-empt,ive schedullng of inclependent periodic tasks, an opt'irrtal alloca,tion of static priorities is the so-called rate-monotonic order, inverse-proportioual to tlie tasks'invocatiot'r periods: if periodi l periodi tlien ;r(r) > n(f ). 3.3 Comrnunication Assune that tasks commulicate asynchronously througli slra.recl objects. In its sirrrplest forr-n. such anObje ct provicles some data,storage that can lre reacl using t,rvo actions (say request and completion) zrncl rloclifiecl, each rvith sortte clela.y rleluy. Let ,L be thc initial value. Object -,i,. 1 pI(I).[-Y(r)' rrl.t(rl,elay).rr](,u).-\ (r)+ (y).e(d c t ay ). -r ( y)l un Suppose ive liave oDj sucli objects and let us rede{ine r?esott,rces to take accottut of both kirrcls of resources: ResorLrces - a"y ll'-l'rPro(:(.ssot'i I lib!-rOb jecti' Bt rvitfi ltutual exclusiorr over sharecl objects, a lower pliority task nlay suspetrcl a higlier prioritl'task. Fol example, if r(i) > ;r(f) ) r(k), llirsk1. rlay secule exclusive access to tlie sharecl objec.t before Task;. Then Tas[,1 has to rvait until Tcrskl, cornpletes and fasfr, ruay be executecl instead (priorily ittt'rrsion). Assune tliat in orcler to use a shared object Object1,7'as11 first t'etluests access fronr thc schecluler by the action regt\j): it u'ill later perfortll t.ii(j) t,, release the object. This requires sorne adclitional folnts of cleclarations for Tas&'1 : -\U) =Ta,riU).nl,.rtl1(t)./', /r(i).-\'(J') I(j,.r) =,'utJj).G; (z)., r/i(j).-\' clel:r-yscaused by thc undelly'ing plocessol's. it neccl uot allPeal <leclarations: deia,ys there' arc only causccl lt1' 1hc sharing of obit'cts (r.esolvecl b1'the scliedrrler) ancl the tinre it takes to access tltelll. The Ilrltecliate Cieililg I,riorily Inheritance Protocol solves thc ltloblern by assiglilg a prior.itl'to an object, tliat is the rnaxitnumof the pliolit.ies of all tasks that shale the object p : 11, obi)- N' Thcn each tiure a task obtairls access to all object. its prioritl.is in-rmecliately raiseci to the ceiling level. Iror a given obiect. * l{ U {I} retuln eithcl the original priorit"'- of the let rlre fulctiop g : As lick, fepres('lits i1 t|tse ll,objl task accessing the object, or I i1 there is no such task. lnitiallr'. g0(/) protocol can rlo\r be aclclt'd to the scliedttlet'. as belou-. f (/, r'. ;') - I. The I Irr,r=t rcql.I(.f[T li],s.r)+ I'"'e1r'r1 r/;' \ flLl il's' ir)+ (i J -r f , sllr I i). t lp( i I i)) + I' rr = 1 j).r(f. i), llL I rls (i) I i))+ L,o )* tre11 I', o'( t,.r ) l) g" t Ifir )' r (/[r'/t]' g' o)] * r 1 I'e11, ( ). I ( ) ) ( u u 1.t (t Irnr:n (j,.f ) lm.rr; 3.4 r ( t)'I ( f lf 0) I i'T I il' g' ") Distribution The ntapping betrveen tasks and l'esources has so fa.r assunrecl ttse of a c.elllralized scheclpler. Suppose instead that ta.sks are partitionecl betrveen ntl > 0 nocles (i\,o|ei ) ar.rangecl into a logical ring and counected by a rnultiple-acccss uett'ork (-\relrirort'). Each nocle provides computing lesoulces iike clocks, tlletnorl' and processor.s and each has a local scheduler. The actions at i\{orle'rvill lle clistinguishecl lrl.tlie srrperscript i, l{odei -ae.1 (TcrsksilRfrotLrces'lSche,/u1e ril\It. Suppose that each ta,sk has a local object (Object; for'faski) to holtl tlie sequences of ruessages to be st'nt.'fhc sencling o{'a utessage rrr is thett tept't'sctttecl b-t' the follorving rleclar;rtious: -\(rrr)' rrrt/i).trt;.rcli(s).i,'J,1i;.II1rrr,.sI -I1(rrr,.s) i licl;.I2(s: .Y.:(') ' ta nr) qi(i).ttl (.)'i'l;(i)..\i Silce we assulne that tasks cannot voluntariiy suspend thet.ttselves, a task cau only invoke a reurote task ancl rvrite to a rentote object. Therefore data messages lia.ve tlre forur n.c.j.r' rvirere tt € [1, ncl] is a node (n I i), c is either introAe or urite,j identifies the task (c - irrt'oAe, i € [1, spo" 1per"f) or object (c = Lurite, J e [1, obj")), and tr is tlie value passed to the ta.sk or rvritten to the object. tlnlike the scheduling of local resources (processors or objects) betlveetr tasks, the schecluling of netrvork traffic (deciding which node is allowed to transmit and for Iow long) cannot be done centrally. We siiall use a simple protocol based on a cir.culating tokern (lofre rr). After receiving the token, y'{ode' nray trausurit ottc message (thc first message of the highest priority task) before passing the token to IYodei*r. A task Tcsl'', u'it,h the highest priority is used to irlplement the protocol on each nocle. This task is spora.dic, iuvoked by action irz" and producing .Given a function lr(i) on [i,nd] which returus eit]rer I ol t'he a result bV ""tl last message receivecl fronr Arode; (40(t) = I), tlie ttetrvork is defiled belorv. l{ etuork -a,y StX(h6).[f(/r) = Do1;y=t out\(r).X(hl, I jD+ t +tt oo o D n11 y7 t 1,r( "' " (lr(j)).r(Atr/i] )] Let tasks be ordered accorcling t,o decreasing value of priority, u'ith tire tasks of tlre same priority orderecl by the uurnber (j). Given j € [1' spo" + per"], lct suc(j) return an irnmediate successor of j or' _L if there is uo successor. 'lheu r deternrines not only the importauce of tasks but also of messages: Task\ -a".1 pI.[X' ...] and I'ask\(token):a"y p]''lX' "'] rvith the cleclalatiotrs belorv. irtr'ocat iotr -\ = irtirp').tttfr.-\r (r) receivecli tokerr ;r: f token lhe n X2(:r:) eise )'l(0) Ir(r) 'tick\.if '"-) j, j.u for us'/ e/se t,hen X3(t) t i.c. -Xa(c, tick\.i Iz(r) f f ' forrvard r -Ys(.r) 'r'elt.oudlr;.-f : X/c, j,u) tick\.if c = inuoke. then X5(j. r) else ,Yo("r, ,) invocation? invoke 7 Xs(l, r) = re{,.;nr'11u;.f write to j Io(-1, r) ': veil(j).rL,ri t,'l.r-1'r(,,).rt : -; release i.el',.X X, priority j)) sn'raller Yt(j) = tick\.\'2$uc( the smallest'i Yz(il ' tick\.if i = Lthen \\ else 7'a(i) elease trud foru'arcl Yj =;i.orir(loken).-Y \'+( j) vdilU) rAj.ra',l4.iift(r).Y!(r, s) 'tick\.if s=e then l'l(j) eiseYo(r,t) = )!(.r,s) tfcfti.)'7(j. s6. s) )'o(-1, ") ' 1'7(j,m,s) 'ffcki.Ii(j. rn. s/) l!(j, rrr. s1 = re(r111.r.7;(").r€i l(i).)'g(rr) ,'e l'1.ottt'r(m)l*" (toke n).I I!(nr) I -;_ j read notnessages'i take tlie liead a'ke the tail rvrite thc t,ail release atitl setrd ,et .\ orlr iltol'" n) be like .\'otlrt lrul n ir h fu.s{'i (lol't n ) replacirrg Tu"l'! arr<l le l,{orler holcl t,he token initiall.v. Tiren for Lc contaiuing actions rirr" ancl otrl", rve have tlie clistribut,ecl s1'stenr (,\'orle l(lolrn)1,\'otle21" ' l-\Ior1r"'rlNelti:orfr)\1,c. f t 3.5 Speciffcation, Verification and Equation-Solving So far rve have shorvn hon, a sirnple timecl process algebra fi'amervork cau be used to builcl a fairll' general rnoclcl for corrrt'uut.ticating systeus rvhich is capable o1' representing resource-linrited executions. We shall now shorv horv the timecl and untin'red properties of such s1'stems can be specified ancl verified. Cionsider tu'o actions. a ancl 6, fol which it is required that x'ltertevet o occurs. b occurs at tnost d later (71fu,b,d)) or d earlier (72(t'b'd)). T\(a,b,cl) =a,1 uZ.lZ 'lr.]Z'(0) A /1,*"lalZl lZ' (t )' (b)1/ [b)Z v V,,(rr)lrA ^ (t) A A,.a-, le(u))Z' (t + u)l A,lulr)z' T2{a,b,d) =a,,1 vZ.[Z 'ltL]Zt(0) A |1,*"lctlZ) [Z'(t).ft]f f tA,7rl")Z'(t) Afi.,r.o[e(u))Z'(t+ tr) lZ' (t )' lb)Z A A,,*blalZ' (t) | t > d) lt < d] A sir-nple functional propert,y, iu contrast,, l'ould state that if tlie value r rec.eivecl b1'action n(r) satisfies a prr>couclition pre(;r), the value y of b(g) nrust satisfl'a post-condition posl(r,y). This, plus the timing requiremeut that b(.y) occurs uo later tharr 11 after n(r), is defineci by the predicate belon'. Ti(t(t:): post(r,y),d) =a".r A =A".,7,'.1"y[(Q))z'(0,r) A',-r''" r"-rla(t:))Z) ( (t (D( tt n e )) lb( y)) Z ) v V" (cr) tt n lZ' . .r' ) =V !),r,o.", (,,u A,,tlr,)Z' (t) A A,,<.r-,[t( u)lz' (t + u)l pre (.r), b(y) : uZ'[Z A /\,,*.lo]zl r eas)' to clefine tliat, o occul's nit.h periocl p ancl ji11er 11. relative thc beginning of each peliocl. Predicates such as this catr be usecl as the l;uilclirtg blocks for typical transact,ions, relating lhe timing and values of task ittteractious. A fransactions s,ill typically rcla,te the input to a. task (Taski) u'itli t'he otttPut frorl another t,ask (Tn.si';) which tnay uot be on the satrle trode. Let 7irs,t; be loca.ted at .Nor/e",'l'cr.ski aI ltiode"' and after the action irri(:r:) irr ivhich rr' satisfies the pre-conclition prr(r), action ou{'(U) must occur no earlier tlian d1 ancl no later than t/2 ancl *'ith g satisfying the post-conclition posl(.r', y)' It is also Transaction -a,.s Tl(itt'i'@) : pre(r),"".{'@): post(.r:, y), ri 1)A 6) , A., ., Trli''o' 1'1' ^'{i' '12) With lirnited conrputing resoufces and in the absence of assumptious about hon' often inf' arrives. it is in general impossible to nteet this transaction. Let d2 be the minimum intet'-a.rrival time for action zrrl': .'1.)-stl?7?pl io?? =,ir.f A,,,, T2( i n'l ( r: )' rinl' ( y), rl z ) Tlren given a real-tirrrc s)'steln (Sysl,etn), the properties of trausactions trrust orrly be verifie'cl rvheu tlrc assrtmptiotrs ale satisfiecl. ,9yslent l,,lssurrrptiort s l0 )'f ransrLctiort s And if System has the form clescribed earlier, verification will take a full account of the constraint,s imposed by the underlying irardrvare, for centralizecl and distributecl systems lespectively. a s k sl Re sow' ce sl S che dule r) \I l,4ss'rrrnpl rio??s + T r an s act i on s (l{oclel (token)lli!_tl,{ocle'lhretuork) \r. F Ass'uttptiorts:+ Tr(i.tlsacti.orts (T One more advanta.ge of representing resource coustraittts syrrtactica.lly is t,hc possibility of fincling a feasible scheduler (if one exists) automatically, as tlie rvell-knolvn equation-solving problem. Tlie problen has attracted sotne attention [21, 17] and algorit,hnric solutions have been plol;osecl ancl inrplerttcnted [10]. Fault-Tolerance for Unlimited Resources In Sect,ion 2 rve introduced a general framenork for desclibing and reasouiug about distributed ancl rea.l-time systems and in Section ll showed horv to represeut and verify systems rvhich can only rely on limitecl (in terms of the nun.rber aucl speed) set of hardu'are resources. And we made it very specific of iiorv hardrvare (processors, memor)'. clocks or communication media) should behave in older for propelties of tlie overall systen to hold. \\/e non' shorv horv to reasou about systems that ale designed to sustain anticipated hardrvare failures. to shorv that thel'are provably fault-tolerant. \Ve continue clesclibing faults ancl their effect on t,he semant,ics of TCICS, and then show horv rve can prove fault-tolera.uce. fot' given assumption about fa.ults and first for unlintitecl rcsollrces. 4.L Faults and their Effect The fault-tolerance of a system is ofterr verifiecl by synt:rctically' tlansfolnring it irrto its fault-affecled version and then verifying its properties as if no faults are present [18]. This method allows sta,ndarc] techniques to be usctl for proving fault-tolerance, so \\re begin by examining irou' it ca.u be rtsed itt our logic. For a process Q, assume that a 'faulty' declaration f, iu geueral diffelelt from 'normal' declarations being part of the syntax of Q, is used to specify arrticipa.ted faults. Let Q be transformed intoT(Q,P) to represent the eflects of such fauits. The transformatiou is clefined as follorvs. :a"1 T (0,\I/) 0 T(pX.A.V) :d"r tlx.(AfiV) T(o.Q,i!) -a"1 a.T(Q,V) T(Qt + Qz,V) =a"r T(Qr.\I') + T(Q2.{r) T (Q tlQz,i!) :a,,J T (Q r,{/)lT (Qr.{/ ) T(Q\ L.tI/) =a".rT(Q,f) \ I T(Qlel.v) =a" 1 T(Q,V)lsl Assrune that 7(Q,f) is well-definecl. i.e. all coustant,s declared. ancl sinct' [rZ](I) ale t'itlier prefixecl ll1-r or a.ltr farrlts are autononrous, a.ll expressions ll a surllnation of such expressions. Such a 'faulty'' declaration {' is gcnerated b1' tlre abstract syntax V ::: r O c I tZ.} V. Sorrle exatllples are a processor rvhiclt 1ra-r- clecide to tick early, a tirtrer n,hic.h tnay tirneortt late. a shared object rvhich sornetimes fails to remernber a. rvritten value aud a ttetwork ivhich mav lose nlessages. Such faults are rePresentecl by the rleclarations below. r (.a) [X(t) 'prl6(j)'I(.r) +:( speetll - 1)'l1cA','-f (i)] Vri,,rr, =tte.1 r Q)lX" = l1l;.tr'rnrotrt;.-Y + lzirnel(u).f '('u)l Vobje,:t =d"1 r 6r [I(r) : rr!..llel,tu1.r'rl12).I(r)f ul(y).e(rleln9).I(,r)] V,etttork =d".J r (l [f (ft) = f (lr[f/j])] V1.,o,,,,o, =,t,1 Horvever, given a process Q, a spt:cification I of fault's artcl a propelty' ,F tlrat rnust liold ciespite these laults. verifying T(Q,V) I f, is not suflicient to prove that Q is lault-tolerant Ii2]. It is uecessary' to take iuto accoutrt that faults are unpredictable: after proving coLLecttress for a uumber of anticipated fauits. correctlless for any sul)set of tliese fault rnust be (provably) guaranteed. This. I- (in thc pl'esence horvever. is not the case for Q and f belolv because T(Q,V) = (in of faults). the abseuce of all faults) but Q F ,F, Q :a,f gf. [X = 6.I"] [X'i c.I"-+ 6.I" + r.X"') rlt :de.f r r.-, [-I 3 -I/] f --ac,t [e](o)ti [X" = b.I][I"' ? u..X" * r.I] 'f[e leason is action r-r rvhiclr is only'possiblc itr tlie plesenct'of faults. But if onll sonre of thc alrrl T(Q,II/ + O) ? l', t; rna,Ji no lougel holcl present' (T(Q,V) f F), as belorv. farrlts are = everi if Q Q I- :a,7 pI. [I ' r.X'+ b.I"] [I' = a,.X" +b.X") [x,,= 6.r][r"' - u.x") - {///] €) [X/'/ = I] rlt =de.f 7 r.-r [-I/ <P =ae.f ?- f :tte.f [e](b)tt '_lhis is beca.use the faults 0. may lesult irr the state X"'but therr action b is only possible in the preseuce of @. The property of faull-monolotticily ts nol assured in this logic or in many ot,hel senra.ntical tlieories for brauching tirne (bisimulatious, testing equivalence, etc). 'Ib defile a fault-monotonic version of the logic, hou,eveL, lr,e tteecl to first, definc tlie fault-a.ffectecl semantics of the language explicitly. \\re clo so rrsitrg r.elation r;+ for f-a.ffectecl trausitions. rvith r;-+ defirrecl sirnilarll'1e --+ . but nith one aclditional transition rule: E,#E';,i€I ' '' ' .= Er?E' [v](r ){!i^lY} F " all lirr + P' I't1Ei.iel E t-+ (I ll' e r/orn(f )) I2 ancl 1 € {o,4} in Table 2 4.2 Proving Fault-Tolerance Any transition w'hich is possible in the abseuce of faults ( ----+ ) is also possible in their presence ( r;+ ). But in a fault-nronotonic versiou of the logic, transit'ions rvhich are onlg possible in the presence of faults require specia.l attention as t,liey must be tolerated lvherr they occ.ur but, like faults, they catttrot be reliecl upon to occur. The first step torvards t,his is to retnove negation frorn the logic. ff I Z I V,rtF. I Aierre | (6).nr | [a]l7e V ::= lZ' Felllz' Fe)Y F :::lr I fJ' lvZ.Y I pZ.Y I V,.rf' I Arerf' | (a)r | [a]r Ite ::=tt I 'Ihe next step is to removethe symrnetry between modalities,so that (cr),|/ is verifiecl accorciing to the transitions ----+ and [cr]fl according to ;+;the latter rvill ensure that such transitions are tolerated and the forurer that they are uot' reliecl upon. Given f. the senrautics is belorv (Q llv F iff Q € [f]). [f f\t =a,t a [tt]5 =a,1 P [Ai.r 4no =a".r f-lie r[4nr [zno -a"1 6(2) [pZ'v\a:a,J ){5' ] [V]r' c 6'](Z) [!;.7 F.no =a'.r Ure r[Fi],'. ^[uZ'Yl6:a".r U{6' | 6/ c [Vn6'](Z) [(a)f], :a".1 {Pl}r,,,P 4 P' r\t = 6 A P' e [Fnr] [ta]r'I, :a".1 {PlYp,.(P # P' AT:6) + P'e fiflno] (3) to the rvay the refittetrtt'ttt 1tt't'its nioclal characterisation Il-1].'fhe rnot,ivatiol there is cliffelent: ----+ a,re transitions of the specification that the irnpieruentatiou tllust perfornl. and r;+ are transitions that lllay ol'tlla1'tlot be performe<I. (N,IPL ancl fault-tolerance are discussed again in Section 6). ,Lhis treatutent of modalities colrespouds orcler of \{oclal Process Logic [16] receivecl Fault-Tolerallce for Lirnited Resources A realist,ic analysis of the tirr.ring properties of a syst'em must take int'o accouttt the linritatiotrs of the underlying hardware. This is even rnore Ireedecl if har<lware fa.ilures are to be toleratecl. Fault-tolerance recluires reduudancy aclditional components (hardrvare redundancy), instructions (soft'rvare redundancy) or executions (tirne reclundancy) and leduudancy requires resollrces and time. R,esources r-nust be assigned rvhen a fault occurs (e.g. for roliback recovery') aucl also to enable run-time recovery, e.g. for perioclic checkpointing and for votiug on t,lie orrtcottte of N-moclular exec-utiotrs. We shall no*'combine consicleration of lesource iimitat,ions aird faults ancl sho*' ho'w tht' tirning properties of fault-tolerant aucl resourcc*litt-tited s1''st'erlrs can be analysecl. A nrajor issue, like before, is the allocation of tasks to resources. But now n'e shall use clynamic allocation accortling to the urgeucy of tasks ancl ai.'ailabilitv of resources. _t ') 5.1 Proving Fault-Tolerance for Bounded Resources As befolc, let a s1'stem consist of a nurnber of tasks. Ttrsl;,s. sotue of theln lterioclic others spora.clic, each rvith its oil'n titner, executed on a cetrtra.lizecl set ol resources, Resonrces, including plocessors and protected shared objects. Let a sclrecluler, Scheduler, ntap tasks into resources in a lvay tliat ellstll'es tliat the tiniing constraints Tinting are tnet clespite harclrvare failures Vrrror,rr"r- alrl \/, lF*,. "., ",... T i m i n q Tinirtg nta)'contain a nuntber of lecluiremetits but rvitlr lirritecl ( Tc sk s l.Re.so rn' ce sl'9 c h e d'u /er ) cotttprrtitrg r.esources ancl w'ith llo assurnptions about horv often sPoladic ta.sks at'rive, t<r satisfy tlrern ntay'llot be possible. But Timittg cottttritis uo uegatiotr (to eustrrer fault-nolotonicity) and thus cannot e-\pless inrplication. This basic problellr results frortr tlie ua.ture of verifying the timing propelties of resoulce-ltouutl systems in t.he preseltce of faults. We shall assullte thtr.t resortt'ces, ,flesottlces, al.e rlot shart'd rvith ta.sks lvhich are part of the environtt"teut.. Therefolc the interarrival tilie ol tlie sporadic tasks (perhaps iuvoked by these euvirottnreut tasks) *,ill never cleperrd on fzrilures of these resources. 'fhe solutiolt is thel to first verifl' assurnptions in the absence of fa.ults (l) and if they liolcl tlien to also velify' tlansactions in the presence of anticipated faults (|Fs,.""",.."). (TasA's|-Resou'cesl,|cheduler) ( Tns l.s I Re.so rn' ce sl,S c \tr l'4sstnnptiotts hedzle r ) \.r. Fv,.. " ", "" " thetr T r t n s u r:l i ott s 5,2 Dynamic Best-Effort Scheduling In or.cler to nrake decisions after the occurrence of a fa,ult. a scheclulet' tttttst ltavt: irrformation about the resources available at that tirtte. I"ol exattrple, collsidet' a fail-stop assumptiou [25] and the actions crashT, aucl repairedp b1' rvhich a sclreduler. is inforlted of the status of Proctssorp, a.ssrttttitrg that rellair takes time re pntirl: :dt.t r,., ([f (t) t f uitr,.e(repairt).]'] [y 'i:l:paire.d6. Ii ir"lt(,r).f U)]) Let the furiction g I ll,pro] - {I, T} for k € fl,pro) return T if Proc:essor1, is operative ancl I otherwise (initially so(k) = T). Then iu orcler to uou-preeltpt,ively schedule inciependent tasks in the preseuce of faults. n'e have the '"] where schcclulel tr,\(.f0, So).[X(/, S) = -I(/.s) = I1101=.,_ req;.X(f lT li), s)+ V.f ntt-rto11 pr 1r' ( i )'-r ( f lk I il' g )+ s( kr=r ailk '-r(/' y[r/r'])+ Is1r'y=rnr,g ,,!t1.v1 f (1il r' X ( flL I'f -' ( A )]' .l[r/k] )+ Do1* y=tnou,,,s1.v v f Lrr o.r(i,.f ),rk futur=r /r rt q(.f I re YttLir'7,'f A (f' g[f/]]) t4 Another consequence of the preseuce of fa.ults is that the static allocation of pliorities t,o tasks is then usually ineffective. Consicler the n'ell-knorvu earliestdearlline-first (EDF)policy: t,l're closer the task's cleadlint, the higlier its lrliority. This policy is easy' to implement for tasks rvit,lt incliviclual deacllines. Let r/ : [1, spo*per] ---- R1 clenote such deacllines and for all i such that /(i) I I (i.e . for all invokecl tasks) let ll(r) return the time that, TasA; has beeu irrvokecl; initially h0(i) = 0. \\re introduce a nery prefix operator ctOl.Pe to t'epresetrt the clelay before the action c is offered a.ncl assurning {,hat Pe c.ontaius the tirne variabie t, rve have the rules ar@t.PeS feTOlt) and a@1. I'c'tdl a@t.Pelt + dltl 126). Finalll', let the preclicate rrr.in(i, f ,h,t) hold if anrong the susl>eticlt:rl tasks, 7'n.sfri is the closest to violating its tleadline: nztin(zi, f ,h,t) =a,.r f(i) - T n (/("r) : T + r1(i) + A(t) - t < d(j) + A(j) - l). Then EDIr carr be implenrentecl by' /rI(/b, go, ho,[)).[f (/, g, h,t)' . . .] n'here X (f ,s,/r, l)' I.rrol=,- reqi(01.-r-(/[I l i],s, hlt l il,t)+ I/101g1t,t1 re/,:@l'I(/[ L I i]' s' h' t)-l Lu,,,1,..s,o,r1nkt..rns( ! )ns(r,t=r z'lr (i)'at 'X ( 'f lk li)' g ' lt ' t)I Ig1u v=. n n g,," 1 1 f ail x @l'I("f' clL / k), lt, t )+ r Irlu ;=.n0.,,,s1 1 y f ailk@t' x ( flL / f - ( k )l' s[I/]1, h' l )+ Ir1^.1=r repctirp@t.x (f ' g[T /k]. h ' t) 1r1 The UDF policy is optinra.l for indepenclent tasks on a, single fault-free 1;ro- cessol a.ucl a besl eft'ort policy in general [23]. 5.3 Dynamic Planning-Based Scheduling A planning-ltn.serl schecluler, in coutra.st, will onll'schedule a task if its cleacllines can be guaranteecl. Let each task request a processol by'sencling an upper botrncl bou.nd(r) on t,lie nurnber of basic machine cycles to cor.rrplet,e a.n invoca.tion (r is a pararneter) and let acceptance and rejection oftasks be representecl b1'thcactions occi and rejr respectively. Tl.ren for sporadic tasks we have: Taski -4"7 gI. (licA1 O C) [I' fnl(r). reqi$ound(r)).(acc1.]-(.r) * rej;.I)] lZ(r)' reto.oul(z).X) A planning-based scheduler will maintain a schecluie of all tasks that rvill guarantee their timely cornpletion provided no processors fail in the meanrvhile. Tlre sclieclule is represettted by the function h : [1 .pro] - fl,per ] s,po]* rvhich returns t,he sequence of t,a.sks that are scheduled to be executed otr each processor' (l (ft)o is ctrrrently executetl on Processor'1, alld initially lo(lr) : e ). In aclclition. ,'\r to retuln the upper bouticl on tlie tiuurber of n'e apply b : ll,pet'+ spo) of each active task (initialll'De(i) = 0). curreut iuvocation cvcles for t,Irt' nrachine liach tirne a task cornpletes. the next task is t,ahen for execution aud u-hert a nerv t,a.sk arrives, tlie schc<lulel w'ill try to accorrrnrodate its executiou in the existing schechrle. 'I'his is clorre by'looking fol atr operative Pi'ocr.s.sor';. u'hiclt is L.t fa.st euough to guarantee the additional task's deadline (c is the uulrber of cy'cles ancl strrn returtrs the srttn of all rrurt-rbels in tlte seclrieuce): /sl(r. k,h.c) =a"1 (sunr(h(*))*c)*speedl S d(t).The task is acceptecl rf such a processor exists. hi case a processor fa.ils, the schedulel lvill try to relocate all its t.asks {br execution on other operative processors. This, horvever, may not alrvaJ-s succecd and the sclie<luler then t'r'rters a degracle<l nrocle of operation in rvhich tasks n'hich cauuot be accornnroclated rvill be dropperl from t.he execution. Each tinre tliis happens. tlrc a.ctiorr A;g;nAi is perforrttecl, auuottuciug the nurubel of the task. I("/,9,/,.b) t I11ny=_, reql(c).I1(/,g,h.b.c,i)l Iylo 1s1 t.t1 rel;' X 2( flL I i)' u' tUr (.f (i))' I f ()l' Ioluy=. f uilp'X3(f 'slLlk)' l' ll' r)+ Iu1o1=t lcTrazr'1'-I(/,glf Ik),lt'b) ,\' I ( /. y. lt. b., . i) = I-rrr )-r+ ../ -r(i.a .n.,.t ru i,.'\' ( [. g. h. l,Ja I Xz(1,s, h, b. A,) " Ie1,t)=r'n1"r(i,Ar,h.c) ilrci'X2(f h(k) = s V /(lr(ft)o) I T if lAe rr 6' /( ;) )+ lT li)' s' hlh(k) : ilkl'blcli)' k) -Y(/. lt,h,b) elst Ttrtp(h(k)o).r(/[A/lr(,t)6]. s. lr, b) ' if h(k) = e then I(/,g, h,b) else X+(f ,9, lr.lr.l'. ft(fr)o) X+(f ,g,/t, b, A, i) i I,rrt=rn.lsr(i,r.h,i(r:)) x'zU,u,hb(k)'lk,h(l) : illl'b'l)+ X:1.f ,g,h, b,,() Ir1, 1=t* -.f r(i,t. h.i,( i) ) de g r nde (i )' X :)( /' 9' h [/r ( f' )' / k)' b' k) xL(.f, g,h,b, k) " iJ h(kJ : € v f(h(k)o) * T llrerr -\3( /, S. h, b) e I se 1n't 1,(h (k)0 ).f3(./[A'/ir ( t' )6]. a. /r. 6) s ,\s n.e can see. the planrring-basccl policl'al;ove u'ill only provirlc guaratrtt't's il no failure occllls aftel tasks ale allocat,t'cl but n-ill otlteru'ise degladt'glaccftrlll'' if some tasks cannot be accornmoclat,eci. tlncler sufficientll' stlorlg ;tssuttrptions it rr-ray be possible to plovicle guarantees in the presetrce of atil' farrlts, but the issues of feasibility (assurnptions) ancl utilization (r'esoulces) tnal trtalie such a solrrtion impractical. The graceful ciegradation. ltonever', rvill make it possible to share the loacl a.rnoirg the different nodes of a. clistriltut,ecl systettt. ancl to relocate the tasks for u'liich the deacllines cannot be guaranteed. We havc alreacly shorvu horv to schedule netu'ork traffic to consicler the urgency' of nressages. A sirnilar replication of objects and tasks can also be usecl to ensure resiliencl' to uocle aucl memory failures. Thc issue is then to ensure t,hat tlie replicas are consisteut. 6 Conclusions To analyse the timing properties cif a distributecl sy'steur, it is essential to cousidcr the iimit:rtionsof the resources of the. syst,em artrl the wa\'fesourc(.s at'e a.llocatecl to tasks.'fhe existing forrnal tcchniclues ale eillrel based on the nt:r-xitttal pat'- or provicle verl' basic Ineans ol resolving conrpetition for rcsourcesj bv sttrtically assigning priorities to actious. lf. in aclclitiou. lrarclrvare fa.ilures ale to be consiclerecl. then to staticallv detertuiue the task executiott orclel is rrsualll.inappropriate. In this paper. rve have shou'u horv llie sirrtple a.llelism assrrnrption l6 framework of Timecl CCS can be used for a geueral n-rodel fot' resource-basecl executiols. We have also clemonstratecl the use of different techniques for t'ask schecluling - non-pre-en'rptivt-' and pre-emptive, static ancl clynan'ric, best-effot't and planning-basecl arrcl show'ecl ho'rv to handle priority'inversiou ancl to scheclule network tra{fic. Since faults are unplediclable, reasoning about fault-tolera,nce tnust, be faultrnonotonic: after proving correctness for a nttmber of faults. correcttress for sotrie of thep must be guaranteecl. Nlost te<:hniques for provable fault'-tolerance al'e based o1 a sy'nt,actic replcsentation of faults. Using modal p-calculi a.ncl the aclditiolal trapsitions t,o moclel the effects of faults, we have cletnonstratecl that t'his colnlrioll technique rvill not ensule fault-n'ronotonicity. The first step in a solrttiotr is to clea.rly separat,e design clecisions an<i envirotrmeut assutt-tptions atr<l this rvas done by provicliug the explicit fault-affected semautics of the process languagtr. '.fhe semantics is usecl in the second step, n'here the logic is refined into its faultmonotorric versiou using the timecl ancl moclal exteusiotts of the ]lennessy-XIilner' logic. The logic can verifl' fault-t,olerance an<l lve have deuroustratecl tliat it can be usecl to specify sin-rple trarrsactious. Our work has been based on the timed extetrsion of CICS, Timed CCIS [26], and this rl'as chosen as the sirnplest fi'anrework to suit our purposes. TCCS has beerr furtliel extendecl to allorv loose specificatious, in Tinred Nloclal Specifications [7] rvhich follon'\Ioclal Process Logic [16]. It is possible to rrse N,IPL to specify ancl ver.ify fault-tolerance [4]. N,IPL and its refinetnetrt orclt'r'ing u'oulcl also pern'rit ferver faults tharr tlie maximun.t to occur. apply'ing adrnissible transitiols t,o sltecify tlietn. flut n'ithout separtrtiug design constt'aitrts (tlansitions 11-hich are admissible but unnecessary) and ertrriLotttttettt assutlrptions (tl'arlsit,iols rv|ic| rnodel faults), N'IPL ca.nnot, rvit,hout risking lealizability problerns [], 2], support refinement t,on'ards an inclea.sing ttumber of faults. As tieu'clesign clecisions are rnacle ancl tlie neeci for ne'iv ha.rdrvare or the higher reliabilitJ'arisc, it rnay be necessary' to tolerate nerv faults that coulcl uot have been a.nticipatecl earlier. For untimecl s1'sterns ancl unlirnited resources, this rvas desct'ibecl in [12]: for tirnecl syster-r'rs ancl limitecl resources. this will be subject of a colttpatriotr I)aper. The idea is to provide two u.ays of refiuement to take accoutit of' au increasing number of anticipated faults: the rich-man's refineuent' proceecls to tolerate ail anticipated faults,'cleating'new resources rvhenever neecled to satisfy dea,cllines; the poor-man's refinement proc.eeds until the level of redundancy required exceeds u,hat is available iu the set of resources. References t. N{. Ahadi a.ncl L. Lanrport. (lomposing spe<:ifications. --1 (.1 .\[ Tran,sar:t.iorts ort Pto- gramnilr.g [,crngtrrtgcs cnrcl ,Systetns, 15(1):73 132, 1993. N'|. Abacli, L. Lanrport. ancl P- \\iolper. Rea.lizable anrl unrealizable specificatiols of rea<:tive systclns. Lir'C,J. 372:1-l7. 1989. 3. A.A. Rertossi ancl L.\;. N{ancini. Scheduling algorit,hms for fault-tolerzrn<:e iIr ltarclrr:irl-tinre s-vstems. Rtul-Tine ,9gstetns. T(31229 24i-r' 1994. l7 .1. A. Borjessorr, Ii.Cl. l,arsen, ancl A. Skou. Generalitv in clesign and compositional verification usiug TA\,'. Forntal x[ethods in.5y.sttrn Drsiqn.6(3):23!t-258, 19!)5. 5. A. Burls au(l A. Wellings. A computational model for fixed priorit-v scheduling. In N{. Joseph, r:ditor, Real-Titne ,9ystems: ,9pecification, \|erificot.ion uttl Analysis. Prent,icc-Hall. 1 9[)5. ancl (i. \\'inskcl. C.l(lS rvith prioritl' choice. In.f ortnation artcl (.lorrtPtLt crti ort. 176:2(i--37. 1 995. [i. Cerans. J.C]. Cioclskesen, ancl Ii.Ci. Larsen. 'l'irnecl mo(lal specificat,ions. lt\-(1,5, 715 253 267,1993. R. ('leravelancl ancl t\'1. IIenness.y. Priorities iu proce,'ss algebras. lnfotntatiort artrl. 6. J. (ia.nrilleri 7. 8. Corttptttcrtiott, 8T:58 77. 1990. resourcc-basecl prioritized bisirnulation for real-tittte s1'stcrrrs. 1n/orrrt ati.ort attd (lotn'lttt'lr.ttion, l lil:102 l'12. 1994. J.(j. (loclskesen et a.l. Ep.si,lon - User's ]t[a.n.uol. I)epartrnent of Nlathenratics ancl (.lourputer Scieuce. Ilniversity of Aalborg' 19!]3. l,I. Ilennessr, ancl T. Regan. i\ process alge bra for tirned s)'stems. Technical t t:Port. llniversitv of Sussex. l!191. 'f . Janorvski. Bisimrrlrftion and Fa.ult-Tolercmce. PhD thesis, l)epartment of ('ornputer Science, ITniversit)' of \\Iarrvick, 1995. li. Larsen. Proof s)'stenls for Hennr:ssr'-N'Iilner logic rvitli recrtrsion. l.\ C:'5, !). It. Gerber alcl I. l,ee. A 1iJ. I 1. 12. 1:1. 299:21it 230, 1988. 14. Ii. Larsen. NIodal specifications. LN(:'9,407. 1990. 15. K.(i. Larscu. Conte.r:t-DeperuJenl. Bi.simulution Betu:een Pt'occsse's. PlrD thesis. Ilniversit-v of Edinburgh. Scotlancl, 1986. 16. Ii.Ci. Larsen alcl B. Thornsen. A modal process logic. ln Proc:..'Jrrl .'lrtttttal 'Syrttposittrrt on [,ogic in. CotttPtrtt;r,5cienr:e, pagt:s 20i] 210, BB. 1?. X. Liu. ,9pecificatiort nncl I)t:corttltositiort irt (loncurrency. l'hf) thesis, I)t'parttttcnt of N{athernatics ancl Clontputer Sciencc. I-lnivcrsit}' of Aalborg. 1992' 18. Z. Lil alci X{. Joseph. Transformat,ions of programs for fzmh-toletattt:t. Fortrtal Aspects o.f Contputirtg. 4:442 469, 1{)92. l g. R. X{ilner. (lornntttniccrt.iott, ctnd (lon,crn'rettcy. Prentice-FIall International, 1.()89. 20. F. Nloller ancl cj. Tofts. A tenrporal calculus of cornm. s-vstems. lN("5, .158. 90. 21. J. Parrow. Subrnoclule constrnction as ecltation solr,ing in CCIS. Tl'tcorL-ticrtl (lontputer,9cie.rtce. 68:175 202, 89. 22. P. Pleinevaux. Real-tirne fault toicrant operation of the 802.5 token ring. f ime,9yst.enrs,8:79 91, Renl- 1995. 23. I{. Ramamritharn. Dynamic priority sc}recluling. In M. Joseph, eclitor, Reol-Ti,nte ,gystcnts: SpeciJicatiort, lierif ctttion a.ncl Analysis. Prentice-Hall, 19!15' 24. A,. Salrvicki antl T. N{iildner. On the algorithniic properties of concrtrrent progranrs. INCl,9. 125. 1981. ancl lf.B. Schneicler. Fail stop processors: An approzrch to clesigning fault-tolera.ut compnting svsterns. A(tll Trans. on (iomP. '9ys'. 1(:1), 1983' 25. R.D. Schlichting ,gystetns. PhD thesis. Depa,rttnent of C)ompttter Liniversit]' of Ter:hnologr'. 199I 26. Warrg Yi. I Cirlcuftts o.f Real'I'ime Scie:nce. Clhalnrers 'l'his a.rtic'k,r!as l)t.oc(.sse(l rrsing tlre LNIi,{ rna.cro Packa.ge rvit}r LLN(lS stvle l6