Extremum Seeking Control for Real-Time Optimization Miroslav Krstic UC San Diego IEEE Advanced Process Control Applications for Industry Workshop Vancouver, 2007 Example of Single-Parameter Maximum Seeking f * * asin t f '' f ( ) f * 2 * ˆ k s Plant y 2 s sh sin t 1 Example of Single-Parameter Maximum Seeking * f - unknown! f ( (t)) 2 Topics - Theory • History • Single parameter ES, how it works, and stability analysis by averaging • Multi-parameter ES • ES in discrete time • ES with plant dynamics and compensators for performance improvement • Internal model principle for tracking parameter changes • Slope seeking • Limit cycle minimization via ES 3 Topics - Applications • PID tuning • Internal combustion (HCCI) engine fuel consumption minimization • Compressor instabilities in jet engines • Combustion instabilities • Formation flight • Fusion reflected RF power • Thermoacoustic coolers • Beam matching in particle accelerators • Flow separation control in diffusers • Autonomous vehicles without position sensing 4 History • Leblanc (1922) - electric railways • Early Russian literature (1940’s) - many papers • Drapper and Li (1951) - application to IC engine spark timing tuning • Tsien (1954) - a chapter in his book on Engineering Cybernetics • Feldbaum (1959) - book Computers in Automatic Control Systems • Blackman (1962 chap. in book by Westcott) - nice intuitive presentation of ES • Wilde (1964) - a book • Chinaev (1969) - a handbook on self-tuning systems • Papers by[Morosanov], [Ostrovskii], [Pervozvanskii], [Kazakevich], [Frey, Deem, and Altpeter], [Jacobs and Shering], [Korovin and Utkin] - late 50s - early 70’s • Meerkov (1967, 1968) - papers with averaging analysis • Sternby (1980) - survey • Astrom and Wittenmark (1995 book) - rates ES as one of the most promising areas for adaptive control 5 Recent Developments • Krstic and Wang (2000, Automatica) - stability proof for single-parameter general dynamic nonlinear plants • Choi, Ariyur, Wang, Krstic - discrete-time, limit cycle minimization, IMC for parameter tracking, etc. • Rotea; Walsh; Ariyur - multi-parameter ES • Ariyur - slope seeking • Tan, Nesic, Mareels (2005) - semi-global stability of ES • Other approaches: Guay, Dochain, Titica, and coworkers; Zak, Ozguner, and coworkers; Banavar, Chichka, Speyer; Popovic, Teel; etc. • Applications not presented in this workshop: o o o o o Electromechanical valve actuator (Peterson and Stephanopoulou) Artificial heart (Antaki and Paden) Exercise machine (Zhang and Dawson) Shape optimization for magnetic head in hard disk drives (UCSD) Shape optimization of airfoils and automotive vehicles (King, UT Berlin) 6 ES Book 7 Tutorial Topics Covered in the Book • Introduction, history, single-parameter stability analysis • Plant dynamics, compensators, and IMC for tracking parameter changes • Limit cycle minimization via ES • Multi-parameter ES • ES in discrete time • Slope seeking • Compressor instabilities in jet engines • Combustion instabilities • Formation flight • Anti-skid braking • Bioreactor • Thermoacoustic coolers • Internal combustion engines • Flow separation control in diffusers • Beam matching in particle accelerators • PID tuning • Autonomous vehicles without position sensing 8 Basic Extremum Seeking - Static Map f * * ˆ k s asin t f * minimum of the map f " second derivative (positive - f ( ) has a min.) ˆ estimate of * y 2 s sh sin t k adaptation gain (positive) of the integrator y output to be minimized * unknown parameter f '' f ( ) f * 2 * Plant 1 s a amplitude of the probing signal frequency of the probing signal h cut-off frequency of the "washout filter" / modulation/demodulation s sh 9 How Does It Work? f * * f '' f ( ) f * 2 * ˆ asin t Estimation error: k s Plant y 2 s sh sin t * ˆ 2 2 a f " f " a f" 2 y f* % af "%sin t cos 2 t 4 2 4 Loc. Analysis - neglect quadratic terms: 2 2 a f " f " a f" 2 y f* % af "%sin t cos 2 t 4 2 4 10 How Does It Work? f * * ˆ asin t f '' f ( ) f * 2 * k s Plant y 2 s sh sin t 2 2 a f " a f" * % y f af " sin t cos 2 t 4 4 2 s a2 f " a f" * [y] f af "%sin t cos 2 t sh 4 4 11 How Does It Work? f * * asin t f '' f ( ) f * 2 * ˆ k s Plant y 2 s sh sin t Demodulation: 2 s a f" 2 sin t [y] af "%sin t cos 2 t sin t sh 4 a2 f " % a2 f " % a2 f " cos 2 t sin t sin 3 t 4 4 8 12 How Does It Work? f * * f '' f ( ) f * 2 * ˆ Plant asin t k s y 2 s sh sin t Since * ˆ then ˆ & 13 How Does It Work? f * * asin t f '' f ( ) f * 2 * ˆ k s Plant y 2 s sh sin t k a2 f " % a2 f " % a2 f " cos2t sin t sin 3t s 4 4 8 high frequency terms - attenuated by integrator 14 How Does It Work? f * * f '' f ( ) f * 2 * ˆ asin t k s Plant y 2 s sh sin t ka 2 f " % 4 Stable because k,a, f " 0 15 Stability Proof by Averaging f * * f ( ) f * Plant f '' * 2 * ˆ y 2 e f* asin t ˆ k s t s sh h y sh sin t Full nonlinear time-varying model: d % k f" % asin d 2 2 e sin 2 d h f" % e e asin d 2 16 Stability Proof by Averaging f * * f ( ) f * f '' * 2 Plant y 2 * ˆ e f* ˆ k s asin t s sh t h y sh sin t Average system: d % kaf " % av av d 2 d h f " %2 a 2 eav eav av d 2 2 Average equilibrium: av 0 a2 f " eav 4 17 Stability Proof by Averaging f * * f ( ) f * f '' * 2 Plant y 2 * ˆ e f* asin t ˆ k s s sh t h y sh sin t Jacobian of the average system: kaf " 0 2 J av h 0 18 Stability Proof by Averaging f * * f ( ) f * f '' * 2 Plant y 2 * ˆ e f* ˆ asin t k s s sh t h y sh sin t Theorem. For sufficiently large there exists a unique exponentially stable periodic solution of period 2/ and it satisfies a2 f " 1 2 / (t) e2 / (t) O , t 0 4 Speed of convergence proportional to 1/, a2, k, f " 19 Stability Proof by Averaging f * * f ( ) f * f '' * 2 Plant y 2 * ˆ e f* asin t ˆ k s s sh t h y sh sin t Output performance: 1 y f * f "O 2 a 2 20 PID Tuning Using ES Based on contributions by: Nick Killingsworth Background & Motivation Proportional-Integral-Derivative (PID) Control • Consists of the sum of three control terms - Proportional term: - Integral term: - Derivative term: uP (t ) Ke(t ) K t u I (t ) e( s)ds TI de(t ) u D (t ) KTD dt e(t) = r(t) – y(t) r(t) reference signal y(t) measured output • Often poorly tuned (Astrom [1995], etc.) 22 Background – PID We use a two degree of freedom controller The derivative term only acts on y(t) • This avoids large control effort when there is a step change in the reference signal r Cr u + - y G Cy 1 Cr K 1 TI s 1 C y K 1 TD s TI s 23 Tuning Scheme Step function r(t) Continuous Time Cr k - u(t) J(k) y(t) G C y k k Extremum Seeking Algorithm Discrete Time 24 Extremum Seeking (k ) cos(k ) y (k ) J ( ) z 1 z 1 zh cos(k ) Simple - three lines of code 25 Extremum Seeking Tuning Scheme Implementation 1. 2. Run Step response experiment with ZN PID parameters Calculate J T 1 2 J ( k ) e( k ) dt T t0 t0 26 Extremum Seeking Tuning Scheme Implementation 1. 2. 3. Run Step response experiment with ZN PID parameters Calculate J Calculate next set of PID parameters using discrete ES tuning method (k ) h (k 1) J (k 1) ˆi (k 1) ˆi (k ) i i cos(i k )J (k ) (1 h) (k ) i (k 1) ˆi (k 1) i cos(i (k 1)) 27 Extremum Seeking Tuning Scheme Implementation 1. 2. 3. 4. Run Step response experiment with ZN PID parameters Calculate J Calculate next set of PID parameters using discrete ES tuning method Run another step response experiment with new PID parameters 28 Extremum Seeking Tuning Scheme Implementation 1. 2. 3. 4. 5. Run Step response experiment with ZN PID parameters Calculate J Calculate next set of PID parameters using discrete ES tuning method Run another step response experiment with new PID parameters Repeat 2-4 set number of times or until J falls below a set value Repeat 29 Implementation – Cost Function Cost Function J(k) Used to quantify the controller’s performance Constructed from the output error of the plant and the control effort during a step response experiment Has discrete values at the completion of each step response experiment T 1 2 J ( k ) e ( ) dt k T t0 t0 where T is the total sample time of each step response experiment is a vector containing the PID parameters: K , TI , TD 30 Implementation – Cost Function Cost Function J(k) t0 is the time up until which zero weightings are placed on the error. This shifts the emphasis of the PID controller from the transient phase of the response to that of minimizing the tracking error after the initial transient portion of the response T 1 2 J ( k ) e ( ) dt k T t0 t0 to 31 Example Plants Four systems with dynamics typical of some industrial plants have been used to test the ES PID tuning method 1. Time delay G1 ( s ) 2. 3. 1 e 5 s 1 20 s Large time delay 1 G2 ( s ) e 20 s 1 20 s Single pole of order eight G3 ( s ) 4. 1 (1 10s )8 Unstable zero G4 ( s) 1 5s (1 10s)(1 20s) 32 Results • Ziegler-Nichols values used as initial conditions in the ES tuning algorithm • Results compared to three other popular PID tuning methods: - Ziegler-Nichols (ZN) - Internal model control (IMC) - Iterative feedback tuning (IFT, Gevers, ‘94, ‘98) 33 Results - 1 G1 ( s ) e 5 s 1 20 s a) Evolution of Cost Function b) Evolution of PID Parameters c) Step Response of output d) Step Response of controller 34 Results - G2 ( s ) a) Evolution of Cost Function c) Step Response of output 1 e 20 s 1 20 s b) Evolution of PID Parameters d) Step Response of controller 35 Results - G3 ( s ) 1 (1 10s )8 a) Evolution of Cost Function b) Evolution of PID Parameters c) Step Response of output d) Step Response of controller 36 Results - G4 ( s) a) Evolution of Cost Function c) Step Response of output 1 5s (1 10s)(1 20s) b) Evolution of PID Parameters d) Step Response of controller 37 Results – Cost Function Comparison Step Response of output The following cost functions were minimized using ES: T 1 ISE e( k ) 2 dt T 0 38 Results – Cost Function Comparison Step Response of output The following cost functions were minimized using ES: T 1 ISE e( k ) 2 dt T 0 T 1 ITSE te( k ) 2 dt T0 39 Results – Cost Function Comparison Step Response of output The following cost functions were minimized using ES: T 1 ISE e( k ) 2 dt T 0 T 1 ITSE te( k ) 2 dt T0 T 1 IAE | e( k ) | dt T0 40 Results – Cost Function Comparison Step Response of output The following cost functions were minimized using ES: T 1 ISE e( k ) 2 dt T 0 T 1 ITSE te( k ) 2 dt T0 T 1 IAE | e( k ) | dt T0 T 1 ITAE t | e( k ) | dt T0 41 Results – Cost Function Comparison Step Response of output The following cost functions were minimized using ES: T 1 ISE e( k ) 2 dt T 0 T 1 ITSE te( k ) 2 dt T0 T 1 IAE | e( k ) | dt T0 T 1 ITAE t | e( k ) | dt T0 T 1 2 Window e ( ) dt k T t0 t0 42 Actuator Saturation Saturation of 1.6 applied to control signal for plant G1 1 G1 ( s ) e 5 s 1 20 s ES and IMC compared with and without the addition of an anti windup scheme Tracking anti-windup scheme 43 Actuator Saturation Step response of output Control signal during step response 1.6 1.2 1.4 IMC IMCtracking 1.2 ES EStracking 1 1 u(t) y(t) 0.8 0.6 0.8 0.6 0.4 IMC IMC ES EStracking 0.2 0 0.4 tracking 0 20 40 Time(sec) 60 80 0.2 100 0 0 20 40 60 80 100 Time(sec) 44 Effects of Noise Band-limited white noise has been added to output Power spectral density = 0.0025 Correlation time = 0.2 Independent noise signal for each iteration Simulations on plant G1 1 5 s G1 ( s) e 1 20s 45 Effects of Noise a) Evolution of Cost Function c) Step Response of output b) Evolution of PID Parameters d) Step Response of controller 46 Selecting Parameters of ES Scheme Must select , perturbation step size , adaptation gain , perturbation frequency h, high-pass filter cut-off frequency (k ) ˆ(k ) i J (k ) J ( ) i i z 1 (k ) z 1 zh cos(k ) Looks like have more parameters to pick than we started out with! However, ES tuning is less sensitive to parameters than PID controller. 47 Selecting Parameters of ES Scheme G2 ( s) 1 e 20s 1 20s ES Tuning Parameters 0.06,0.30,0.20T , 2500,2500,2500 T i 0.8i h 0.5 , 2 , 10 , 2 10 K Ti Td 1.01 31.5 7.16 1.00 31.1 7.6 1.01 31.3 7.54 1.01 31.0 7.65 48 Selecting Parameters of ES Scheme Need to select an adaptation gain and perturbation amplitude for EACH parameter to be estimated In the case of a PID controller, = [K, Ti, Td], so we need three of each. The modulation frequency is determined by: i ai * where 0 < a < 1 The highpass filter (z-1)/(z+h) is designed with 0<h<1 with the cutoff frequency well below the modulation frequency . i Convergence rate is directly affected by choice of and , as well as by cost function shape near minimizer. 49 Example of ES-PID tuner GUI Evolution of step response under ES tuning 1 iteration (ZN) 5 iterations 10 iterations 50 iterations 1.1 Step tracking response 1.05 1 0.95 0.9 0.85 0.8 0 10 20 30 40 50 Time (sec) 60 70 80 90 100 50 Punch Line ES yields performance as good as the best of the other popular tuning methods Can handle some nonlinearities and noise. The cost function can be modified such that different performance attributes are emphasized 51 Control of HCCI Engines Based on contributions by: Nick Killingsworth (UCSD), Dan Flowers and Sal Aceves (Livermore Lab), and Mrdjan Jankovic (Ford) HCCI = ? HCCI = Homogeneous Charge Compression Ignition Low NOx emissions like spark-ignition engines High efficiency like Diesel engines More promising in near term than fuel cell/hydrogen engines 53 HCCI Engine Applications Distributed power generation Automotive hybrid powertrain 54 What is the difference between Spark Ignition, Diesel, and HCCI engines? Categories of Engines Compression Ignition Spark ignition Homogeneous charge HCCI Spark ignition engine Inhomogeneous charge Diesel Direct injection engine 56 Spark Ignition Engine Basic engine thermodynamics: engine efficiency increases as the compression ratio and =cp/cv (ratio of specific heats) increase engine indicated efficiency 0.7 0.6 1.4 0.5 SI engines 0.4 1.3 0.3 0.2 1 Engine Efficiency 1 CR 1 0.1 = 1.35 for fuel and air mixture 0.0 = 1.4 for air 1 3 5 7 9 11 13 15 17 19 compression ratio 57 Diesel Engine Highly efficient because they compress only air ( is high) and are not restricted by knock (compression ratio is high) engine indicated efficiency 0.7 Diesel engines 0.6 1.4 0.5 SI engines 0.4 1.3 0.3 1 Engine Efficiency 1 CR 1 0.2 = 1.4 for air 0.1 0.0 = 1.35 for fuel and air mixture 1 3 5 7 9 11 13 15 17 19 compression ratio 58 HCCI Engine Compression ratio not restricted by “knock” (autoignition of gas ahead of flame in spark ignition engines) a efficiency comparable to Diesel engine indicated efficiency 0.7 0.6 1.4 0.5 Diesel and HCCI engines SI engines 0.4 1.3 0.3 Engine Efficiency 0.2 1 = 1.4 for air 0.1 0.0 1 CR 1 = 1.35 for fuel and air mixture 1 3 5 7 9 11 13 15 17 19 compression ratio 59 HCCI Engine Potential for high efficiency (Diesel-like) Low NOx and PM (unlike Diesel) BUT, no direct trigger for ignition - requires feedback to control the timing of ignition! 60 Experiment at Livermore Lab Caterpillar 3406 natural gas spark ignited engine converted to HCCI Set up for stationary power generation (not automotive) 61 Actuators Mixing “Tees”(x 6) Combustion timing (output) is very sensitive to intake temperature (input) Cooled Intake Air Valve Actuators Heated Intake Air Controlled Intake Temperature to Individual Cylinders Cold Manifold Hot Manifold 62 Overall Architecture: Sensors and Software User interface Valve Position Real-Time Controller PC running Labview RT OS Cylinder Pressure Crank Angle Position 63 ES used to MINIMIZE FUEL CONSUMPTION of HCCI engine by tuning combustion timing setpoint + S e CA50 - PI CA50SP Tintake HCCI Engine CA50 . mfuel Extremum Seeking 64 ES delays the combustion timing 6 crank angle degrees, reducing fuel consumption by > 10% 65 Larger adaptive gain: ES finds same minimizer, but much more quickly 4.8 16 CA50 ave CA50_setpoint mass flow rate of fuel 4.7 12 4.6 10 4.5 8 4.4 6 4.3 4 4.2 2 4.1 0 4 0 50 Fuel Consumption [g/s] Combustion Timing CA50 [CAD ATDC] 14 100 150 200 250 300 350 400 450 500 550 Time (sec) 66 Axial Flow (Jet Engine-Like) Compressor Control • Active controls for rotating stall only reduce the stall oscillations but they do not bring them to zero nor do they maximize pressure rise. • Extremum seeking to optimize compressor operating point. Experimental Results Extremum seeking stabilizes the maximum pressure rise. Pressure Rise Problem Statement time bleed valve Caltech COMPRESSOR Motivation Smaller, lighter compressors; higher payload in aircraft Pressure rise EXTREMUM SEEKER Air Injection Stall Controller 1 s sin t washout filter Combustion Instability Control Problem Statement • • Experiment on UTRC 4MW combustor Rayleigh criterion-based controllers, which use phaseshifted pressure measurements and fuel modulation, have emerged as prevalent ext. seeking suppresses oscillations The length of the phase needed varies with operating conditions. The tuning method must be non-model based. time Pressure Motivation fuel • • Tuning allows operation with minimum oscillations at lean conditions Reduced engine size, fuel consumption and NOx emissions COMBUSTOR Phase-Shifting Controller 1 s sin t Frequency/ amplitude observer washout filter EXTREMUM SEEKER Formation Flight Engine Output Minimization Tune reference inputs yref and zref to the autopilot of the wingman to maximize its downward pitch angle or to minimize its engine output 69 Simulation of C-5 Galaxy transport airplane for a brief encounter of “clear air turbulence” 70 Thermoacoustic Cooler (M. Rotea) Electric energy Acoustic energy Hot-end heat exchangers Stack Cold-end heat exchangers Heat pumping Resonance tube Pressurized He-Ar mixture Electro-dynamic driver Standing sound wave creates the refrigeration cycle Solid surface (stack plate) Gas particle in a standing wave QH 3 2 QL 1 4 Heat Pumping 1-2: adiabatic compression and displacement 2-3: isobaric heat transfer (gas to solid) 3-4: adiabatic expansion and displacement 4-1: isobaric heat transfer (solid to gas) 71 Thermoacoustic Cooler Neck (mass) Volume (stiffness) Moving piston (varying resonator’s stiffness) Electro-dynamic Driver Heat exchangers Helmholtz resonator Tuning Variables • Piston position (acoustic impedance) • Driver frequency 72 ES with PD compensator ax sin(xt x ) sin(xt ) LPF Integrator PD + POS Command a f sin( f t f ) sin( f t ) LPF HPF + Integrator Qc Cooling Power Calculation PD + Tunable Cooler + FREQ Command 1 Tf s 1 1 Tx s 1 73 Experiment – Fixed Operating Condition 4 141 22.65 142 29.92 143 35.67 144 28.63 145 21.25 142 144 15.89 34.12 145 39.68 146 35.12 148 19.34 140 4.95 142 9.00 144 18.55 145 146 23.86 35.99 147 41.28 148 38.00 149 30.36 150 19.36 146 16.34 148 33.34 149 41.21 150 40.70 151 153 34.69 19.63 151 32.16 152 35.60 153 31.74 5 6 7 8 Cooling Performance with ESC Cooling Power (Watt) POWER W 60 Piston Position (in) FREQ Hz 8 40 20 0 ESC ON 0 50 100 150 200 250 300 0 150 50 100 150 200 250 300 50 100 150 Time (sec) 200 250 300 6 4 2 Driving Frequency (Hz) POS in. 145 140 0 ESC quickly finds optimum operating point (41.3W, 147Hz, 6.2in) 74 Cooling Power (Watt) Experiment – Varying Operating Condition 100 50 0 0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400 150 200 250 300 350 400 350 400 Piston Position (in) 12 10 8 6 Driving Frequency (Hz) 4 155 150 ESC ON 145 140 0 50 100 Flow Rate Change 100 Flow Rate (ml/s) ESC tracks optimum after cold-side flow rate is increased 80 Cold Side Hot Side 60 40 20 0 50 100 150 200 250 300 Time (sec) 75 ES for the Plasma Control in the Frascati Fusion Reactor Contribution by Luca Zaccarian (U. Rome, Tor Vergata) Optimization Objective Framework: Additional Radio Frequency heating injected in the plasma by way of Lower Hybrid (LH) antennas: plasma reflects some power Goal: Optimize coupling between the Lower Hybrid antenna and tha plasma, during the LH pulse Reflected Power Map Reflected power: Convex fcn of edge density Convex fcn of edge position Possible approaches to optimize: 1. Move the antenna (too slow!) 2. Move the plasma (viable – adopted here) Probing not Allowed - Modified ES Scheme Knob Extracted Input sinusoid Extracted Output sinusoid Experimental results with medium gain K = 300 Safety saturation limits performance Control action is quite aggressive. Experimental results with lower gain K = 200 (Antenna has been moved) Graceful convergence to the minimum reflected power Gain too high - instability K = 350 Instability Gain is too large Experiments - Summary Input/output plane representation: K = 300: saturation prevents reaching the minimum K = 200: graceful convergence to minimum (slight overshoot) K = 350: gain too high – all the curve is explored