Estimating the Change Point of Binary Profiles
with a Linear Trend Disturbance
Alireza Sharafi
(corresponding author) M.S.C. Department of Industrial Engineering, Amirkabir University. a.sharafi@aut.ac.ir,
Majid Aminnayeri
Associate Prof. Department of Industrial Engineering, Amirkabir University. mjnayeri@aut.ac.ir,
Amirhossein Amiri
Assistant Professor. Department of Industrial Engineering, Faculty of Engineering, Shahed University. amiri@shahed.ac.ir,
Mohsen Rasouli
M.S. student. Department of Industrial Engineering, Isfahan University of Technology. mohsen.rasouli@in.aut.ac.ir,
sooner and corrective action can be implemented more
quickly. Identifying the real time of the process change
is known as change-point estimation problem.
Abstract
Identification of a real time of a change in a process,
when an out-of-control signal is present is significant.
This may reduce costs of defective products as well as
the time of exploring and fixing the cause of defects.
Another popular topic in the Statistical Process
Control (SPC) is profile monitoring, where knowing
the distribution of one or more quality characteristics
may not be appropriate for discussing the quality of
processes or products. One, rather, uses a relationship
between a response variable and one or more
explanatory variable for this purpose. In this paper, the
concept of Maximum Likelihood Estimator (MLE)
applied to estimate of the change point in binary
profiles, when the type of change is drift. Simulation
studies are provided to evaluate the effectiveness of
the change point estimator.
Change point problems are mainly classified according
to change types including step, drift and isotonic
shifts. To find the real time of a change, many authors
have suggested several methods such as Maximum
Likelihood Estimator (MLE), Cumulative Sum
(CUSUM), Exponentially Weighted Moving Average
(EWMA) and intelligent method, (artificial network,
clustering and decision tree). For example Pignatiello
and Samuel [1- 4] proposed a Maximum Likelihood
Estimator in different control charts to find the real
time of change point under a step shift. Perry and
Pignatiello [5, 6] used this method to estimate the
change point in the mean of normal and poisson
distribution with a linear trend disturbance,
respectively.
Keywords: binary profiles; change point; maximum
In the area of SPC, usually the quality of a process or a
product was represented by the distribution of one (or
more) quality characteristic and monitored by
univariate (or multivariate) control chart. However,
Kang and Albin [7] presented a popular topic with
widespread application in SPC namely, profiles
monitoring, in which a relationship between a
response variable and one or more independent
variables, known as profiles is monitored over time.
Scientists classify profiles based on the type of
relationships into the linear (simple, multiple and
polynomial), nonlinear and geometric profiles. Profiles
are studied in Phases I and II. In Phase I, Mestek et al.
likelihood estimator (MLE); statistical process control
(SPC), control chart
1- Introduction
Statistical Process Control (SPC) is used to monitor
and reduce the variation of a process. Control charts
are the most important tools in SPC which are used to
monitor quality characteristics. These charts can detect
any changes or shifts in a process; however, a shift
usually occurs much earlier before it is detected. When
a control chart signals a special cause, quality
engineers should identify and remove the cause(s) of
variation and return the process to in-control state.
However, knowing when a process has changed would
help quality engineers to limit the time window within
which they should search for assignable causes.
Consequently, the assignable causes can be identified
2
[8] used a T based control charts to monitor the
regression parameters of a simple linear profiles.
Mahmoud and Woodall [9] used a global F-test for
monitoring the regression coefficients in conjunction
with a univariate control chart to monitor the variance.
Mahmoud et al. [10] applied a likelihood ratio test to
monitor simple linear profiles. In Phase II, Kang and
1
Albin [7]
proposed
two
methods including a
2- Logistic Regression Model
2
multivariate T chart and a combination of EWMA-R
chart to monitor simple linear profiles. Kim et al. [11]
recommended three independent univariate EWMA
control charts to monitor shifts in parameters of a
simple linear profile.
Many categorical response variables have only two
categories: for example, whether you take public
transportation today (yes, no), or whether you have
had a physical exam in the past year (yes, no). Denote
a binary response variable by y and E ( y )   . The
value of  can vary as the value of x changes, and
we replace  by  ( x ) . For n independent
observations, the number of successes has the
binomial distribution specified by the index n and
parameter  . The relationships between  ( x ) and
Identifying the change point in profiles has been
studied by some researchers. Mahmoud et al. [10] used
a likelihood ratio based method to identify the real
time of a step change in Phase I monitoring of a simple
linear profiles. Zou et al. [12] proposed a method
based on likelihood ratio statistics to find the change
point in parameters of a simple linear profile in Phase
II. Kazemzadeh et al. [13] used the same method to
estimate the change point in polynomial profiles under
a step shift in Phase I. Note that the change point
problem in profile data is under a different sampling
framework from that of the other models. In the profile
applications, multiple data sets are collected over time
in a functional data sampling framework and a
possible change point is take place after any data set.
x are usually nonlinear and the S -shaped curves are
often realistic shapes for this relationship which are
called the logistic regression models.
In a logistic regression model, there are n independent
experimental sets, with p predictor variables for each
set, which is shown by
xi  ( xi1 , xi 2 ,, xip )T as well
as corresponding Bernoulli response variables namely
zi for i  1, 2,..., n . The probability of success in
Most researches in profile monitoring, assume the
response variable is continuous (usually Normal) and
characterize profiles with linear or nonlinear models.
However, in many industrial applications the response
variable is discrete such as binary (as in the case of a
product can be classified as defective or non defective)
or countable (as the number of defect products or
number of patients in a hospital). Yeh et al. [14]
studied binary profiles in Phase I. They proposed
each set is denoted by
i
and each
i
is a function of
xi . In the logistic regression model this function is
characterized by the link function g  i  , defined as
g  i  
2
different T control charts for monitoring logistic
regression profiles. Shang et al. [15] proposed a
control scheme based on EWMA-GLM to monitor the
relationship between the binary response and random
explanatory variables in Phase II. Their approach
assumes that explanatory variables are random
variable and takes different values in each profile
sample. To the best of our knowledge, Only Sharafi et
al. [16] suggested a method to identify the real time of
a step change in Phase II monitoring of binary profiles
and there is no more researches in this area. In this
paper we propose a Maximum Likelihood Estimator to
find the real time of a change in Phase II monitoring of
logistic regression profiles. Here, we assumed that the
type of the disturbance is drift. The rest of the paper is
organized as follows: Section 2 illustrates logistic
regression model and explains the steps of estimating
the model parameters. Section 3 presents the change
point model and assumptions of the problem. The
performance of the proposed model is investigated in
Section 4. Conclusions and some future researches are
provided in the final section.
log  i 
1  log  i 
 1 xi1   2 xi 2  ...   p xip

where   1 ,  2 ,...,  p

T
(1)
is the model parameters
vector. It is usual to set xi1  1 in order that 1 be the
intercept of the model. Equation (1) simplifies to
i 
exp( xiT  )
1  exp( x  )
T
i

exp(i )
1  exp(i )
T
In this equation i  xi  
(2)
.
p
 x
k ik
. We also assume
k 1
that data are grouped so that for the i th setting of the
predictor variables, there are
i  1,2,..., n .
mi observations,
n
M   mi
is
total
number
of
i 1
mi
observations. The response variable is
yi   zik ,
k 1
where
2
zik is the k th observation (0 or 1) in i th
yi follows a binomial
distribution with parameters mi ,  i . Albert and
predictor variable settings,
Initialize the estimate of   ˆ (0) . Set i  0
Anderson [17] proposed MLE method to estimate the
model parameters for this purpose. They used the
following likelihood function:
m 
y
m y
L (, y)    i   i  i 1   i  i i ,
i 1  y i 
ˆ (i )
Calculate ˆ (i ) , ˆ (i ) , ˆ (i ) and W

(i )
ˆ (i )
ˆ (i )  W
qˆ  
n
(3)
Where    1 ,  2 ,...,  n  and y   y1 , y2 ,..., yn 
Taking the logarithm of the Equation (3) and using
i  xiT  
i
p
 x
 log
k ik
k 1
1  i

ˆ (i ) X
ˆ ( i 1)  XT W
l (, y ) 
 mi 
n
( i 1)
ˆ
p
i
Yes
(4)
(i)
ˆ  ˆ
Figure I. The procedure of estimation of logistic
regression parameters
Taking derivative of Equation (4) with respect to  ,
and using iterative weighted least square estimation
method suggested by McCullagh and Nelder [18], the
logistic regression parameters can be estimated as
follows:

ˆ
  XT WX
In
the

ˆ
XT Wq
Equation
3- MLE Change Point Estimator
Here, it is assumed that the underlying process initially
operates in a state of statistical control, with
observations coming from a poisson distribution with
the known parameters    0 (  is a p-dimensional
(5)
(5)
X   x1 , x2 ,..., x n  ,
T
ˆ  diag  m ˆ 1  ˆ  , m ˆ 1  ˆ  ,..., m ˆ 1  ˆ  ,
W
1 1
1
2 2
2
n n
n
vector);
ˆ 1  y  ˆ  and ˆ   m ˆ , m ˆ ,..., m ˆ  .
qW
1 1
2 2
n n
f yij
 
T
The procedure iterations is described in the figure I.
McCullagh and Nelder [18] proved that as
large
or
mi
is
constant,

is



1
so,
the
mass
probability
 mi  y
m y
    i ij 1   i  i ij where
y
 ij 
function
is
yij is the
value taken by the response variable for the i th value
of the predictor variable in the j th profile.
n becomes
distributed
The change type is also assumed to be drift or linear
trend change in the parameter  i . Consider an in-
asymptotically as a p-dimensional normal distribution
N p , XT WX

No

1
ˆ ( i )q ( i )
XT W
k ik

 p

 mi log 1  exp   k xik  ,


i 1
 k 1
 


1
i 1 k 1
i
n

(i )
( i 1)
ˆ  ˆ
 log  y    y  x
i 1
(i )
, one can reexpress
the log-likelihood as
n
1
Update the estimation of 
i  i 1
T
T
  y  ˆ 
 . This procedure will be used in
control process with independent observations coming
from a binomial distribution with parameters mi and
MLE change point estimator described in Section 3.
i .
After elapsing an unknown amount of time, the
parameter  i changes from its in control state of  i 0
to an unknown out-of-control state of  i1 ,  i1
  i0 ,
and the function of  i1 is given as
 i1   i 0  b  j   
3
j    1,  2,..., T ,
(6)
where
Ln L ( , b y )
b is the slope (or magnitude) of the linear trend
i .
T
During the formulation of profiles j  1,2,..., the
process parameter
i
become larger with linear trend
some unknown value  i1
T

b and equal to
  i1  b  j    
  1    b j   

 
j  1 i 1 
i1

yij
1   i1  b  j    
n

n
mi

(7)
n
j  1 i 1

T
  i0 

1   i 0 
n
j 1 i 1

n
 m Ln 1   
i
j 1 i 1
i0
(10)
   b  j   
   yij Ln  i1

j  1 i 1
1   i1  b  j    
T
n
T

   b j   

(8)

i1  b  j    

  mi Ln 1   i 0     yij Ln 1  i1
j 1 i 1
b , and denote it as b̂ .
ˆ  arg max { yij Ln 
 mi   n
  
   yij Ln  i 0  

1   i0 
 yij  j 1 i 1
T


 j  




b
j





i1

Substituting b for b in Equation (8) and computing
Equation (8) over all possible change points  in
search of the maximum log-likelihood function yields
Ln L( , b y )    Ln 
j 1 i 1
i
MLE of the rate parameter,
The MLE of  is the value of  that maximizes the
likelihood function in Equation (7). Taking the
logarithm of Equation (7)


n
  m  1  
In Equation (9) there is no closed-form solution for b .
Perry and Pignatiello [6] proposed the Newton’s
method (also known as the Newton–Raphson method)
to find the MLE of the rate parameter, b . Newton’s
method is one of the Numerical analysis which is used
to find successively better approximations to the roots
of a function. We use Newton’s method to find the
n
n
yij 
j  1 i 1
  i 0  b  j   , where
 mi   n   i 0 
mi
L( , b y )   
 yij  1    1   i 0 
j 1 i 1 
i0 
 j 1 i 1 
T

 j  

 1   i1  b  j      i1  b  j      (9)




T is the last profile sampled in which the control chart
signaled an out-of-control state. Two unknown
parameters in model are  and b , representing the
last profile taken from an in-control process and the
rate of the linear trend, respectively. To estimate these
unknown parameters, this paper used a MLE approach,
similar to Perry and Pignatiello [6]. We denote the
MLE estimator of the change point as ˆ . The
likelihood function is as follows:


n
j  1 i 1
is equal to its known in-control
value. For profiles j    1,  2,..., T the parameter
i

b
disturbance in
n
 m Ln 1  
j  1 i 1
i
i1
 b  j    }
where ˆ is the MLE for the last profile taken from the
in-control process. Whenever this chart signals an outof-control state, the real time of a change can be
estimated via Equation (10).
n
  mi Ln 1   i1  b  j    ,
j  1 i 1
To determine the unknown parameters in Equation (8),
(  and b ), an expression is required for b in terms of
 which maximize the log-likelihood function in
Yeh et al. [14] introducted five Hotelling T control
charts to monitor binary profiles in Phase I. The
plotting statistic for profile j is defined as
Equation (8), defined as b . Obtaining this expression
is not a trivial task. The partial derivative of Equation
(8) with respect to b is given by
T j2   j  
2


T


S 1  j   ,
(11)
where ˆ j is estimator of the logistic regression
parameters in
j th profile and S is the variance
covariance matrix of ˆ j . Any of these
T 2 charts uses
a different method to estimate the mean vector and
4
Now, suppose an out-of-control process whose
parameter vector  i
shifts from  i 0
to
2
covariance matrix. They showed TI control chart
which estimate the covariance matrix by averaging the
covariance estimates of each given sample, provides
more effective way to detect step shifts and drift.
These control charts also can be used in Phase II. So
 i1   i 0  b  j    . A Monte Carlo simulation
study is accomplished to examine the performance of
the estimator. The change point is simulated at profile
50 (   50 ). For profiles j  1, 2,...,50 the
independent observations is produced by binomial
distribution with parameters 50 and  i 0 . Starting at
2
we used TI control chart to detect the out-of-control
state in Phase II. Covariance matrix in Phase II is
computed using the following equation:
  ( XT WX ) 1 ,
where
profile 51, observations are simulated from the out-of-
(12)
W
is
equal
control process with  i1
to
  i 0  b  j  50 until the
diag{m 1 (1   1 ), m 2 (1   2 ),, m n (1   n )}
TI2 chart signals an out-of-control. At this time, the
. The upper control limit for the proposed control chart
change point estimator is used and the real time of the
process change is determined. This procedure is
repeated 10,000 times for each magnitude of the
change under study, which is described in the figure II.
is also equal to  2, ,the  percentile points on the
2
chi-square distribution with 2 degrees of freedom.
The results are summarized in Tables I and II and
Figure III. Table I shows the expected length of each
simulation run E  T  which is the expected value of
4- Performance of the MLE Estimator
To estimate the performance of the MLE estimator, an
example is given. In this example, number of
explanatory variables is equal to 2, (p=2). Thus the
link function is simplified as g  i   1   2 xi ,
the number of samples taken until the first alarm is
given by the control chart, i.e. E T   ARL  50 .
Table I also shows the average change point estimate
and the standard deviation of the change point
estimator under different magnitude of the shifts
considered. Because the actual change is at time 50,
the average change point estimate,  , should be close
to 50.
where 1 and  2 are the intercept and the slope of the
regression function respectively and is shown by the

vector   1 ,  2

T
We set the matrix X as:
T
1
...
1
 1

X =
 ,
 log  0.1 log  0.2  ... log  0.9  
It is assumed that the number of experiments in each
predictor variable is constant and equal to 50 ,
Produce 50 independent incontrol profiles with the
vector of  i 0
( mi  50 for i  1,2,...,9 ) and the in-control  i 0 is
 0.04,0.06,0.13,0.27,0.48,0.68,0.82,0.89, 0.91T .
initial  is estimated as  2.5, 3.46 
T
from historical
2
2,0.005
 10.59 .
i
The
dataset in Phase I. The upper control limit for the TI
control chart is equal to
Make a change in the vector of
10,000
times
repeat
2
The
covariance matrix of the logistic regression parameters
(  ) in Phase II is computed by Equation (12) as
follows:
Estimate the logistic parameters and
depict control chart until issuing an outof control signal
Calculate  for all profiles
Determine the largest  as the
change point
Figure II. The flowchart of the simulation procedure
  (X T WX) 1
 0.06627
 
 0.07693
0.07693 

0.1179 
5
Table I. Expected number of samples until the signal and
standard deviations of the change point estimators
with 10,000 simulations runs when   50
b
E T 

se(ˆ)
0.003
56.13
51.82
2.39
0.005
55.01
51.21
1.73
0.007
53.86
50.86
1.32
0.01
53.02
50.58
1.01
0.015
52.43
50.44
0.74
0.02
52.04
50.29
0.51
0.025
51.72
50.21
0.41
Figure III. The performance of the MLE estimator
Table II shows the results of proportion of 10,000
simulation runs that the estimator lies within a
specified tolerance of the real change point value. For
example if b  0.01, the estimated probability that ˆ
lies within 1 or less from the real change point is 0.81.
Also in this case, in 46% of the simulation runs the
estimator correctly identifies the real time of the
change.
As shown in Table I, for drift rate parameter equal to
0.003 the expected number of samples taken until the
signal is 56.13, the average change point estimate is
51.82 and the standard deviation of the change point
estimator is 2.39. Moreover, the performance of the
estimator improves significantly with increases in
magnitude of the drift rate parameter  b  . Figure III
also indicates this issue. As perceived from this figure,
with increasing b , the expected number of samples
taken until the signal and average change point
estimator are becoming near to 51 and 50,
respectively. Furthermore, the standard deviation of
the change point estimator, that is shown shady, is
becoming smaller.
Table II shows that the performance of the MLE
estimator improves significantly with increases in
magnitude of the shift in the slope, b .
Table II. Estimated precision performances over a range of
 i with 10,000 simulations runs when   50
b
0.003
0.005
0.007
0.01
0.015
0.02
0.025
p̂  ˆ    0
0.21
0.30
0.36
0.46
0.57
0.72
0.83
p̂  ˆ    1
0.47
0.59
0.70
0.81
0.85
0.97
0.99
p̂  ˆ    2
0.64
0.81
0.89
0.94
0.99
1.00
1.00
p̂  ˆ    3
0.79
0.92
0.98
1.00
1.00
p̂  ˆ    4
0.90
0.98
1.00
p̂  ˆ    5
0.97
1.00
p̂  ˆ    6
1.00
6
[10] Mahmoud, M. A., Parker, P. A., Woodall, W. H.,
Hawkins, D. M. (2007) A change point method for linear
profile data. Quality and Reliability Engineering
International, 23(2), 247-268.
5- Conclusions
In this paper, we provided the MLE method to estimate
the change point in phase II monitoring of logistic
regression profiles, when the type of change is drift. The
results of this study showed that the performance of the
proposed estimator is fine to identify the real time of
change point under different magnitude of drift rate.
[11] Kim, K., Mahmoud, M. A., Woodall, W. H. (2003). On
the monitoring of linear profiles. Journal of Quality
Technology, 35(3), 317-328.
Developing this method to the other distributions of the
exponential family such as Poisson and Gama would be
future researches in this area. Furthermore the other types
of the change, including step changes and isotonic
changes could be investigated by researchers.
[12] Zou C, Zhang Y, Wang Z. Control chart based on
change point model for monitoring linear profiles. IIE
Transactions 2006; 38(12), 1093-1103.
References
[14] Yeh, B., Huwang, L., Li, Y. M. (2009). Profile
monitoring for a binary response. IIE Transaction, 41(11),
931-941.
[13] Kazemzadeh, R. B., Noorossana, R., Amiri, A. (2008).
Phase I monitoring of polynomial profiles. Communications
in Statistics - Theory and Methods, 37(10), 1671-1686.
[1] Pignatiello, J. J., Samuel, T. R. (1998a). Identifying the
time of a step change with X control charts. Journal of
Quality Engineering, 10(3), 521-527.
[15] Shang, Y., Tsung, F. and Zou, C. (2011). Phase II
profile monitoring with binary data and random predictors.
Journal of Quality Technology, 43(3), 196-208.
[2] Pignatiello, J. J., Samuel, T. R. (1998b). Identifying the
time of a step change in a normal process variance. Journal
of Quality Engineering, 10(3), 529-538.
[16] Sharafi, A., Aminnayeri, M. and Amiri, A. (2012).
Identifying the time of step change in binary profiles. To
appear in the International Journal of Advanced
Manufacturing Technology, DOI: 10.1007/s00170-0123899-4.
[3] Pignatiello, J. J., Samuel, T. R. (1998c). Identifying the
time of a step change in a poisson rate parameter. Journal
of Quality Engineering, 10(3), 673-681.
[4] Pignatiello, J. J., Samuel, T. R. (2001). Estimation of the
change point of a normal process mean in SPC
applications. Journal of Quality Technology, 33(1), 82-95.
[17] Albert, A., Anderson, J. A. (1984). On the existence of
maximum likelihood estimates in logistic regression models.
Biometrika, 71(1), 1-10.
[5] Perry, M. B., Pignatiello, J. J. (2005). Estimation of the
change point of a poisson rate parameter with a linear
trend disturbance. Quality and Reliability Engineering
International, 22(4), 371-384.
[18] McCullagh, P., Nelder, J. A. (1989). Generalized
linear models, 1nd ed. London, UK: Chapman & Hall.
[6] Perry, M. B., Pignatiello, J. J. (2006). Estimation of the
change point of a normal process mean with a linear trend
disturbance. Quality Technology and Quantitative
Management, 3(3), 101-115.
[7] Kang, L., Albin, S. L. (2000). Online monitoring when
the process yields a linear profile. Journal of Quality
Technology, 32(4), 418-426.
[8] Mestek, O., Pavlik, J., Suchnek, M. (1994). Multivariate
control charts: Control charts for calibration curves.
Freseniusí Journal of Analytical Chemistry, 350, 344-351.
[9] Mahmoud, M. A., Woodall, W. H. (2004). Phase I
analysis of linear profiles with calibration applications.
Technometrics, 46(4), 380-391.
7