Improvements to the Nonintrusive Load Monitor
by
Christopher Donovan Salthouse
Submitted to the Department of Electrical Engineering and Computer
Science
in partial fulfillment of the requirements for the degree of
Master of Engineering in Electrical Engineering and Computer Science
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
May 2000
@ Massachusetts Institute of Technolog
.
T
2000. All rights reserved.
Author.
Department of Electrical Engineering and Computer Science
May 21, 2000
C ertified by .,-
.......
..............
.........................
Steven B. Leeb
Associate Professor
Thesis Supervisor
Certified by..........
...................................
Steven R. Shaw
Post-Doctoral Associate
Supervisor
Accepted by .
Arthur C. Smith
Chairman, Department Committee on Graduate Theses
A55IVIASAHS§IN5S3TITUTE
OF TECHNOLOGY
ENO
JUL 2 72000
LIBRARIES
_
Improvements to the Nonintrusive Load Monitor
by
Christopher Donovan Salthouse
Submitted to the Department of Electrical Engineering and Computer Science
on May 21, 2000, in partial fulfillment of the
requirements for the degree of
Master of Engineering in Electrical Engineering and Computer Science
Abstract
This thesis is a number of improvements to the Nonintrusive Load Monitor. First a
preprocessor, the NILM stage that emphasizes the features of power data useful for
load classification, was implemented in software. Then the effect of voltage phase on
load transients is examined as a means to improve the NILM training. The porting
of the system to a PC/104 system increased the installation flexibility. Finally, a
toolbox of utilities for reporting data in terms of the behavior of physical loads within
a monitored building is presented.
Thesis Supervisor: Steven B. Leeb
Title: Associate Professor
Thesis Supervisor: Steven R. Shaw
Title: Post-Doctoral Associate
2
Acknowledgments
I would like to thank Prof. Steven B. Leeb, Prof. Les Norford, and Dr. Steven Shaw
for their great help throughout this project.
3
Contents
1
Introduction
8
..........................
NILM Implementation ......
1.2
Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
1.2.1
Preprocessor.
. . . . . . . . . . . . . . . . . . . . . . . . . . .
10
1.2.2
Transient Variation . . . . . . . . . . . . . . . . . . . . . . . .
11
1.2.3
Miniature NILM
. . . . . . . . . . . . . . . . . . . . . . . . .
11
1.2.4
R eporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
2 Preprocessor
2.1
Downsampling.
12
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
. . . . . . . . . . . . . . . . . . . . . . . .
14
2.1.1
Low Pass Filtering
2.1.2
Parameter Estimation
2.1.3
Downsampling
. . . . . . . . . . . . . . . . . . . . . .
14
. . . . . . . . . . . . . . . . . . . . . . . . . .
15
2.2
Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . .
16
2.3
Setting the Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
2.4
Combining Data From Multiple Reads
. . . . . . . . . . . . . . . . .
19
2.5
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
3 Phase Variation
3.1
4
8
1.1
21
The Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
3.1.1
28
C lustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Miniature NILM
32
4
4.1
Components ................
32
4.1.1
Parvus Complete Scalable CPU II
33
4.1.2
CardPC . . . . . . . . . . . . . .
34
4.1.3
DiskOnChip . . . . . . . . . . . .
34
4.1.4
Real Time Devices DM6420 A/D
35
4.2
Assembly
. . . . . . . . . . . . . . . . .
36
4.3
R esults . . . . . . . . . . . . . . . . . . .
37
5 Postprocessing
6
38
5.1
On and Off Matching . . . . . . . . . . .
39
5.2
See . . . . . . . . . . . . . . . . . . . . .
41
5.3
Tee2 . . . . . . . . . . . . . . . . . . . .
41
5.4
T im er . . . . . . . . . . . . . . . . . . .
41
5.5
Prune ...
42
5.6
Chansift . . . . . . . . . . . . . . . . . .
.............
......
42
Conclusion
43
6.1
Preprocessor . . .
. . . . .
43
6.2
Training . . . . .
. . . . .
43
6.3
Reporting
. . . .
. . . . .
44
6.4
Miniature NILM
. . . . .
45
A Prep.c for PCL818
49
B DM6420.c
65
C smartlog.c
84
D gSmartlog.c
91
E see
100
F tee2
102
5
G timer
104
H prune
105
I
106
Chansift
107
J Miniature NILM Parts
6
List of Figures
1-1
Overview of NILM System . . . . . . . . . . . . . . . . . . . . . . . .
9
2-1
Preprocessor Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . .
13
2-2
Downsampling of Data . . . . . . . . . . . . . . . . . . . . . . . . . .
16
3-1
an incandescent light bulb turned on at different voltage phases
. . .
22
3-2
a computer turned on at different voltage phases . . . . . . . . . . . .
23
3-3
a rapid start fluorescent light turned on at different phases . . . . . .
23
3-4
an instant start light turned on at different phases . . . . . . . . . . .
24
3-5
an incandescent light bulb turned on at different voltage phases
. . .
24
3-6
a computer turned on at different voltage phases . . . . . . . . . . . .
25
3-7
a rapid start fluorescent light turned on at different phases . . . . . .
25
3-8
an instant start light turned on at different phases . . . . . . . . . . .
26
3-9
inductive m otor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
3-10 an instant start light turned on at different phases . . . . . . . . . . .
27
3-11 Clusters created with k-means plotted over the entire data . . . . . .
29
3-12 Demonstration of the limitations of k-means . . . . . . . . . . . . . .
29
3-13 The result of modifications to k-means . . . . . . . . . . . . . . . . .
30
3-14 Clusters created with modified k-means plotted over the entire data .
31
7
Chapter 1
Introduction
A nonintrusive load monitor (NILM) is a device that identifies the operating schedule
of individual loads within a building by monitoring the power entry [14, 13, 11, 19,
1, 12]. The use of a computer to disaggregate the loads within a building creates
an alternative to the standard load monitoring technique of instrumenting each load
of interest. In addition to the original goal of easing the burden of collecting data
associated with the operating schedule of electrical loads, the NILM has also found
other applications where knowledge of load activity is useful. It could be used with
a security system to signal an alarm if the lights turn on or the garage door opener
is operated [9]. Similarly, the NILM can be used as a verification tool for a building
management system.
1.1
NILM Implementation
Broadly, a nonintrusive monitors works by identifying times when the power usage of
a building changes, and then attributing that change to the operation of a load within
the building. One basis for building a nonintrusive monitor is to examine changes in
the steady state real and reactive power [9]. This approach has been modified by a
number of researchers to include an additional variable to model harmonic content
[21, 3, 7, 18]. This thesis presents a number of enhancements to the NILM, a classifier
based on transient behavior [14, 13, 11, 19, 1, 12.
8
diag
Diagnostic Stream
diagnostic output
xNELM
,
_
V(t)
Preprocessor
Graphics Stream
NILM
--
--
w3nilm
web output
I(t) --
Text Stream
Load behavior log
smartlog
Figure 1-1: Overview of NILM System
This NILM system is modular so that it can be configured for a variety of applications [19]. Figure 1-1 shows its components. Raw current and voltage measurements
are digitized by an analog to digital card (A/D). The preprocessor converts this data
into short time estimates of the spectral content, spectral envelopes, which are pattern matched by the NILM [1, 11, 12] . It generates a variety of outputs that can
either be displayed or further processed by additional utilities.
The current and voltage at the power entry of a building are first converted to
signal level currents for measurement. These current are digitized at the NILM computer by a data acquisition card. This thesis includes the development of a driver for
a data acquisition card, the DM6420. Data from either the DM6420 or the PCL818,
another A/D card, is then processed by a preprocessor. The preprocessor develops
estimates of the harmonic content ak(t) and bk(t) for the interval from 0 to T,
-
x(t - T + s) cos (k
bk(t) = -
x(t - T + s) sin (k
ak(t) =
T lo
T[
[13].
9
2(t
T
T
- T + s))ds
(1.1)
(t - T + s))ds
(1.2)
For steady state values of current and voltage, a1 is proportional to real power
and b1 proportional to reactive power. When the power is changing, as in the case
of the transients used for NILM classification, the curves no longer correspond to the
well defined quantities of real and reactive power, but they continue to capture many
of the key features of AC loads.
The classification data from NILM is sent to a variety of programs as data
streams [19]. These streams are available in three formats: text, graphics, and diagnostics. The text stream is a plain text stream that contains the name, time, scale,
and error of each event. The graphics stream contains the same information as well
as the preprocessed data used for classification. The diagnostic stream provides unprocessed data for detailed analysis by other programs. A variety of programs have
been developed that use these streams to answer questions about the physical devices
within buildings.
1.2
Contributions
This thesis describes contributions to the NILM system. A preprocessor has been
implemented in software on the NILM processing computer. Laboratory studies of
the repeatability of transients have been made. This data was used to improve the
choice of exemplars for classification. Then the system was ported from a personal
computer to a PC/104 class computer. Finally, a tool box of utilities was developed
for interpreting event data in terms of the activity of physical loads in the monitored
building.
1.2.1
Preprocessor
There are a variety of ways for a preprocessor to create the spectral envelope estimates
described by Equations 1.1 1.2. Early preprocessors used an analog phased lock loop
to synchronize to the voltage waveform. A multiplying digital to analog converter then
multiplied the current by sine waves synchronized with the voltage [13]. To increase
flexibility and reduce cost, the preprocessor was later implemented on a digital signal
10
processor(DSP) using a similar algorithm [20]. The preprocessor is now implemented
in software on the system that performs the NILM classification. The algorithm was
also changed from the DSP version to use the Fast Fourier Transform to minimize
the computational cost of preprocessing.
1.2.2
Transient Variation
The NILM works by matching the current transient when a load turns on with a stored
transient. This technique can be optimized by matching only those components of
a transient that are repeatable across variation in the voltage phase at transition.
An algorithm for finding these portions of transients was developed based on data
collected for a variety of loads as they were powered on at various points in the voltage
cycle.
1.2.3
Miniature NILM
NILM has been developed on personal computers. This offers a great deal of flexibility
in development, but it limits where NILM can be installed because of ruggedness,
size, and ventilation issues. These are overcome by porting the system to a PC/104
computer.
1.2.4
Reporting
The NILM generates a list of events while the user is often interested in the operation
of one or more physical loads in the building. This gap is bridged by a variety of
utilities that generate the reports for a user.
11
Chapter 2
Preprocessor
The first step of classifying event transients is preprocessing, the extraction of important features from the raw data. Because the voltage provided by the utility is
periodic, the current can be meaningfully decomposed into harmonics of that period.
This creates two data streams, a and b, for each of the k harmonics of current, x,
during time 0 to T as define by:
ak(t)
= 2
bk(t) =
T 0
j
T fo
x(t -
T + s) cos (k 2(t - T + s))ds
T
(2.1)
x(t -
T + s) sin (k 2(t - T + s))ds
T
(2.2)
[13]. For a periodic signal, these values are well defined: a1 is real power drawn by
the load; b1 is the reactive power.
In previous versions of the NILM preprocessor, the input current was decomposed
into these spectral envelope estimates by multiplying the current by sine waves [13,
20].
The new preprocessor is software (prep.c) running on the NILM computer.
It uses the Fast Fourier Transform(FFT), a more efficient algorithm for computing
harmonics on a computer [15]. In order to create a preprocessor using an FFT, the
data must be processed before and after the FFT, Figure 2. The FFT decomposes
its input into harmonics of the window length. Those harmonics are also harmonics
of the voltage frequency if the window length equals the voltage period. This con-
12
S Downsampling
v(t)
LPF
i(t)
LPF
91
--
>
FFT
im
Phase
T
Period
Measurement
:
Measure
Downsampling
FFT
Rotate
Spectral
Estimates
Figure 2-1: Preprocessor Flow Chart
dition is met by resampling the current data. The output of the FFT has both real
and imaginary components for each harmonic. These correspond to the a and b of
Equations 2.1 and 2.2 if the input window has no phase offset from the voltage, i.e.
the corresponding voltage window begins with a positive zero crossing. The window
position is not set in this algorithm, so phase information must be reintroduced into
the FFT output. In the harmonic domain, phase is a rotation of the complex vector
associated with each harmonic. So, phase is reintroduced by rotating each harmonic
by an angle calculated from the voltage data.
2.1
Downsampling
The FFT is performed on 128 points that contain exactly one voltage cycle. The
length was set to 128 points because the FFT requires fewer operations for data sets
with lengths that are a power of two than for other similar lengths. The data must
contain one voltage cycle in order for the harmonics to correspond to ak(t) and bk(t).
Theoretically, these conditions could be met by setting the sampling rate of the A/D
to 128 times the line frequency, but the A/D cards used for the NILM set the sampling
rate by dividing down a high frequency clock. This limits the available sampling rates.
So a sampling rate of 128 samples per line cycle is created by downsampling. The A/D
is set to sample at 7936 samples per second, as close to the desired 7680 samples per
second as possible. Then the data is low pass filtered to ensure that the bandwidth is
below the downsampled level. The downsampling ratio is calculated by measuring the
13
period of the voltage waveform. Finally, current and voltage data are downsampled.
2.1.1
Low Pass Filtering
A low pass filter is included to prevent aliasing from the downsampling. An FIR filter
is used because its linear phase, corresponding to a time delay, does not change the
time relationship between the harmonics. The filter coefficients were calculated using
the firl command in Matlab and hard coded into the preprocessor in lines 19 through
34 of prep.c. The filter is applied directly to the sequence in prep-filter.
2.1.2
Parameter Estimation
The number of samples per line cycle must be measured in order to set the downsampling ratio. The basic algorithm is to count the number of samples in an integer
number of cycles and divide by the cycle count. Two techniques are used to improve the accuracy of this approach. First, the cycles are counted starting at average
crossings because the derivative is the greatest at these points. Second, a linear fit is
performed at these zero crossings to average out noise. The algorithm for calculating
the downsampling ratio for a block of voltage data as implemented by prep-update
in prep.c, Appendix A, is:
1. Calculate a rough average by averaging the first 20 seconds of data.
2. Estimate the period by finding the positive average crossings.
3. Recalculate the average by averaging the data over an integer number of periods.
4. Recompute the period based on the improved average.
5. Calculate the resampling ratio based on the positive zero crossings.
Since the period is changing very slowly, the estimate of the period can be improved by looking at many measurements taken over a long period of time. An
estimate of the downsampling ratio is made for a block of data as described in the
14
previous paragraph. The period estimate is then combined with measurements from
earlier blocks of voltage data using a recursive least-squares algorithm. The general
form of this method is described citemath by
k
+ Pk~ksk
k-1
Ek
~tk
Pk
=
-
(2.3)
k Ok-1
(Pk1
Pk-1#kqPk_1
--
k-1
)
The parameter estimateOk, is adjusted by the product of the uncertainty associated with the estimate,Pk, and the error, Ek, projected by the regressor,
is the current measurement,
#O'k_1.The uncertainty
'Yk
#k.
The error
minus the value predicted by the parameter estimate,
is adjusted by a factor, A, that adjusts the relative weighting
of measurements.
Since the regression is only applied to one variable, the downsampling ratio, Equation 2.3 is simplifed to
0
k=
k
Pk
k-1 + PkEk
~k
(2.4)
~- k-1
1
A
- -(Pk1
-
P2
-i)
A + Pk_1
This is implemented at the end of prep-find_dsratio in prep.c, Appendix A.
2.1.3
Downsampling
The downsampling is done by linear interpolation. An original index is calculated for
each downsampled index by multiplying by the downsampling ratio,
Vresampled[XI = Voriginai[x *
15
dsratio].
(2.5)
I
I
I I
Iine~
I
line
x
0.9
x
6.8
-e
x
-
.
0.6
a)
xX
6.5 F-
xx
6.4
0.3
xeX
6.2
0.1
0
2
4
6
8
10
12
14
16
1
Time
Figure 2-2: Downsampling of Data
The value at that index is calculated by a linear interpolation between the samples
on either side of the desired value, i.e.
V[x] = V[ floor(x)] + (x - floor(x)) * (V[ceil(x)] - V[ floor(x)])
(2.6)
This operation is performed by prep-downsample in prep.c, Appendix A.
2.2
Fast Fourier Transform
A Fast Fourier Transform (FFT) is applied to the data to decompose it into sine
and cosine waves. The FFTW library developed by M.I.T.'s Laboratory of Computer
Science (LCS) was chosen because of its level of support and platform flexibility. An
impressive web site contains an online manual as well current versions and release
16
20
notes
[6].
The supported operating systems include a variety of UNIX systems
for personal computers, Windows, MacOS, and others. It is called from prepft,
Appendix A.
2.3
Setting the Phase
The output of the FFT has real and imaginary components for each harmonic, but
these only correspond to a and b if the block of data that the FFT is applied to is
properly aligned. One way to determine a and b from the output of the FFT would be
to perform this alignment in the time domain. Each block of data would have to be
chosen so that it begins at a maximum of the voltage sine wave. But, this placement
is necessarily based on only a few points near the beginning of the window.
A better method is to apply the FFT to a block of data with arbitrary phase
and then to apply a transformation to the real and imaginary components of each
harmonic to determine ak(t) and bk(t). This transform can be calculated by noting
that the real and imaginary output of the FFT for each harmonic form a complex
number that specifies a magnitude and phase for each sine wave. The phase of the
FFT output for the current can be corrected by subtracting the phase of the voltage
FFT output.
The algorithm can be presented more formally by finding an angle, a, that the first
harmonic of the voltage waveform needs to be rotated by to make it completely real.
If the first harmonic of the current waveform is rotated by a, the real component of
the first harmonic will become a 1 (t) and the imaginary component will become bi (t).
In a given amount of time, the second harmonic will cover twice the angle. Thus, it
must be rotated by 2a.
A rotation matrix can be written in terms of a,
M
cos a
- sin a
sin a
cos a
17
.
(2.7)
The inverse of this matrix is
M- 1 =
cos a
sin a
-sina
cosa
I
(2.8)
Given the constraint that the rotation matrix should rotate the first harmonic of
voltage so that it is completely real, the rotation matrix can be written in terms of
PV, the real part of the first harmonic of voltage, and qv, the imaginary part of the
first harmonic of voltage:
I
M
Multiplying both sides by M- 1 ,
[
PV
/p v+q
(2.9)
0
1
I"
=M
.
(2.10)
I
(2.11)
oJ
Substituting in the value of M- 1 ,
[
V1
po +q2J
cos a
-sin a
Substituting Equation 2.11 into Equation 2.7,
PV
qv
V p+qV2
qv
VFp-2v +qv2
Nfp 2V+qV2
PV
V -2 +q 2
v
(2.12)
The second harmonic is rotated by the matrix M 2 , the third harmonic by M 3 ,
etc.
18
1. Normalize Real(V) and Imaginary(V) so that Magnitude(V) = 1.
Real (Vnormalized)
Imaginary (Vnormalized) =
Real(V)
j(Real(V))2 + (Imaginary(V)) 2
(2.13)
Imaginary
(V)
mgnayV
2
I(Real(V)) + (Imaginary(V))2
(2.14)
2. Recursively calculate the rotation matrix, Mk(t) by repeatedly calculating
Real(changing)
Real(V) * Real(changing) - Imag(V) * Imag(changing)
(2.15)
Imag(changing) = Real(V) * Real(changing) - Imag(V) * Imag(changing)
(2.16)
3. ak(t) and bk(t) are calculated using the same equations as above.
4. Steps 2 and 3 are repeated for each harmonic required.
2.4
Combining Data From Multiple Reads
The preprocessor operates on one block of data at a time. A modified overlap and save
method is used to combine data from multiple reads. The overlap and save method
saves N-1 points, where N is the number of points required by the system to generate
a single output point, from the end of one data block for inclusion at the beginning
of the next data block [15]. A modification is required because the preprocessor does
not process all of the data block it operates on. More data is read from the A/D
card than will be used, because there is some uncertainty before the downsampling
process about the number of points that will be used. So the save block must include
all the data which was not used. The algorithm used in prep.c is
1. Save raw data from the end of the current and voltage streams prior to processing (prep-save).
19
2. After processing, the index of the last data point used to create create the
output data is stored (prep-load).
3. The index is converted to an index in the raw data by:
OriginalIndex=
ProcessedIndex
DownSamplingRatio
filterlength
2
(2.17)
4. The saved data following the original index is copied to the beginning of the
buffer for processing in the next function call.
2.5
Results
This software preprocessor has been tested both in the lab and in the field. It has been
installed on both personal computers and the miniature systems described in Chapter
4. The graphs have similar shapes to those generated with previous preprocessors.
The suitability of this preprocessor for load classification is demonstrated in Chapter
3 where the phase variation of loads is examined.
20
Chapter 3
Phase Variation
It has been argued that the nonintrusive classification process should be based on
steady state data [9]. This argument is based on three observations: sampling rates
must be higher to capture transients, "off" events do not generally have transients,
and steady state signatures are additive. It also argues that transients vary depending
on the "exact point in the voltage cycle at which the switch opens or closes." These
arguments have been quoted in other papers on nonintrusive monitors [17, 18].
However, a NILM has been developed that successfully uses transient information
[11, 13]. Higher sampling rates and initial processing were demonstrated to be feasible
by building a NILM with custom hardware and a DSP. This system could discriminate
between a variety of loads that draw similar power levels, a problem for the steady
state system. The problem of overlapping loads, solved in steady state systems by
the additive nature of loads, was resolved by demonstrating the ability of the NILM
to recognize transients on top of near constant power draw from other loads.
The NILM requires that features from the power transient be chosen for matching
in the classification process
[11, 19]. An investigation of transient variation with
voltage phase can lead to an improved algorithm for developing these exemplars.
21
16000
14000 -
12000 10000 -
2
8000 -
6000 4000 -
2000 -
0
-2000
0
5
I
10
I
15
I
20
I
25
I
30
35
time
Figure 3-1: an incandescent light bulb turned on at different voltage phases
3.1
The Experiment
A computer controlled, phase programmable switch was used to turn loads on and off
while a NILM preprocessor recorded the current transients. The switch was controlled
by a c program, phase2.c, that closed it at different points in the voltage cycle. A
variety of loads were connected to the switch, and the preprocessed data was stored
in data files.
This data was analyzed in Octave, a clone of Matlab. Analysis starts with a call
to dissect.m. This script scans the data file for adjacent points that differ by more
than a threshold value. It stores the set of points surrounding that activity into an
output matrix. The classification algorithm used in the NILM adds a step to this
process. It can slide the matching pattern a few points in time to make a better match
[19]. This same quality of alignment can be achieved with dissect.m by adjusting the
threshold value. The success of this approach can be seen in Figures 3-1 3-2 3-3 3-4.
Figures 3-1 3-2 3-3 3-4 show the composite of a number of tests of four loads.
These figures demonstrate that many of the features in transients remain unchanged
22
I
12000
10000
-
8000
-
I
I
I
I
I
I
25
30
35
40
- 6000 -
4000
2000
0
-2000
0
10
5
15
20
45
time
Figure 3-2: a computer turned on at different voltage phases
1600
1400
I-
1200 1000
800
600
400
200
0
-200
0
50
I
I
I
100
150
200
250
time
Figure 3-3: a rapid start fluorescent light turned on at different phases
23
7000
6000 -
-
5000
4000 -
3000 -
2000
F-
1000
0
-1000
0
10
20
30
40
50
60
70
80
90
time
Figure 3-4: an instant start light turned on at different phases
16000
14000 -
12000 -
Ii
ii
10000 8000 6000 1*
4000
$ 0*
ij * *
++ + +
+ +
2000
0
-2000
- +
0
+
+
+
+
+
+
+
+$ +
I
I
I
5
10
15
I
I
I
20
25
30
- --
--
--
----
-
35
time
Figure 3-5: an incandescent light bulb turned on at different voltage phases
24
12000
II
II4
10000 -
+
+
+
+
8000 -
,6000
-
4000 -
I
2000
+
~*4~~4++44444444
0
-+
+
4 + +
-2000
0
+ ..
.
+
I
I
I
I
I
I
I
5
10
15
20
25
30
35
40
45
time
Figure 3-6: a computer turned on at different voltage phases
1600
1400
1200 -
1000 -
9
+
800 +
600
-
4
400 +
++
200 -
0
-200
0
50
150
100
200
250
time
Figure 3-7: a rapid start fluorescent light turned on at different phases
25
7000 700
I
I
I
I
I
I
I
6000 -
5000 -
4000 -
.
3000 -
2000 -
1000 -
+
+
0 *
0
+++++++++*
-1000 1
0
10
I
I
I
I
I
I
20
30
40
50
60
70
80
90
time
Figure 3-8: an instant start light turned on at different phases
with voltage phase variation. By looking at the individual points in these graphs, as
shown in figures Figures 3-5 3-6 3-7 3-8, a pattern in the variation is clear. Those
times at which the greatest variation between trials occur are the same points where
the greatest variation within a given trial occurs.
This leads directly to an algorithm for determining which points should be used for
classification of loads. Areas where there is little change in the values of the spectral
envelope estimates are of no use in identification and may make it difficult to detect
two events that occur in close temporal proximity. Areas where a great deal of change
is occurring are also a concern because they can lead to a great deal of variability
between trials. The portion of the algorithm that removes areas of high change was
written into a matlab script called findbad.m. This was used to circle the points
that should be removed in Figure 3-9. This code was then added to vsection, the
program that creates the exemplars.
26
4500
!i
4000 3500 -
3000
S
-
I
2500
I
2000
I
C1500
S
S
1000
I
I; ~
I
500
++..+++++.
0
-500
0
40
30
20
10
70
60
50
time
Figure 3-9: inductive motor
2500
-
2000
-
.
1500
1000
500
0
-500
0
10
20
30
40
50
60
70
80
90
time
Figure 3-10: an instant start light turned on at different phases
27
3.1.1
Clustering
Figure 3-10 shows an example of a transient with fewer repeatable features. There is
much more variation in this set of curves than in the others shown, but there are still
repeatable features. A few groups of curves will similar shapes can chosen. These
groups tend to peak, creep down, plateau, and then reach a steady state value. The
difficulty in creating exemplars for this data is that the height and length of the
plateau varies. This can be resolved by creating groups of transients for this load
that are similar and creating exemplars for a few of those groups.
The groups could be created by hand, but that would be a tedious process. A
clustering algorithm automates this process. It also averages the curves within each
group to create a representative curve. The first attempt was to use the k-means
algorithm. Each trial consists of 1000 points on each of 8 channels. For clustering
this is seen viewed as one point in 8000 space. K-means begins with a choice of the
number of clusters the data will be placed into. For this application, the choice is
made by looking at the plot of all the data and counting the observed clusters. A few
clusters are added to this count to account for hidden clusters and clusters which will
combine in the process. The operation of k-means then iterates between calculating
the center of each cluster by averaging all the points in that cluster and reassigning
points to the cluster with the nearest center [10].
The octave script clustermation.m is an implementation of this algorithm that
plots the graphs of each of the clusters at each iteration to help with visualization of
the process. Figure 3-11 shows the results of running clustermation on the compact
fluorescent data plotted in Figure 3-10. The one concern is that a cluster is not being
created for one of the curves that is evident when the data is viewed. The first curve
in the data set is the curve that peeks and then immediately descends to its final
value. While this is not representative of a large number of trials in our experiment,
it would be useful to train the system to identify such events in the future.
The reason that curve is not represented in the clusters can be seen in a two
dimensional example. Figure 3-12 shows a two dimensional system where the outlier
28
2500
2000 -
1500 -
1000
500
0
-500
100
80
60
40
20
0
120
time
Figure 3-11: Clusters created with k-means plotted over the entire data
12
10
8
6
4
2
0
+
+
+
X
+
+
X
X4
W
)K
x
x
333
X3
X
X
x
x
X
X
0
0
0
-2
-8
-6
-4
-2
0
2
4
6
8
x
Figure 3-12: Demonstration of the limitations of k-means
29
12
10 -
8
6
4
2
x
x
x
x
x
x
+
+
+
+
+
+
0
+
+
+
0
0
X
-
x
X
X
X
+
-2
-8
I
I
-6
-4
I
-2
0
I
2
4
6
8
X
Figure 3-13: The result of modifications to k-means
is poorly represented by the clusters. The outlying point does little to pull the center
towards it since it is competing with the more numerous points in the central group.
This mechanism explains the behavior seen when working with the actual data.
Two modifications could be made to k-means to solve this problem. A revised
technique for determining the center of a cluster would solve the problem while preserving many of the benefits of k-means.
This system would still fall within the
general framework of k-means and it allows for convergence of the system in the
sense that k-means converges by reaching a state where no changes are warranted.
The disadvantage of this solution is the difficulty of choosing an algorithm for placing
this center. The center is traditionally found by taking the mean of the points in the
cluster, thus the name of the technique. This is done because it gives the least square
error. If the distance to each point were raised to a higher power, the cost of a long
distance would be higher. Unfortunately, this forces a more complicated formula for
placing the center.
Instead a technique of removing one of the clusters that is most similar to another cluster and placing the center of that cluster at the data point furthest from
30
2500
2000
1500
-z
1000
0
00 1
0
I
20
40
1
so
80
100
i20
time
Figure 3-14: Clusters created with modified k-means plotted over the entire data
any cluster center was used. This is a simple algorithm to implement, as done in
cpmation.m. The result of this algorithm can be seen on the two dimensional case
in figure 3-13 and on actual data in figure 3-14.
31
Chapter 4
Miniature NILM
The Nonintrusive Load Monitor has been developed on personal computers because
they are cheap, flexible, and offer a mature development environment. Unfortunately,
PCs are unsuitable for installation at many locations because they are large, fragile,
and require fan cooling. The PC/104 architecture, a standard for embedded computer
systems based on personal computers, solves these problems while requiring only
minor changes to the NILM distribution.
4.1
Components
A number of other groups have developed small nonintrusive monitor installation
packages [17, 8, 2]. But these systems send partially processed data to a PC for further
analysis. One of the advantages of NILM is that the information is available minutes
after the event occurs [19]. This places the burden of doing all of the computations
on the installed system and requires a method of broadcasting the analyzed data.
Th creates a substantial list of requirements for the system: computing power to
handle the analysis; storage to hold data, exemplars, and programs; analog inputs for
current and voltage sensors; and an Ethernet network interface for broadcasting data.
These could be met with a variety of systems, but Parvus's Complete Scalable CPU
II (CSCII) offers all of these features as well as a great deal of compatibility with the
PC system [16]. The system runs the same software being used on the PC system.
32
It utilized the PC/104 standard for its bus, allowing for the addition of peripherals
such as an A/D card. The system also supports two useful standards for storage: an
IDE port allows connection of conventional hard drives and a DiskOnChip connector
supports M-Systems's flash RAM technology for solid state drives. In addition to
the immediate gain of being able to quickly move over the existing NILM software
to the new platform, the system prevents a forking of the development tree, allowing
improvements to the desktop development system to become immediately available
to the embedded system.
4.1.1
Parvus Complete Scalable CPU II
The Complete Scalable CPU II(CSCII) is a motherboard that connects a variety of
embedded technologies. This project uses: the Embedded All-in-one System Interface
(EASI) to connect a Pentium 233 processor; a PC/104 Bus to connect an A/D card;
10 Base T Ethernet to connect the system to the internet; a 5 Volt power connection
to power the computer; and standard PC connectors for programming and debugging
the system. Other features that we do not use include SVGA and Alphanumeric LCD
connectors and analog and digital inputs [16].
The peripherals that were used in this project all worked with standard drivers
in Linux. The Ethernet chip is an NE2000 compatible. The VGA hardware worked
with standard VGA drivers. The only issue was connectors. All of the connectors
on the CSCII are IDC connectors. The manual provides pinouts of these connectors
which can be used to construct cables to convert to industry standard connectors. A
complete set of such cables is available from Parvus and is highly recommended as
a starting point because they allow for simple connection of the CSCII to standard
peripherals.
Two connectors had to be made in order to use the CSCII. A power connector
provides the 5 Volts needed to power the CSCII and other PC/104 boards. The 5
volts was also provided in a custom IDE connector. Although the CSCII provides a
standard 2.5" IDE connector, it does not provide enough power for many 2.5" IDE
hard drives. The Parvus supplied cables were then used for all of the other cables.
33
4.1.2
CardPC
CardPCs are processors with attached memory that conform to the Embedded Allin-one System Interface as specified by the Card-Sized PC Standardization Group.
Epson and Cell Computing are the two primary manufacturers of these CPU cartridges [16].
For this project they are simply inserted into the CSCII where they
function as expected.
4.1.3
DiskOnChip
The DiskOnChip is the most difficult component to setup. There are a two problems. First is the space limitation. DOCs are quite limited in capacity, the largest is
currently 144 Megs, with the price decreasing dramatically as one moves to smaller
capacities. Our installation is Linux based, but the need to select components for a
minimal installation would be similar with most operating systems. M-Systems has
a variety of documents and tools that help with this process. A process of starting
with the minimal system recommended by M-Systems and adding on components as
required for our functionality was used. This system was mostly successful, although
authentication does not work as intended on the system.
Second, is the difficulty of setting the DOC up to run with Linux
[4]. These
steps are detailed in an application note available from M-Systems, but a few of the
difficult steps should be pointed out. The configuration of the DiskOnChip can be
done a PC using a DOC socket that plugs into the ISA bus. The computer must be
booted to DOS initially so that a utility may be used to switch the DOC out of boot
mode so that Linux can boot. The computer is then rebooted to Linux and the kernel
is patched to support the DOC. When Linux is rebooted with this patched kernel,
the DOC can be mounted as an additional drive and files can be copied over. The
computer can then be rebooted off of the DOC after the DOS utility is used to set
the DOC back into boot mode. This process took a week the first time. Even with
that experience, the second time still took close to ten hours.
The installation process detailed in the M-Systems documentation results in a
34
system that boots off of the DOC with a minimum amount of functionality, leading
to the lengthy process of adding packages and deleting components. At a minimum
the system had to hold the preprocessor, NILM, and a web based output program.
These required that it also include PERL and Octave. For remote administration,
SSH was also included. Initially, a c compiler and emacs were also considered for
inclusion, but they were deemed to large. Jove, Jonathan's Own Version of Emacs,
a small emacs style editor was used in place of emacs. A list of all the files installed
on the machine is included as an appendix.
4.1.4
Real Time Devices DM6420 A/D
The Complete Scalable CPU II offers some A/D capabilities, but a separate card was
chosen for a variety of reasons. It allowed for development in a PC, using a PC/104 to
ISA conversion card, separate from any issues associated with the PC/104 computer.
It allowed the selection of a card that could meet higher specifications. And, the Real
Time Devices card has very good documentation and sample code that supported
the development of a Linux driver compared to the meager offerings of the on board
system.
Steven Shaw spent a great deal of time developing excellent drivers for the PCL818
series of A/D cards. The DM6420 was carefully chosen because it had a number of
features that assisted with driver development. The most important lesson learned
from the PCL818 is the importance of finding a card designed for continuous operation, the PCL818 was not and required a great deal of tweaking in order to work
with the NILM. Much of the work on the PCL818 driver was also reused because
of similarities in the operation of the two cards. The third factor was the excellent
documentation for the DM6420 [5].
Dual DMA is the key feature for a card to be useful for continuous operation.
Direct Memory Access is a process where the A/D card can write its data directly
into memory where it can be copied by programs. DMA is supported by A/D cards
in two ways. In single DMA, the card fills up a buffer and then sets a bit signifying
its completion. The program than resets the card and the card restarts collecting
35
data. This is the way the PCL818 operates. It is difficult to do continuous operation
this way because the bit must be reset before a sample is missed. In dual DMA, the
card has two buffers. The card sets a completion bit when it fills up a buffer just as
with a single DMA, but it then continues writing data, now into the second buffer.
The card alternates between the two buffers. The program then copies the data out
of the buffer to which data is not being written.
A decision to reuse much of the PCL818 code was made to reduce development
time and increase compatibility. This was at the cost of the features unique to the
DM6420 such as its more general method for specifying the ways in which channels
are sampled.
4.2
Assembly
Two miniature NILMs were assembled. The first system used a DiskOnChip for
storage; the second system uses a 2.5" hard disk designed for use in a notebook
computer. In the completed systems, each contained power, Ethernet, and sensor
connectors.
The PC/104 specification should have allowed the DM6420 to plug directly into
the CSCII, but the electrical components on each of boards protrude enough to prevent mating of the connectors. This was solved by purchasing an adapter board sold
as a PC/104 Double Height Adapter.
There are a variety of enclosures available for this form factor. Because the devices
run at low power levels there is no need for ventilation and the enclosure is simply
a box.
The Can-tainer, an extruded aluminum box, was chosen because it cost
substantially less than the other prebuilt options. It has rubber supports inside the
box that secure the computer and machinable metal plates on both ends for placement
of connectors.
36
4.3
Results
Preliminary results on the DiskOnChip system were promising. It ran for weeks at a
time serving the web page of NILM results. After a few months it stop responding.
Work on the hard disk system was also problematic. One hard drive did not boot
on the miniature NILM after installation on a personal computer because of large
disk BIOS issues. The replacement drive draws more power than can be provided by
the 2.5" IDE interface, so power is taken directly from the miniature NILM power
connector.
The system rebooted at unexpected times, often during periods of hard disk access.
Powering the miniature NILM from a laboratory power supply, it was determined
that the system requires an input voltage over a hundred millivolts above the 5 Volt
nominal value. It is hypothesized that the additional load of the hard disk may bring
the available voltage down below the reboot level. This problem has not yet been
solved.
37
Chapter 5
Postprocessing
The NILM can provide a variety of data to different users. It can provide valuable
information for load forcasting to utilities. It can also help with there efforts at demand side management, programs to encourage users to use less peak power. It can
also help building managers determine the duty cycle of key components in HVAC
systems [9, 13, 14]. All of this data requires an additional level of processing following the NILM output of event identifications. This layer of processing is called
postprocessing.
Instead of developing specialized programs to report data in formats specific to
various applications, a toolbox of applications have been developed that can be combined on the command line to create a variety of reports. This is a continuation of
the intelligent data flow methodology used throughout NILM development [19]. Each
program operates on data and passes on the minimum amount of data for the next
stage to operate. This saves later stages from having to wade through large amounts
of data. This modular design can be seen in the preprocessor that takes raw data
and passes harmonic data to NILM. The NILM then passes tags with different levels
of information to display programs.
The goal of post processing is to take that display data and reformat it in different
ways. The classification of "off" transients can be improved by matching them to "on"
transients. When a devices turns on, it draws power in a characteristic way as it stores
energy: an incandescent light bulb heats up it filament; a motor accelerates its rotor
38
[11, 13].. This provides the turn on transient with which we identify loads. When a
device turns off, there is only a step change in power as the device no longer consumes
power. Some distinctions can be made based on the magnitude of the step and the
harmonic content. But, there is much less information than with the associated turn
on transient, and the classification is thus more error prone. The accuracy of these
"off" transient classifications can be improved by requiring that an "off" event be
matched to an "on" event of a corresponding magnitude. This is done by smartlog,
which operates on a text stream, and gsmartlog, which operates on graphics tags.
Users might also be interested in being warned if the system believes that more
loads are on than the building contains. An example is a building with three lights
that the system is describing as having four lights on. In this situation it is difficult
for the computer to determine the cause of the error, but it can flag the error so that
the user can examine the stream more carefully.
A collection of smaller utilities allow the piping of data to files. In order to send a
stream of data to multiple programs, tee2(Appendix F) is used to echo its input to
multiple streams. If raw or preprocessed data is stored to a file, the file will quickly
become very large. The routine timer(Appendix G) can be used to save data for a
preset number of seconds. The size of stored files can be further reduced by only saving
every Nth line of data, where N is some integer. This is done by prune(Appendix
H). With preprocessed data, the file size can be minimized by only saving data for
selected channels using chansift (Appendix I).
5.1
On and Off Matching
Each of these post processing utilities increases the utility of NILM data by adding
additional information about the building electrical system. Much of the information that we are adding, such as the number of each class of load in a building or
the importance of duty cycle requires customization for each installation. But, the
statement that a load must first turn on before it can turn off is universal. Even the
slightly stronger statement that is used in this utility, that a load must turn on with
39
a scale factor s before it can turn off with a scale factor s, is true for nearly all loads.
That is why smartlog was the first utility written.
The originial version of smartlog operates on a text stream. Information from
the tag about NILM's classification of the event and its scaling is transformed by
smartlog into a device description and device scaling and then the revised tag is
sent to the output. This allows for the piping of data through this program and on to
other programs that deal with text tags. A later version, gsmartlog, performs the
same operations on graphics tags.
NILM sends out tags every time it classifies an event. These tags fall into two
groups: devices turning on and devices turning off. When a device turns on, the
system correctly specifies the device in most cases. When a device turns off, the
system often makes mistakes.
For instance, the instant start light turning off in
the test setup are often mistagged as incandescent lights turning off. The key to
distinguishing an incandescent light turning off from an instant start light turning off
is two fold. First, in cases where only on of the devices was turned on the correct
tag can be easily chosen. Second, if the magnitudes of the loads are different, that
magnitude can be used to distinguish the events.
Smartlog does this by taking a tag from its input and matching it to a tag in
its memory. That tag will have type, "on" or "off", and a device name associated
with it. If it is an "on" tag, then that device name is added to the list of devices
which are currently on with the scale factor of the incoming tag. If it is an "off" tag,
then there may be a list of multiple devices with which it may match. Smartlog will
look at each of those possible devices which is listed as being on. It finds an error
associated with each device equal to the magnitude of the event transient divided
by the expected transient for each device. If this number is greater than one, the
reciprocal is taken. The answer closest to one is returned and that device is removed
from the list of on devices.
This program requires a table that relates events to devices. It reads this data
from a translation table stored in a file. Each line of the file contains a device name,
tag name, tag type, and scale factor. The file can be created by hand or by using a
40
utility called trans. It can be run on a set of tags that represent a single device being
turned on and off repeatedly. It creates the necessary entries for the translation table
file including scaling factors.
5.2
See
In many cases, the number of a given type of load in a building is known.
For
instance, a building may have one hot water heater. If NILM states that two hot
water heaters are on, then it has made an error. A small utility called see highlights
these discrepencies in a log by placing an asterix in front of it.
5.3
Tee2
Tee is a standard UNIX utility for splitting a text stream in much the same way
the plumbing joint of the same name splits a pipe. It reads in a line from its input
and prints that line to each of the streams in its command line. Unfortunately, this
program does not work with binary streams because the line it is reading in does not
end. Tee2 solves this problem by reading and echoing a fixed number of characters.
5.4
Timer
Data can be processed for a preset amount of time by using timer, a PERL script
that echos its input to its output for a preset number of seconds and then exits.
This will stop execution of the entire command line which executed the timer. For
instance,
dd if=/dev/pcl818 I tee2 "Iprep stdin I timer 3600 > saved"
I nilm
stdin exemplars -g "w3nilm -p 6060" &
will run the NILM sending its classification information to a web server running on
port 6060 and save preprocessed data to a file called saved. After an hour, 3600
seconds, NILM and the data logger will both exit.
41
5.5
Prune
The size of a data file can be reduced by only saving every Nth line of data, where N
is an integer. This operation is performed by the PERL script prune(Appendix H).
5.6
Chansift
Another technique for reducing file size when saving preprocessed data is to only
record selected channels. This is automated by using chansift(Appendix I).
42
Chapter 6
Conclusion
The projects described in this thesis have been combined to develop a NILM system
for installation. The NILM, preprocessor, training, and reporting software are all
installed on a "gold" machine. The hard disk contents of this machine are archived
to a recordable compact disc, CD-R, that can be used to create NILM machines for
installation. The same CD-R should be usable to create hard disk based PC/104
systems, but as discussed in Chapter 4, these systems have reliability problems that
prevent their use.
6.1
Preprocessor
The preprocessor has performed well during use in the laboratory and at an installation in a dormitory. Errors have been demonstrated with processors slower than 200
Mhz created by the inability of the processor to reset the A/D card between reads.
These problems are easily avoided, however, by using faster processors.
6.2
Training
The work on transient repeatability led to the creation of a variety of tools for training
the NILM. An Octave script called vsection is used to generate exemplars for NILM
classification, suggesting portions of the transients which should be repeatable. For
43
loads with multiple transient behaviors, cpmation, a clustering script, can generate
representative transients for training with vsection. These scripts are not as user
friendly as much of the rest of NILM, in particular their error messages are often
cryptic; but, they have been used to train the NILM in a straight forward manner.
Reporting
6.3
The toolbox of reporting tools developed in Chapter 5 is also included on the "gold"
system. A few examples of the combination of reporting tools to create the desired
report format are presented in scripts stored on the machine.
dd if=/dev/pcl818 I tee2 "Iprep stdin Ipuddle -p publichtml" Inilm stdin
exemplars14 -g "w3nilm -p 6060" -t "smartlog transtab > matched.data"
&
demonstrates many of the functions. Data is read from the A/D card using the
Linux function dd. The script tee2 described in Chapter 5 is used to send the
data to prep and NILM. The data is preprocessed as described in Chapter 2 by
prep. Continuously updating graphs of preprocessed data are created by puddle.
The program nilm classifies the transients and sends tags to w3nilm and smartlog.
These tags are provided to web browsers by w3nilm. Finally, smartlog matches
"on" and "off" events.
The data logging functions are demonstrated by
dd if=/dev/pcl818 I tee2 "I prep stdin I puddle -p public-html" "prep
stdin I timer 60 1prune 30 1chansift 1 2 > data" I nilm stdin exemplars14
-g "w3nilm -p 6060" &
This command line includes the same dd,puddle, and NILM commands as previously presented. The new portion begins with timer, this script is set to echo its
input to its output for 60 seconds. Then every 30th line of data is passed through
prune. Finally, the first two channels, ip and 1q, are passed through chansift. This
44
stream is saved to a file called "data". These command lines have been used to test
the reporting functions, and they have been shown to work. As NILM systems are
installed, the ability of the toolbox to create the desired reports will be tested.
6.4
Miniature NILM
The miniature NILM systems were very promising. The DiskOnChip system could
be used to develop a NILM system with no moving parts. The hard disk version
would create a physically small NILM system while requiring only minor variations
to the standard install. Unfortunately, both of these systems presented unexpected
difficulties.
The small capacity of the DiskOnChip limits both the programs and data that can
be stored on the NILM machine. Since training of the NILM will often involve storing
a large data file for offline analysis, the DiskOnChip based systems would have to be
trained on a separate NILM. THE DOC also has a limited number of write cycles,
creating a potential failure mechanism for these miniature NILM systems.
The hard disk version would avoid these problems associated with the DiskOnChip.
Unfortunately, attaching a hard disk to the miniature NILM exposes some of the
weaknesses in the design of the Complete Scalable CPU II. First, the IDE interface
does not provide enough power for the hard disk, so a special cable must be developed
to power the hard disk directly from the external power supply. Second, the hard disk
version has the troublesome behavior, as noted in the end of Chapter 4, of rebooting
at unexpected times. Preliminary work suggests that the rebooting may be caused by
the power supply voltage dipping below the required value. While a miniature NILM
system has been demonstrated, work remains to be done to improve its stability.
45
Bibliography
[1] C. B. Abler. Spectral envelope estimation for transient event detection. Master's
thesis, MIT, May 1998.
[2] Agnim I. Cole and Alexander Albicki. Algorithm for non-intrusive identification
of residential appliances. In Circuits and Systems. Proceedings of the 1998 IEEE
InternationalSymposium on, volume 3, pages 338-341, 1998.
[3] Agnim I. Cole and Alexander Albicki. Data extraction for effective non-intrusive
identification of residential power loads. In Instrumentation and Measurement
Technology Conference, 1998. Conference Proceedings, IEEE., volume 2, pages
812-815, 1998.
[4] Using
the
DiskOnChip
with
Linux
OS.
http:/www.m-
sys.com/files/appNotes/doc/IM-21_Linux.pdf, 2000.
[5] DM6420HR User's Manual. Real Time Devices, Inc., 1997.
[6] FFTW Web Site. http://www.fftw.org, 2000.
[7] Gerhard P. Hancke and Deon Vrey. Electric load monitoring and control in
the domestic environment.
In Instrumentation and Measurement Technology
Conference, Advanced Technolgies in I & M, volume 2, pages 560 - 562, 1994.
[8] George W. Hart. Residential energy monitoring and computerized surveillance
via utility power flows. IEEE Technology and Society Magazine, pages 12-16,
June 1989.
46
[9] George W. Hart. Nonintrusive appliance load monitoring. Proceedings of the
IEEE, 80:1870-1891, December 1992.
[10] John Hartigan. Clustering Algorithms. Wiley, New York, 1975.
[11] S. B. Leeb.
A Conjoint Pattern Recognition Approach to Nonintrusive Load
Monitoring. Phd, MIT, Department of Electrical Engineering and Computer
Science, February 1993.
[12] S. B. Leeb and J. L. Kirtley. A Transient Event Detector for Nonintrusive Load
Monitoring. U. S. Patent Number 5,483,153, Issued January 1996.
[13] S. B. Leeb, S. R. Shaw, and J. L. Kirtley. Transient event detection in spectral envelope estimates for nonintrusive load monitoring. IEEE Transactions on
Power Delivery, 7(3):1200-1210, July 1995.
[14] L. K. Norford and S. B. Leeb. Nonintrusive electrical load monitoring in commercial buildings base on steady-state and transient load detection algorithms.
Energy and Buildings, 24(1):51-64, May 1996.
[15] Alan V. Oppenheim and Ronald W. Schafer. Discrete-Time Signal Processing.
Prentice-Hall, 1989.
[16] User Manual: Scalable CPU II with Ethernet and I/O. Parvus Corporation,
1999.
[17] Hannu Pihala. Non-intrusive appliance load monitoring system based on a modern kWh-meter. PhD thesis, Technical Research Centre of Finland, 1998.
[18] J.G. Roos, I.E. Lane, E. C. Botha, and G. P. Hancke. Using neural networks for
non-intrusive monitoring of industrial electrical loads. In Instrumentation and
Measurement Technology Conference Prcoeedings, IEEE, pages 1115-1118, 1994.
[19] S. R. Shaw.
System Identification Techniques and Modeling for Nonintrusive
Load Diagnostics. Phd, MIT, Department of Electrical Engineering and Computer Science, February 2000.
47
[20] S. R. Shaw et al. Instrumentation for high performance nonintrusive electrical
load monitoring. Journal of Solar Energy Engineering, 120, August 1998.
[21] F. Sultanem. Using appliance signatures for monitoring residential loads at meter
panel level. IEEE Transactions on Power Delivery, 6(4), October 1991.
48
Appendix A
Prep.c for PCL818
This is the preprocessor described in Chapter 2.
/* compile gcc prep.c -lm -ilinfit -lfftw -lrfftw */
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
<stdio.h>
<err.h>
<fftw.h>
<rfftw.h>
<math.h>
<sys/types.h>
<sys/stat.h>
<fcntl.h>
<string.h>
<unistd.h>
<errno.h>
"prep.h"
10
#define FREQ 60
static const int __prep.nfilter__ = 16;
static const double -- prep-filter_] =
-0.00063046914864,
-0.00181856812428,
-0.00256194161246,
-0.00158749399440,
0.00236951266897,
0.00833249697835,
0.01180361285504,
0.00675929677933,
-0.00917451199773,
-0.02973090688604,
-0.03981645226642,
-0.02230164763869,
{
20
30
49
0.03102796590725,
0.11114350049251,
0.19245540210071,
0.24373020388648};
static __inline__ void prep-splitsequence(prep-t *p);
static __inline__ void prep-save(prep-t *p);
static __inline__ void prep-filter(double *buffer, int *nBuffer,
const double *filter,const int nFilter);
static __inline__ void prep-update(prep-t *p);
static __inline-- double prep-rough-average(prep-t *p);
static __inline__ void prep-find-zeros(prep-t *p, double avg);
static __inline__ void prep-accurate-average(prep-t *p);
static __inline-_ double prep find-dsratio(prep-t *p);
static __inline__ void prep downsample(prep_t *pdouble *raw,
double *sampled);
static __inline__ void prepifft(prep-t *pdouble *Adouble *B);
static __inline__ void prep-rotate(prep-t *p, double *E,int ide);
static double *prep rotation-multiply(double *bardouble *changing, int n);
static __inline__ void prep-load(prep-t *p);
40
50
/*
Initialize the preprocessor :
The device dev must be opened in non-blocking mode.
*7
void prep-init(prep-t *p,
const char *dev,
/* the device/file *7
60
int Nbasis,
number of elements per basis function (no samples per line cycle) */
int shift,
/* the number of samples to shift between rows */
int *harm,
pointer to the desired harmonics (first must be the first harmonic) *7
int Nharm,
The number of desired harmonics *7
int rows)
The number of rows of preprocessed data we'd like to retrieve at a time *7
/* rows must be > =2 or the unclaimed buffer fails */
/*
/*
/*
/*
{
extern int errno;
p->rate=8000; 7*7920;*7
7*
inits
/*The
number of samples per second*7
*/
50
70
p->fudge=0;
p->inCurrent=0;
p->iCurrent=8;
p->vCurrent=0;
p->harm=harm;
80
p->shift=shift;
p->nHarm=Nharm;
p->nBasis=Nbasis;
p->rows=rows;
p->nBuffer= 10*p ->rate/FREQ+(rows)*p->rate/FREQ*shift/Nbasis;
p->dsratio=0;
p->dsforget=0.999;
p->dsuncertainty=0.005;
p->average=0;
p->avgforget=0.999;
p->avguncertainty=0. 1;
90
/* talk to fftw */
p->plan=rfftw..create-plan(Nbasis,FFTWREALTOCOMPLEX,
FFTWMEASURE);
p->in = fftw-malloc(Nbasis*sizeof(fftw-real));
p->out = fftwmalloc(Nbasis*sizeof(fftw-real));
100
7*
*7
p->iBuffer=(double *)malloc(sizeof(double) *(p->nBuffer+ 10));
p->vBuffer= (double *)malloc(sizeof(double)* (p->nBuffer+ 10));
p->inBuffer=(word *)malloc(sizeof(word) *2*p->nBuffer);
allocate spaces
p->pzeros=(double *)malloc(sizeof(double)*(rows*p->shift/p->nBasis+5));
p->nzeros=(double *)malloc(sizeof(double)*(rows*p->shift/p->nBasis+5));
p->unclaimedi=(double *)malloc(sizeof(double)* (5+20*p->rate/FREQ));
p->unclaimedv=(double *)malloc(sizeof(double)*(5+20*p->rate/FREQ));
p->B=(double *)malloc(sizeof(double)* (p->rows*p->shift+p->nBasis+2));
p->A=(double *)malloc(sizeof(double)*(p->rows*p->shift+p->nBasis+2));
110
p->iHarm=(double *)malloc(sizeof(double) *2*p->rows*p->nHarm);
p->vHarm=(double *)malloc(sizeof(double)*2*p->rows);
if(!p->in I !p->out |
!p->iBuffer II !p->vBuffer |
I I !p->pzeros I I !p->nzeros
I I !p->unclaimedi I I !p->unclaimedv
!p->A
!p->B II !p->iHarm I !p->vHarm) {
!p->inBuffer
prep-close(p);
errx(1,"%s : memory allocation error at line %d
-FUNCTION__,
_-LINE__);
}
if((strcmp(dev,"--")==0) 11 (strcmp(dev,"stdin")==0))
51
120
p->nilm=fieno(stdin);
else {
p->nilm=open(dev,O-RDONLY);
-1)
if(p->nilm =
errx(1,"Xs : line %d : error %s opening %s",
_-FUNCTION__, __LINE__, strerror(errno), dev);
130
/* fixme here */
/*
}
*7
fcntl(p->nilm, F-SETFL, ONONBLOCK);
}
7*
close the preprocessor *7
#define FREE(x) if((x))
{
free((x)); (x)
NULL;
}
140
void prep-close(prep-t *p)
{
FREE(p->Buffer)
FREE(p->vBuffer)
FREE(p->unclaimedi)
FREE(p->unclaimedv)
FREE(p->nzeros)
FREE(p->A)
FREE(p->B)
FREE(p->pzeros)
FREE(p->inBuffer)
150
if(p->plan)
rfftw-destroy-plan(p->plan);
if(p->in)
fftwfree(p->in);
if(p->out)
fftw_free(p->out);
160
p->nBuffer=0;
if(p->nilm != -1)
close(p->nilm);
}
170
52
/*
Obtain some preprocessed data.
Data is written to matrices E (for envelopes) and A (for raw data),
if data is available. The number of rows for E was previously set
by a call to prep-init. lde, lda is the leading dimension of the
respective arrays. E has Nharm*2 columns. A has 1 column (for
current, later, 3 columns for three-phase), and has rows*Nbasis
rows.
180
Return values:
0 not enough data yet to "fill-out" the dest matrices
1 destination matrices are filled with data.
-1 End of file / Overflow
190
int prep-read(prep-t *p, double *E, int ide, double *A, int Ida)
{
extern int errno;
int rval;
int toRead;
if(A==NULL)
{
A=p->A;
ida=p->rows;
200
}
toRead=sizeof(word)*
(2*p->nBuffer-p->inCurrent-p->iCurrent-p->vCurrent);
rval = read(p->nilm, p->inBuffer+p->inCurrent, toRead);
if(rval == 0 ) /* EOF *7
return -1;
if(rval == -1)
/* error *7
{
210
if (errno==EAGAIN)
return 0;
if (errno==EINTR)
return 0;
else {
warnx("Xs : unknown error", __FUNCTION__);
return -1;
}
53
}
{
else
220
p->inCurrent += rval/sizeof(word);
if(rval == toRead)
{
prep-splitsequence(p);
prep-save(p);
prepfilter (p->vBuffer,&p->vCurrent,_prep-filter--,--prepnfilter_);
prep-filter(p->iBuffer,&p->iCurrent,_prep-filter-,__prep-nfilter__);
230
prep-update(p);
prep-downsample(p,p->iBuffer,A);
prep-downsample(p,p->vBuffer,p->B);
if(E! =NULL)
{
prep_fft(p,A,p->B);
prep-rotate(p,E,de);
240
}
prep-load(p);
p->inCurrent
rval = 1;
=
0;
}
else {
rval
}
=
0;
}
250
return rval;
I
static __inhine__ void prepsplitsequence(prepAt *p)
{
int last,i,warned=0;
word data;
260
last=-1;
for(i=0;i<p->inCurrent;i++)
{
data=*(word *)(p->fudge+(char *)p->inBuffer+2*i);
/* this is an ugly hack to try and realign bytes
data=p->inBuffer[i];*/
/*
54
*/
if((data & OxF)==0)
{
if(last! =0)
last=O;
p->vBuffer[p->vCurrent++]=(double) (data> >4);
if (p->vCurrent>=p->nBuffer+10)
270
{
p->vCurrent=p->nBuffer+9;
warnx("%s : line %d : error buffer overflow",
_FUNCTION__,
}
_LINE--);
}
else
warnx("Sequence Error: two data points from channel 0");
280
}
else if ((data & 0xF)==1)
{
if (last != 1)
last =1;
p->iBuffer[p->iCurrent++]=(double) (data> >4);
if (p->iCurrent>=p->nBuffer+10)
{
p->iCurrent=p->nBuffer+9;
warnx("Xs : line %d : error buffer overflow",
_FUNCTION__,
}
290
_-LINE--);
}
else
warnx("Sequence Error: two data points from channel 1");
}
else
{
if (warned>5)
300
{
warned=0;
if(!p->fudge)
p->fudge-=1;
else
p->fudge=0;
}
else
warned+=1;
warnx("Sequence Error: data is from unknown channel");
}
55
310
}
}
/* linfit takes data,an array of 7 points, starting at index and finds the
point where the line of best fit through those 7 points crosses val.
static -- inline-_ double linfit(double* data,int index,double val)
*/
320
{
double x,y,a,b,c,d,e,fg,h[7];
double t;
h[0] = 0.57143;
h[1]= 0.42857;
h[2]=0.28471;
h[3]=0.14286;
a = h[]*data[index];
b = (h[1])*data[index+1];
t = (data[index+2] - data[index+6]);
t
=
C =t;
d =
x =
330
h[2] * t;
(h[3])*(data[index+3] - data[index+5]);
a + b + c + d;
e= ((double)-0.10714) * (data[index] - data[index + 6]);
f= ((double)-0.07143) * (data[index+1] - data[index+5]);
g= ((double) -0.03571) * (data[index+2] - data[index + 4]);
y = e + f + g;
340
if (y==O)
{
warnx("\nError: Y==0");
val=3.5;
}
else
val=(val-x)/y-1;
return (val);
350
}
7*
saves roughly the last 20 cycles of data from current
and voltage so that unfiltered data can
be replaced in the buffers if it is not used. *7
static __inline_ void prep-save(prep-t *p)
{
56
p->iSave-p->vCurrent- (int) (20*p->rate/FREQ);
p->vSaved=p->vCurrent-p->iSave;
p->iSaved=p->iCurrent-p->iSave;
memcpy(p->unclaimedv,p->vBuffer + p->iSave,p->vSaved*
360
sizeof (double));
memcpy(p->unclaimedi,p->iBuffer + p->iSave,p->iSaved*
sizeof (double));
}
static __inline_ void prep-filter(double *buffer, int *nBuffer,
const double *filter,const int nFilter)
{
370
int ij;
double temp;
for(i=O;i<(*nBuffer-2*nFilter);i++)
{
temp=0.0;
for(j=zO;j<nFilter;j++)
temp+=filter[j] *(buffer[i+j]+buffer[i+2*nFilter-j]);
380
buffer[i]=temp;
}
*nBuffer-=31;
}
static __inline_ void prep-update(prep-t *p)
double avg;
avg=prep-rough-average(p);
prep-find-zeros(p,avg);
prep-accurate-average(p);
prep-find-zeros(p,p->average);
prep-find-dsratio(p);
390
}
static -- inline_
double prep-rough-average(prep-t *p)
{
int i;
double avg = 0.0;
400
if(p->average!=0)
return p->average;
else
I
for(i=0;i< (3*p->rate/FREQ) ;i±+)
57
avg+=p->vBuffer[i];
avg*=FREQ/(3*p->rate);
}
return avg;
410
}
static _-inline__ void prep-find-zeros(prep-t *p, double avg)
{
int i=O;
int moved=0;
while(p->vBuffer[i] >avg)
{
moved=1;
420
i++;
}
if (moved)
{
i+=5;
/* If I had to advance through a zero crossing I want to get past
that before I look for the next zero crossing. *7
}
p->cp=O;
p->cn=O;
while(i<p->vCurrent)
430
{
while((i<p->vCurrent)&&(p->vBuffer[i] <avg))
i++;
if (i<p->vCurrent)
p->pzeros[p->cp++]=i+linfit(p->vBuffer,i,avg);
i+=5;
if (p->cp>p->rows*p->shift/p->nBasis)
/* once I have my rows I don't want to mess with the end.*7
break;
440
if (p->cp>1)
if ((p->pzeros[p->cp-1]-p->pzeros[p->cp-2])<100)
warnx("Short Cycle! avg=Xf length=%f cycle= %d",(float)avg,
(float) (p->pzeros[p->cp- 1] -p->pzeros[p->cp-2]),p->cp- 1);
while((i<p->vCurrent)&&(p->vBuffer[i] >avg))
i++;
if (i<p->vCurrent)
p->nzeros[p->cn++] =i+linfit(p->vBuffer,i,avg);
if(p->cn >= p->rows*p->shift/p->nBasis+4)
{
/* may want a warning here *7
58
450
p->cn= p->rows*p->shift/p->nBasis;
I
i+=5;
I
p->cp--;
p->cn--;
return;
7*
7*
since I advanced cp and cn after reading the value I need to*/
subtract the extra *7
460
I
static _-inline-_ void prep-accurate-average(prep-t
*p)
{
double avg = 0.0;
int i;
i=p->pzeros[0];
while(i<p->pzeros[p->cp])
{
470
avg+=p->vBuffer[i];
i++;
}
if (p->cp<2)
{
warnx("Error: No Positive Zero Crossings Found.\n");
if (p->average==0)
p->average=2000;
}
480
else
{
avg/=(p->pzeros[p->cp]-p->pzeros[0]+1);
if(p->average==0)
{
p->average=avg;
I
else
{
p->avguncertainty/=(p->avgforget+p->avguncertainty);
p ->average+ =p ->avguncertainty *(avg- p->average);
490
}
}
return;
}
static _-inline-- double prep-find-dsratio(prep-t *p)
{
double period;
double dsratio;
500
59
if(p->cp<2)
{
warnx("Error: No Positive Zero Crossings Found.\n");
}
else
{
period=(p->pzeros[p->cp]-p->pzeros[0])/(p->cp);
dsratio=period/p->nBasis;
if(p->dsratio==O)
{
510
p->dsratio=dsratio;
}
else
{
p->dsuncertainty/=(p->dsforget+p->dsuncertainty);
p->dsratio+=p->dsuncertainty*(dsratio-p->dsratio);
dsratio=p->dsratio;
}
}
520
return p->dsratio;
}
static -- inline-- void prep-downsample(prep-t *pdouble *raw,
double *sampled)
{
double temp;
int i;
for(i=O;i< (p->rows*p->shift+p->nBasis);i++)
530
{
temp=i*p->dsratio;
sampled [i] =raw[(int)floor(temp)] +(temp-floor(temp))
*(raw[(int)floor(temp)+1] -raw[(int)floor(temp)]);
}
}
/* applies an ift to blocks of data */
540
static __inline_ void prepifft(prep-t *p,double *A,double *B)
{
int ij;
for(i=O;i<p->rows;i++)
{
for(jz=O;j <p->nBasis;j++±)
60
p-->inDj]=BUj+i*p->shift];
rfftw-one(p->plan,p->in,p->out);
p->vHarm[i*2]=p->out[1];
p->vHarm[i*2+ 1]=p->out[p->nBasis- 1];
for(j =O;j <p->nBasis;j++)
p->inj]=AU+i*p->shift];
550
rfftw-one(p->plan,p->in,p-->out);
for(j=O;j <p->nHarm;j++)
{
p->iHarm[i*2*p->nHarm+j*2]=p->out[(int)p->harmDj]];
p->iHarm[i*2*p->nHarm+2*j+1]=
p->out[(int)(p->nBasis - p->harm[j])];
}
560
}
p -> lastclaimed=-(p ->rows) *p-->shift *p-> dsratio;
}
static __inline__ void prep-rotate(prep-t *p, double *E,int Ide)
{
int ij;
double pqchange[2],pqbar[2],pqi[2];
double mag;
double *pqtemp;
570
for(i=O;i<p->rows;i++)
{
pqbar[]=p->vHarm[i*2];
pqbar[1]=p->vHarm[i*2+ 1];
mag=sqrt(pqbar[O]*pqbar[o]+pqbar[1]*pqbar[1]);
pqbar[O]/=mag;
pqbar[1]/=mag;
for(j=O;j<p->nHarm;j++)
580
pqi[]=p->iHarm[i*p->nHarm*2+2*j];
pqi[1]=p->iHarm[i*p->nHarm*2+2*j+1];
pqchange[O]=pqbar[O];
pqchange[1] =pqbar[1];
pqtemp=prep-rotation-multiply(pqbar,pqchange,p->harm]);
*(E+i+(2*j)*lde)=(pqi[o]*pqtemp[O]+pqi[1]*pqtemp[1]);
*(E+i+lde*(2*j+1))=(pqtemp[]*pqi[1] - pqi[O]*pqtemp[1]);
}
}
}
590
static double *prep-rotation-multiply(double *bar,double *changing,int n)
{
double temp[2];
61
if(n==1)
return (bar);
temp[O] =bar[O] *changing[o] -bar[1] *changing[1];
temp[1]=bar[O] *changing[1] +bar[1] *changing[o];
if(n==2)
600
{
changing [0] =temp [0];
changing[1] =temp[1];
return changing;
I
return (prep-rotation-multiply(bar,temp,n- 1));
}
610
7* replaces the unused data saved by prep-save
static -_inline__ void prep-load(prep-t *p)
*7
{
int i;
i=(p->lastclaimed)-p->iSave;
/*i=(p->lastclaimed)-p->iSave;*/
if (i<0)
{
620
warnx("Error negative i\n");
i=0;
p->iCurrent=O;
p->vCurrent=O;
}
else
{
memcpy(p->vBuffer,p->unclaimedv+i,(p->vSaved-i)*sizeof (double));
memcpy(p->iBuffer,p->unclaimedi+i,(p->iSaved-i)*sizeof(double));
p->iCurrent=p->iSaved-i;
p->vCurrent=zp->vSaved-i;
630
}
}
7*
return the file handle associated with p so that it can be used
for synchronous I/O multiplexing *7
int prep-handle(prep-t *p)
f
warnx("handle = %d", p->nilm);
62
640
return (p->nilm);
}
/* main program to call from train.m */
#ifdef MAINPROGRAM
void usage(void)
{
650
printf("Usage : prep source [-N number]\n");
exit(1);
}
#define LDAE 100
int main(int argc, char *argv[])
{
prep.t p;
int rval,i,j;
double E[LDAE*8];
7*
660
*/
extern char *optarg;
int lines = 0, linesofar = 0;
int harm[]={1,3,5,7};
int nHarm=4;
if(argc < 2
usage(;
argc > 4)
670
arguments *7
while((rval = getopt(argc-1,argv+1,"N: ")) != EOF)
if(rval == 'N')
lines = atoi(optarg);
else
usageo;
/* process
{
}
prep-init(&p, argv[1],128,64, harm,nHarm,LDAE);
680
rval = prep-read(&p, E, LDAE, NULL, LDAE); /* just read data
*7
do {
rval = prep-read(&p, E, LDAE, NULL, LDAE); /* just read data
if(rval == 1) /* got data */
for(i = 0; i < LDAE; i++)
for(j=0;j<p.nHarm*2;j++)
{
63
*7
printf("7.5e ",E[i+LDAE*j]);
printf(" \n");
linesofar++;
690
{
if(lines && linesofar >= lines)
rval = -1;
break;
}
}
}
while(rval != -1); /* until EOF
*/
prep-close(&p);
700
return 0;
}
#endif
710
64
Appendix B
DM6420.c
A driver for Real Time Devices, Inc. DM6420HR A/D card designed for compatibility
with the PCL818 driver developed by Steven Shaw was developed for the Miniature
NILM system presented in Chapter 4.
* DM6420.C
*
*
Linux kernel module implementing Real Time Devices' DM6420HR
device driver by C. D. Salthouse
*
*
Based on pcl818.c written by S.R. Shaw (sshawnit.edu)
*
*
based in part on code written by and
10
*
*
(c)1995 M. Welsh (mdwcs.cornell.edu)
*
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2, or (at your option)
* any later version.
*
*
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
* the GNU General Public License for more details.
*
*
*
*/
65
20
#define REALLYSLOW-IO
#include
#include
#include
#include
#include
#include
<asm/io.h>
<asm/dma.h>
<asm/irq.h>
<asm/system.h>
<asm/segment.h>
<asm/atomic.h>
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
<linux/fs.h>
<linux/errno.h>
<linux/major.h>
<linux/kernel.h>
<linux/sched.h>
<linux/signal.h>
<linux/module.h>
<linux/mm.h>
<linux/malloc.h>
<linux/ioport.h>
<linux/version.h>
<linux/isdnif.h>
30
40
char kernel-version[] = UTSRELEASE;
50
#include "sdm.h"
#define ALPHA
dm6420 dmd6420;
#define DMAAUTOINIT Ox1O
/*
DM6420 hardware interface ------
60
/* clear the DMA done bit */
static void __inline__ dm6420-resetdma(void)
{
outw-p(0x0004,DM6420_ CLEAR);
inw-p(DM6420_ CLEAR);
}
/* clear the DMA done bit *7
static void __inline__ dm6420_resetirq(void)
{
70
outw-p(0x0040,DM6420_ CLEAR);
inw-p(DM6420_ CLEAR);
}
66
/* tell the dm6420 to reset the Channel Gain Table *7
static void __inline__ dm6420-cleargctable(void)
{
outw-p(0x0008,DM6420_ CLEAR);
inw-p(DM6420-CLEAR);
}
80
/* tell the dm6420 to reset the Channel Gain Table to first entry*/
static void __inline-- dm6420_resetgctable(void)
{
outw-p(OxOO10,DM6420_ CLEAR);
inw-p(DM6420_ CLEAR);
}
/* tell the dm6420 to reset the A/D FIFO *7
static void -inline-- dm6420_resetadfifo(void)
{
outw-p(0x0002,DM6420 CLEAR);
inw-p(DM6420 CLEAR);
90
}
/* Reset the DM6420*/
static void __inline__ dm6420-resetboard(void)
{
outw-p(OxOO01,DM6420_ CLEAR);
inw-p(DM6420_ CLEAR);
}
100
static void __inline__ dm6420_setirq(dm6420 *d, int irq,int source)
{
static unsigned char xlat[] = {0,0,0,0x20,0,0x40,0,0,0,0x60,
0x80,0xA0,0xC0,0,0,0xE0};
if(xlat[irq] == 0)
printk(KERNCRIT "bad irq %d\n", irq);
else {
d->irq-register &= OxFFO;
d->irq-register
110
source Ixlat [irq];
outw-p(d->irq-register, DM6420OIRQREG);
}
7*
start delivering interrupts / data */
static void dm6420- start(void)
{
unsigned long flags;
120
67
unsigned long temp;
unsigned long status;
dm6420 *dmd;
dmd=&dmd6420;
dmd->current dma 1;
DMDEBUG("dm6420_start called\n");
save-flags(flags);
/* these two commands turn off interupts
*7
130
cli(;
* make fifo and dma buffers empty *7
msnd-fifo-make-empty(&(dmd6420.rq));
/
atomic-inc(&(dmd6420.iflag));
dmabuLempty(&(dmd6420.dma-inl));
dmabuLsetup(&(dmd6420.dma-inl),DM6420_DMA1);
dmabuLempty(&(dmd6420.dma-in2));
dmabuLsetup(&(dmd6420.dma-in2),DM6420_DMA2);
140
/*
*7
dm6420_setirq(dmd,DM6420_IRQ,5);
handle the card
dm6420_resetirq(;
dm6420_resetadfifo(;
dm6420_resetdmao;
temp=inw-p(DM6420-TRIGGER);
/*start
dmd6420.startflag = 1;
trigger*/
150
/* wrap up */
restore-flags(flags);
enable-irq(DM6420_IRQ);
}
7*
stop delivering interrupts data
static void dm6420_stop(void)
*7
160
{
unsigned long flags;
DMDEBUG("dm6420_stop called\n");
save-flags(flags);
clio;
68
if(dmd6420.startflag)
inw(DM6420-TRIGGER); /*stop reading*/
170
/*
Disable Interuppts Board
*7
dm6420-setirq(&dmd6420,DM6420_IRQ,O);
/* reset the instance count to zero */
atomic-sub(dmd6420.iflag,&(dmd6420.iflag));
restore-flags(flags);
}
180
7*
interrupt service routine *7
static void dm6420_interrupt(int irq, void *devid, struct pt-regs *regs)
{
unsigned long flags;
int rval;
save-flags(flags);
clio;
DMDEBUG("dm6420-interrupt called\n");
if(dmd6420.iflag)
190
{
DMDEBUG("dm6420_interrupt called with if lag\n");
/* prefetch from the DMA buffer, may want to do something faster here
*7
if (dmd6420.current dma== 1)
{
dmd6420.current-dma=2;
}
rval = msnd-fifowrite(&(dmd6420.rq),
dmd6420.dma-in1.buf,
DM6420ODMABUFSIZE,
0);
200
else
{
dmd6420.current-dma= 1;
rval = msnd-fifowrite(&(dmd6420.rq),
dmd6420.dma-in2.buf,
DM6420_DMABUFSIZE,
0);
}
210
7*
reset the dm card -> this allows the card to resume DMA?
dm6420_resetdma(;
dm6420resetirq();
dmd6420.ireal++;
69
*7
WAKE-UP(&(dmd6420.read-waitq));
} else {
dm6420oresetdma(;
dm6420_resetirq(;
dmd6420.ibogus++;
}
220
restore-flags(flags);
}
/*
Setup the DM6420 card. */
static void dm6420_set(dm6420 *dmd, int *args)
{
int i;
230
unsigned long ADentry;
unsigned long status;
DMDEBUG("dm6420_set called\n");
dm6420_resetboard(;
dm6420_resetdma(;
dm6420_cleargctable(;
dm6420_resetadfifo(;
dmd->controLregister=0;
dmd->triggerregister=0;
dmd->irqregister=0;
dmd->din-register=0;
240
/* Set to 32-bit Pacer Clock *7
dmd->trigger-register 1=0x100;
outw(dmd->trigger-register,DM6420TRIGGER);
/* Prepare to talk to timer chip */
dmd->control-register &= OxFF9F;
outw(dmd->contro-register,DM6420-CONTROL);
/*Program
Counter 0 for mode 2 operation
status= 0x34;
250
*7
outb(status,DM6420OTIMERCTRL);
/*
counter 0, programmable rate gen.
*7
outb(args[18] & OxFF,DM6420_TIMER);
outb(args[18] >> 8,DM6420OTIMER);
/*Program Counter 1 for mode 2 operation */
status=64 + 4 +0x30;
70
260
outb(status,DM6420_ TIMER_ CTRL);
counter 1, programmable rate gen.
/*
*7
outb(args[19] & OxFF,DM6420-TIMER+2);
outb(args[19] >> 8,DM6420_ TIMER+2);
7*
*
Channel setting:
270
*
*
args[O] = start channel
args[1] = stop channel
*
Read the channels that the user wants and write back what the
*
card has in its registers.
Gain setting :
args[2 .. 17] = Gain Code
*
*
*
*
/
if(args[1]<args[0])
280
i=args[1];
args[1]=args[0];
args[O]=i;
I
if (args [0] <0)
args[O]=0;
if(args[1]>15)
args[1]=15;
DMDEBUG("Scanning from channel %dto channel %d\n",args[0],args[1]);
/* Enable Channel Gain Table */
290
dmd->single-ended=O; /* differential *7
dmd->control-register &= OxFFFC;
dmd->control-register 1= Ox01;
outw(dmd->control-register,DM6420_ CONTROL);
for(i=args[0];i<=args[1];i++)
{
ADentry=il(args[i+1]&0x3)<<41dmd->single-ended<<9;
/* with skip bit */
outw(ADentry|0x0800, DM6420_ADTABLE); /* must now double the
frequency but his gives
it time to settle *7
7*
without skip bit */
outw(ADentry, DM6420_ADTABLE);
I
/* disable loading of ADTable and enable use of ADTable*/
dmd->control-register &= OxFFF8;
dmd->control-register 1= 4;
71
300
outw(dmd->controregister,DM6420-CONTROL);
310
/* Set Trigger Repeat to Off *7
dmd->triggerregister&=0xDFFF;
outw(dmd->trigger-register,DM6420_ TRIGGER);
/* Set Start Trigger to Software *7
dmd->triggerregister&=0xFF E3;
outw(dmd->trigger-register,DM6420_ TRIGGER);
/* Set Stop Trigger to Software */
dmd->triggerregister&=0xFF1F;
outw(dmd->trigger-register,DM6420-TRIGGER);
/*
Set Conversion Select to Software
*7
320
dmd->triggerregister&=0xFFFC;
dmd->trigger-register=0x0001;
outw(dmd->trigger-register,DM6420_ TRIGGER);
/*Set
Channel-gain Data Store Enable */
dmd->control-register I-= x0010;
outw(dmd->control-register,DM6420_ CONTROL);
/* tell DM6420 to use DMA 5 and 6 *7
dmd->control-register &= OxOFFF;
dmd->control-register I= 0x9000;
outw(dmd->control-register,DM6420_ CONTROL);
330
/* Set IRQ */
dm6420-setirq(dmd,DM6420_IRQ,5);
}
7*
see the 8254 manual to figure this out
static int dm6420- clockprobe(void)
{
*7
340
return 1;
I
/* Allocate and initialize *7
static int dm6420_init(dm6420 *dmd)
{
DMDEBUG("dm6420init called\n");
atomic-sub(dmd->iflag,&(dmd->iflag));
dmd->readwaitq = NULL;
dmd->startflag = 0;
dmabufLinit(&(dmd->dma_in1));
dmabufLinit(&(dmd->dmain2));
msnd_fifolinit(&(dmd->rq));
72
350
if(dmabuLalloc(&(dmd->dma-inl),DM6420_DMABUFSIZE) == 0) {
if(dmabufialloc(&(dmd->dma-in2),DM6420_DMABUF-SIZE) == 0) {
if(msndfifoalloc(&(dmd->rq), DM6420_READQLENGTH) == 0)
return 0;
360
else
printk(KERNCRIT
"dm6420 : Unable to get space for read fifo!\n");
} else
printk(KERNCRIT "dm6420 : Unable to kmalloc space for DMA!\n");
dmd->currentdma= 1;
}
return -EIO;
}
370
7*
initialize the dm6420 card *7
static void dm6420-initcard(dm6420 *dmd)
{
dm6420Ostopo;
/* no interrupts, please */
DMDEBUG("dm6420_initcard called\n");
380
/* Figure out if the frequency reference is 1 or 10 Mhz... *7
dmd->fMHz = 8;
printk(KERNINFO "dmd6420: io = OxYx, irq %d\n",DM6420-BASE,
DM6420_IRQ);
dm6420-.set(dmd,dm6420_.default-);
printk(KERNINFO "dm6420: Using DMA channels hd and %d.\n",
DM6420_DMA1,DM6420_DMA2);
390
printk(KERNINFO "dm6420: DMA buffer size = %ld.\n",
(long int)DM6420-DMABUFSIZE);
printk(KERN-INFO "dm6420: Read queue size = %ld.\n",
(long int)DM6420-READQLENGTH);
printk(KERNINFO "dm6420: Frequency reference = %dMHz.\n",
dmd->fMHz);
#if defined(__SMP__)
printk(KERNINFO "dm6420: SMP module\n");
#else
400
printk(KERNANFO "dm6420: non-SMP module\n");
#endif
73
}
static void dm6420_free(dm6420 *dmd)
{
DMDEBUG("dm6420_free called\n");
410
dm6420_stopO;
dmabuffree(&(dmd->dmain1));
dmabuLfree(&(dmd->dma_in2));
msnd_fifo_free(&(dmd->rq));
}
/*
--
dm6420 file system interface -------
*7
420
7* read
read works like this:
If non-blocking I/O is selected, never sleeps. Instead returns
-EAGAIN if there is no data ready.
If blocking I/O is used, read sleeps until there is an interrupt,
then returns as much data as is available.
*/
430
Need to protect the fifo.
static int dm6420_read(struct inode *node,
struct file *filp,
char *buf,
int count)
{
int rval, q;
unsigned long flags;
DMDEBUG("read()
440
called \n");
save-flags(flags);
clio;
rval = msnd-fiforead(&(dmd6420.rq),buf,count, 1);
restore-flags(flags);
if(rval == 0)
{
74
if(filp->Lflags & O-NONBLOCK)
{
450
rval = -EAGAIN;
} else {
/* block up to DM6420-TIMEOUT seconds, then caller
gets an EOF if no data *7
q = DM6420-TIMEOUT;
do {
7*
*7
DMDEBUG("read dmaresiude = %d,%d \n", get-dma-residue(5),
getdmaresidue(6));
current->timeout = jiffies+HZ;
460
SLEEPON(&(dmd6420.readwaitq));
current->timeout
=
0;
if(current->signal & ~current->blocked)
rval = -EINTR;
else {
save-flags(flags);
cli(;
rval = msndfiforead(&(dmd6420.rq),buf,count, 1);
restore-flags (flags);
}
}
470
}
while(rval == 0 && (q-- > 0));
}
return rval;
}
480
* ioctl
for the dm6420
*
* The calling program provides an integer array with the desired
* board configuration.
*/
static int dm6420Oioctl(struct inode *inode,
struct file *filp,
unsigned int iocmd,
unsigned long ioarg)
{
int args[64];
int retval;
int majnr = MAJOR(inode->i-rdev);
dm6420_stop(;
75
490
if (majnr != DM6420_MAJOR)
printk(KERNCRIT
{
"dm6420: Wrong device %d (should be %d) at line %d\n",
majnr,DM6420_MAJOR, -LINE__);
return -ENODEV;
500
}
retval = -ENOTTY;
DMDEBUG("ioctl called\n");
if(iocmd == DM6420_IO)
{
retval = copy-from-user(args, (int *)ioarg,sizeof(int)*DM6420-NARGS);
510
dm6420Oset(&dmd6420,args);
} else
DMDEBUG(KERN-CRIT "DM6420 debug : invalid ioctl %d\n",iocmd);
dm6420_start(;
return retval;
}
520
* open
*7
static int dm6420_open(struct inode *inode, struct file *filp)
{
DMDEBUG("open called\n");
if(filp->Lcount > 1)
return -EBUSY;
MOD-INC-USECOUNT;
530
DMDEBUG("Ibogus = %d\n", dmd6420.ibogus);
DMDEBUG("Ireal = %d\n", dmd6420.ireal);
DMDEBUG("Reset interrupt counters to zero.\n");
dm6420-startO;
return 0;
}
7*
close
*7
540
static void dm6420-close(struct inode *inode, struct file *filp)
{
MODDECUSECOUNT;
76
DMDEBUG("close called\n");
DMDEBUG("Ibogus = %d\n", dmd6420.ibogus);
DMDEBUG("Ireal = %d\n", dmd6420.ireal);
dm6420-stop(;
}
550
select
Need to investigate the save-flagso; cli); restore.flags(; option
*7
static int dm6420_select(struct inode *inode,
struct file *filp,
int sel-type,
select-table *wait)
560
{
unsigned long flags;
int rval = 0;
if(sel-type == SELIN)
{
save-flags(flags);
Clio;
if(!msnd-fifo-empty(&(dmd6420.rq)))
rval = 1;
else
select-wait(&(dmd6420.read-waitq),wait);
restore-flags(flags);
570
}
return rval;
}
/*
*7
dmabuf
580
/* reset DMA hardware to currentbuffer *7
static void dmabuLsetup(dmabuf *bunsigned int channel)
{
unsigned long flags;
save-flags(flags);
clio;
590
77
disable-dma(channel);
clear-dma-ff (channel);
set-dma-mode(channel,DMA_MODE-READ Ox10);
set-dma-addr(channel,virt-to-bus(b->buf));
setdma-count(channel,b->length);
enable-dma(channel);
/* Oxi0 auto-init mode *7
restore-flags(flags);
}
600
static void dmabufLinit(dmabuf *b)
{
b->buf = NULL;
b->length = 0;
b->pos = 0;
}
static int dmabufialloc(dmabuf *b, size-t size)
{
610
dmabuLfree(b);
if((b->buf = (char *)kmalloc(size, GFPDMAIGFPKERNEL))
= NULL)
{
b->length = size;
b->pos = 0;
return 0;
}
return -ENOMEM;
620
}
static void dmabufifree(dmabuf *b)
{
if(b->buf) {
kfree(b->buf);
b->buf = NULL;
}
dmabuLinit(b);
630
}
static void dmabuftempty(dmabuf *b)
{
b->pos = 0;
}
78
*
-
*7
fifo
640
/*
* FIFO code from the GPL MSND base drive,
* Copyright (C) 1998 Andrew Veliath
*7
static void msnd-fifo6init(msnd-fifo *f)
{
f->data = NULL;
}
650
static void msnd-fifo6free(msnd-fifo *f)
{
if (f->data)
{
vfree(f->data);
f->data = NULL;
}
}
static int msnd-fifo6alloc(msnd-fifo *f, sizet n)
660
{
msnd-fifo-free(f);
f->data = (char *)vmalloc(n);
f->n = n;
f->tail = 0;
f-->head = 0;
f->len = 0;
if (!f->data)
return -ENOMEM;
670
return 0;
}
static void msnd-fifo6make-empty(msnd-fifo *f)
{
f->len = f->tail = f->head = 0;
}
/* returns 1 if the fifo is empty *7
680
static int msnd-fifo6empty(msnd-fifo *f)
{
return f->len == 0;
}
79
static int msnd-fifowrite(msnd-fifo *f, const char *buf, sizet len, int user)
{
int count = 0;
690
if (f->len == f->n)
return 0;
while ((count < len) && (f->len != f->n))
{
int nwritten;
if (f->head <= f->tail)
{
nwritten = len - count;
if (nwritten > f->n - f->tail)
nwritten = f->n - f->tail;
700
}
else {
nwritten = f->head - f->tail;
if (nwritten > len - count)
nwritten = len - count;
}
if (user) {
}
if (copy-from-user(f->data + f->tail, buf, nwritten))
return -EFAULT;
else
memcpy(f->data + f->tail, buf, nwritten);
710
count += nwritten;
buf += nwritten;
f->len += nwritten;
f->tail += nwritten;
f->tail %= f->n;
}
720
return count;
}
static int msnd-fifo-read(msnd-fifo *f, char *buf, sizet len, int user)
{
int count = 0;
if (f->len == 0)
return f->len;
730
80
while ((count < len) && (f->len > 0))
{
int nread;
if (f->tail <= f->head)
{
nread = len - count;
if (nread > f->n - f->head)
nread = f->n - f->head;
}
else {
740
nread = f->tail - f->head;
if (nread > len - count)
nread = len - count;
}
if (user) {
if (copy-to-user(buf, f->data + f->head, nread))
return -EFAULT;
}
else
750
memcpy(buf, f->data + f->head, nread);
count += nread;
buf += nread;
f->len -= nread;
f->head += nread;
f->head %= f->n;
}
return count;
760
}
/*-----
-
-
kernel
-
static struct file-operations dm6420_fops =
NULL,
dm6420_read,
NULL,
/* iseek
/* Read
/* Write command
NULL,
7*
dm6420-select,
dm6420_ioctl,
NULL,
dm6420-open,
dm6420_close,
NULL,
NULL,
NULL,
select
/* Ioctl
7* mmap
7* open
* close
/* fsync
/* fasync
/* check-media-type
7*
{
*7
*7
*7
*7
readdir
81
*7
*7
*7
*7
*7
*7
*7
*7
*7
770
NULL,
/ * revalidate
I;
780
int init _module(void)
{
printk(KERN-INFO "dm6420: initmodule called\n");
if(dm6420_init(&dmd6420) == 0) {
if(register-chrdev(DM6420-MAJOR," dm6420 ",&dm6420-fops) == 0)
7*
-Get IRQ, DMA channels, and I/O space from kernel
request-region(DM6420_BASE,
Ox20, "dm6420"); /* never fails
{
*7
790
*7
if(request-irq(DM6420_IRQ,dm6420_interrupt, SAINTERRUPT," dm6420",
NULL) == 0) {
if (( request dma(DM6420_DMA1,"dm6420") == 0) &&
(request-dma(DM6420_DMA2,"dm6420")
{
==
0))
dm6420_initcard(&dmd6420);
return 0;
} else
800
printk(KERNCRIT "dm6420: Can't DMA channel %dor %d\n",
DM6420_DMA1,DM6420_DMA2);
} else
printk(KERNCRIT "dm6420: Can't allocate IRQd\n",DM6420_IRQ);
} else
printk(KERNCRIT "dm6420: registerchrdev failed.\n");
dm6420_free(&dmd6420);
}
}
return -EIO;
810
void cleanup-module(void)
{
DMDEBUG("cleanup-module called\n");
if(MOD-INUSE)
printk(KERNCRIT "dm6420: Device busy, remove delayed. \n");
820
7*
stop everything and free up the memory
dm6420_free(&dmd6420);
/* Free ioports, dma and interrupts *7
release-region(DM6420_BASE,0x20);
82
*7
free-dma(DM6420-DMA1);
free-dma(DM6420_DMA2);
free-irq(DM6420-IRQNULL);
/*
Remove.
*/
830
if(unregister-chrdev(DM6420 MAJOR," dm6420")!= 0)
printk(KERNCRIT "dm6420: cleanup-module failed\n");
else
printk(KERNINFO "dm6420: cleanup-module succeeded\n");
}
840
83
Appendix C
smartlog.c
Smartlog is a program described in Chapter 5 that improves the accuracy of "off'
event classifcations by matching "on" and "off' events.
*
smartlog.c
*
*
usage: smartlog transtab
*
*
smartlog takes as input tags and generates an output file of
device onloff time
* example transtab:
*
*
*
*
*
incandescent Incandescent on 1.00000
incandescent IncanOff off 0.990000
instant
InstantStart on 1.00000
instant
IncanOff
off 0.530000
instant
InstantOff off 1.010101
10
*
*
sections of transtab can be created by trans
#include
#include
#include
#include
/*
<stdio.h>
<stdlib.h>
<err.h>
<time.h>
20
example of the data elements in use
tags -> InstantStart: tag -> type = 1 = on
device = instant
scale = 1.00
84
next
null
next ->
IncanOff:
tag
->
30
type = 0 = off
device = incandescent
scale = Q.94
next
->
type = 0 = off
device = instant
scale = 0.53
next
null
*7
40
typedef struct{
int type;
char *device;
double scale;
void *next;
}tag-t; /*a linked list element of a device that matches a tag */
typedef struct{
char *name;
tag-t *tag;
void *next;
}tags-t; /* a linked list element of a tag
50
*7
typedef struct{
char *name;
double scale;
void *next;
}device-t; /* a linked list element of a device
*7
60
tags-t * read-transtable(char *transtable,tags t *tags);
void handle-tags(tags-t *tags);
int main( int argc, char *argv[])
{
tags-t *tags=NULL;
char *transtable;
if(argc! =2)
{
70
perror("Usage: smartlog transtable\n");
return 1;
}
transtable=argv[1];
85
tags=read transtable(transtable,tags);
handle-tags(tags);
}
80
tag-t * newtag(tag-t *next,char *devicechar *onoff,double scale)
{
tag-t *tag;
tag=(tag-t *)malloc(sizeof(tag-t));
if(tag==NULL)
printf("Error allocating tag\n");
tag->next=next;
tag->scale=scale;
tag->device=(char *)malloc(sizeof(char) *50);
90
if(tag->device==NULL)
printf("Error allocation tag->device\n");
strcpy (tag-> device, device);
if(!strcmp(onoff, "on"))
{
tag->type=1;
}
else if(! strcmp(onoff, "of f"))
100
tag->type=0;
}
else
perror("\nError: Unknown tag type.\n");
return tag;
}
tags-t * read-transtable(char *transtable, tags t *tags)
FILE * transfile=NULL;
110
char device[50],tag[100],onoff[30];
double scale;
tags-t *temptags=NULL;
tag-t *temptag=NULL;
int found;
size-t sSize;
transfile=fopen(transtable, "r");
if (transfile==NULL)
return NULL;
while(fscanf(transfile, "Xs %s %s %lf\n",device,tag,onoff,&scale)==4)
{
86
120
found =0;
temptags=tags;
while(temptags! =NULL)
{
if(!strcmp(temptags->name,tag))
{
found=1;
break;
130
}
temptags=temptags->next;
}
if (found)
{
temptag=temptags->tag;
temptags->tag=newtag(temptag,device,onoff,scale);
}
else
{
140
temptags=tags;
sSize=sizeof(tags-t);
temptags==malloc(sSize);
if(temptags! =NULL)
{
temptags->name=malloc(sizeof(char)*50);
if(temptags->name==NULL)
perror("Error mallocing temptags->name\n");
strcpy(temptags->name,tag);
temptags->next=tags;
temptags->tag=newtag(NULL,device,onoff,scale);
tags=temptags;
150
}
else
printf("Error Malloc Failed\n");
}
}
fclose(transfile);
return tags;
160
}
device-t *killdevice(device-t *device)
{
device-t * next;
next=device->next;
free(device-> name);
free(device);
87
170
return next;
}
void handle-tags(tags-t *tags)
{
char cTime[30],cName[50],cScale[20],cError[20],cP[30];
double scale,error;
device-t *devices=NULL;
device-t *tempdevice;
device-t *bestdevice;
device-t *prevbestdevice;
device-t *prevdevice;
double dTime;
double besterror,currenterror;
tags-t *temptags;
tag-t *temptag;
int found;
180
FILE *logfile;
time-t tTime;
190
while(1)
{
if (fscanf(stdin,"YXs:
s
%s :s\n",cTime,cName,cScale,cError,cP)
1=5)
{
perror("Invalid input format");
break;
I
found=O;
dTimez=strtod(cTime,NULL);
200
scale=strtod(cScale,NULL);
error=strtod(cError,NULL);
temptags=tags;
if (temptags==NULL)
perror("Error: No Tags\n");
while(temptags! =NULL)
{
if (!strcmp(temptags->name,cName))
{
found=1;
210
break;
}
temptags=temptags->next;
i
if (found)
88
{
if(temptags->tag->type==1)
{
/* a start tag - add this device to our list*7
tempdevice=devices;
devices=(device-t *)malloc(sizeof(device-t));
devices->next=tempdevice;
devices->scale=scale;
devices->name=(char *)malloc(sizeof(char) *50);
strcpy(devices->name,temptags->tag->device);
logfile=fopen("logf ile"," a");
fprintf(logfile,"%s start %f\n",devices->name,dTime);
220
fclose(logfile);
fflush(stdout);
230
}
else
{
7*
a stop tag
*7
bestdevice=NULL;
besterror=10000;
prevdevice=NULL;
tempdevice=devices;
found=0;
while(tempdevice! =NULL)
240
{
temptag=temptags->tag;
while(temptag! =NULL)
{
if( !strcmp(tempdevice->name,temptag->device))
{
found=1;
currenterror=tempdevice->scale*temptag->scale/scale;
250
if ((currenterror>0) && (currenterror< 1))
currenterror=1/currenterror;
if(currenterror<besterror)
{
besterror=currenterror;
bestdevice=tempdevice;
prevbestdevice=prevdevice;
}
}
260
temptag=temptag->next;
}
prevdevice=tempdevice;
89
tempdevice=tempdevice->next;
}
if (found)
{
logfile=fopen(" logf ile ",a");
if(logfile==NULL)
printf("Error openning logfile = %.s",bestdevice->name);
fprintf(logfile," %s stop Xf\n" ,bestdevice->name,dTime);
fclose(logfile);
if(prevbestdevice==NULL)
devices=bestdevice->next;
else
prevbestdevice->next=bestdevice->next;
free(bestdevice-> name);
free(bestdevice);
270
280
}
else
printf("Error: No appropriate devices found.\n");
}
}
} /*if found */
}
290
300
90
Appendix D
gSmartlog.c
GSmartlog is a program described in Chapter 5 that operates on a graphics stream
to improve the accuracy of "off' event classifcation by matching on and off transients.
7*
* smartlog.c
*
*
usage: smartlog transtab
*
*
*
smartlog takes as input graphics tags and generates an output file of
device onj off time with full graphics tags included
*
* example transtab:
*
incandescent Incandescent on 1.00000
*
incandescent IncanOff off 0.990000
*
instant
InstantStart on 1.00000
*
*
instant
instant
10
IncanOff
off 0.530000
InstantOff off 1.010101
*
*
sections of transtab can be created by trans
*7
#include
#include
#include
#include
#define
#define
#define
#define
<stdio.h>
<stdlib.h>
<err.h>
20
<time.h>
STAGELENGTH 1000
MAXSUBSECTIONS 10
NRECORD 256
THRESHOLD 100
91
#define MAX(A,B) (((A)>(B)) ? (A):(B))
30
/
* example of the data elements in use
tags -> InstantStart: tag
IncanOff:
->
next
->
tag
->
type = 1 = on
device = instant
scale
1.00
next
null
type = 0 = off
device = incandescent
scale = 0. 94
next
->
type
0 = off
device = instant
scale
0.53
next = null
40
*7
typedef struct{
int type;
char *device;
double scale;
void *next;
}tag-t; /*a linked list element of a device that matches a tag */
typedef struct{
char *name;
tag-t *tag;
void *next;
}tags-t; /* a linked list element of a tag
50
*7
60
typedef struct{
char *name;
double scale;
void *next;
}device-t; /* a linked list element of a device
*7
tags-t * read-transtable(char *transtable,tags t *tags);
void handle-tags(tags-t *tags);
70
static void __inline-- mgifiread(void *ptr, sizet size, size-t N, FILE *f)
{
extern int errno;
92
if(fread(ptr,size,N,f) != N)
errx(1,"Xs : %s", __FUNCTION_-,
strerror(errno));
}
80
int mgiLread-sections(FILE *f,
float *a,
int Ida,
int cols)
{
int i,M,N;
float *ptr;
mgiLread(&M,sizeof(int), 1,f);
90
if(M > cols
| M < 0)
errx(1,"NEW %s : bad value %d for M in [%d,%d], wrong mode perhaps?",
__FUNCTION__, M, 0, cols);
for(i = 0; i < M; i++) {
mgifiread(&N,sizeof(int), 1,f);
if(N < 0 11 (2*N+1) > Ida)
errx(1,"/.s : bad value %d for N,
-FUNCTION__,
wrong mode perhaps?",
N);
100
ptr = a+i*lda;
ptr[0] = (float)N;
mgiLread(ptr+ 1, sizeof (float),N,f);
mgifread(ptr+1+N,sizeof (float),N,f);
}
return M;
}
/*
note this mgif-write-section has been modified from Shaw's version */
void mgiLwrite-section(FILE *f, float *a, int Ida, int rows)
110
{
int ij;
fwrite(&rows,sizeof(int), 1,f);
for(i=0;i<rows;i++)
{
j=(int)a[i*ida];
fwrite(&j,sizeof (int), 1,f);
fwrite(a+i*ida+1,sizeof (float),j,f);
fwrite(a+j+i*ida+1,sizeof (float),j,f);
}
120
fflush(f);
}
93
int main( int argc, char *argv[])
{
tags-t *tags=NULL;
char *transtable;
if(argc! =2)
130
{
perror("Usage: smartlog transtable\n");
return 1;
}
transtable=argv[1];
tags=read-transtable(transtable,tags);
handle-tags(tags);
}
140
tag-t * newtag(tag-t *next,char *device,char *onoffdouble scale)
{
tag-t *tag;
tag=(tag-t *)malloc(sizeof(tag-t));
if(tag==NULL)
printf("Error allocating tag\n");
tag->next=next;
tag->scale=scale;
tag->device=(char *)malloc(sizeof(char)*50);
150
if(tag->device==NULL)
printf("Error allocation tag->device\n");
strcpy(tag->device,device);
if(!strcmp(onoff," on"))
{
tag->type=1;
}
else if(!strcmp(onoff, "off"))
160
{
tag->type=0;
}
else
perror("\nError: Unknown tag type.\n");
return tag;
}
tags-t * read-transtable(char *transtable,tagst *tags)
94
{
170
FILE * transfile=NULL;
char device[50],tag[100],onoff[30];
double scale;
tags-t *temptags=NULL;
tag-t *temptag=NULL;
int found;
size-t sSize;
transfile=fopen(transtable, "r");
if (transfile==NULL)
180
return NULL;
while(fscanf(transfile," %s %s %s %lf\n",device,tag,onoff,&scale)==4)
{
found=0;
temptags=tags;
while(temptags! =NULL)
{
if(!strcmp(temptags->name,tag))
{
found=1;
190
break;
}
temptags=temptags->next;
}
if (found)
{
temptag=temptags-->tag;
temptags->tag=newtag(temptag,device,onoff,scale);
I
else
200
{
temptags=tags;
sSize=sizeof(tags-t);
temptags=malloc(sSize);
if(temptags! =NULL)
{
temptags-> name=malloc(sizeof (char)*50);
if(temptags->name==NULL)
perror("Error mallocing temptags->name\n");
strcpy(temptags->name,tag);
temptags->next=tags;
temptags->tag=newtag(NULL,device,onoff,scale);
tags=temptags;
}
else
printf("Error Malloc Failed\n");
95
210
}
}
fclose(transfile);
return tags;
220
I
device-t *killdevice(device-t *device)
{
device-t * next;
next=device->next;
free(device-> name);
free(device);
return next;
230
I
void handle-tags(tags-t *tags)
{
char cTime[30],cName[50],cScale[20],cError[20],cP[30];
float record[16*5121];
double scale,error;
device-t *devices=NULL;
device-t *tempdevice;
device-t *bestdevice;
device-t *prevbestdevice;
device-t *prevdevice;
double dTime;
double besterror,currenterror;
tags-t *temptags;
tag-t *temptag;
int found;
240
FILE *logfile;
time-t tTime;
int lda,cols,M,N,i,j,rows;
250
logfile=stdout;
lda= MAX(STAGELENGTH*2,MAX-SUBSECTIONS*N-RECORD*2) + 1;
cols=16;
rows=1000;
while(1)
{
if (fscanf(stdin,"%
%:
%s : %s\n",cTime,cName,cScale,cError)!=4)
{
perror("Invalid input format");
break;
96
260
}
M=mgif-read-sections(stdin,record,lda,cols);
found=0;
dTime=strtod(cTime,NULL);
scale=strtod(cScale,NULL);
error=strtod(cError,NULL);
temptags=tags;
270
if (temptags==NULL)
perror("Error: No Tags\n");
while(temptags! =NULL)
{
if (!strcmp(temptags->name,cName))
{
found=1;
break;
}
temptags=temptags->next;
280
}
if (found)
{
if(temptags->tag->type==1)
{
7*
a start tag - add this device to our list*7
tempdevice=devices;
devices=(device-t *)malloc(sizeof(device-t));
devices->next=tempdevice;
devices->scale=scale;
devices->name=(char *)malloc(sizeof(char)*50);
strcpy(devices->name,temptags->tag-> device);
fprintf(ogfile,"Xs : %s on : %s : %s\n",cTime,
devices->name,cScale,cError);
mgif-write-section(logfile,record,IdaM);
290
}
else
{
7*
a stop tag
*7
300
bestdevice=NULL;
besterror=10000;
prevdevice=NULL;
tempdevice=devices;
found=0;
while(tempdevice! =NULL)
{
temptag=temptags->tag;
while(temptag! =NULL)
{
310
97
if( !strcmp(temp device->name,temptag->device))
{
found=1;
currenterror=tempdevice->scale*temptag->scale/scale;
if ((currenterror>O) && (currenterror<1))
currenterror= 1/currenterror;
if(currenterror<besterror)
320
{
besterror=currenterror;
bestdevice=tempdevice;
prevbestdevice=prevdevice;
}
}
temptag=temptag->next;
}
prevdevice=tempdevice;
tempdevice=tempdevice->next;
330
}
if (found)
fprintf(logfile,"Xs : %s off : %s : %s\n",cTime,
bestdevice->name,cScale,cError);
mgifiwrite-section(logfile,record,lda,M);
if(prevbestdevice==NULL)
devices=bestdevice->next;
else
prevbestdevice->next=bestdevice->next;
free(bestdevice-> name);
free(bestdevice);
340
}
else
warnx("Error: No appropriate device found.\n");
}
} /*if found */
350
}
98
360
99
Appendix E
see
See is the PERL script discussed in Chapter 5 that flags errors in device operating
schedule reports.
#!/usr/bin/perl
# Function: Prints those lines of file 2 that include a line from file 1
if ($#ARGV != 1) {
die "Usage: $0 watchfile stream\n";
}
($watchfiee,$streamfiee) = @ARGV;
open(WATCH,"$watchf ile");
while(<WATCH>) {
($name,$number)=split('
$\=) \n'1;
chomp($name);
',$_);
10
$\=' ';
chomp($number);
$watch{$name}=$number;
$on{$name}=0;
}
close(WATCH);
open(STREAM, "$streamfile");
while(<STREAM>)
{
($device,$type,$time)=split('
if (exists($watch{$device}))
{
20
',$);
if ($type eq "start")
{
$onf{$device}=$onf{$device}+1;
}
100
if ($type eq "stop")
{
30
$onf{$device}=$onf{$device}-1;
}
if ($on{$device}>$watch{$device})
{ print"**"}
print "$device : $type : $time\n";
}
}
40
101
Appendix F
tee2
This is the tee2 discussed in Chapter 5.
#!/usr/bin/perl
# Tee2 is a binary stream compatible version of tee based on
# tee clone that groks process tees (should work even with old perls)
# Tom Christiansen <tchrist@convex.com>
# 6 June 91
while ($ARGV[0]
/^-(.+)/ && (shift, ($ = $1), 1)) {
next if /^$/;
s/i// && (++$ignore-ints, redo);
s/a// && (++$append, redo);
s/u// && (++$unbuffer, redo);
die "usage tee [-aiu] [filenames]
}
if ($ignore-ints)
10
. . .\n";
{
for $sig ('INT', 'TERM',
'HUP',
'QUIT')
{ $SIG{$sig} = 'IGNORE'; }
}
$mode = $append ?
$fh = 'FHOQO';
>':
%fh = ('STDOUT', 'standard output'); # always go to stdout
$1 = 1 if $unbuffer;
20
for (@ARGV) {
if (!open($fh, (/^[^>I]/ && $mode)
.
$)) {
warn "$0: cannot open $_: $!\n"; # like sun's; i prefer die
$status++;
next;
}
select ((select ($fh) , $1 = 1) [0] ) if $unbuffer;
$fh{$fh++}
=
30
}
102
while ( read STDIN, $buf, 4096 ) {
for $fh (keys %fh) {
print $fh $buf;
}
}
for $fh (keys Xfh) {
next if close($fh) I !defined $fh{$fh};
warn "$0: couldn't close $fh{$fh}: $!\n";
$status++;
}
exit $status;
sub PLUMBER {
warn "$0: pipe to \"$fh{$fh}\" broke!\n";
$status++;
delete $fh{$fh};
}
103
40
Appendix G
timer
Timer is a program described in Chapter 5 that echos its input to its output for a
number of seconds set on its command line.
#!/usr/bin/perl
# tee clone that groks process tees (should work even with old perls)
# Tom Christiansen <tchrist@convex.com>
# 6 June 91
$secs=$ARGV[0];
$start=time;
while ( read STDIN, $buf, 4096 ) {
10
if (time < ($start + $secs))
{
print STDOUT $buf;
}
else
{
die;
}
}
104
Appendix H
prune
Prune is a program described in Chapter 5 that echos every Nth line of its input to
its output where N is an integer set on its command line.
#!/usr/bin/perl
$mod=$ARGV[O];
$count=O;
while(<STDIN>){
$count=$count+1;
if($count==$mod)
{
$count=O;
print STDOUT "$_";
}
10
}
105
Appendix I
Chansift
Chansift is a program described in Chapter 5 that echos selected channels from
stream of preprocessed data.
#! /usr/local/bin/perl
# (c)opyright 2000, S. R. Shaw
# Arguments of this filter specify the columns (starting at 1) of the
# input which are preserved on the output.
while(<STDIN>)
{
@cols = split;
foreach $k (@ARGV)
print $coes[$k-1],
{
10
"
}
print "\n";
I
106
Appendix J
Miniature NILM Parts
Part Name
Part Number
Part Manufacturer
Complete Scalable CPU II
PRV-0835A-01
Parvus Corporation
Complete Scalable CPU II Cable Set
PRV-0840A-01
801-483-1533
DM6420HR-1
Real Time Devices, Inc.
PC/104 Double Height Adapter
HighRel Data Module
814-234-8087
4" Can-Tainer
CT104 4"
Tri-M Systems
Can-Tainer End Cap
CT-EC05
1-800-665-5600
2mm Wiremount Socket
MSA44A-ND
Digikey
1mm Ribbon Cable
MC44L-51ND
1-800-344-4539
0.1" 10 pin IDC Connector
MSC10K-ND
3365 Series Cable
MC1OG-5-ND
5 pin DIN Connector
CP-1250-ND
107