Matlab DIY
Lesson 3: Parsing data
Today's Lesson
• “parse”= analyze into its parts
o Sort the data you want MATLAB to use
o Designate different segments of the data
o Tell MATLAB to do different things with different
parts of the data
• “batch” = process a group of records as a single unit
(one command processes multiple files)
Why Parse?
Let's look at a few example files:
- data output may not be in evenly spaced rows
- you might be interested in a subset of the data and want
to separate it from the rest
- you might want to run different analysis on different
parts of the data
Your first .m program
During previous lessons you typed individual
commands and entered them in the command window
.m allows you to access a series of commands by
entering the name of the program in the command
window
Open MATLAB
 set current directory
open current directory window
double click on L3_csvread_pc.m
L3_csvread_pc.m
review commands from previous lessons
Parse option 1: choose rows and columns of
data where Matlab begins to read data
Reading in the data file
data = csvread([inpath,infilename],1,1);
tsvread
xlsread
dlmread
These all read in numbers NOT text
0,0 = first cell in upper left corner
1,0 = can be used to eliminate header data
10,5 = starts reading on 10th line, 5th column
Common csvread errors
>> d = csvread([inpath,filename],1,0);
??? Error using ==> csvread
Filename must be a string.
Step 1: see what Matlab thinks filename is:
1) type filename in command window
2) check class of filename in workspace
Step 2: adjust so that Matlab sees filename as a string
1) redefine filename variable by adding ‘’ or []
2) type filename{1} this turns it into a string
d=csvread([inpath,filename{1}],1,0);
Common csvread errors
>> u = csvread([inpath,filename{1}],10,0);
??? Error using ==> csvread
File not found.
Step 1: check the path
-is the name what you think it is?
-have you been consistent in spelling for
inpath and filename?
Step 2: check your data file
-have you used the right name?
-should it be in a folder?
-have you used the right suffix?
u = csvread([data_inpath,filename{1}],10,0);
Common csvread errors
>> c = csvread([data_inpath,filename{1}],0,0);
??? Error using ==> textscan
Mismatch between file and format string.
Trouble reading number from file (row 1, field 1) ==>
time,
Step 1: open data file and check for letters in the data
Step 2: adjust to avoid problem area
--adjust where Matlab begins reading or
--adjust organization of the file
Parse option #2
Create a new smaller array with a subset of data
>> c = csvread([data_inpath,filename{1}],1,1);
>> k = csvread([data_inpath,filename{1}],2:8,1:10);
??? Error using ==> textscan
Header lines must be a scalar integer.
Error in ==> csvread at 45
m=dlmread(filename, ',', r, c);
>> k = c(2:8,1:10);
Parse option #3:
Naming segments of the data
x_pos=data(:,1);
y_pos=data(:,2);
z_pos=data(:,3);
mus1=data(:,4);
mus2=data(:,5);
mus3=data(:,6);
event=data(:,7);
time=data(:,8); %notice that we started reading the data at 1,1 so the file
data is missing the first row and first column, each of these column numbers
have been adjusted accordingly
Now you can perform simple formulas
mn_x=mean(x_pos);
SD_y=std(y_pos);
Two ways to find more info on abbreviations for mathematical
functions
1) command window
>>help mean
2) Go to Matlab help and search for mean
Parse option #4:
create program to select a subset of data
onstamp=find(event>4);
ontime=onstamp(1);
offstamp=find(event<0);
offtime=offstamp(1);
mv_time=(offtime-ontime)*1/84; %sampling rate =84Hz
subset of rows
all columns
eyes_open=data(ontime:offtime,:); %this creates a subset of
data from rows that begin with event onset and end with event
offset, the : means all columns are included
Printing Data to a File
outfilename=[date,'L3_output.csv'];
out_fid=fopen([outpath,outfilename],'w');
fprintf(out_fid,'filename,mn_x,std_y,mv_time,mn_x_EO\n');
fprintf(out_fid,'%s,%s,%s,\n',infilename,mn_x,SD_y,);
L3.m
%Outputting just a String, useful for Headers.
fprintf(out_fid,‘filename,mn_x, SD_y,\n');
%Outputting data from variables.
fprintf(out_fid, '%s,%s,%s\n', filename,mn_x, SD_y,);
Format specifier for how information is to be written,
%s= string
%d= decimal
Values to go in spaces
%f=fixed point decimal
Look these up by typing “ help fprintf”
'\n'?
Special Characters you will likely use:
• \n = The newline character
• \t = The tab character
• \\ = The backslash character
fprintf(out_fid, '%s,%s,%s\n', subj_num,group,AP_RMS);
fprintf(out_fid, '%s\t%s\t%s\n', subj_num,group,AP_RMS);
Output per data set
One line vs. multiple lines
fprintf(out_fid, '%s,%f,%5.3f,,/n', filename, mn_x,MT,);
fprintf(out_fid, '%g,%g,/n', eyes_open(1,:), eyes_open(2,:));
Single variable output
entire column of data
from an array
Run the .m file
1) Save any changes to .m program
2) Type the name of the program (but not .m) in command window
3) Enter
4) Did you get any errors?
5) Did you get any messages at all?
6) Check your workspace
7) Are all your variables in place?
8) Check your desktop files
9) Did Matlab create an output folder?
10)Did Matlab create any output files?
11)Open the files
12)Do they look correct?
13)Which number specifier do you want to use? f, g, e, d?
Reading data with textscan
infile=[inpath,infilename];
fid=fopen(infile,'r') ;
if fid == -1
[infile,' could not be opened']
return
end
d=textscan(fid, '%s', 'delimiter', '\n'); %scans textfile and loads up as a string,
delimiter chops it by each new line
fclose(fid); %line above read in contents and stored it in "d",
d is array of all info line by line
textlines=d{1}; %this puts a wrapper on the data, begins with the first piece
that is my data set
Creating a data array with textscan
for p=2:length(textlines)
herenow=strread(textlines{p},'%s','delimiter',',');%grabs the text lines and
reads the whole line kind of turns it into csv with text)
for q=1:length(herenow)
big_array((p-1),q)=herenow(q); %(i-1) so it creates the new file from
row 1 otherwise have a blank where old labels were
end
end
Creating data columns with textscan
subj_num = cell2mat(big_array(:,1));
x_pos_t= str2num(cell2mat(big_array(:,2)));
Processing multiple data files
Open L3_combine_batch_textscan.m
This is a template file
It is not set up to work on a current data set
It has multiple inpaths to combine different files
It has multiple subj_nums to allow batch processing
Processing multiple files
inroute='/Users/saavedra/Desktop/CP_eye_hand/key_variables/';
inroute2='/Users/saavedra/Desktop/CP_eye_hand/left_overs/';
outroute='/Users/saavedra/Desktop/CP_eye_hand/data_combine/';
if exist(outroute)==0
mkdir(outroute);
end
subj_num={'u11','t04','t19',};%creates an array with 1 row and a cell with
a string for each subject name, you must have a folder for each subject
for subj_index=1:length(subj_num)%creates an index of all the subjects
inpath=cell2mat([inroute,subj_num(subj_index),'/']);% creates a string
with subjnum imbedded
infilename=cell2mat([subj_num(subj_index),'results.csv']); %creates an
infilename with subj_num imbedded
Practice making errors
• Counting is critical for output files
• Play with some common things that create weirdness
in your output file
1) More headers than variables
2) Less headers than variables
3) More % s (or d or f) than variables
4) Put less %s than variables
5) Take the /n off the end of fprintf on data file
Practice opening files
• Open one of your own data files
• Label and sort data
• Keep records of the error messages
• Bring a record of errors to class
• Bring solution if you found it
• IF you did NOT find solution please send the error
to Wayne and Sandy before 10 am Monday
morning.