Data mining and its aplication
Week 1: introduction to data mining (hoofdstuk 1-4)
1. The need for data mining
2. The data mining process and its stakeholders
3. Exploratory data analysis hoofdstuk 1.1.3
4. Data visualization hoofdstuk 4.3
1: the need for data mining
‘data mining is making in a graphic or non-graphic way a summit on an extraordinary
amount of data’
Problems of data mining addresses:
Risk assessment (the loan)
Demand prediction (taxis)
Fraud detection(bank transactions)
Anomaly detection
Large lanquage models (klantenservice bot)
2: the data mining process and its stakeholders
Data process workflows
enrichment the value of data
Data mining stakeholders
Business user: Has a sound understanding of the business domain targeteted by the
data mining project.
Project sponsor: in most cases the initiator or driver of the data mining project
Project manager: this person is in charge of the data mining project implementation
and is concerned with meeting goals for quality, time and budget targets.
Business intelligence analyst: this person act as the bridge between the data and the
business view of the targeteted problem. Driving activities related to the key
performance indicators (KPI).
 The data administrator,integrator and engineer are working closely on the
technical side and share relevant code and documentation.
Data administrator and integrator: a technical person with sound data management
competences and data preparation and solution delivery.
Data scientist/engineer: data moddeling and evaluation (analysing methods and
tools).
The data mining project workflow
Key problem categories
Conceptual moddeling/cognitive mabs
3: Exploratory data analysis
Getting the data -> ETL : Extraction, Transformation, Loading
Getting data is not always easy bc of:
 Different resources: flat files, different databases etc.
 Integration is cumbersome: missing/not uniwue id’s
 Sometimes also privacy concerns
Data needs to be transformed:
 Type conversions ( omzettingen bestandstypen)
 Missing value correction/clean up/imputation
 Generation of new values
EXPLORATORY DATA
 Attributes and instances
Numeric can be:
1. Discrete – age, count, represented as integer values
2. Continuous: height, weight, distance, represented as values.
3. The scale of numeric can be: interval(date, ook onder nul als goed is). Ratio
scale(distance so with a 0 value or more) and absolute scale (counting).
Accuracy: the closeness between the value in the data and the true value.
Completeness: data quality issues
\
4: data visualization
Visualisation charts
According to the visualization principles
- some visualization examples
Week 2: Linear Regression (hoofdstuk 8 ongeveer)
Linear regression hoofdstuk 8.3:
Polynomial regression (H9.1.3.2?):
Linear Regression take-aways
Concluie van trees
Overfitting / underfitting
Why time series analysis?
examp
Why use time-series?:
Soorten time lines; seasonality
Time series forecasting
Graphical analysis :
ARIMA modelling:
1. Introduction to ARIMA
2. ARIMA models
3. ARIIMA model selection
4. ARIMAX
1. Expotential smoothing vs. ARIMA
2. ARIMA MODELS : general framework
Week 3: clustering (hoofdstuk 7: finding patterns)
Clustering algoritm:
Hoofdstuk 7.4: density-based clustering
Hierarchal clustering: hoofdstuk 7.1
Prototype- and Model-Based (k-means..)Clustering: H 7.3
Density based clustering: hoofdstuk 7.4
Clustering evaluation:
Week 4: hierboven lijkt erop alleen practicum
Week 5: classification: decision trees, model validation(H5)
- Decision trees hoofdstuk 8.5
- Pruning hoofdstuk 8.1.3/ 7.6.3
- Model validation hoofdstuk 5.5
CART 1 is cost-complexity pruning.
2: pruning: het proces van het verwijderen of verkleinen van een dataset door
irrelevante, redundante of laagwaardige data te elimineren.
Technieken voor pruning (4):
3: model validation
mock exam classification
vragen
3: Model validation: tyopes of model errors and strategies for more reliable model
evaluation (4)>
Paar van de laatste slides van week 5 classification.
Process mining:
Multiple Choice Questions (MCQ) – max 30 points (50% of the exam marks)
1. Which of the following statements can be true for linear regression?
A. The achieved performance always improves when including more attributes
B. It’s best to reduce the number of data records to avoid overfit
C. It is inferior to logistic regression when the predicted output is a logarithmic function of
the input
D. Its performance can be assessed by the root mean squared prediction error
2. We wish to visualise the dispersion of average earnings in the groups (a) 0-20 years (b) 2140 years (c) 41-60
years (d) 61-80 years (e) 80-100 years (f) 100-120 years Which of the following charts would
be most
appropriate?
A. Bar chart for with a bar for each group
B. Histogram showing the distribution for each group
C. Series of boxplots for each group
D. Scatter plot for each group
3. Which of the following cases of clustering outcomes is the best?
A. We cannot tell that a single of the different results is the best
B. Average inter-class distance is 5 and average intra-class distance is 10
C. Average inter-class distance is 10 and average intra-class distance is 8
D. Average inter-class distance is 7 and average intra-class distance is 6
4. The frequency of a discrete (non-numerical) variable can be represented by:
A. Scatter plot;
B. Box plot (box-and-whisker);
C. Bar chart;
D Heat map.
5. A data set contains 1000 records of 4 input attributes (x1, x2, x3, x4) and one output
attribute (y). The x1
value is missing in 5 records, the x2 value is missing in 150 records, the x3 value is missing in
800 records, and
the x4 attribute has no missing values. The input and output attributes are all numerical and
the problem is to
estimate the value of y from the input attributes. The exact response depends on the exact
details of the data,
but without further information, which approach to handling missing values appears as
appropriate?
A. Deleting all records with missing values
B. Deleting attribute x3 and then deleting all the records with missing values from the other
attributes
C. Deleting all attributes with more than 5% of missing values
D. Replacing all missing values with the most common attribute value
6. Which of the following best describes different categories of data states?
A. Data stored in hard drives; data transmitted over the network
B. Data stored in data bases; data distributed over the web
C. Data in use; data on the move
D. Labelled data, unlabelled data
7. In Time Series analysis we wish to understand if there is seasonality in the attribute values.
In other words, we
wish to understand if there are ________ in the attribute values
A. Outliers
B. Patterns
C. Repeated cycles
D. Change points
8. Which of the following is not a data pre-processing activity?
A. Data transformation
B. Clustering of data to identify patterns
C. Time stamping alignment of data records
D. Data de-noising
9. A financial asset management company needs to make stock values predictions. While it
can afford small
prediction error variations, even a few cases of very large prediction errors can lead to
financial disaster. Which
of the following prediction metrics would be appropriate for selecting among different
prediction models?
A. Mean squared error = 100; Mean absolute average error = 10
B. Mean squared error = 60; Mean average error = 20
C. Mean squared error = 40; Mean average error = 25
D. Mean squared error = 70; Mean average error = 10
10. How can be overfitting avoided in decision tree training:
A. By choosing an appropriate evaluation measure
B. By pruning
C. By k-fold cross-validation
D. By using recursive partitioning
11. The estimated number of infections in a disease epidemic are known to be strongly
depended on the
infections of the immediately previous dates, while less influenced by dates further in the
past. For the time
series model employed for predicting infections numbers, which of the following would be
more appropriate?
A. A model that returns the average value of the past 15 days
B. A model that returns the value of the latest record
C. A weighted moving average model of the last 3 records, with 0.5, 0.35, and 0.15 being the
corresponding weight for the t, t-1, and t-2 values
D. A random walk model based on the latest value of the infection numbers
12. Information gain can be used to develop a ______ model
A. Linear regression
B. K-Means
C. Decision Tree
D. ARIMA
13. In a data mining project, a _________ has appropriate skills for data analysis and data
management and is
able to communicate results to the relevant stakeholders, based on a sound understanding of
the problem
A. Business Intelligence Analyst
B. Data Scientist / Engineer
C. Data Administrator & Integrator
D. Data Architect
14. In a classification task you obtain True Positives (TP) = 15, False Positives (FP) = 10, True
Negatives (TN) = 70,
False Negatives (FN) = 5. What is the overall accuracy?
A. 75%
B. 87.5%
C. The provided information is insufficient to fully calculate the overall accuracy
D. 85%
15. Which of the following is not considered as a data mining task?
A. Web-based payment transactions
B. Outlier detection
C. Data-driven prediction
D. Labelling of data instances
16. What is the reason for wishing to prune a Decision Tree?
A. To save storage space when storing the Decision Tree
B. To save computational time when testing the Decision Tree
C. To reduce overfitting
D. To reduce the error on the training dataset
17. Clustering can be used in order to:
A- Detect outliers
B- Identify meaningful associations between attributes
C- Determine appropriate attributes to include for a classification model
D- Classify data between different given categories
18. Which of the following statements may correspond to an appropriate assessment of the
performance of a
logistic regression model?
A- Room mean squared error is 0.9
B- Cohen’s Kappa is 0.6
C- Silhouette coefficient is 0.9
D- None of the above
19. We need to estimate the daily traffic on a certain junction and plan accordingly for future
works. The data
records available are: Period (P) 1: Traffic = 20000; P2=15000; P3=25000; P4=40000;
P5=60000; P6=50000;
P7=70000. Which of the following would offer the best prediction for the P7 value?
A. A 3-period moving average predictor
B. A weighted moving average with weights 0.8 and 0.2 for the last two values in time order?
C. A predictor the returns the average value of all previous records?
D. A predictor that returns the previous record value
20. Which of the following statements is correct for a decision tree classifier
A. The decision tree should continue growing until all data are correctly classified
B. Pruning the decision tree is advisable, even if this reduces the rate of correctly classified
training data
C. The optimal decision tree is the one that produces the minimum room mean squared
prediction error
D. A decision tree is can be the best choice when other clustering methods have failed
21. We wish to assign the sentiment in tweet posts to one of the following categories: anger;
annoyance;
indifference; understanding; backing; enthusiasm. What type of data mining problem does
this correspond to?
A. Clustering
B. Classification
C. Regression
D. Exploratory Data Analysis
22. The vibration measurement is the Root Mean Squared (RMS) value of the acceleration
measurement
amplitude. As long as the machine condition deteriorates, the RMS value increases, however
the average
vibration remains zero (0). Which of the following could describe the type of the time series?
A. The time series is stationary
B. The time series is non stationary
C. The time series is noisy
D. None of the above
23. Categorical variables can appropriately be used in a regression task if:
A. They are ordinal
B. They are coding no more than two categories
C. They are numerical
D. There are sufficient data records for each category
24. While studying the response of customers to a marketing campaign, you identify 5
customer types which are
valuable to your study. You therefore decide to find all customers who are most similar to
each one of the above
types. The type of algorithm most appropriate for your study is ___________________.
A. Linear regression
B. Decision tree
C. k-nearest neighbours
D. Association rules
25. Which of the following is true for unsupervised learning algorithms?
A. Learning algorithms with no control over the quality of their outcomes
B. Learning algorithms which are fully autonomous, without human oversight
C. Learning algorithms which have no pre-defined class outcomes as part of the learning
process
D. None of the above
26. Which of the following can be best for understanding the range and dispersion of
numerical data?
A. Mean and median values
B. Minimum and maximum values together with the mean value
C. Minimum and maximum values together with standard deviation
D. Minimum and maximum values together with the median value
27. Which of the following statement about outliers is the one that is best justified?
A Outliers should be detected and excluded from the set of data records to be used for
training, to avoid
them dominating the model training process
B For numerical attributes, the outlier attribute value should be replaced by the mean
attribute value
C The problem must first be analysed to determine an appropriate approach for handling
outliers
D Outliers should be detected and excluded from the test set of data records as reporting
performance on
outliers is misleading
28. Which of the following statements can be true for linear regression?
A. The achieved performance always improves when adding more data records
B. The performance can be poorer when non linear associations between attributes and
outcomes exist
C. It’s best to reduce the number of attributes to avoid overfit
D. The performance can be assessed via the Cohen’s kappa.
29. We wish to understand if we have a significant proportion of our customers earning
between 70-100keuros
annually. What type of data mining problem does this task best correspond to?
A. Clustering
B. Classification
C. Exploratory Data Analysis
D. Process Mining
30. Which of the following is not considered as a data mining task?
A. Data cleaning
B. Data patterns identification
C. Authentication for data access
D. Data records selection