Big Data in IoT
Complete Exam Preparation Guide
All 6 Chapters · All 11 Labs · Theory + Practical Cheat Sheet
FORMAT
20 multiple-choice questions (A/B/C/D) + Practical Jupyter notebook tasks: import libraries ·
load CSV · clean data · compute stats · create plots. The practical is described as simpler
than the full lab tasks.
PART 1 — Theory Q&A (from all 6 chapters + labs)
TIP
Questions are ordered by chapter. The 20 on your exam will be a selection from topics like
these — definitions, named lists, comparisons, and 'which algorithm / chart type / tool does X?'
Q1. What are the four 'Vs' of Big Data?
Answer: Volume, Velocity, Variety, Veracity.
Volume = how much data. Velocity = how fast it is generated. Variety = the types of data. Veracity = ensuring
data accuracy (preventing corrupt data from spoiling the dataset). IBM stated it takes 200–600 Terabytes to
qualify as Big Data.
Source: Ch.1
Q2. What is the definition of Big Data?
Answer: Data that is so vast, fast, or complex that it cannot be stored, processed, and analyzed
using traditional data storage and analytics applications.
Growing exponentially (not linearly), driven by IoT devices, sensors, mobile networks. Traditional RDBMS cannot
keep up.
Source: Ch.1
Q3. What is Open Data?
Answer: Any content, information, or data that people are free to use, reuse, and redistribute
without any legal, technological, or social restriction.
Defined by the Open Knowledge Foundation. Contrasts with Private Data, which is regulated by
government/expectations of privacy.
Source: Ch.1
Q4. What is the difference between Structured and Unstructured Data?
Answer: Structured = fixed fields in a database/spreadsheet; Unstructured = raw data with no
fixed schema (photos, audio, video, emails, etc.).
Structured data is easily entered, classified, queried, and analyzed (stored in relational databases). Unstructured
data includes photos, video, blogs, Word documents, PDFs — roughly 80% of all data generated today.
Source: Ch.1
Big Data in IoT — Exam Prep | Page 1 of 15
Q5. What is the difference between Data at Rest and Data in Motion?
Answer: Data at Rest = static data stored in a physical location (flow:
Store→Analyze→Notify→Act). Data in Motion = dynamic data processed in real-time before it
becomes irrelevant (flow: Analyze→Act→Notify→Store).
IoT sensor streams are Data in Motion — analysis must happen before data is stored, or the opportunity to act is
lost.
Source: Ch.1
Q6. Which database is described as 'the most widely deployed database in the world'?
Answer: SQLite.
SQLite is simple, embedded, and requires no server. Used in mobile apps, browsers, and IoT devices. It is the
underlying engine for the SQL lab in this course.
Source: Ch.1
Q7. What was Hadoop (HDFS) created to solve?
Answer: To deal with big data volumes — it is a redundant, distributed file system that stores
data across many computers.
Hadoop is NOT a single application — it is an ecosystem of applications working together. MapReduce is its
distributed processing framework.
Source: Ch.1 / Ch.6
Q8. What are the three types of analytics? Describe each.
Answer: Descriptive (historical reporting), Predictive (forecasts future trends), Prescriptive
(recommends best actions).
Descriptive = what happened. Predictive = what might happen. Prescriptive = what should we do about it. These
represent increasing levels of analytical sophistication.
Source: Ch.2
Q9. What are the 6 steps of the Data Analysis Lifecycle?
Answer: 1-Gather data, 2-Prepare data, 3-Choose a model, 4-Analyze data, 5-Present results, 6Make decisions.
The cycle then repeats, beginning again with gathering new data. Results from step 5 drive organizational
strategy changes in step 6.
Source: Ch.2
Q10. What does ETL stand for, and what is it used for?
Answer: Extract, Transform, Load — a process for collecting data from various sources,
transforming it, and loading it into a database.
ETL is a fundamental data engineering process. Extract = pull from source. Transform = clean/reshape/convert.
Load = insert into target database for analysis.
Source: Ch.2
Q11. What does the DIKW model stand for, and what does it represent?
Answer: Data, Information, Knowledge, Wisdom — it shows the transitions data undergoes to
gain enough value to inform wise decisions (= Business Intelligence).
Big Data in IoT — Exam Prep | Page 2 of 15
Raw data becomes information when organized. Information becomes knowledge when understood. Knowledge
becomes wisdom when applied to make good decisions.
Source: Ch.2
Q12. What is the CIA Triad in the context of Big Data?
Answer: Confidentiality, Integrity, Availability — a guideline for data security in an
organization.
Data protection regulations vary by country. The four general cloud security controls are: Deterrent, Preventive,
Detective, Corrective.
Source: Ch.2
Q13. What are the three Python libraries used throughout this course, and what does each do?
Answer: NumPy (arrays/math), Pandas (DataFrames/data manipulation), Matplotlib
(plotting/visualization).
NumPy adds mathematical array operations. Pandas adds tables and time-series support for cleaning large
datasets. Matplotlib creates everything from simple line plots to 3D charts. Jupyter Notebook is the interface used
to run them.
Source: Ch.2
Q14. What is the difference between Correlation and Causation?
Answer: Correlation = two things change at a similar rate. Causation = one thing directly
causes a change in another. 'Correlation does not imply causation.'
Example: Ice cream sales and drowning rates are correlated (both rise in summer), but ice cream does NOT
cause drowning. This is one of the most important principles in data science.
Source: Ch.3
Q15. What are Descriptive vs. Inferential Statistics?
Answer: Descriptive = summarizes the data you have (mean, median, std). Inferential = uses a
sample to make generalizations or predictions about a population.
Descriptive stats describe your dataset. Inferential stats let you draw broader conclusions — e.g. testing whether
a drug works for all patients based on a trial sample.
Source: Ch.3
Q16. What are the three characteristics of a statistical sample?
Answer: Distribution, Centrality (mean/median/mode), Dispersion (variability/spread).
Distribution = how values are spread out. Centrality = the 'typical' value. Dispersion = how spread out the data is
(std deviation, range, variance).
Source: Ch.3
Q17. What are the two types of variables, and their subcategories?
Answer: Categorical (Nominal = names/labels; Ordinal = ordered categories) and Numerical
(Continuous, Ratio, Discrete).
Nominal: country names, gender. Ordinal: shirt sizes (S/M/L), star ratings. Continuous: temperature, speed.
Discrete: number of devices, count of crimes.
Source: Ch.3
Big Data in IoT — Exam Prep | Page 3 of 15
Q18. What is NaN in pandas, and what is NaT?
Answer: NaN = Not a Number — represents missing or undefined numeric values. NaT = Not a
Time — represents missing datetime/timestamp values.
Both are pandas missing-value sentinels. Detect with df.isnull(). Handle with df.dropna() to remove rows, or
df.fillna(value) to replace. NaT appears when a datetime column has no value.
Source: Ch.3
Q19. What are the two main categories of Machine Learning?
Answer: Supervised (uses labeled data; solves regression and classification) and
Unsupervised (no labels; solves clustering and association).
Supervised = you train on known input-output pairs. Unsupervised = the algorithm finds patterns on its own
without predefined answers.
Source: Ch.4
Q20. Name three classification algorithms covered in this course.
Answer: k-Nearest Neighbor (k-NN), Support Vector Machine (SVM), Decision Tree.
k-NN uses distance between examples as similarity. SVM finds the hyperplane that best separates groups.
Decision Trees split data by feature values using if/else rules. All are supervised classifiers.
Source: Ch.4
Q21. What are the 6 steps of the Scientific Method as used in model validation?
Answer: 1-Ask a question, 2-Perform research, 3-Form a hypothesis, 4-Test the hypothesis, 5Analyze data/draw conclusion, 6-Communicate results.
This is the same flow used in the exam screenshot question. The FIRST step is always 'ask a question about an
observation' — not forming a hypothesis.
Source: Ch.4
Q22. What are the four types of Validity in research?
Answer: Construct (does it measure what it claims?), Internal (was the experiment designed
correctly?), External (can conclusions apply elsewhere?), Conclusion (are conclusions
reasonable from the data?).
Validity = are you measuring the right thing, in the right way, and drawing the right conclusions? All four must be
considered when evaluating any ML model or experiment.
Source: Ch.4
Q23. What are the three types of measurement error?
Answer: Gross (mistakes in instrument/recording), Random (random factors → normal
distribution around mean), Systematic (environmental/instrument shift → shifts distribution in
one direction, harder to fix).
Random errors average out over many measurements (they form a bell curve). Systematic errors are dangerous
because they consistently push results in the wrong direction and are harder to detect.
Source: Ch.4
Q24. What is the Bias-Variance tradeoff in machine learning?
Big Data in IoT — Exam Prep | Page 4 of 15
Answer: Bias = error from wrong model assumptions. Variance = error from sensitivity to small
data fluctuations. You cannot minimize both simultaneously.
High bias = underfitting (too simple model). High variance = overfitting (too complex model). The goal is to find
the sweet spot in between — often controlled with max_depth in Decision Trees.
Source: Ch.4
Q25. What is a Data Anomaly?
Answer: A value that deviates far from the expected values — an outlier, frequently removed
from data sets.
In the Anomaly Detection lab, Euclidean distance from the mean is used to identify anomalies. Points beyond a
decision boundary threshold are flagged. The decision boundary appears as a sphere in 3D feature space.
Source: Ch.4 / Lab
Q26. When should you use each of the 5 main chart types?
Answer: Line = trend over time; Column = compare values across categories; Bar = long
category names; Pie = composition (totals 100%); Scatter = correlation or distribution/outliers.
Column and Bar charts: avoid more than 7 categories; always start axis at zero; column spacing ≈ half column
width. Pie: segments total exactly 100%. Scatter: useful for spotting clustering and anomalies.
Source: Ch.5
Q27. What is the difference between Deductive and Inductive reasoning?
Answer: Deductive = top-down, general-to-specific, uses facts/premises to guarantee a
conclusion. Inductive = bottom-up, specific-to-general, creates conclusions from
observations/patterns (can be wrong).
Deductive example: 'All mammals have eyes. Humans are mammals. Therefore, humans have eyes.' Inductive
example: sampling a population and concluding something about all similar populations.
Source: Ch.5
Q28. What is Folium, and what JavaScript library does it use?
Answer: Folium is a Python library that combines Python scripts with the mapping abilities of
Leaflet.js to create interactive geographic maps in Jupyter Notebooks.
Used in the SF Crime lab to display crime incidents on an interactive map with CircleMarkers. A 'tileset' is the
underlying map image data (raster or vector).
Source: Ch.5 / Lab
Q29. What is Fog Computing?
Answer: Processing data near the source (on a gateway device) instead of sending everything
to the cloud — used when more sensors are added or shorter response time is needed.
Architecture: Device → Gateway → Network → Cloud. The gateway processes data locally, reducing latency.
Contrast with pure Device → Network → Cloud (all processing in cloud).
Source: Ch.6
Q30. What are the three main cloud service models?
Answer: SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a
Service).
Big Data in IoT — Exam Prep | Page 5 of 15
SaaS = use the app (e.g. Gmail). PaaS = use the platform to build apps (e.g. Google App Engine). IaaS = rent
raw infrastructure — compute, storage, networking (e.g. AWS EC2).
Source: Ch.6
Q31. What are the three parts of the Big Data Pipeline?
Answer: 1-Data Ingestion (Kafka), 2-Data Storage (Cassandra), 3-Compute/Processing (Spark).
Kafka ingests real-time streaming data using transaction logs. Cassandra is a NoSQL distributed DB using CFS
(NOT master-slave). Spark is the processing engine, running on Hadoop using HDFS + YARN.
Source: Ch.6
Q32. What is Lambda Architecture?
Answer: A data processing architecture that uses BOTH stream processing (real-time) AND
batch processing to maintain accurate views of live and historical data.
Stream processing handles fresh, fast data. Batch processing reprocesses all historical data periodically.
Combining both gives you speed AND accuracy.
Source: Ch.6
Q33. What does Virtualization do, and what is a Hypervisor?
Answer: Virtualization separates the OS from the hardware. A Hypervisor is software that
creates and runs virtual machine (VM) instances.
Types: Storage virtualization (combines multiple physical storage devices into one virtual device), Network
virtualization (virtual networks within a virtualized infrastructure), Containers (specialized virtual areas).
Source: Ch.6
Q34. A data analyst processes a dataset with pandas and notices a NaT. Which data type is
expected for the missing data?
Answer: B. timestamp (datetime64).
NaT = Not a Time. It is the missing-value marker for datetime columns in pandas, equivalent to NaN for numeric
columns. It appears when a timestamp value is missing or undefined.
Source: Ch.3 / Screenshot
Q35. Which type of learning algorithm can predict the value of a loan interest rate based on
other variables?
Answer: B. Regression — predicts a continuous numeric value.
Regression finds a mathematical relationship between independent variables (inputs) and a dependent variable
(output). Classification predicts categories; Clustering groups unlabelled data; Association finds co-occurrence
rules.
Source: Ch.4 / Screenshot
PART 2 — Practical Python Cheat Sheet
HINT
The teacher said: import libraries · load CSV · remove columns · create a diagram. Your friend
added: connect libraries, .csv file, remove columns, create a diagram. This covers ALL of that
and more.
Big Data in IoT — Exam Prep | Page 6 of 15
STEP 1 — Import Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sqlite3
%matplotlib inline
# arrays, math, polyfit
# DataFrames, CSV I/O
# all plotting
# heatmaps
# SQL database
# show plots in notebook
# ML libraries (used in later labs):
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
from scipy.optimize import curve_fit
from mpl_toolkits.mplot3d import Axes3D
STEP 2 — Load & Inspect Data
df = pd.read_csv('file.csv')
# load CSV
df = pd.read_csv('./data/file.csv', names=['col1','col2'])
df.head()
df.tail()
df.head(10)
display(df.head())
df.shape
df.columns
df.dtypes
df.info()
df.describe()
len(df)
# custom headers
# first 5 rows
# last 5 rows
# first 10 rows
# nicer Jupyter output
# (rows, columns) tuple
# list of column names
# data type of each column
# dtypes + non-null count (spot NaN quickly)
# count/mean/std/min/max for all numeric cols
# number of rows
STEP 3 — Clean Data
# Drop columns (axis=1 = columns, axis=0 = rows)
df = df.drop(['col1','col2'], axis=1)
df.drop('col', axis=1, inplace=True)
# modify in-place
del df['col']
# delete one column
df = df.drop(columns=['Unnamed: 0'])
# remove auto-index artifact
# Handle NaN / NaT
df.isnull().sum()
# NaN count per column
df.isnull().sum().sum()
# total NaN in entire DataFrame
df = df.dropna()
# remove any row with NaN
df['Age'].fillna(df['Age'].mean(), inplace=True) # fill with mean
df.fillna(0, inplace=True)
# fill all NaN with 0
# Rename columns
df = df.rename(columns={'Old Name (unit)': 'new_name',
'Download (Mbit/s)': 'download'})
# Change type of a column
df['col'] = df['col'].apply(lambda x: float(x))
# Reorder columns
df = df.reindex(columns=['col_b','col_a','col_c'])
# Save clean data
df.to_csv('clean.csv', index=False)
Big Data in IoT — Exam Prep | Page 7 of 15
STEP 4 — Add & Transform Columns (Feature Engineering)
# Arithmetic on existing columns
df['ping_rate'] = 1000.0 / df['Ping (ms)']
df['speed_kbps'] = df['Mbit/s'] * 1000
# ms → 1/second
# Extract part of a date/time string
df['Month'] = df['Date'].apply(lambda row: int(row[0:2]))
df['Day']
= df['Date'].apply(lambda row: int(row[3:5]))
# Parse datetime
df['date_only'] = df['Datetime'].apply(lambda s: pd.to_datetime(s).date())
# Encode categorical → numeric (for ML)
df['Gender'] = df['Gender'].apply(lambda v: 0 if v == 'male' else 1)
# Compute Euclidean distance from mean (Anomaly Detection)
mu = df.mean()
df['distance'] = np.sqrt(np.square(df - mu).sum(axis=1))
STEP 5 — Filter, Slice & Query
# Boolean filter
df[df['Month'] == 8]
df[df['Category'] == 'BURGLARY']
df[(df['x'] > 10) & (df['y'] < 50)]
df[(df['a'] == 1) | (df['b'] == 2)]
# AND
# OR
# .query() — cleaner syntax for complex conditions
df.query('Month == 7 and Day == 4')
# Count unique value occurrences
df['Category'].value_counts()
# sorted descending
df['Category'].value_counts(ascending=True)
# Unique values
np.unique(df['column'])
df['column'].unique()
# Access by position / label
df.iloc[0]
# first row by position
df.iloc[0:5]
# rows 0-4
df.loc[idx, 'col']
# by index label and column name
STEP 6 — Statistics & Correlation
# Basic stats
df.describe()
df['col'].mean()
/ df.mean()
df['col'].median()
df['col'].std()
df['col'].min() / df['col'].max()
df['col'].sum() / df['col'].count()
# all stats at once
# mean of one / all cols
# median
# standard deviation
# min / max
# sum / count
# Find index of min/max
idx = df['col'].argmin()
df.iloc[idx]
# position of minimum
# get that row
Big Data in IoT — Exam Prep | Page 8 of 15
# Frequency distribution
freq = df['col'].value_counts().reset_index()
freq.columns = ['value', 'count']
# Pearson correlation matrix
df.corr(method='pearson', numeric_only=True)
sns.heatmap(df.corr(numeric_only=True))
# visualize as heatmap
plt.show()
STEP 7 — Visualize Data
# ── HISTOGRAM ─────────────────────────────────────────
plt.figure(figsize=(8,5))
plt.hist(df['col'], bins=25)
plt.xlabel('Label', fontsize=14)
plt.ylabel('Frequency', fontsize=14)
plt.title('Title')
plt.show()
# ── LINE PLOT ──────────────────────────────────────────
plt.figure(figsize=(8,5))
plt.plot(x, y, linewidth=2)
plt.xlabel('X'); plt.ylabel('Y')
plt.show()
# ── SCATTER PLOT ───────────────────────────────────────
plt.scatter(df['x'], df['y'])
plt.scatter(df['X'], df['Y'], c=df['code']) # color-coded
# ── HEATMAP (correlation) ──────────────────────────────
import seaborn as sns
sns.heatmap(df.corr(method='pearson', numeric_only=True))
plt.show()
# ── 2×2 SUBPLOT GRID ───────────────────────────────────
fig, ax = plt.subplots(2, 2, figsize=(10,10))
ax[0][0].hist(df['col1'], bins=20)
ax[0][1].hist(df['col2'], bins=20)
ax[1][0].plot(x, y)
ax[1][1].scatter(df['x'], df['y'])
plt.tight_layout(); plt.show()
# ── REGRESSION LINE ────────────────────────────────────
m, b = np.polyfit(x, y, 1)
# slope and intercept
plt.scatter(x, y)
plt.plot(x, m*x+b, 'r-', linewidth=2) # regression line
plt.plot(x.mean(), y.mean(), 'k*', markersize=14) # centroid
plt.show()
STEP 8 — SQL with SQLite
import sqlite3
# Connect (creates file if not exists)
conn = sqlite3.connect('mydb.db')
cur = conn.cursor()
# Write DataFrame to SQL table
df.to_sql('table_name', conn, if_exists='replace', index=False)
# Execute queries
cur.execute('SELECT * FROM table_name LIMIT 10')
Big Data in IoT — Exam Prep | Page 9 of 15
rows = cur.fetchall()
row = cur.fetchone()
# list of tuples
# single tuple
# Common queries from lab:
cur.execute('SELECT AVG(Download) FROM LA_wifi_speed_UK')
cur.execute('SELECT * FROM speed JOIN pop ON speed.code = pop.code')
cur.execute('SELECT * FROM table ORDER BY col LIMIT 10')
# Read result into DataFrame
df_result = pd.read_sql('SELECT * FROM table_name', conn)
# Add a row (pandas 2.0+ — df.append() is deprecated!)
new_row = pd.DataFrame([{'col1': v1, 'col2': v2}])
df = pd.concat([df, new_row], ignore_index=True)
conn.commit()
conn.close()
STEP 9 — Machine Learning
# ── LINEAR REGRESSION (NumPy) ──────────────────────────
m, b = np.polyfit(x, y, 1)
# slope, intercept
y_pred = m * x + b
# ── POLYNOMIAL REGRESSION ──────────────────────────────
p = np.poly1d(np.polyfit(x, y, 2)) # degree 2 (quadratic)
y_pred = p(x)
# evaluate
# ── EXPONENTIAL FIT (scipy) ────────────────────────────
from scipy.optimize import curve_fit
def my_exp(x, a, b):
return a * (b ** x)
params, cov = curve_fit(my_exp, x, y)
# ── ERROR METRICS ──────────────────────────────────────
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
r2 = r2_score(y, y_pred)
# 1.0 = perfect
mse = mean_squared_error(y, y_pred) # penalizes large errors
mae = mean_absolute_error(y, y_pred) # average absolute error
# ── DECISION TREE CLASSIFICATION ───────────────────────
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
columns = ['Fare','Pclass','Gender','Age']
X = df[columns].values
y = df['Survived'].values
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.40, random_state=0)
clf = DecisionTreeClassifier(criterion='entropy', max_depth=3)
clf.fit(X_train, y_train)
print('Accuracy:', clf.score(X_test, y_test))
predictions = clf.predict(X_test)
STEP 10 — Full Exam Template (copy & adapt)
# ── IMPORTS ───────────────────────────────────────────
import numpy as np
import pandas as pd
Big Data in IoT — Exam Prep | Page 10 of 15
import matplotlib.pyplot as plt
%matplotlib inline
# ── LOAD ──────────────────────────────────────────────
df = pd.read_csv('data.csv')
display(df.head())
# ── CLEAN ─────────────────────────────────────────────
df = df.drop(['Unnamed: 0', 'extra_col'], axis=1)
df = df.dropna()
df = df.rename(columns={'Old': 'new'})
# ── FEATURE ───────────────────────────────────────────
df['new_feature'] = df['col'] * 2
# ── STATS ─────────────────────────────────────────────
print(df.describe())
print('Mean:', df['col'].mean())
# ── PLOT ──────────────────────────────────────────────
plt.figure(figsize=(8,5))
plt.hist(df['col'], bins=20)
plt.xlabel('Label', fontsize=14)
plt.ylabel('Frequency', fontsize=14)
plt.title('Distribution')
plt.show()
PART 3 — Key Terminology (60 terms)
Term / Concept
Definition
4 Vs of Big Data
Volume, Velocity, Variety, Veracity
Big Data (def.)
Data so vast/fast/complex traditional tools cannot handle it
Open Data
Free to use, reuse, redistribute — no legal/tech/social restriction
Structured Data
Fixed fields in a database (easily queried and analyzed)
Unstructured Data
No fixed schema — photos, audio, video, emails, blogs
Data at Rest
Static stored data — flow: Store→Analyze→Notify→Act
Data in Motion
Real-time data — flow: Analyze→Act→Notify→Store
ETL
Extract, Transform, Load — collect, clean, then load into DB
DIKW
Data→Information→Knowledge→Wisdom (= Business Intelligence)
Descriptive Analytics
Relies on historical data to report what happened
Predictive Analytics
Uses past data to forecast what might happen
Prescriptive Analytics
Recommends actions that will produce the greatest benefit
Population
Group of similar entities sharing common characteristics
Big Data in IoT — Exam Prep | Page 11 of 15
Sample
A representative subset of a population
Descriptive Stats
Summarize the values of a dataset (mean, std, min, max, etc.)
Inferential Stats
Use a sample to generalize or predict about a whole population
Distribution
Variable vs. its frequency or probability
Centrality
Mean, median, and mode
Dispersion
Variability in the distribution (std dev, range, variance)
Nominal
Categorical variable with no inherent order (e.g. country names)
Ordinal
Categorical variable with ordered categories (e.g. shirt sizes S/M/L)
Continuous
Numeric variable along a continuum (e.g. temperature)
Discrete
Numeric variable with specific values from a finite set
NaN
Not a Number — missing/undefined numeric value in pandas
NaT
Not a Time — missing datetime/timestamp value in pandas
Correlation
Two things change at similar rates (does NOT imply causation!)
Causation
One thing directly causes a change in another
Supervised ML
Learns from labeled data; solves regression + classification
Unsupervised ML
Finds patterns without labels; solves clustering + association
Regression
Predicts a continuous numeric output (e.g. price, speed, rate)
Classification
Predicts a discrete category label (e.g. survived: yes/no)
Decision Tree
ML model using if/else splits on feature values (criterion=entropy)
k-NN
k-Nearest Neighbor — uses distance between examples as similarity
SVM
Support Vector Machine — finds best hyperplane separating classes
train_test_split
Splits data into training and test sets to evaluate on unseen data
np.polyfit(x,y,1)
Fits a linear (degree-1) regression; returns [slope, intercept]
np.poly1d(coeffs)
Wraps polynomial coefficients into a callable function p(x)
R²
Coefficient of determination — 1.0 = perfect fit
MSE
Mean Squared Error — penalizes large errors more than small ones
MAE
Mean Absolute Error — average magnitude, robust to outliers
Bias
Error from wrong model assumptions (wrong choice of model)
Big Data in IoT — Exam Prep | Page 12 of 15
Variance
Error from sensitivity to small fluctuations in training data
Anomaly / Outlier
Value that deviates far from expected — often removed or flagged
Euclidean Distance
Straight-line distance in n-dimensional space
Deductive Reasoning
Top-down: general premises → specific guaranteed conclusion
Inductive Reasoning
Bottom-up: specific observations → general conclusion (can be wrong)
Formal Fallacy
One or more premises are false
Informal Fallacy
Premises do not adequately support the conclusion
Line chart
Shows trends over time (time on x-axis)
Column chart
Compares values across similar categories (max 7 categories)
Bar chart
Like column but horizontal — used when category names are long
Pie chart
Shows composition of a whole (segments total 100%)
Scatter plot
Shows correlation or distribution; identifies clusters/outliers
Folium
Python library for interactive maps using Leaflet.js
Tileset
Raster/vector data that displays a map on screen
Pyplot
Matplotlib module with style functions for creating/customizing plots
Plotly
Online tool to generate shareable interactive visualizations
SQLite
Most widely deployed database — simple embedded SQL engine
HDFS
Hadoop Distributed File System — redundant, distributed storage
MapReduce
Distributed processing framework — parallelizes algorithms at scale
Kafka
Streaming platform for real-time data ingestion (uses transaction logs)
Cassandra
Open-source NoSQL distributed DB using CFS — NOT master-slave
Spark
Open-source distributed processing engine; runs on Hadoop+YARN
Lambda Architecture
Uses BOTH stream processing AND batch processing
Fog Computing
Data processed near source at gateway, not in cloud
Virtualization
Separates OS from hardware
Hypervisor
Software that creates and runs virtual machine instances
SaaS / PaaS / IaaS
Software / Platform / Infrastructure as a Service
CIA Triad
Confidentiality, Integrity, Availability — data security guideline
Big Data in IoT — Exam Prep | Page 13 of 15
PART 4 — Important Named Lists to Memorize
List Name
Items
4 Vs of Big Data
Volume · Velocity · Variety · Veracity
3 Types of Analytics
Descriptive (historical) · Predictive (forecast) · Prescriptive
(recommend action)
6-Step Data Analysis Lifecycle
Gather → Prepare → Choose model → Analyze → Present results
→ Make decisions
6-Step Scientific Method
Observe → Ask question → Form hypothesis → Test → Analyze →
Communicate
6-Step ML Process
Prepare data → Create learning set → Create test set → Create
loop → Test → Implement
4 Types of Validity
Construct · Internal · External · Conclusion
4 Types of Reliability
Inter-rater · Test-Retest · Parallel-Forms · Internal Consistency
3 Types of Measurement Error
Gross (mistakes) · Random (normal dist.) · Systematic (shifts
distribution)
3 ML Classifier Algorithms
k-NN · SVM (Support Vector Machine) · Decision Tree
3 Main Cloud Service Models
SaaS · PaaS · IaaS
3 Parts of Big Data Pipeline
Ingestion (Kafka) · Storage (Cassandra) · Compute (Spark)
5 Most Popular Chart Types
Line · Column · Bar · Pie · Scatter
3 Python Libraries in Course
NumPy · Pandas · Matplotlib
3 Characteristics of a Sample
Distribution · Centrality (mean/median/mode) · Dispersion
2 Types of Reasoning
Deductive (top-down, general→specific) · Inductive (bottom-up,
specific→general)
2 Types of ML
Supervised (regression/classification) · Unsupervised
(clustering/association)
2 Components of Prediction
Error
Bias (model choice) · Variance (measurement error)
CIA Triad
Confidentiality · Integrity · Availability
3 Types of Variable
(Numerical)
Continuous · Ratio · Discrete
2 Types of Variable
(Categorical)
Nominal · Ordinal
Big Data in IoT — Exam Prep | Page 14 of 15
3 SQL Sub-Languages
Data Definition Language · Data Manipulation Language · Data
Query Language
PART 5 — Lab Topics Quick Reference
Lab Topic
Key Functions & Concepts
SF Crime Analysis
pd.read_csv, value_counts(), boolean filter, lambda on apply, folium map,
CircleMarker
Internet Meter SQL
sqlite3, df.to_sql(), SELECT/AVG/JOIN, cursor.execute(), fetchall()
Descriptive Statistics
mean, median, std, value_counts(), frequency distribution, normal
distribution
Correlation Analysis
df.corr(method='pearson'), seaborn sns.heatmap(), filter by gender,
numeric_only=True
Internet Meter
Visualization
isnull().sum(), dropna(), plt.subplots(2,2), plt.style, to_csv(), argmin/argmax
Simple Linear Regression
np.polyfit(x,y,1) → m,b; predict y=mx+b; centroid at (mean_x, mean_y)
Decision Tree
Classification
DecisionTreeClassifier, train_test_split, fillna(mean), encode gender,
fit/score/predict
Regression Error Metrics
r2_score, mean_squared_error, mean_absolute_error, np.poly1d, orders 13
Internet Traffic
Regression
Polynomial vs exponential fit, scipy curve_fit, MSE bar chart, overfitting
demo
Anomaly Detection
Euclidean distance from mean, normalize 0-1, alarm rate sweep, 3D
scatter+sphere
Data Acquisition (IoT)
speedtest-cli, subprocess.Popen, csv.writer, datetime.now(), split Datetime
column
You have done all the labs. You know this subject. Good luck tomorrow, Orest!
Big Data in IoT — Exam Prep | Page 15 of 15