1
Residency Project 1
Devesh Sushilkumar Bachhawat
Ph.D. in Information Technology, University of Cumberlands
MSDS 631-M30: R Programming
Dr. Bindu George
October 18, 2024
2
Residency Project 1
Introduction
Data quality management is essential in any data analysis to ensure reliable and
meaningful insights. This project investigates the diamonds dataset from the ggplot2 package in
R, focusing on detecting potential outliers, noise, and managing missing values. Specifically, we
examine diamonds with carat values near 3 and 5 to identify whether any of these diamonds are
underpriced relative to their size. By using scatterplots, we explore patterns and potential
anomalies, and we discuss strategies for handling missing values. This analysis aims to
distinguish outliers from noise and apply suitable data handling techniques to maintain data
quality.
The diamonds dataset contains 53,940 observations and 10 variables describing the
attributes of diamonds. Below is a breakdown of the key variables:
carat: Weight of the diamond.
cut: Quality of the cut (Fair, Good, Very Good, Premium, Ideal).
color: Diamond color, with D being the best (colorless) and J being the lowest quality.
clarity: Measure of diamond inclusions (I1, SI1, SI2, VS1, VS2, VVS1, VVS2, IF).
depth: Total depth percentage (z / mean(x, y)).
table: Width of the top of the diamond relative to its widest point.
price: Price in US dollars.
x, y, z: Length, width, and depth measurements in mm.
The summary statistics of the diamond dataset is as follows:
Carat: Range: 0.2 – 5.01 | Median: 0.7
Price: Range: $326 – $18,823 | Median: $2,401
3
Cut: Majority are Ideal and Premium cuts.
Color: Most diamonds are rated G, H, or F in color.
Clarity: Common clarity levels are SI1 and VS2.
Q1) Identifying Cheap Diamonds with 3 and 5 Carats
Diamonds with carat values between 2.95–3.05 and 4.95–5.05 were extracted for
analysis. For these diamonds, we calculated the 1st quartile (25th percentile) of their prices to
detect cheap diamonds that might be potential outliers. These cheap diamonds were flagged for
further investigation to determine if their low price is justified by factors like cut or clarity. The
plot below zooms in on diamonds near 3 and 5 carats, with cheap diamonds highlighted in red.
Q2) Using Interactive Scatterplots to Identify Outliers
4
Visualizing the data through scatterplots helps to detect patterns and anomalies. We
created a scatterplot below for the overall relationship between carat and price.
Q3a. How to Tell If the Data is an Outlier or Important?
Determining whether a data point is an outlier, or an important observation requires
further investigation. In the case of cheap diamonds, if attributes such as cut, clarity, and color
are low, the lower price may be justified, making these diamonds valid observations. However, if
the diamond's attributes do not explain the low price, it might indicate a data entry error or an
anomaly. Domain knowledge about the typical relationship between diamond quality and price
plays a crucial role in making this distinction.
Q3b. Difference Between Noise and Outliers
5
While outliers represent extreme data points that deviate from the norm, they can still
provide meaningful insights if valid. In contrast, noise refers to random variations or errors in
data that obscure patterns. For example, cheap diamonds could either represent meaningful
outliers or be the result of data entry mistakes (noise). Proper handling of both noise and outliers
ensures that meaningful patterns are retained while errors are minimized.
4. Pros and Cons of Handling Missing Values
Managing missing data is crucial to maintain the reliability of analysis. Two common
strategies to handle missing data are elimination of data objects and estimation of missing values
(imputation), each with its advantages and limitations.
The elimination of data objects involves removing records with missing values to ensure
data consistency. This method is straightforward to implement and works well when missing
data is infrequent and random. However, it can reduce the dataset size, potentially weakening the
performance of statistical models and machine learning algorithms. Additionally, if the missing
data is not random, elimination can introduce bias, skewing the results and limiting the
representativeness of the analysis.
On the other hand, estimation of missing values (imputation) aims to preserve the
structure and size of the dataset by filling in the missing values with estimated ones. Imputation
minimizes bias compared to elimination, especially when missing data is more prevalent.
However, imputed values are inherently approximations, which can introduce inaccuracies.
Furthermore, more sophisticated imputation methods, such as multiple imputation, can be
computationally intensive, requiring significant time and resources to implement effectively.
Choosing the right strategy depends on the nature of the dataset and the goals of the analysis.
Q5. Limitations of Analyzing Real-World Data with Missing Values
6
Real-world datasets often contain missing values and other imperfections, making it
impossible to fully replicate the original data. Missing data introduces bias and uncertainty,
especially when the missingness is not random. Even with advanced imputation methods, the
results are only approximations, leading to a loss of accuracy. These limitations highlight the
need for careful data management practices to mitigate the impact of missing values.
Appendix (R code)
# Install and load necessary packages
install.packages(c("ggplot2", "plotly", "dplyr"))
library(ggplot2)
library(plotly)
library(dplyr)
# Load the diamonds dataset
data("diamonds")
# Filter diamonds with carat close to 3 or 5
target_diamonds <- diamonds %>%
filter((carat >= 2.95 & carat <= 3.05) | (carat >= 4.95 & carat <= 5.05))
# Calculate the 1st quartile price
price_threshold <- quantile(target_diamonds$price, 0.25)
# Identify cheap diamonds below the 1st quartile
cheap_diamonds <- target_diamonds %>% filter(price < price_threshold)
# Display the cheap diamonds identified
print(cheap_diamonds)
# Create overall scatterplot of carat vs price
7
overall_plot <- ggplot(diamonds, aes(x = carat, y = price)) + geom_point(alpha = 0.5) +
ggtitle("Overall Scatterplot of Carat vs Price") + xlab("Carat") + ylab("Price")
# Convert to interactive plot
ggplotly(overall_plot)
8
References
Grolemund, G., & Wickham, H. (2020). R for Data Science: Import, Tidy, Transform,
Visualize, and Model Data. O’Reilly Media.