BD201_Big data and business analy4c
Module 1:
What is Data Analysis?**
- Defini4on: Data analysis is the process of inspec4ng, transforming, and modeling data to
discover useful informa4on and support decision-making.
- Example: Imagine you have survey data about favorite ice cream flavors. Data analysis helps
you figure out which flavor is the most popular.
Roles of a Junior Data Analyst**
Discuss the day-to-day ac4vi4es and responsibili4es of a junior data analyst.
- Example: A junior data analyst might clean and organize data, create charts to show trends,
and help make recommenda4ons based on the data.
Key Analy4cal Skills**
- Data Cleaning: Removing errors and inconsistencies from datasets.
- Data Analysis: Extrac4ng insights and paTerns from data.
- Data Visualiza4on: Presen4ng data in a visual format for easy understanding.
Example: If you have a list of temperatures, data cleaning means making sure they're all in the
same unit (e.g., Celsius), analysis might involve finding the average temperature, and
visualiza4on could be a graph showing temperature changes over 4me.
- **Data Cleaning Example:**
- Scenario: Imagine you have a list of students' test scores, and some scores are entered
incorrectly (e.g., 150 instead of 15).
- Solu4on: Data cleaning involves reviewing the scores and correc4ng any errors to ensure
accurate analysis.
**Exercise 1: Analyzing Data**
- **Exercise:** Given a set of test scores (20, 18, 22, 25, 150), find the average and create a bar
chart to represent the data.
**Exercise 1: Analyzing Data**
- **Exercise:** Given a set of test scores (20, 18, 22, 25, 150), find the average and create a bar
chart to represent the data.
- **Solu4on:**
- Average = (20 + 18 + 22 + 25 + 150) / 5 = 235 / 5 = 47. - Bar Chart: [Visualiza4on of the scores]
Tools for Data Analysis**
Spreadsheets (like Excel), SQL (for databases), R programming (for advanced analysis), Tableau
(for visualiza4on).
- Example: Excel is great for organizing and performing calcula4ons on data, while SQL helps
extract specific informa4on from large databases.
- **Explana4on:**
- Spreadsheets (like Excel) allow you to organize and analyze data using formulas and
func4ons.
- SQL (Structured Query Language) is used to extract specific informa4on from
databases.
-
- R programming is a powerful tool for advanced data analysis and sta4s4cal modeling.
- Tableau is a data visualiza4on tool that helps create interac4ve and informa4ve charts
and graphs.
Data Life Cycle**
**Collec4on:** Gathering data from various sources (e.g., surveys, sensors).
**Cleaning:** Reviewing and correc4ng data for accuracy and consistency.
**Analysis:** Extrac4ng insights and paTerns from cleaned data.
**Visualiza4on:** Crea4ng visual representa4ons (charts, graphs) for easier understanding.
**Interpreta4on:** Making sense of the results and drawing conclusions.
Analy4cal Thinking**
- Defini4on: The ability to solve problems by breaking them down into smaller parts and
iden4fying paTerns.
Data analysis is a crucial part of the data-driven decision-making process. It helps businesses
and organiza4ons make informed choices based on evidence from their data.
- **Example:**
- Imagine a retail store using data to decide which products to stock more of, based on customer
buying paTerns.
Job Opportuni4es**
•- Entry-level data analysts can work in various industries, such as marke4ng, finance,
healthcare, and sports, helping organiza4ons make data-driven decisions.
Effec4ve Ques4oning Techniques**
- Explain the importance of asking the right ques4ons in data analysis.
- **Example 1:** In a retail store, asking "Which products sell best during holiday seasons?" can
guide marke4ng strategies.
**Exercise 1: Asking the Right Ques4ons**
- **Scenario:** You run a small bakery. What ques4ons would you ask to improve sales?
- **Sample Ques4ons:**
- "What are our best-selling pastries?"
- "Which items are popular during weekends?"
Data-Driven Decision-Making**
- Define data-driven decision-making and its impact on businesses.
**Exercise 2: Case Study Analysis**
- **Case Study:** A tech company uses customer feedback to modify a sonware feature.
Analyze the data and explain the impact on customer sa4sfac4on.
- **Solu4on:** [Student's analysis]
Presen4ng Findings**
-data analysts present their findings effec4vely.
- **Example 3:** Comparing two presenta4ons: one with clear visuals and another with
cluTered slides.
Real-World Business Scenarios**
- Introduce various business scenarios where effec4ve ques4oning and decision-making are
crucial.
- **Example 4:** A retail store deciding which products to discount based on historical sales
data.
**Exercise 4: Scenario Analysis**
- **Scenario 1:** A hotel manager aims to improve guest experience. What ques4ons should
they ask about customer feedback data?
- **Solu4on:** Ques4ons could include: "What are the most common complaints?" and "What
improvements are suggested?"
- **Scenario 2:** A city council is planning public transporta4on routes. What data would be
important for decision-making?
- **Solu4on:** Data on popula4on density, traffic paTerns, and popular des4na4ons.
Importance of Spreadsheets**
- Explain why spreadsheets are essen4al tools for data analysts.
- **Example 5:** Using Excel to track monthly expenses and analyze spending paTerns.
Structured Thinking for Problem-Solving**
- Define structured thinking and its role in developing solu4ons.
Managing Stakeholder Expecta4ons**
- Discuss strategies for effec4ve communica4on with stakeholders.
- **Example 7:** A data analyst successfully communicates complex findings to a non-technical
audience.
1. Effec4ve Ques4oning Techniques:
- Importance of asking the right ques4ons in data analysis.
- Example: In a retail store, asking about best-selling products during holidays.
2. Data-Driven Decision-Making:
- Defini4on and impact on businesses.
- Example: Using website traffic data to launch a product.
3. Presen4ng Findings:
- Importance of clear and effec4ve communica4on.
- Example: Comparing two presenta4ons with different visual clarity.
4. Real-World Business Scenarios:
- Various scenarios emphasizing effec4ve ques4oning and decision-making.
- Example: Discoun4ng products based on sales data.
5. Importance of Spreadsheets:
- Spreadsheets as essen4al tools for data analysis.
- Example: Using Excel to track expenses and analyze spending paTerns.
6. Structured Thinking for Problem-Solving:
- Defini4on and applica4on in developing solu4ons.
- Example: Structured approach to troubleshoo4ng a computer issue.
7. Managing Stakeholder Expecta4ons:
- Strategies for effec4ve communica4on with stakeholders.
- Example: Successfully communica4ng complex findings to non-technical audience.
**Example:** Consider a manufacturing company facing a decline in product quality. Instead of
approaching the issue as a single, overwhelming problem, structured thinking would involve:
Identifying Key Factors:** Determine factors affecting product quality, such as raw materials, production processes, and quality
control measures.
Analyzing Each Factor:** Break down each identified factor further. For instance, under 'production
processes,' consider sub-components like machine calibration, operator training, and maintenance protocols.
Evaluating Relationships:** Assess how these factors interrelate. For example, poor machine calibration might impact product quality,
which is related to operator training and maintenance practices.
Developing Action Steps:** Based on the analysis, devise specific actions to address each factor. This could involve revising
training protocols, implementing stricter quality control checks, or upgrading machinery.
Monitoring Progress:** Implement the proposed solutions and continually monitor their impact on product
quality. Adjust strategies as needed based on ongoing evaluations.
This structured approach ensures that the problem is addressed comprehensively,
with targeted solutions for each contributing factor.
Structured thinking provides a strategic framework for problem-solving, enhancing the effectiveness and efficiency of decisionmaking processes.
Managing Stakeholder Expectations**
Common strategies for effective communication with stakeholders.
1. Active Listening: Actively engage with stakeholders to understand their concerns, perspectives, and expectations. This demonstrates empathy and ensures that their input is
valued.
2. Clear and Concise Messaging: Communicate information in a clear, straightforward manner, avoiding jargon or technical language that may be unfamiliar to stakeholders.
3. Tailored Communication: Customize your message to suit the specific needs, interests, and preferences of different stakeholders. This ensures that the information is relevant
and resonates with them.
4. Regular Updates: Provide consistent and timely updates on project progress, milestones, and any potential challenges. This maintains transparency and keeps stakeholders
informed.
5. Visual Aids and Presentations: Use visual tools such as charts, graphs, and presentations to illustrate data and key points. Visual aids enhance comprehension and retention of
information.
6. Addressing Concerns Proactively: Anticipate potential concerns or questions that stakeholders may have and address them proactively in your communication.
7. Feedback Mechanisms: Establish channels for stakeholders to provide feedback or ask questions. This fosters a collaborative and open communication environment.
8. Setting Realistic Expectations: Be transparent about what can be achieved within a given timeframe and resources. Avoid over-promising and under-delivering.
9. Crisis Communication Plan: Have a plan in place for communicating in high-pressure or crisis situations. This ensures a coordinated and effective response.
10. Engagement Platforms: Utilize various communication channels such as meetings, emails, presentations, and collaboration tools to reach stakeholders through their preferred
means.
Big data Definition
• Big data refers to extremely large and complex sets of data that cannot be easily
managed, processed, or analyzed with traditional data processing tools or techniques. It
is characterized by what is often referred to as the "3Vs":
1.Volume: Big data involves vast amounts of data. This could range from terabytes to
petabytes or even exabytes of information.
2.Velocity: Big data is generated and collected at an extremely high speed. This could be
from various sources such as social media interactions, sensors, or online transactions.
3.Variety: Big data comes in various forms. It can be structured (like data in databases),
semi-structured (like XML or JSON files), or unstructured (like text documents, images,
videos, etc.).
Structured Data:
• Definition: Structured data refers to data that is highly organized and follows a specific format or schema. It fits neatly
into tables or databases with well-defined rows and columns.
• Examples:
• Relational Database Records: Information stored in SQL databases, where each entry is organized into rows and
columns. For example, an employee database with fields like Name, Age, Department, etc.
• Spreadsheets: Excel files with clear headers and rows of data, where each column represents a specific attribute.
• CSV (Comma-Separated Values) Files: Data stored in a tabular format where each value is separated by commas.
• Use Cases:
• Financial Records: Transaction data in banking or accounting systems.
• Inventory Management: Product details stored in a database.
• Customer Relationship Management (CRM): Customer profiles and interactions
Semi-Structured Data:
• Definition: Semi-structured data does not adhere to a fixed structure, but it does
have some level of organization. It often contains tags, attributes, or markers that
provide some level of hierarchy.
• Examples:
• JSON (JavaScript Object Notation) Files: Data format that organizes information
using key-value pairs, allowing for nested structures. It is commonly used in web
applications and APIs.
• XML (Extensible Markup Language) Files: Similar to JSON, XML uses tags to
define elements and their relationships, making it easier to structure data
hierarchically.
• NoSQL Databases: Document-oriented databases like MongoDB or CouchDB that
store data in semi-structured formats.
• Use Cases:
• Web APIs: Data from web services, like social media feeds, often use semistructured formats.
• Log Files: Server logs may contain structured information but with some
variability.
Unstructured Data:
• Definition: Unstructured data lacks a specific format or organization. It doesn't fit neatly into traditional
databases or tables and can include a wide variety of content types.
• Examples:
• Text Documents: Articles, emails, social media posts, and other textual content without a predefined
structure.
• Images and Videos: Media files that contain vast amounts of information but lack inherent organization.
• Audio Recordings: Speech, music, or other audio files that do not follow a standardized structure.
• Use Cases:
• Social Media Feeds: User-generated content in the form of posts, comments, images, etc.
• Multimedia Archives: Collections of images, videos, and audio files.
• Sensor Data: Readings from IoT devices, which can include text, images, and other data.
Additional Note:
Mixed Data Types: In practice, many
datasets contain a mix of structured,
semi-structured, and unstructured
data. For instance, a social media
platform may have structured user
profiles, semi-structured posts with
tags, and unstructured comments.
Processing Challenges: Analyzing and
extracting insights from semistructured and unstructured data can
be more challenging than structured
data. Advanced techniques like
natural language processing (NLP) and
machine learning are often used.
**
Correla'on Analysis**:
correla4on_matrix = df.corr()
import seaborn as sns
import matplotlib.pyplot as plt
sns.heatmap(correla4on_matrix, annot=True)
plt.show()
or
import pandas as pd
print(df.corr())
Differences:
•
Code 1 uses Seaborn to create a visual representa4on of the correla4on matrix using a
heatmap. This provides an easy-to-interpret graphical overview of the correla4ons.
•
Code 2 directly prints the correla4on matrix as a table in the console. While it gives you
the same informa4on, it's presented in a tabular form without any visual representa4on.
In summary, Code 1 offers a visual representa4on of correla4ons using a heatmap, which can be
more intui4ve for iden4fying paTerns, while Code 2 provides the raw correla4on coefficients in
a tabular format for easy access to numerical values. The choice between them depends on
your preference for visualiza4on or raw data presenta4on.
### Types of Correla.on:
Correla4on measures the strength and direc4on of a linear rela4onship between two variables.
Here are the types of correla4on:
1. **Perfect Posi1ve Correla1on (r = 1)**:
- When two variables have a perfect posi4ve correla4on, it means that they have a linear
rela4onship where the change in one variable is directly propor4onal to the change in the other.
- Example: The rela4onship between height in inches and height in cen4meters.
2. **Strong Posi1ve Correla1on (0.8 < r < 1)**:
- This indicates a strong posi4ve linear rela4onship. When one variable goes up, the other
tends to also go up, but it may not be perfectly in sync.
- Example: The rela4onship between hours of study and exam scores.
3. **Weak Posi1ve Correla1on (0 < r < 0.5)**:
- In this case, there's a posi4ve correla4on, but it's not very strong. The variables tend to
increase together, but the rela4onship isn't as clear-cut.
- Example: The rela4onship between the number of years of experience and salary.
4. **No Correla1on (r = 0)**:
- This means that there's no linear rela4onship between the variables. Changes in one variable
do not predict changes in the other.
- Example: The rela4onship between shoe size and favorite color.
5. **Weak Nega1ve Correla1on (-0.5 < r < 0)**:
- In this case, there's a nega4ve correla4on, but it's not very strong. As one variable goes up,
the other tends to go down, but the rela4onship isn't highly predictable.
- Example: The rela4onship between temperature and ice cream sales.
6. **Strong Nega1ve Correla1on (-1 < r < -0.8)**:
- This indicates a strong nega4ve linear rela4onship. As one variable goes up, the other tends
to go down, and vice versa.
- Example: The rela4onship between hours spent watching TV and exercise levels.
7. **Perfect Nega1ve Correla1on (r = -1)**:
- When two variables have a perfect nega4ve correla4on, it means that they have a linear
rela4onship where one variable always decreases when the other increases.
- Example: The rela4onship between temperature in Celsius and temperature in Fahrenheit.
Understanding the type of correla3on between variables is important
for interpre3ng the rela3onship and making accurate predic3ons or
decisions based on the data.
**Data Visualiza'on**:
Imagine you're plotng a map for your family's treasure hunt, where you want to mark the
star4ng point and the final des4na4on. Similarly, this code creates a scaTer plot to visualize the
rela4onship between two numerical aTributes (`'Numeric_Column_1'` and
`'Numeric_Column_2'`). ScaTer plots are great for visually inspec4ng rela4onships.
### 1. Line Plot:
A line plot is used to visualize the rela4onship between two con4nuous variables. It's useful for
showing trends or paTerns over 4me.
import matplotlib.pyplot as plt
# Example usage:
plt.plot(df['Column_1'], df['Column_2'])
plt.xlabel('Column_1')
plt.ylabel('Column_2')
plt.4tle('Line Plot')
plt.show()
### 2. ScaKer Plot:
A scaTer plot is used to visualize the rela4onship between two con4nuous variables. It displays
individual data points on a two-dimensional graph.
import matplotlib.pyplot as plt
# Example usage:
plt.scaTer(df['Column_1'], df['Column_2'])
plt.xlabel('Column_1')
plt.ylabel('Column_2')
plt.4tle('ScaTer Plot')
plt.show()
### 3. Histogram:
A histogram is used to visualize the distribu4on of a single numerical variable. It displays the
frequency of data points within specified intervals (bins).
import matplotlib.pyplot as plt
# Example usage:
plt.hist(df['Numeric_Column'], bins=20)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.4tle('Histogram')
plt.show()
### 4. Bar Plot:
A bar plot is used to compare the frequency or value of different categories or groups. It's
suitable for categorical variables.
import matplotlib.pyplot as plt
# Example usage:
plt.bar(df['Category'], df['Value'])
plt.xlabel('Category')
plt.ylabel('Value')
plt.4tle('Bar Plot')
plt.show()
### 5. Box Plot:
A box plot (or whisker plot) is used to display the distribu4on of a numerical variable, showing
the median, quar4les, and poten4al outliers.
import matplotlib.pyplot as plt
# Example usage:
plt.boxplot(df['Numeric_Column'])
plt.xlabel('Column')
plt.ylabel('Value')
plt.4tle('Box Plot')
plt.show()
### 6. Pie Chart:
A pie chart is used to represent the distribu4on of a categorical variable as a "pie" divided into
slices.
import matplotlib.pyplot as plt
# Example usage:
plt.pie(df['Category'].value_counts(), labels=df['Category'].value_counts().index,
autopct='%1.1f%%')
plt.4tle('Pie Chart')
plt.show()
### Differences:
- Line plots and scaTer plots are used to visualize rela4onships between two con4nuous
variables. Line plots connect the data points, while scaTer plots do not.
- Histograms are used to visualize the distribu4on of a single numerical variable, while bar plots
are used to compare categories or groups.
- Box plots provide informa4on about the distribu4on of a variable, including median, quar4les,
and poten4al outliers.
- Pie charts display the distribu4on of a categorical variable as a "pie" divided into slices,
showing the propor4ons of each category.
Replace `'Column_1'`, `'Column_2'`, `'Numeric_Column'`, `'Category'`, and other placeholders
with the actual column names from your dataset. These examples should help you create
visualiza1ons for your exis1ng dataset.
Data Analysis
Import Pandas library
[]
import pandas as pd
Load the data frame
[]
df = pd.read_csv('/content/Car_Sales.csv')
#df = pd.read_json('data.json')
[]
df
[]
df.corr()
[]
correlation_matrix = df.corr()
import seaborn as sns
import matplotlib.pyplot as plt
sns.heatmap(correlation_matrix, annot=True)
plt.show()
output
Return one or more specified row(s)
[]
print(df.loc[1])
Return row 0 and 1
[]
print(df.loc[[0,1,7]])
Pandas the maximum numbers of rows returned
[]
print(pd.options.display.max_rows)
[]
print(pd.options.display.max_columns)
Increase the maximum number of rows to display the
entire DataFrame
[]
pd.options.display.max_rows = 9999
dfmax = pd.read_csv('/content/Car_sales.csv')
print(dfmax)
Viewing the Data
[]
df
Get a quick overview by printing the first 10 rows of the
DataFrame:
[]
print(df.head())
[]
print(df.head(10))
The tail() method returns the headers and a specified
number of rows, starting from the bottom.
[]
print(df.tail())
Info About the Data
[]
print(df.info())
Apply filter to Asian Countires
[]
Asian_Countries = df.query("continent == 'Asia'")
[]
Asian_Countries
Get the number for asian countries
[]
len(Asian_Countries)
Get the total exports for Asia
[]
Asian_Countries['total_exports'].sum()
Apply filter to African countires
[]
df.af = df.query("continent == 'Africa'")
[]
df.af
[]
len(df.af)
[]
df.af['total_exports'].sum()
Apply filter for North American countires
[]
NA = df.query("continent == 'North America'")
[]
NA
[]
len(NA)
[]
NA['total_exports'].sum()
Apply filter for South American countries
[]
SA = df.query("continent == 'South America'")
[]
SA
[]
len(SA)
[]
SA['total_exports'].sum()
Apply filter for Europe
[]
EU = df.query("continent == 'Europe'")
[]
EU
[]
len(EU)
[]
EU['total_exports'].sum()
Apply filter for Oceania
[]
OCE = df.query("continent == 'Oceania'")
[]
OCE
[]
len(OCE)
[]
OCE['total_exports'].sum()
Set the names of column 1&2
[]
data = pd.read_csv('/content/World_data.csv', names=['DR','AYAT','A','B','C','D','E','F'])
[]
data
Remove rows that contain empty cells.
[]
new_df = df.dropna()
[]
new_df
only replace empty values for one column, specify the
column name for the DataFrame
[]
Column_null = df["total_exports"].fillna(400)
Column_null
[]
df
Set all the Null values 999
[]
fill_null_value = df.fillna(999,inplace = True)
df
Replace Using Mean, Median, or Mode
[]
x = df["total_exports"].mean()
df["total_exports"].fillna(x, inplace = True)
df
[]
y = df["total_exports"].median()
df["total_exports"].fillna(y, inplace = True)
df
[]
z = df["total_exports"].mode()[0]
df["total_exports"].fillna(z, inplace = True)
df
Replacing Values
Set "area" = 222 in row 3
[]
df.loc[3, 'area'] = 777
df
Delete rows where "Density" is less than 50
[]
for x in df.index:
if df.loc[x, "Density"] < 50:
df.drop(x, inplace = True)
df
Discovering Duplicates
[]
print(df.duplicated())
Removing Duplicates
[]
df.drop_duplicates(inplace = True)
Colab paid products - Cancel contracts here