Task 2 Assignment – Introduction to Programming
Task: Formulate three hypotheses about the factors that might be related to lung cancer based
on the data in the file "/content/lung_cancer_dataset.csv".
My Approach: My approach: In Colab, I explained the assignment to Gemini. Gemini provided
the answer below. My three hypotheses are listed below and explained in the section that
follows.
§
§
§
Hypothesis 1: Heavy consumers of alcohol are more likely to be diagnosed with lung
cancer.
Hypothesis 2: Individuals with higher pack years are more likely to be diagnosed with
lung cancer.
Hypothesis 3: Individuals exposed to second-hand smoke are more likely to be
diagnosed with lung cancer.
Link: Colab Lung Cancer Analysis Assignment
Gemini’s answer:
1. Understand the data
Subtask: Review the columns and their descriptions to understand what each variable
represents.
2. Explore relationships
Subtask: Use visualizations and descriptive statistics to explore potential relationships
between variables. Look for patterns, correlations, and diLerences between groups.
Reasoning: Visualize the relationship between categorical variables and lung cancer using bar
plots and the relationship between numerical variables and lung cancer using box plots to gain
a clearer understanding of potential relationships.
1
3. Formulate hypotheses
Subtask: Based on your understanding and exploration, formulate three testable hypotheses
about the factors that might be related to lung cancer in this dataset.
Reasoning: Based on the previous data exploration, I will formulate three testable hypotheses
about factors related to lung cancer.
2
Hypotheses about factors related to lung cancer:
1. Individuals with higher pack years are more likely to be diagnosed with lung cancer.
2. Individuals with exposure to radon, asbestos, or secondhand smoke are more likely to
be diagnosed with lung cancer.
3. Individuals with a history of COPD or a family history of lung cancer are more likely to be
diagnosed with lung cancer.
4. Summary
Data Analysis Key Findings
§
§
§
§
Descriptive statistics show a wide range for age and pack years, with some potentially
unusual values like negative pack years and very high maximums.
The average pack years is higher for patients diagnosed with lung cancer compared to
those without.
Contingency tables and visualizations suggest potential relationships between lung
cancer and exposure to radon, asbestos, and secondhand smoke, with a higher
proportion of lung cancer cases in categories indicating exposure.
There appears to be a potential relationship between lung cancer and a history of COPD
or a family history of lung cancer, with more cases observed in individuals with these
conditions.
Insights or Next Steps
§
§
The presence of unusual values in 'pack_years' (negative values, very high maximums)
indicates a need for data cleaning or validation before further analysis.
The initial exploration supports formulating hypotheses related to smoking history,
environmental exposures, and medical history as potential risk factors for lung cancer.
Alcohol consumption vs Lung cancer
Hypothesis 1: Heavy consumers of alcohol are more likely to be diagnosed with lung cancer.
The logic behind this is that drinking alcohol causes smoking cigarettes.
Alcohol Consumption Distribution:
Alcohol consumption
Heavy
Moderate
Count
17532
17431
Contingency table for alcohol
consumption and lung cancer:
Alcohol consumption
Heavy
Moderate
Lung cancer
No
Yes
10825
24244
5580
11851
3
The contingency table shows the counts of individuals with and without lung cancer for each
category of alcohol consumption (Heavy and Moderate).
The bar plot visually represents these counts. By comparing the proportion of 'Yes' to 'No' lung
cancer cases within each alcohol consumption category, we can observe if there appears to be
a relationship.
Looking at the output, it seems that among those with lung cancer, a larger number are in the
'Heavy' alcohol consumption category compared to the 'Moderate' category. However, to
determine if this is a statistically significant correlation, further analysis would be needed, such
as a chi-squared test.
Based on this initial exploration, there appears to be a potential association, but further
statistical testing is required to confirm a correlation.
Let's perform a chi-squared test to see if there's a statistically significant correlation between
alcohol consumption and lung cancer.
The key value to look at here is the p-value. A small p-value (typically less than 0.05) indicates
that the observed relationship is unlikely to have occurred by random chance, suggesting a
statistically significant association.
§
§
Chi-squared statistic: 39.04571193368931
P-value: 4.1399744944755154e-10
4
§
Degrees of freedom: 1
Expected frequencies table:
Alcohol consumption
Heavy
Moderate
Lung cancer
No
Yes
5341.385579040701
12190.6144209593
5310.614420959299
12120.3855790407
In this case, the p-value is 4.1399744944755154e-10, which is a very small number (much less
than 0.05). This suggests that there is a statistically significant association between alcohol
consumption and lung cancer in this dataset.
The chi-squared statistic and degrees of freedom are also part of the test output, but the pvalue is the most direct indicator of statistical significance.
Conclusion: Therefore, based on this chi-squared test, the data shows a statistically significant
correlation between alcohol consumption and lung cancer.
Cleaning the data
Prompt: For the boxplot "age vs. Lung Cancer" please only consider ages in the range
between 0 and 100 and for the box-lot and "pack_years vs. Lung Cancer" please only
consider pack years in the range between 0 and 100.
Inisghts
§ Looking at the box plot for 'age vs. Lung Cancer', we can see that 50% of people
diagnosed with lung cancer are aged between 40 and 80. In other words, there is a 50%
chance of developing lung cancer between the ages of 40 and 80.
5
§
Looking at the box plot for 'pack_years vs. Lung Cancer' we can see that 50% of people
diagnosed with lung cancer have a pack-year score between 32 and 78. In other words,
there is a 50% chance of getting lung cancer if the pack years amount to 32 or higher.
Pack years vs Lung cancer
Hypothesis 2: Individuals with higher pack years are more likely to be diagnosed with lung
cancer.
Average pack years by lung cancer
status:
Lung cancer
No
Yes
Pack years
48.363194
63.068524
6
Conclusion: The fact that the box and median line for the 'Yes' lung cancer group are generally
higher than those for the 'No' lung cancer group visually supports the hypothesis that
individuals with higher pack years are more likely to be diagnosed with lung cancer. Combined
with the observation that the average pack years are higher for those with lung cancer, we have
evidence from both descriptive statistics and visualization supporting this hypothesis.
Secondhand smoke exposure vs Lung cancer
Hypothesis 3: Individuals exposed to secondhand smoke are more likely to be diagnosed with
lung cancer.
Secondhand smoke exposure
distribution:
Contingency table for secondhand
smoke exposure and lung cancer:
Secondhand smoke exposure
No
Yes
Secondhand smoke
exposure
No
Yes
Count
25017
24857
Lung cancer
No
Yes
8859
7546
15998
20097
7
The contingency table shows the counts of individuals with and without lung cancer for each
category of secondhand smoke exposure (Yes and No).
The bar plot visually represents these counts. By comparing the proportion of 'Yes' to 'No' lung
cancer cases within each secondhand smoke exposure category, we can observe if there
appears to be a relationship.
Looking at the output, it seems that among those with lung cancer, a larger number are in the
'Yes' secondhand smoke exposure category compared to the 'No' category. However, to
determine if this is a statistically significant correlation, further analysis would be needed, such
as a chi-squared test.
Based on this initial exploration, there appears to be a potential association, but further
statistical testing is required to confirm a correlation.
Let's perform a chi-squared test to to determine if there is a statistically significant correlation.
Chi-squared statistic for secondhand smoke exposure: 446.4609436911833
P-value for secondhand smoke exposure: 4.2493142381346436e-99
Degrees of freedom for secondhand smoke exposure: 1
§
§
§
Expected frequencies table for secondhand smoke exposure:
Lung cancer
Secondhand smoke exposure
No
Yes
No
7765.009023
7814.990977
Yes
17091.990977
17202.009023
Let's interpret these results. Again, we'll focus on the p-value.
The p-value for secondhand smoke exposure is 4.2493142381346436e-99. This is an extremely
small number, much, much less than the typical significance level of 0.05.
This very small p-value indicates strong evidence to reject the null hypothesis of independence
between secondhand smoke exposure and lung cancer. In other words, the observed
association is highly unlikely to be due to random chance.
Therefore, based on this chi-squared test, there is a statistically significant correlation between
secondhand smoke exposure and lung cancer in this dataset. This supports the hypothesis that
individuals exposed to secondhand smoke are more likely to be diagnosed with lung cancer.
Prompt: Are elderly people more susceptible to lung cancer if they were exposed to
secondhand smoke?
Gemini: That's an interesting question that looks at the combined eLect of age and
secondhand smoke exposure on lung cancer susceptibility. To investigate this, we need to
look at the data in a way that considers all three factors together.
This will require a few steps to analyze. I can generate a plan to explore this relationship.
§
Categorize age: Create age groups, including a category for "elderly" people.
8
§
§
§
§
Group data and calculate rates: Group the data by age category and secondhand
smoke exposure, and calculate the lung cancer diagnosis rate within each group.
Visualize results: Create visualizations (e.g., grouped bar plots) to compare lung
cancer rates across age groups and secondhand smoke exposure levels.
Interpret findings: Analyze the calculated rates and visualizations to understand if
elderly individuals exposed to secondhand smoke have a higher lung cancer
susceptibility compared to other groups.
Finish task: Summarize the findings regarding the susceptibility of elderly people
exposed to secondhand smoke to lung cancer.
Lung cancer rates for all age categories:
age_category secondhand_smoke_exposure lung_cancer_numeric
0
Child
No
NaN
1
Child
Yes
NaN
2
Adult
No
0.559923
3
Adult
Yes
0.646817
4 Middle-aged Adult
No
0.652632
5 Middle-aged Adult
Yes
0.752077
6
Elderly
No
0.702274
7
Elderly
Yes
0.784901
Lung cancer rates for Elderly with and without secondhand smoke exposure:
9
6
7
age_category secondhand_smoke_exposure lung_cancer_numeric
Elderly
No
0.702274
Elderly
Yes
0.784901
Prompt: Explain the "lung_cancer_numeric"
Gemini: The lung_cancer_numeric column was created to help us calculate the lung cancer
diagnosis rate.
Here's what it represents:
§ We converted the original 'lung_cancer' column, which had values like 'Yes' and 'No',
into a numerical format.
§ 'Yes' diagnoses were assigned a value of 1.
§ 'No' diagnoses were assigned a value of 0.
By converting 'lung_cancer' into a numerical column (0s and 1s), we can then easily
calculate the mean of this column for diLerent groups (like age categories and secondhand
smoke exposure levels). The mean of a 0/1 variable is equivalent to the proportion or rate of
the category assigned to 1 (in this case, the lung cancer diagnosis rate).
So, lung_cancer_numeric is a temporary numerical representation of the lung cancer
diagnosis status that allows us to calculate rates.
Data Analysis Key Findings
Based on the analysis of the lung_cancer_rates DataFrame and the bar plot:
§
§
§
Elderly individuals exposed to secondhand smoke have a higher lung cancer rate
(approximately 0.78) compared to elderly individuals not exposed (approximately 0.70).
The lung cancer rate for elderly individuals exposed to secondhand smoke
(approximately 0.78) appears to be the highest among all age categories and exposure
levels, with the exception of Middle-aged Adults exposed to secondhand smoke
(approximately 0.75).
Therefore, the data suggests that elderly individuals exposed to secondhand smoke are
more susceptible to lung cancer than other groups, although Middle-aged Adults
exposed to secondhand smoke also show a high susceptibility.
10