UCLouvain
Introduction to Bioinformatics
LGBIO2010: Introduction to Bioinformatics
Vincent Branders
September, 2024
Substituting Pierre Dupont
Outline
2 / 38
1. Motivations
2. A Review Quizz on Molecular Biology
3. Course Organization
Outline
3 / 38
1. Motivations
2. A Review Quizz on Molecular Biology
3. Course Organization
Bioinformatics
4 / 38
• Interdisciplinary field that combines :
biology,
medicine,
▶ computer science,
▶ statistics,
▶ applied mathematics
▶
▶
• Also known as computational biology
• Theoretical models and practical data processing tools for the study of
biological systems
Bioinformatics transforms biological data into knowledge,
leading to a deeper understanding of life and health
From biology to computational biology
5 / 38
• The genome (full DNA) of thousands of
species has already been sequenced
• DNA sequences just form the “raw data”
and require interpretation to gain
meaningful insights
• Massive data volume makes manual
analysis impractical
• Specialized algorithms and data analysis
techniques are essential to extract
knowledge from the vast biological data
Feb 2001
Exponential data growth
6 / 38
logarithmic y-scale!
Practical algorithms and statistical methods
7 / 38
Objectives:
• Analyze real biological data
• Apply appropriate algorithms and statistics to:
Uncover hidden patterns
not visible through basic analysis
▶ Automate repetitive tasks
saves time and reduces human error
▶ Make predictions using machine learning
on new data, based on past data
▶
Challenges
8 / 38
• Computer scientist/statistician
▶
need to avoid to apply/design algorithms blindly without considering the
underlying biology
• Molecular biologist
▶
▶
need to avoid to consider algorithms or softwares as black boxes
go beyond the “click on the web” methodology
• Biomedical engineer
▶
reconcile both worlds
A win-win partnership
9 / 38
• The final validation must come from understanding the biology and from a
proper field assessment (e.g. a clinical trial)
• New algorithms need to be designed to address biological questions
• Further research in machine learning and bio-statistics is required, e.g. for
precision medicine
Big Data Analytics applied to Medicine
10 / 38
Machine Learning
• the science of getting computers to act without being explicitly
programmed
• a data-driven approach: models built from observed data before making
predictions on new data
Precision Medicine
An area of medicine where decisions, treatments and follow-ups are aimed to be
tailored to each individual patient
Example 1: diagnosis
Early diagnosis of rheumatoid infections
11 / 38
• Rheumatoid arthritis
• Lupus
• Psoriatic rheumatism
• Microcristalline arthritis
• Inflammatory osteoarthritis
Data
Medical records + some biological information from a collection of already
diagnosed patients
Objective
Find a model, based on some key markers (to be identified), to correctly classify
(= diagnose) new patients
Example 2: prognosis
12 / 38
Data
Medical records + some biological information from a collection of patients
already treated and followed over several years
Objective
Find a model, based on some key markers (to be identified), to correctly assign
new patients to specific risk groups (low risk versus high risk of relapse) and to
choose treatments accordingly
Example 3: response to treatment prediction
13 / 38
Predict patient response to an immuno-therapy
against melanoma before treatment
Data
Medical records + some biological information from a collection of observed
patients to be responders or non-responders to the therapy
Objective
Find a model, based on some key markers (to be identified), to correctly classify
new patients into responders versus non-responders
A unifying view
14 / 38
• All three tasks, diagnosis, prognosis, and treatment response prediction, are
similar:
▶
they involve building models using historical data to make predictions for
new cases
• In medical practice, the term predictive model is often used for treatment
response prediction.
• In machine learning and statistics, all three tasks are considered types of
predictive modeling
Outline
15 / 38
1. Motivations
2. A Review Quizz on Molecular Biology
3. Course Organization
Cells are the fundamental units of life
16 / 38
All living organisms are made of cells, whether they form a colony of bacteria, a
carnivorous plant or a panda
Cells vary in shape and size but they share a similar chemistry and the same
basic principles
Questions
• Among bacteria, yeast and humans which are prokaryotes or eukaryotes?
• What is the main structural difference between prokaryotic cells and
eukaryotic cells?
Prokaryotic and eukaryotic cells
17 / 38
Model organism
• Escherichia coli
(bacterium)
almost all
prokaryotes are
unicellular
Model organisms
• Saccharomyces
cerevisae (yeast)
unicellular
• Homo sapiens
37 trillion cells,
hundreds of cell
types
• All cells are enclosed by a plasma membrane
• The genetic material (DNA) is floating inside prokaryotic cells while it is
packed in the nucleus and some organelles (e.g. mitochondria) for
eukaryotes
Question
• How many cells in E. coli, yeast, humans?
Chromosomes
18 / 38
• Most prokaryotes have their
genetic material in a single circular
chromosome
• The DNA of eukaryotes is packed
into several linear chromosomes
Questions
• How many chromosomes do humans have?
• How is encoded the genetic information in DNA?
• What is Watson-Crick pairing?
Human Karyotype
19 / 38
Humans have 46 chromosomes
• 22 pairs of chromosomes
• 1 pair of sex chromosomes (XX or XY)
DNA: Deoxyribonucleic Acid
20 / 38
• 4 different nucleotides, differing by their
base
Adenosine (A)
Thymine (T)
▶ Cytosine (C)
▶ Guanine (G)
▶
▶
• Double stranded molecule, two chains of
nucleotides forming a double helix
• Each strand can be seen as sequence on a
4 letter alphabet
• One strand is the reverse-complement of
the other running in opposite direction
Watson-Crick base pairing
21 / 38
Question
Why is such pairing particularly useful when DNA replicates?
DNA replication
22 / 38
Since A pairs only with T, and G only with C, each DNA strand acts as
template for its own duplication
Questions
• What is a gene and how does it relate to a protein?
• Which are the notions of transcription and translation?
• What are introns versus exons?
• Where is RNA playing a role?
Protein synthesis
23 / 38
• A gene is a fragment of DNA coding for a protein
• A gene is first transcribed into RNA and then
translated into a protein
• For eukaryotes, introns are spliced out from
pre-mRNA and only exons are included in the
actual m(essenger)-RNA
Question
• How does RNA differ from DNA?
RNA: Ribonucleic Acid
24 / 38
• Different sugar-phosphate
backbone
• Uracil instead of Thymine
• U pairs with A
• Single-stranded molecule
Questions
• How is mRNA translated into a protein?
• What is the primary structure of a protein?
What is a protein made of?
• What is a codon? What is a stop codon?
• Why do we say that the genetic code is redundant?
mRNA translation
25 / 38
• mRNA is translated into a protein whose primary structure is a polypeptide chain
of amino acids
• A codon is a substring of 3 consecutive nucleotides coding for a specific amino
acid
• The genetic code is redundant as they are 64 codons for 20 amino acids
• Translation starts with Methionine and ends on a stop codon
Question
• How do the various amino acids differ structurally?
Amino acids
26 / 38
• The 20 amino-acids differ by the nature of their R group
• Each polypeptide has a unique amino acid sequence defining its primary structure
Question
• What are secondary/ternary/quaternary structures of proteins?
Protein secondary structure
27 / 38
Due to complex chemical interactions the amino acid chain has local structures
such as α-helix, β-sheet and u-turns
Protein ternary/quaternary structure
28 / 38
• Due to complex interactions, the local structures folds into a 3-D structure called
its conformation
• Quaternary structure: some proteins consists of several polypeptides
▶
hemoglobin has 4 polypeptide subunits
• The 3-D structure is considered to determine the protein function
Question
• Which are some common protein functions?
Examples of protein functions
29 / 38
• catalyze metabolic reactions ⇒ enzymes
• carry signal (= information) from cells to cells
• transport small molecules such as oxygen
• form cellular structures (tissues)
• regulate cell processes, including the synthesis of proteins with transcription
factors and RNA polymerase
• cut apart molecules no longer needed ⇒ degradases
Question
• What is called the central dogma in molecular biology?
Central dogma
30 / 38
Outline
31 / 38
1. Motivations
2. A Review Quizz on Molecular Biology
3. Course Organization
Course objectives
32 / 38
• Understand the basic concepts of molecular biology for appropriate use of
bioinformatics tools
• Formulate informed decisions between several computational methods that
are available for solving biological questions
• Use the information available in major sequence databases with a critical
mind
• Master a software environment such as R (Bioconductor) to implement
bioinformatics methods
• Evaluate results from a statistical viewpoint
Expected Schedule
33 / 38
Week
Date
Location
Monday 16:15-18:15
1
20/09
MERC04
Introduction to Bioinformatics
2
27/09
3
04/10
MERC04
Sequencing and Assembly
4
11/10
MERC04
Sequence Statistical Analysis
MERC04
R tutorial
Deadline + feedback
-
5
18/10
MERC04
Follow-up
6
25/10
MERC04
Pairwise Alignment Algorithms
7
01/11
8
08/11
MERC04
Molecular Phylogeny
9
15/11
MERC04
Follow-up
10
Project Start
P1 : Sequence Statistics
P1
-
MERC04
Follow-up
10
22/11
MERC04
Hidden Markov Models
11
29/11
MERC04
Multiple Alignment – Profile HMMs
12
06/12
MERC04
Gene Expression Analysis – Marker Discovery
MERC04
Follow-up
13
13/12
MERC04
Follow-up
14
20/12
MERC04
Predictive Modeling + Final Review
14
23/12
Inginious
Feedback P3
P2 : Phylogeny
P2
P3: Large Scale Expression Analysis
P3
Mandatory material + evaluation
34 / 38
The required material includes all documents (lecture slides, project assignments,
…) available on the Moodle website for this course
https://moodle.uclouvain.be/course/view.php?id=1839
Evaluation
• Closed book exam (75% of the final grade)
• Written exam (on paper or on a computer)
• Projects in groups of (max.) 2 students (25% of the final grade)
Projects to be submitted on
https://inginious.info.ucl.ac.be/course/LGBIO2010
▶ Project deadlines are strict
▶ Project grades are fixed after feedback
▶ Projects cannot be re-implemented for the second session
▶
Get help, help the other students
35 / 38
• The course schedule includes follow-up sessions to help you
• You are welcome to post (and to answer to) questions on the Moodle
Student Forum
• Find the right balance
sharing/publishing source code/reports on which you are evaluated is not
allowed (beyond your teammate)
▶ sharing/discussing questions about what you are expected to produce is very
welcome!
▶
• Each group of students is expected to submit their own solution to each
project
Use resources and cite them!
36 / 38
• The use of public resources (e.g. stackoverflow.com), including generative
AIs (e.g. chatGPT ) is permitted, as long as each (fragment of) code
submitted by the group mentions all the resources used
• The distribution or exchange between groups of students of (fragments of)
code is not authorized by any means (GitHub, Facebook, Discord, etc.),
even after the project deadlines.
• Failure to comply with these rules for any project may result in an overall
grade of 0 for all projects.
Recommended books
37 / 38
Additional resources on molecular biology
38 / 38
• LGBIO1111 - Part 1 on cellular biology and genetics
http://moodle.uclouvain.be/course/view.php?id=1910
• Molecular Biology of the Gene, Watson et al., 7th Edition, Pearson, 2013
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Sequencing and Assembly
Vincent Branders
October, 2024
Substituting Pierre Dupont
Outline
2 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
Outline
3 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
DNA replication: reminder
4 / 60
• The basic mechanisms of DNA
replication are similar across organisms
• Basic idea: each strand acts as a
template for the synthesis of a new,
complementary strand
• How is this process carried out in a
cell?
DNA polymerase
5 / 60
• DNA polymerase is responsible for the synthesis of DNA
▶
It adds nucleotides one by one to the growing DNA chain, incorporating
only those that are complementary to the template
• The DNA polymerase:
Requires a template
Only adds nucleotides to the 3’-end
of strands
▶ Can only elongate a preexisting chain
called a primer
▶
▶
• Provides a 3’-end to the DNA
polymerase
▶
Remove the nucleotides that are
wrongly added to the chain
Polymerase Chain Reaction (PCR)
6 / 60
• A biotechnique to make many copies of a specific DNA region in vitro
▶
▶
It might be a gene whose function is to be understood
A genetic marker used to match with known DNA samples
…
• The goal is to have enough DNA for downstream analyzes
• Through cycles of temperature changes, copies of the target region are
produced
Key ingredients:
Steps of PCR
7 / 60
Repeated cycles of heating and cooling
1 Denaturation
2
Annealing
3
Extension
PCR in practice
8 / 60
• The cycle of heating and cooling is repeated typically 30-40 times
• The whole process takes a few hours and produces up to billions of copies
of the target sequence
• PCR provides many copies of a specific DNA region
• The next step is to determine the precise order of nucleotides in the DNA
Sequencing
9 / 60
• Determining the precise order of nucleotides within a DNA or RNA
molecule
▶
Actually the primary structure of nucleic or amino acids
• Sequencing is crucial for understanding genetic information, locating genes,
identifying mutations, and studying biological functions, evolutionary
relationships …
1
First Generation
▶
▶
2
Sanger
Maxam-Gilbert
Next-Generation (NGS)
Illumina
Ion Torrent
▶ 454 sequencing
…
▶
▶
3
Third Generation
▶
▶
Oxford Nanopore
Pacific Biosciences
Sequencing generations: overview
10 / 60
• First Generation: High accuracy, but low throughput and expensive
• Next Generation (NGS): High throughput and cost-effective, but short
read lengths
• Third Generation: Long reads, real-time sequencing, but higher error rates
Outline
11 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
Genome assembly
From sequencing limitations to solutions
12 / 60
• Sequencing limitation:
DNA sequences (genomes) are too long to be read in one go
▶
Solution
1
Fragment the DNA sequences
▶
2
3
DNA is fragmented into shorter pieces that can be
sequenced
Pieces are sequenced into reads
Reads are assembled together
▶
These short reads must be computationally
reassembled to reconstruct the original genome
sequence
The Genome assembly problem
13 / 60
• Reconstructing the original genome from short reads is challenging:
Tens of thousands of reads
Sequencing errors and low coverage complicate assembly
▶ Repetitions in sequences cause ambiguity
▶
▶
• Approaches to solve this challenge:
▶
▶
Reference-based assembly: Align reads to a known reference genome
De novo assembly: Align reads without reference genome
Outline
14 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
Using a template
15 / 60
• “Unrelated” humans have genomes that are 99.8-99.9% similar
▶
depending on exactly how you count
ATTATTTTCCCCTCCCACTCCGATCGACGTACTGATGCTACGTAGCTGACTGATCGGATCTAGTAGCTGAGC
TCTCATCAGGCAGTCAGCGCAGGCGGAGATCTACGCGCGCCGGGCGAGGCTCATCGCGGATACAACCCCCGC
ACCCAGCACACACTTTCCCCTCCCACTCCATATCTGCATCTATCGCGTACACCAGTCCAAGCGCAGGCCGCT
CCATACCCGGAGCCTCCCACTCCATTCGCTGCATCGTACTCTCGCGGATATTCTATGTGCGAACCAACCAAA
ACCCAGCACACACTTTCCCCTCCCACTCCGTTTCCCATCTATATATATGGCGACACCAGTCAGCGCAGGCCG
CCATACCCGGAGATATCTGGTTTCCCCTCCCACTCCGGCGTCGCTGCATCGTACTCTGCGAACCAACCAACT
ACCCAGCACACACTTTCCCCTCCCACTCCGTTTCCCCTCCCACTCCCTGCATCGCGTATATAGCGCAGGCAT
CCATACCCGGAGATATATTGCGACACCAGTCCGCTGCATCGTACTCTGCATGCGAACCAATGCACCAAACCG
• Any human genome can serve as a template for aligning reads from another
human genome, despite minor differences
Reference-based Assembly
16 / 60
• Using a known reference genome to guide the assembly of short reads
▶
▶
Try to match all the reads to the reference genome
The spot where it matches most closely is our best guess as to where it
originated
• Pinpoints the origin of reads
▶
To study mutations, population diversity,
and disease association
• A suitable reference genome may not always be available (but we’ll discuss
that afterward)
The read alignment problem
17 / 60
• There are plenty of references available:
• Human genome,
• Cow genome,
• Fruit fly genome,
• Chicken genome,
• Mouse genome,
• Corn genome,
• Honey bee genome,
…
Read alignment problem
• Given a sequencing read, and given a reference genome, how do we find
where the read best matches?
• Where does it align most closely to the reference genome?
Aligning reads
18 / 60
• To align reads to a reference geneome, we take advantage of the fact that
different individuals have very similar genome sequences:
▶
Repeatedly align a short read to the reference genome
• This is more difficult that you might thing at first!
Challenges in aligning reads
19 / 60
• One alignment per sequencing read
Major challenges
• Short read lengths
▶
100 to 1000 bases long
• Large reference genome size
▶
Billions base pairs
• High volume of reads
▶
Millions of reads
• Repetitive sequences
• Sequencing errors
• Genomic variability
The size of the Human genome
20 / 60
• The Human genome is 3.2 Gbp = 3.2 billions of bases pairs
• this is quite long...
Seeing reads and genomes as strings
21 / 60
• Aligning billions of reads to a genome spanning billions of base pairs is a
complex task
• Positive note : reads and genomes can be viewed as strings
▶
There is a lot of work in computer sciences about how to work effeciently
with strings
• Pattern searching
• String indexing
• Data compression
...
• Despite these tools, read alignment remains a challenging problem
• Current solution: seed-and-extend algorithm, also used by BLAST
▶
A method to align short sequencing reads to a (or multiple) reference(s)
genome(s) by finding exact matches and then refining them
Overview of the seed-and-extend approach
22 / 60
Seed-and-extend
1
Seeding:
▶
2
Break the read into small segments (seeds)
Hit Detection:
Find exact matches (hits) for these seeds in an index of the reference
genome
▶ Use the pigeonhole principle to account for possible errors
▶
• Ensures that even if there are a few mismatches, at least one part of the read
will still match exactly
3
Extension:
▶
▶
4
Extend the matching seeds to align the whole read to the genome
Looks for the best way to align the entire read to the genome around the
seeds, allowing for mismatches, insertions, and deletions
Result:
▶
Select the best alignment for each read based on the match quality
Outline
23 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
Reference-based vs. de novo assembly
24 / 60
• If we do not have (or use) a reference genome, we perform a de novo
genome assembly
▶
▶
There is no reference genome to assemble the reads
We need to find reads with patterns that overlap
The de novo assembly problem
• Reconstruct a complete DNA sequence from short, fragmented sequences
• Involves the alignment and merging of these fragments to form a genomic
sequence
• Different set of specialized algorithms and data structures to solve this
inherently difficult problem
The de novo shotgun assembly
25 / 60
• de novo: there is no reference genome
• shotgun: reads are randomly drawn from the genome
• Our problem:
Assembly in practice
26 / 60
• We actually have:
▶
▶
a set of reads coming from an unknwon location in the genome
no sequence for the genome
How to assemble reads together?
27 / 60
• Assume two different reads coming from the same region
TCTATATCTCGGCTCTAGG
TATCTCGACTCTAGGCC
• A suffix of one is (very similar to) a prefix of the other
▶
▶
long match and nearly perfect
this is a hint that reads originate from overlapping regions of the genome
TCTATATCTCGGCTCTAGG
||||||| |||||||
TATCTCGACTCTAGGCC
Importance of overlaps
28 / 60
• If a suffix of read A is similar to a prefix of read B
TCTATATCTCGGCTCTAGG
||||||| |||||||
TATCTCGACTCTAGGCC
• then A and B might overlap in the genome
TCTATATCTCGGCTCTAGG
TCTATATCTCGGCTCTAGGCC
TATCTCGACTCTAGGCC
• Read alignment and overlap allow to assemble reads together
▶
Yet if A and B originates from the same region of the genome, why would
we observe a mismatch?
Mismatches in reads from overlapping regions
29 / 60
TCTATATCTCGGCTCTAGG
||||||| |||||||
TATCTCGACTCTAGGCC
1
Sequencing errors
From the polymerase
From the sequencing technology
▶ From the base caller
▶
▶
2
Polyploidy
▶
▶
Each chromosome comes in multiple examples
Human genome is diploid: reads might differ as one originates from the
mother while the other comes from the father
Importance of coverage
30 / 60
• The coverage is related with the amount of redundant information we have
• When building the consensus sequence, larger coverage implies more
evidences for the bases at a particular position
• More coverage leads to more and longer overlaps
De Bruijn graphs
31 / 60
• We have a set of sequencing reads
• …that partially overlap
• How to build the complete genome sequence from that?
de Bruijn graphs
• Advantage: Efficiently represents complex sequence overlaps
• Challenges:
▶
▶
Repetitive regions cause multiple “solutions”
Sequencing errors lead to fragmented assembly
Building a de Bruijn graph
32 / 60
Split a collection of reads into all possible k-mers
2 For each, consider the left and right k – 1-mers
1
3
Consider all the unique left and right k – 1-mers of all k-mers as nodes
4
For each k-mer, draw an edge from the left k – 1-mer to the right k – 1-mer
• One edge per k-mer
• One node per distinct k – 1-mer
Using the de Bruijn graph
33 / 60
• Would you be able to build the original sequence from the de Bruijn graph?
• A reconstruction of the original genome corresponds to a walk through the
graph
▶
Start at some node, move through the graph, from node to node, following
the edges and their direction
• The walk that allows us to reconstruct the genome is the one that uses
each k-mer eactly once
Reconstruction of the genome
34 / 60
• A reconstruction of the original genome is a walk that uses each k-mer
eactly once
▶
▶
Equivalently, a walk that crosses each edge exactly once
This is an Eulerian walk
• Some graph does not have an Eulerian walk, if a graph does it is called an
Eulerian graph
• Note: there are a few assumptions to get an Eulerian graph from
sequencing reads
1
2
Reads corresponds to all the k-mers of the genome
Each k-mer is sequenced exactly once, without error
Never met in practice, but still we can do things with the graph
A long long long time
35 / 60
• Let’s try with the genome “a_long_long_long_time” and k = 5
• Assuming each 5-mer is generated exactly once with no error
• a_lon
• ong_l
• _long
• ng_lo
• long_
• _long
• ng_lo
• long_
• g_lon
• ong_t
• long_
• g_lon
• ong_l
• _long
• ng_ti
• Consider all 4 letters prefixes and
suffixes
• For each unique, draw a node
• for each 5-mer, draw an arrow
• g_tim
• _time
A long long long time
36 / 60
• Let’s try with the genome “a_long_long_long_time” and k = 5
• Assuming each 5-mer is generated exactly once with no error
• To recover the original sequence :
1
Check that the graph is eulerian
All nodes have as much in-edge as
out-edge
▶ Or one has one in-edge fewer and one has
one fewer out-edge
▶
2
Start from node with one fewer in-edge (or
any edge if no such node)
3
Follow any path as long as there are edges
4
If there is no edge, report the node and
backtrack
Example
37 / 60
• Let’s do an example
Example (2)
38 / 60
• Start at a node with more out-edges than in-edges
Example (3)
39 / 60
• Follow any out-edge, here towards 1
Example (4)
40 / 60
• Follow any out-edge, here towards 3
Example (5)
41 / 60
• Follow any out-edge, here towards 4
Example (6)
42 / 60
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4
Example (7)
43 / 60
• We backtracked to 3
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4, 3
Example (8)
44 / 60
• We backtracked to 1
• As there are available options, follow any out-edge
Solution : 4, 3
Example (9)
45 / 60
• Follow any out-edge, here towards 2
Solution : 4, 3
Example (10)
46 / 60
• Follow any out-edge, here towards 1
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4, 3, 1
Example (11)
47 / 60
• We backtracked to 2
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4, 3, 1, 2
Example (12)
48 / 60
• We backtracked to 1
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4, 3, 1, 2, 1
Example (13)
49 / 60
• We backtracked to 0
• As there are no available options, store the current node in the solution and
backtrack
Solution : 4, 3, 1, 2, 1, 0
Example (14)
50 / 60
• There is no node left to backtrack from
• Read the solution in reverse order!
Solution : 4, 3, 1, 2, 1, 0
Another example
51 / 60
• Let’s try with the genome “ZABCDABEFABY” and k = 3
• Assuming each 3-mer is generated exactly once with no
error
• There are two eulerian walks!
▶
▶
ZA → AB → BE → EF → FA → AB → BC → CD → DA → AB → BY
ZA → AB → BC → CD → DA → AB → BE → EF → FA → AB → BY
Another example (continued)
52 / 60
• Let’s try with the genome “ZABCDABEFABY” and k = 3
• Assuming each 3-mer is generated exactly once with no
error
• We haven’t escaped the ambiguity of repetitive regions making assembly
difficult
• De Bruijn graphs for repetitive regions will have many different eulerian
walks
▶
▶
One will correspond to the original sequence
All others are incorrect shuffling of the genome
• De Bruijn graphs are still a common and usefull way to (internally)
represent assemblies
Real-world aligners
53 / 60
• When the overall graph is ambiguous
▶
Which is almost always true because of repetitive DNA
• …we cannot reconstruct the original genome unambiguously
• Instead, identify portions of the graph that can be reconstructed
unambiguously and assemble them into contigs
• Rather than reporting a single assembled genome sequence, the assembler
will report a set of contigs
• This is the best we can do
• Simple (bactrial) genome can be reconstructed
• Apart from that, nearly all genomes are fragmented into pieces
Gaps in the human genome
54 / 60
• Human genome: most studied genome
• Latest reconstruction: May 09 2022
▶
Still 875 gaps remaining!
Outline
55 / 60
1. Sequencing
2. Genome assembly
2.1 Reference-based assembly
2.2 The de novo assembly
3. The human genome is repetitive
Genomes result from evolutive processes
56 / 60
• The human genome, as observed through sequencers, and any other
genome, is the end result of a complex evolutionary process
• This process introduces patterns into the genome that make it very different
from a sequence as we would get from a uniform generation process
• Real genomes are very unlike “random” genomes
Mobile genetic elements (MGEs)
57 / 60
• The human genome is highly repetitive
▶
Far more than what we would get by chance with random generation
Transposable elements (Transposons; TE)
• Nucleic acid sequence that can change its position within a genome
▶
Sometimes creating or reversing mutations and altering genome size
• Transposition often results in duplication of the
same genetic material
• Major part of eukaryotes DNA and a driving force
of evolution and biodiversity
Importance of transposons
58 / 60
• Such repetitive regions in the genome affect whether we can design good
algorithms for solving the read alignment and assembly problem
• Alu sequences are 300 bp sequences repeated thoughout the human
genome
▶
Cover 11% of the human genome
The problem of repetitive regions
59 / 60
• Alu sequences are 300 bp sequences repeated thoughout the human
genome
▶
Cover 11% of the human genome
• If a read comes from inside a copy of Alu, it will be difficult to determine
from which Alu repeat the read actually come from
▶
It will most probably be impossible
Repeats make assembly difficult
60 / 60
• Repetitive sequences can make it difficult or even impossible to correctly
assemble the original genome
• Repetitive sequences cause ambiguity: multiple reconstructions possible
▶
Regardless of the used algorithm (exact, greedy, …)
• Typical errors:
▶
▶
Over-collapsing the DNA string
Switching elements positions
Still, repetitive elements will make the assembly problem very difficult
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Sequence statistics
Vincent Branders
October, 2024
Substituting Pierre Dupont
Outline
2 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Outline
3 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Abstracting the genome
4 / 40
All models are wrong, some are useful.
(G.E.P. Box, British statistician)
ATTATTTTCCCCTCCCACTCC
TCTCATCAATACAACCCCCGC
ACCCAGCACACACACACCGCT
CCATACCCCGAACCAACCAAA…
Outline
5 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Base composition
6 / 40
Haemophilus influenzae (NC_000907)
• First full bacterial genome ever sequenced (in 1995)
• 1,830,138 bp
A
567623
C
350723
G
347436
T
564241
Question
Does the data look right?
Well...
567623 + 350723 + 347436 + 564241 = 1830023 ̸= 1830138
Base composition
7 / 40
Haemophilus influenzae (NC_000907)
• First full bacterial genome ever sequenced (in 1995)
• 1,830,138 bp
A
567623
C
350723
G
347436
T
564241
K
14
K
M
N
R
S
W
Y
G or T
A or C
any base
A or G
G or C
A or T
C or T
=
=
=
=
=
=
=
M
11
N
46
R
10
S
12
W
11
Y
11
Relative base frequencies
8 / 40
A
31.02%
C
19.16%
G
18.98%
T
30.83%
0.40
Haemophilus influenzae (window=90,000 bp)
0.15
0.20
0.25
0.30
0.35
A
C
G
T
0
500000
1000000
1500000
GC-content
9 / 40
A-T
61.8%
G-C
38.2%
Haemophilus influenzae (window=90,000 bp)
0.3
0.4
0.5
0.6
0.7
A−T
G−C
0
500000
1000000
1500000
A closer look at GC-content
10 / 40
Haemophilus influenzae (GC content anomaly)
A−T
G−C
0.3
0.3
0.4
0.4
0.5
0.5
0.6
0.6
0.7
0.7
A−T
G−C
0.8
0.8
Haemophilus influenzae (window=20,000 bp)
0
500000
1000000
1500000
1500000
1550000
1600000
This anomaly is attributed to an ancient insertion of viral DNA
1650000
1700000
Bacteriophage lambda GC-content
11 / 40
Bacteriophage Lambda
0.3
0.4
0.5
0.6
0.7
A−T
G−C
0
10000
20000
30000
40000
• Enterobacteria phage lambda: a virus that infects bacteria
• First half is GC-rich, second half is AT-rich
• AT-rich regions denature at lower temperatures
• It is believed that the ability to quickly denature DNA facilitates the
insertion in the bacterial cell being infected
Outline
12 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Dimer frequencies
13 / 40
Haemophilus influenzae
A
31.02%
A*
C*
G*
T*
C
19.16%
*A
0.1202
0.0665
0.0514
0.0721
*C
0.0505
0.0372
0.0522
0.0518
G
18.98%
T
30.83%
*G
0.0483
0.0396
0.0363
0.0656
*T
0.0912
0.0484
0.0499
0.1189
1 = 0.0625 of the time
• Equally likely dimers would appear 16
• AA and TT look particularly frequent
• CC, CG and GG look particularly rare …do they?
Are those statistical biases informative?
14 / 40
A
31.02%
A*
C*
G*
T*
C
19.16%
*A
0.1202
0.0665
0.0514
0.0721
*C
0.0505
0.0372
0.0522
0.0518
G
18.98%
*G
0.0483
0.0396
0.0363
0.0656
T
30.83%
*T
0.0912
0.0484
0.0499
0.1189
• AA and TT look particularly frequent but A and T alone are also the most
frequent
• what would be the expected frequency of dimers observed by chance?
▶
need for a background model to characterize the non-uniform distribution of
individual nucleotides
Multinomial background model
15 / 40
A
31.02%
C
19.16%
G
18.98%
T
30.83%
• Estimate the probability of each nucleotide independently
P̂(A) = f(A) = number of A’s = 0.3102
sequence length
• Use those estimates in a random generative model
AAGTTGACATAATTTGCT…
• The expected frequency of a dimer XY:
E[f(XY)] = P̂(X)P̂(Y) = f(X)f(Y)
Odd ratios
16 / 40
• Now we can look at what is surprising:
Ratio between observed frequency and expected frequency
f(XY)
f(XY)
=
E[f(XY)]
f(X)f(Y)
A*
C*
G*
T*
*A
1.2490
1.1180
0.8735
0.7540
*C
0.8495
1.0119
1.4348
0.8762
*G
0.8209
1.0892
1.0074
1.1202
*T
0.9533
0.8189
0.8525
1.2504
• The most over-represented dimer is GC ( ̸= G+C-content !)
• CC, CG and GG are not particularly rare
• The rarity of TA is quite universal
• A statistical test could tell us whether we depart significantly from 1.0
Multinomial model = random permutation
17 / 40
f(XY)
frandom (XY)
• the expected frequency E[f(XY)] = f(X)f(Y) with a multinomial model can
be replaced by the observed frequency frandom (XY) in a random sequence
generated from this model
• equivalent to a random permutation of the original sequence
Original
TATGGCAATTAAAAT
Permuted
CAAGATTGATAATAT
Odd ratios generalized to k-mers
18 / 40
Ratio between observed frequency and expected frequency
f(X . . . Xk )
f(X1 . . . Xk )
= ∏k1
E[f(X1 . . . Xk )]
i=1 f(Xi )
• Useful when looking for frequent patterns of k consecutive characters
• Two most frequent 10-mers in Haemophilus influenzae:
AAAGTGCGGT and ACCGCACTTT occur more than 500 times
Are those 10-mers informative? Addressing this question requires:
a more sophisticated background model (e.g. permutation respecting codon
structure or a set of “reference” sequences)
▶ a statistical test to assess significance
▶ biological validation
▶
Outline
19 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Outline
20 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Eukaryotes versus prokaryotes
21 / 40
Illustrations from Molecular Biology of the Cell (© Garland Science 2008)
Reading frames
22 / 40
By convention, the reading frame -2 along the backward strand is in front of the reading
frame +2 along the forward strand. For example, the first STOP codon along the backward
strand is on reading frame -2 (and not -3)
Illustration from Molecular Biology of the Cell (© Garland Science 2008)
Standard genetic code
RNA alphabet
23 / 40
Mitochondria
24 / 40
Illustration from Molecular Cell Biology, 5e (© WHFreeman 2004).
• Mitochondria include their own DNA
• The protein synthesis process is similar to the one of prokaryotes
• Some pecularities in the genetic code (hence not fully standard!)
Outline
25 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
The basic algorithm
26 / 40
ORF finding
repeat along the sequence
1
look for a first START
(or the next START after a STOP on the same frame)
2
look for the next STOP on the same reading frame
• Consider each reading frame:
3 on the forward strand
3 on its reverse complement
(the same problem, not the same solution!)
▶ respect the +/- convention (e.g. -2 in front of +2)
▶
▶
• After a START, you may find other codons for Met before a STOP
▶
▶
those are not true start
An ORF is a longest strech of DNA between a START and a STOP,
without being interrupted by another STOP on the same frame
Is an ORF actually a coding gene?
27 / 40
The DNA stretch found between a START and a STOP codon might be
due to chance
2 The ORF might be there but the gene not expressed
1
▶
▶
3
a trace of the past (or future) evolution
we are ignoring here regulations at the transcription/translation levels
Some gene sequences do not strictly follow the standard ORF structure
▶
we are ignoring some rare exceptions in the genetic code (e.g. some
fragment of stop codon may play the role of full STOP)
• We address question 1 through a statistical test procedure
• We leave questions 2,3 for a careful biological validation
Outline
28 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
A concrete example
29 / 40
• Suppose you want to assess the efficiency of a new pain-killing drug versus
the current best alternative
• You consider a representative sample of patients receiving drug 1 (the new
drug) and another sample receiving drug 2 (the control)
• You assess how many minutes it takes for each patient suffering from a
headache to feel better after taking either drug
Sample 1
2
3
6
5
1
Sample 2
7
8
5
6
7
• You want to know whether there is any significant difference of efficiency
between both drugs
Simplifying assumptions
30 / 40
Caution
We deliberately ignored many important related questions
• what is a representative sample, how big should it be and how to collect it?
• can feeling better be accurately casted into a yes/no answer?
• is the number of minutes before relief a relevant criterion and can it be
accurately evaluated?
• cost?
• side-effects?
• existing patents?
• competitors?
• regulations?
• …
Reformulate the question
31 / 40
Compute the sample means m1 , m2 and sample variance s21 , s22
2 Check whether the difference between both means is significant or else
should be attributed to randomness in our respective samples
1
Sample 1
2
3
6
5
1
m1 = 3.4
s21 = 4.3
n1 = 5
Sample 2
7
8
5
6
7
m2 = 6.6
s22 = 1.3
n2 = 5
Statistical test
32 / 40
• Null hypothesis: H0 : µ2 = µ1
• Alternative hypothesis: Ha : µ2 ̸= µ1
• Test statistics : T = m2 – m1 = µ̂2 – µ̂1
• It is known that T approximately follows a
Student t distribution
T=
(µ̂2 – µ̂1 ) – (µ2 – µ1 )
√ 2
s1
s2
+ n2
n1
2
• Fix a significance threshold α (e.g. 5%)
(6.6–3.4)–(0)
T = √ 4.3 1.3 = 3.0237 > t0.975 = 2.43 ⇒ reject H0
5
+ 5
⇒ claim the drugs are not equally effective
p-value of a test
33 / 40
Instead of fixing a priori a significance threshold α = 5%, one can report the
p-value of the test
• p-value = the smallest α that would lead to
reject the test
• here T = 3.0237 ⇒ p-value = 0.02229
▶
the true means could still be equal (=
the drugs could still be equally effective),
but the probability of our conclusion to
be wrong is 2.2%
• the lower the more significant the result
• a p-value of max 5% (or 1%) is still often considered for the test to be deemed
(highly) significant
• we will revisit this question when discussing multiple testing
Assessing the statistical significance of ORFs
34 / 40
Hypothesis
• Significant ORFs in an actual sequence should be longer than ORFs
observed by chance
• The NULL model (= control) is typically made of a random permutation of
the original sequence
Algorithm
1
Find all ORFs in a random permutation of the original sequence
2
Report the length distribution of random ORFs
Accept as significant ORFs, any ORF in the original sequence longer than a
prescribed threshold
3
▶
▶
the maximal ORF length observed at random
the 99% percentile of the random ORF length distribution
⇒ permutation test with a p–value = 1%
ORF finding in H influenzae
35 / 40
H influenzae (red) versus permuted version (blue)
0
5
2000
4000
10
6000
15
8000
20
H influenzae (red) versus permuted version (blue)
0
100
200
300
400
1000
ORF length (bp)
2000
3000
4000
ORF length (bp)
• Maximum random ORF length = 528 ⇒ 1252 actual ORFs are longer
99% percentile = 204 ⇒ 2219 actual ORFs are longer
• 1765 genes are annotated (on NCBI), including some pseudo-genes or
hypothetical protein coding
5000
Statistical note
36 / 40
5
10
15
20
H influenzae (red) versus permuted version (blue)
1000
2000
3000
4000
ORF length (bp)
Parametric two-tailed
t-test
Non-parametric one-tailed
permutation test
5000
Outline
37 / 40
1. Simple genome statistics
1.1 Change point analysis
1.2 Finding unexpected k-mers
2. Identification of Open Reading Frames
2.1 Brief review of the underlying biology
2.2 ORF finding algorithm
2.3 Significance assessment
3. Your first project
Your first project
38 / 40
1
Register as a group of (max.) 2 students on
inginious.info.ucl.ac.be/course/LGBIO2010
(normally already done when checking the A0.X tutorial)
Project 1 on sequence statistics: solve the A1.X tasks on Inginious
3 Get your hands on real biological data and real software
2
▶
4
you will have to program a bit
Do not wait the last minute! The hard deadline is specified on Inginious
Anti-plagiarism
39 / 40
I hereby certify that the results and code that I will submit for the projects is coming
from my own work and that of my teammate (if any in my group). The submitted
works will not be (even partial) copy/paste from information found on the internet or
from the work of other groups.
I also certify that I will not distribute any answer or code related to these projects, in
person or on any repository (github, bitbucket, Facebook groups, ....) accessible to
anybody beyond my teammate, even after the deadlines.
Any violation of the above statements will be considered as cheating and will be reported
as such to the President of the Jury.
Note 1: data mining tools exist to detect similarities between codes or textual files
Note 2: This should not prevent fruitful exchanges between students!
Post your clarification questions/answers on the Moodle student forum
Enjoy team work
40 / 40
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Molecular Phylogeny
Vincent Branders
October, 2024
Substituting Pierre Dupont
Outline
2 / 46
1. Introduction
2. UPGMA
3. Neighbor Joining
Outline
3 / 46
1. Introduction
2. UPGMA
3. Neighbor Joining
Phylogeny
4 / 46
• A phylogeny, or evolutionary tree, represents the relationships among biological
species believed to have evolved from a common ancestor
• Taxonomists used anatomy and physiology (e.g. number of legs, presence of
feathers) to group and classify organisms (taxa, sing. taxon = taxonomic unit)
• Molecular phylogenetics uses molecular sequences (genes, proteins, …) to
construct phylogenetic trees ⇒ each taxon is represented by a specific sequence
Why use molecular traits?
5 / 46
• Discrete Changes
Molecular sequences are built from discrete units (nucleotides or amino
acids), simplyfying the comparisons
• Rich Data Source
▶
Genomes contain thousands of genes and proteins, providing a broad dataset
to study evolutionary relationships
• Objective Metrics
▶
Molecular data reduce subjectivity compared to traditional anatomical traits
• Universal Presence
▶
Genes are shared across all forms of life, enabling phylogenetic studies from
close relatives to vastly divergent species
• Neutral Mutations
▶
Many genetic changes are unaffected by natural selection, reducing bias in
phylogenetic reconstructions
Tree components
6 / 46
• Each tree leaf represents a taxon
• Each internal node represents a common ancestor
• Branch lengths are sometimes interpreted as proportional to evolution time
• Multifurcating node (polytomy)
▶
▶
Often interpreted as uncertainty about the evolution order
Typically approximated by consecutive bifurcating nodes
⇒ binary trees
• Tree inference methods: only the taxa are observed and the tree must be inferred
(= reconstructed)
Tree symmetry
7 / 46
Only the relationships between taxa matters: (A, (B, (C, D)))
Rooted versus unrooted trees
8 / 46
• Some tree inference methods (e.g. UPGMA) produce rooted trees, other methods
(e.g. Neighbor-Joining) produce unrooted trees
• For unrooted trees, the direction in which evolution occurred is undefined but the
branch lengths still represent evolution time
Rooting with an outgroup
Note: only tree structures are represented
here, not accurate branch lengths
(2n–5)!
with n ≥ 3 taxa
2n–3 (n–3)!
n = 10 ⇒ Nu ≈ 2 × 106
Nu =
(2n–3)!
with n ≥ 2 taxa
2n–2 (n–2)!
n = 10 ⇒ Nr ≈ 35 × 106
Nr =
Tree inference methods
9 / 46
• Distance based methods
Start from a matrix of pairwise distances between taxa
The distances are derived from pairwise alignments between sequences
▶ Progressively cluster taxa by picking closest (groups of) taxa first
▶
▶
• Parsimony
▶
Search for a tree minimizing the total number of mutations
• Maximum Likelihood
▶
Search for a tree maximizing P(data|tree) according to some probabilistic
model of evolution
Sequence Alignment
10 / 46
Goal
• Sequence alignment arranges two DNA sequences side-by-side for
comparison
• It helps identify similarities (matching bases) and differences (mismatches)
at each position
• Each base from sequence xi is aligned with each base from sequence xj
• 17 matches
• 4 mismatches
• 1 gap or indel
Evolutionary distance between sequences
11 / 46
1
2
compute an alignment between a pair (xi , xj ) of e.g. DNA sequences
compute the relative frequency f of sites u (= positions in the sequence)
where xui ̸= xuj (= mismatches)
the frequency is normalized by the number of matches (xui = xuj ) + the
number of mismatches (xui ̸= xuj )
▶ the number of gaps, if any, are ignored when computing this relative
frequency
▶
3
transform such a frequency f in an evolutionary distance dij
▶
some mutations are unobserved while several mutations occurred
Examples:
2 mutations, 0 difference: A → C → A
3 mutations, 0 difference: A → C → T → A
1 mutation,s 1 difference: A → G
3 mutations, 1 difference: A → C → T → G
Jukes-Cantor distance
12 / 46
(
)
4
f = 34 1 – e– 3 dij
f
dij = d(xi , xj ) = – 43 ln(1 – 4f
3)
dij
Properties
• When xi versus xj include so many changes that they should be considered fully
unrelated, there are still 14 of the sites matching by chance ⇒ f = 34 and dij = ∞
• This model assumes that each site can mutate independently of the others
• This model assumes a fixed rate of change at any site for any residue
Illustration from Inferring Phylogenies, Felsenstein, 2004
DNA evolution models
13 / 46
Kimura model: (biological) transitions (α) are
more likely than transversions (β)
Jukes-Cantor model is a Markov
model with transition probability
matrix:
A
C
G
T
[ 1 –A3α
α
α
α
C
α
1 – 3α
α
α
G
α
α
1 – 3α
α
T
α
α
α
1 – 3α
]
Constant mutation rate α for all
residues
A
C
G
T
[ 1 – 2βA – α
β
α
β
C
β
1 – 2β – α
β
α
G
α
β
1 – 2β – α
β
T
β
α
β
1 – 2β – α
]
• Both models lead to an equal probability of appearance of the 4 nucleotides in
the long run, which could be incorrect (cfr. GC-content)
• See Felsenstein, 2004, chap. 13 for more sophisticated evolution models
Outline
14 / 46
1. Introduction
2. UPGMA
3. Neighbor Joining
UPGMA
15 / 46
7.15
Algorithm UPGMA
Input: S a set of sequences x1 , . . . , xm
Input: dij a distance measure between xi and xj
Output: A phylogenetic tree T
// Initialize a set C of clusters C1 , . . . , Cm
C ← {{x1 }, . . . , {xm }}
// Initial clusters are tree leaves
T ← a partial tree whose leaves are the xi ’s, all at height zero
A
while |C| > 1 do
Choose pair of clusters (Ci , Cj ) in C such that dij is minimal
Define a new cluster Ck = Ci ∪ Cj and compute dkl for all l ̸= i, j
Add node k as parent of nodes i and j, and place it at height dij /2
Add Ck to the set C, and remove Ci and Cj from it
return T
6
4.5
2
B
D
C
E
The final tree is interpreted as a phylogenetic tree and the branch length is representative of
evolution time
Distance between clusters
16 / 46
• The original distance dij is defined between pair of sequences
• After grouping sequences into clusters, the distance needs to be defined
between clusters
• UPGMA uses the average distance between sequences from each cluster
dij =
1
|Ci ||Cj |
∑
dpq
p∈Ci ,q∈Cj
UPGMA stands for Unweighted Pair Group Method using Arithmetic mean
• When Ck = Ci ∪ Cj and Cl is another cluster (l ̸= i, j)
dkl =
|Ci |dil + |Cj |djl
|Ci | + |Cj |
UPGMA Example
17 / 46
A
B
C
D
E
A
B
12
C
16
14
D
20
12
10
E
12
12
4
A
8
B
C
D
E
0
UPGMA Example (2)
18 / 46
A
B
C
D
E
A
2
B
12
C
16
14
D
20
12
10
E
12
12
4
A
8
B
D
C
E
UPGMA Example (3)
19 / 46
A
B
C
D
E
A
2
B
12
C
16
14
D
20
12
10
E
12
12
4
d(CE, A) =
A
8
|C|×d(C,A)+|E|×d(E,A)
= 1×16+1×12
= 14
|C|+|E|
1+1
B
D
C
E
UPGMA Example (4)
20 / 46
4.5
A
B
CE
D
2
A
B
12
CE
14
13
D
20
12
A
9
B
D
C
E
UPGMA Example (5)
21 / 46
4.5
A
B
CE
D
2
A
B
12
CE
14
13
D
20
12
d(DCE, A) =
A
B
9
|D|×d(D,A)+|CE|×d(CE,A)
= 1×20+2×14
= 16
|D|+|CE|
1+2
D
C
E
UPGMA Example (6)
22 / 46
6
4.5
2
A
B
DCE
A
B
12
DCE
16
12.7
A
B
D
C
E
UPGMA Example (7)
23 / 46
6
4.5
2
A
B
DCE
A
B
12
DCE
16
d(AB, DCE) =
12.7
A
B
D
|A|×d(A,DCE)+|B|×d(B,DCE)
= 1×16+1×12.7
= 14.3
|A|+|B|
1+1
C
E
UPGMA Example (8)
24 / 46
7.15
6
4.5
2
AB
DCE
AB
DCE 14.3
A
B
D
C
E
UPGMA Example (9)
25 / 46
7.15
6
4.5
2
A
B
D
C
E
Properties of the reconstructed tree
26 / 46
7.15
A
B
C
D
E
6
A
4.5
B
12
C
16
14
D
20
12
10
E
12
12
4
2
8
A
B
D
C
E
Additivity
• Tree is additive as the distance between any pair of leaves (i, j) is the sum of the
distances (= branch lengths) between them and their common ancestor k
dij = dik + dkj
Molecular clock
• Tree is ultrametric (by algorithm design) as the distance between any pair of leaves (i, j)
and their common ancestor k are equal: dik = djk
• Since branch lengths represent evolution time, it assumes the same rate of evolution
from any ancestor to each of its children
Ultrametric condition
27 / 46
7.15
A
B
C
D
E
6
A
4.5
B
12
C
16
14
D
20
12
10
E
12
12
4
2
8
A
B
D
C
E
• UPGMA is guaranteed to reconstruct the correct tree iff the original
distance is an ultrametric
Symmetry: dij = dji
Distinguishability: dij > 0 ⇔ i ̸= j
▶ Triangular Inequality: d ≤ d + d
ij
ik
kj
▶ Ultrametricity: for any triplet (i, j, k) they can be renamed such that
dij ≤ dik = dkj (2 distances are equal and the last one is smaller or equal)
▶
▶
• The distance metric from the example is not an ultrametric
some pairwise distances are not preserved
e.g. d(A, E) = 12 ̸= 14.3
▶ the final tree is incorrect
▶
Ultrametric condition (ctd.)
28 / 46
• Tree correctly reconstructed by UPGMA
A
B
C
D
E
A
5
B
8
C
10
10
D
8
2
10
E
4
8
10
4
2
1
8
B
D
A
E
• Incorrect tree
original
Illustration from Biological Sequence Analysis, Durbin, 1998
reconstructed
C
Agglomerative clustering
29 / 46
• Despite its strong assumption (molecular clock), UPGMA is commonly used
because of its simplicity
• UPGMA is a particular case of agglomerative clustering used e.g. to group gene
expression measurements (instead of taxa)
▶
the distance d can be defined e.g. between vectors d(xi , xj ) of expression
values across samples of various patients
• Several variants of agglomerative clustering: Ck = Ci ∪ Cj
UPGMA
|C |d +|C |d
dkl = i|C il|+|Cj | jl
i
j
average-link rule
WPGMA
d +d
dkl = il 2 jl
weighted average link rule
(each example has the same influence)
dkl = min(dil , djl )
single link rule
dkl = max(dil , djl )
complete link rule
Outline
30 / 46
1. Introduction
2. UPGMA
3. Neighbor Joining
The Neighbor-Joining Algorithm
31 / 46
Basic ideas
• No longer assume molecular clock but keep additivity to build an unrooted
tree
• Start with taxa representing the tree leaves and aggregate first neighboring
leaves
Additivity
For any 3 leaves i, j, m there is a node k where the branch to them meet ⇒
additivity implies:
dim
djm
dij
=
=
=
dik + dkm
djk + dkm
dik + dkj
Consequently:
dkm =
)
1(
dim + djm – dij
2
Finding neighboring leaves
32 / 46
Looking for pair i, j with a minimal dij is not enough to find neighbors
d12 < d13
but
(1, 3) are neighbors not (1, 2)
Corrected pairwise distance
• To find neighbors, subtract average distances to other nodes
Dij = dij – (ri + rj ) where
∑
1
▶
▶
ri = |L|–2 k∈L dik
|L| is the number of leaves
• Compute pair (i, j) for which Dij is minimal
The NJ Algorithm
33 / 46
D
5.75
A
7.65
6
Algorithm Neighbor-Joining
4.25
Input: S a set of sequences x1 , . . . , xm
4.35
Input: dij a distance measure between xi and xj
B
Output: An unrooted phylogenetic tree T
// Initialize T to be the set L of leaf nodes, one for each sequence
L=S
while |L| > 2 do
Pick a pair i, j in L for which the corrected distance Dij is minimal
Define a new
( node k (parent
) to i and j)
Set dkl = 12 dil + djl – dij
(∀ l ̸= i, j)
// for any leaf l other than i or j
(
)
Add k to T with branch lengths: dik = 12 dij + ri – rj ; djk = dij – dik
Remove i and j from L and add k to it
// Termination: L consists of two leaves i and j
Add a final branch to T connecting i and j with length dij
return T
8
1.25
7
3.25
.75
E
C
Neighbor-Joining Example
34 / 46
D
A
B
C
D
A
E
A
B
12
C
16
14
D
20
12
10
E
12
12
4
C
8
B
E
Neighbor-Joining Example (2)
35 / 46
D
A
B
C
D
E
A
A
B
12
C
16
14
D
20
12
10
E
12
12
4
8
A
B
C
D
C
B
E
E
|L| = 5 ⇒ |L| – 2 = 3
A
B
−24.7
C
−18.7 −17.4
D
−16.7 −21.4 −21.4
E
−20
−16.7 −22.7 −20.7
rA = 60
= 20 ; rB = 50
= 16.7 ; rC = 44
= 14.7
3
3
3
36
rD = 50
=
16.7
;
r
=
=
12
E
3
3
DAB = 12 – (20 + 16.7) = –24.7
DAC = 16 – (20 + 14.7) = –18.7
DAD = 20 – (20 + 16.7) = –16.7
...
⇒ join neighbors A and B
Neighbor-Joining Example (3)
36 / 46
D
A
B
C
D
E
A
7.65
A
C
6
B
12
C
16
14
D
20
12
10
E
12
12
4
6
C
4.35
E
8
D
6
C
9
D
10
10
E
6
4
8
B
E
Let 6 be the parent node of {A, B}
d6C = 12 (dAC + dBC – dAB ) = 12 (16 + 14 – 12) = 9
d6D = 12 (dAD + dBD – dAB ) = 12 (20 + 12 – 12) = 10
d6E = 12 (dAE + dBE – dAB ) = 12 (12 + 12 – 12) = 6
dA6 = 12 (dAB + rA – rB ) = 12 (12 + 20 – 16.7) = 7.65
dB6 = dAB – dA6 = 12 – 7.65 = 4.35
Neighbor-Joining Example (4)
37 / 46
D
A
6
C
D
7.65
E
C
6
6
C
9
D
10
10
E
6
4
4.35
E
8
B
|L| = 4 ⇒ |L| – 2 = 2
= 12.5 ; rC = 23
= 11.5 ; rD = 28
= 14 ; rE = 18
=9
r6 = 25
2
2
2
2
D6C = 9 – (12.5 + 11.5) = –15
D6D = 10 – (12.5 + 14) = –16.5
D6E = 6 – (12.5 + 9) = –15.5
DCE = 4 – (11.5 + 9) = –16.5
...
⇒ join neighbors C and E (or, alternatively, 6 and D)
Neighbor-Joining Example (5)
38 / 46
D
A
6
C
D
7.65
E
6
6
7
C
9
D
10
10
E
6
4
6
4.35
8
7
6
7
5.5
D
10
7
B
D
3.25
.75
E
Let 7 be the parent node of {C, E}
d76 = 12 (dC6 + dE6 – dCE ) = 12 (9 + 6 – 4) = 5.5
d7D = 12 (dCD + dED – dCE ) = 12 (10 + 8 – 4) = 7
d7C = 12 (dCE + rC – rE ) = 12 (4 + 11.5 – 9) = 3.25
d7E = dCE – d7C = 4 – 3.25 = 0.75
C
Neighbor-Joining Example (6)
39 / 46
D
A
6
7
7.65
6
D
7
3.25
6
7
5.5
D
10
4.35
7
B
|L| = 3 ⇒ |L| – 2 = 1
r6 = 15.5 ; r7 = 12.5 ; rD = 17
D67 = 5.5 – (15.5 + 12.5) = –22.5
D6D = 10 – (15.5 + 17) = –22.5
D7D = 7 – (12.5 + 17) = –22.5
⇒ join neighbors 6 and 7 (or 6 and D, or 7 and D)
.75
E
C
Neighbor-Joining Example (7)
40 / 46
D
5.75
A
7.65
6
6
7
D
4.25
8
1.25
6
7
5.5
D
10
4.35
7
B
7
3.25
C
.75
E
Let 8 be the parent node of {6, 7}
d8D = 12 (d6D + d7D – d6,7 ) = 12 (10 + 7 – 5.5) = 5.75
d68 = 12 (d67 + r6 – r7 ) = 12 (5.5 + 15.5 – 12.5) = 4.25
d78 = d67 – d68 = 5.5 – 4.25 = 1.25
Note: joining {6, D} or {7, D} instead of {6, 7} would lead to the same tree
(by additivity)
NJ properties
41 / 46
• The NJ algorithm is guaranteed to recover the correct unrooted tree if the
actual distances are additive
• Real data will only be at best approximately additive
• The tree can be rooted if need be with an outgroup taxon
UPGMA vs Neighbor-Joining
42 / 46
A
B
C
D
E
A
B
12
C
16
14
D
20
12
10
E
12
12
4
8
UPGMA
Neighbor-Joining
D
7.15
5.75
6
A
7.65
6
4.5
4.25
8
1.25
4.35
2
A
B
D
C
E
B
7
3.25
.75
E
C
Summary
43 / 46
• Inference of phylogenetic trees aims at reconstructing the evolutionary tree
from a set of taxa, a set of (genomic) sequences representative of various
species
• The pairwise distances between taxa typically come from a pairwise
alignment
• Some mutations are not observed (because of additional intermediate
mutations)
▶
Jukes-Cantor or Kimura evolutionary models correct for this
Summary (2)
44 / 46
• UPGMA is a distance based method assuming additivity and ultrametricity
(molecular clock) and producing a rooted tree
• Neighbor-Joining is a distance based method assuming additivity
producing an unrooted tree
• Maximum parsimony methods (not detailed here) look for tree with a
minimal number of mutations
• Probabilistic methods (not detailed here) require a probabilistic model of
evolution (such as JC or Kimura) to compute a tree maximizing
P(data|tree) (Maximum likelihood) or P(tree|data) (Bayesian inference)
Further reading
45 / 46
Chap. 7, (8)
Chap. 11, (13), …
Software tools
46 / 46
• R package
cran.r-project.org/web/packages/ape/
• Phylip
evolution.genetics.washington.edu/phylip.html
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Pairwise alignment algorithms
Vincent Branders
November, 2024
Substituting Pierre Dupont
Outline
2 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Outline
3 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Notions of homology and identity
4 / 43
Homology
• Common ancestry between a pair of structures, genes or proteins
• A pair of species can have a common ancestor and/or one being the
ancestor of the other
• Homology needs not imply similar function
Identity
• Proportion of pairs of identical residues between two aligned sequences
▶
17 matches
▶
4 mismatches
▶
1 gap or indel
Identity = 17
22 = 77%
Comparing homologous sequences
5 / 43
• Discover matching parts between homologous sequences
• Trace back the mutations (but also transfers, repeats, inversions) that
could explain the relationships between sequences
• Matches between an arbitrary pair of sequences may be due to chance
rather than homology ⇒ statistical significance required
Dotplot
A visual assessment of similarity based on identity
6 / 43
The plot is often smoothed by
averaging over sliding windows through
both sequences
Note: check matrix orientation to
interpret the diagonal correctly
Dotplot use
7 / 43
• Genome comparison revealed inversion of gene positions
• Dotplot is a visual aid but it does not provide an exact alignment
illustration of possible matches but several matches
could appear along the same line or column
▶ consecutive matches are not connected
▶ no consideration for gaps
▶
Outline
8 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Pairwise alignment
9 / 43
Objective
Find a best (to be scored) way to align 2 sequences including matches,
substitutions (or mismatches) and possible gaps (insertions or deletions)
• Both sequences need not have the same length
• Scoring matrices define similarity scores between letters (e.g. nucleotides or
amino acids)
The higher the score the more similar a given pair of letters
smatch > smismatch
▶ A gap penalty (negative score) is also defined
▶
• Look for a pairwise alignment with maximal cumulative score
Gap penalties
10 / 43
Linear gap penalty
Affine gap penalty
γ(g) = –dg
g: number of consecutive gaps
γ(g) = –d – e(g – 1)
g: number of consecutive gaps
d: gap penalty (e.g. 8)
d: gap open penalty (e.g. 12)
e: gap extension penalty (e.g. 2)
• Affine is more relevant from a biological viewpoint but more complex to compute
with
• Affine reduces to linear whenever e = d
Global versus local alignment
11 / 43
Global Alignment
12 / 43
Input: a sequence x of length n and a sequence y of length m
Input: an appropriate scoring matrix S and a gap penalty γ(g)
Output: an alignment between x and y with maximal cumulative score
Example: {
s(xi , yj ) =
1
0
if xi = yj
if xi ̸= yj
γ(g) = –8g
s(x, y) = 0 + 0 – 8 + 0 – 8 + 0 = –16
• How many such alignments exist?
• How to compute an optimal alignment?
Number of gapped alignments
13 / 43
k diagonal, 0 ≤ k ≤ min(n, m)
n – k horizontal
m – k vertical
n + m – k total
How many ways to combine k, n – k, and m – k operations?
min(n,m)
N=
∑
k=0
(n + m – k)!
4n
≈ √
k!(n – k)!(m – k)!
πn
Assuming n ≈ m (sound for a global alignment): n = 300 ⇒ N ≈ 1.10179
Outline
14 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Scoring matrices
15 / 43
• Nucleotide scoring matrices
▶
unitary (1,0) or e.g. BLAST(1,-3)
• Protein scoring matrices for substitution
between AAs
the genetic code: number of mutations to get
one AA from the other
▶ physico-chemical properties of amino acids
▶ observed amino-acid substitutions in related
proteins: PAM, BLOSUM
▶
• Scoring system scale is often arbitrary ⇒
alignment scores can only be interpreted in
relative terms
BLOcks SUbstitution Matrix (BLOSUM)
16 / 43
BLOSUM matrices are based on residue substitutions seen in conserved regions
from the BLOCKS database
substitution scores depend on the observed frequencies of substitutions of an
AA by another AA across clusters of similar sequences
▶ clustering according to (at least) X % of identity prevents a bias due to
similar sequences which would otherwise be over represented
▶ actual scores are log odd ratios between observed frequencies of
substitutions and expected frequencies of observing a match by chance
▶
BLOSUM50 Scoring matrix
17 / 43
BLOSUM50
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
PAM versus BLOSUM matrices
18 / 43
• Percent (or Point) Accepted Mutation (PAM) matrices are based on
evolutionary model between proteins and derived from phylogenetic trees
• PAM1 proteins with one mutation per 100 residues
• PAM250 proteins with 250 mutations per 100 residues
(a position can mutate several times)
• PAM X the larger X the more distant the proteins
• BLOSUM X the larger X the more similar the proteins
(clusters with ≥ X % identity)
PAM 120
PAM 160
PAM 250
BLOSUM 80
BLOSUM 62
BLOSUM 45
• BLOSUM 62 is the default value for finding conserved domains among
averagely related proteins
Outline
19 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Dynamic programming
20 / 43
Key observations
1
we look for a maximal cumulative score
▶
2
a sum of independent individual scores: s(xi , yj ) or gap penalties
an optimal global alignment between x and y is made of optimal alignments
between subsequences (e.g. prefixes)
▶
▶
decomposition into optimal solutions to sub-problems
each sub-problem needs to be computed only once
Score of a partial alignment
21 / 43
• F(i, j) cumulative score to align x1 . . . xi with y1 . . . yj
• F(0, 0) = 0
Alignment initialization
22 / 43
3 ways to extend a partial alignment
23 / 43
1
xi aligned to yj
2
xi aligned to a gap
3
yj aligned to a gap

 F(i – 1, j – 1) + s(xi , yj )
F(i, j) = max
F(i – 1, j) – d

F(i, j – 1) – d
A backpointer stores which of the 3 possibilities is optimal (= the max)
Multiple solutions might exist
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
• The table is filled from top left to bottom right (prefixes!)
• F(n, m) = the final alignment score
• The actual alignment is found following backpointers
n
4
• Computational complexity (with n ≈ m) in O(n2 ) instead of O( √
)
n
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998); BLOSUM50
Outline
25 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Global versus local alignment
26 / 43
Global
Local
Look for conserved segments only
Naïve algorithm
27 / 43
Look for a global alignment
between subsequences
1
Apply NW from any starting point
2
For each starting point, look for the
maximal ending score
3
Look for the maximum score overall
Computational complexity
28 / 43
1
Apply NW from any starting point
2
For each starting point, look for the
maximal ending score
• O(n2 ) starting points
(with n ≈ m)
• Each NW takes O(n2 )
3
Look for the maximum score overall
• Globally O(n4 )
Smith-Waterman
29 / 43
Key idea
• reset to initial score = 0 from anywhere
• choose best between reset or pursuing current alignment
4 ways to extend a partial alignment
30 / 43
1
xi aligned to yj
2
xi aligned to a gap
3
yj aligned to a gap

0



F(i – 1, j – 1) + s(xi , yj )
F(i, j) = max
 F(i – 1, j) – d


F(i, j – 1) – d
A backpointer stores which of the 3 last possibilities is optimal, otherwise 0
Smith-Waterman algorithm
31 / 43
• End of the local alignment: position (i, j) such that F(i, j) is maximal
• Start of the local alignment: follow backpointers till 0
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
Smith-Waterman algorithm
32 / 43
Requirements:
• scores s(., .) for strong mismatches (dissimilar residues) need to be < 0 (= worse
than reset) otherwise long stretches of unrelated subsequences could be aligned
• score s(., .) for matching similar residues need to be > 0, otherwise 0 everywhere
Computational complexity (with n ≈ m) in O(n2 )
(same as Needleman-Wunsch, 11 years later!)
Outline
33 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Semi-global alignment
34 / 43
Do not penalize gaps at the beginning or at the end of either sequences
• Initialize F(i, 0) = F(0, j) = 0 ; 0 ≤ i ≤ n ; 0 ≤ j ≤ m
• Compute the global recurrence
F(i, j) = max

 F(i – 1, j – 1) + s(xi , yj )
F(i – 1, j) – d
 F(i, j – 1) – d
• End of the alignment: max value among the entries in the last column F(i, m), ∀i
or the last line F(n, j), ∀j
Local alignment with repeats
Look for several local matches of y into x (no longer symmetric between x and y)
35 / 43
Illustration from Biological Sequence Analysis
(© Cambridge University Press 1998)
dots (…) represent unaligned fragments of x
F(0, 0) = 0
F(i, 0) = max
{

 F(i,0)
F(i – 1, 0)
F(i – 1, j) – T
∀j = 1, . . . , m
F(i, j) = max
• Find local matches scoring higher than T (here T = 20)
• F(i, 0) stores the best end score so far (–T)
• F(n + 1, 0): total score of k matches –kT
9 = 49 – 40 = 21 + 28 – 2 ∗ 20
F(i – 1, j – 1) + s(xi , yj )
 F(i – 1, j) – d
F(i, j – 1) – d
Gap penalties
36 / 43
So far: a linear gap penalty γ(g) = –dg
1
xi aligned to yj
2 xi aligned to a gap
3 yj aligned to a gap

 F(i – 1, j – 1) + s(xi , yj )
F(i, j) = max
F(i – 1, j) – d

F(i, j – 1) – d
• an affine gap penalty γ(g) = –d – e(g – 1) is more relevant
• we need to distinguish whether we consider the first gap or subsequent
gaps, either in x or y
Affine gap penalty
37 / 43
Instead of a single state F(i, j), one distinguishes 3 states
{
M(i, j) = max
{
Ix (i, j) = max
{
M state : xi aligned to yj
Ix state : xi aligned to a gap
Iy state : yj aligned to a gap
Iy (i, j) = max
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
M(i – 1, j – 1) + s(xi , yj )
Ix (i – 1, j – 1) + s(xi , yj )
Iy (i – 1, j – 1) + s(xi , yj )
M(i – 1, j) – d
Ix (i – 1, j) – e
M(i, j – 1) – d
Iy (i, j – 1) – e
Outline
38 / 43
1. Sequence comparison
2. The alignment problem
3. Scoring matrices
4. Global alignment: Needleman-Wunsch algorithm
5. Local alignment: Smith-Waterman algorithm
6. Several alignment variants
7. Significance assessment
Statistical significance of the alignment score
39 / 43
Procedure
Compute pairwise random alignment
scores
3
Compute histogram of random alignment
scores
(normalized by length)
p-val = 1– percentile of actual score
between x and y
0.10
0.08
0.06
0.04
2
0.02
between x and random sequences
(e.g. 500 permutations of y)
▶ between y and random sequences
(e.g. 500 permutations of x)
▶
Histogram of scores
p−val = 0.02
0.00
1
40
50
60
70
80
Computational complexity
40 / 43
• Sound procedure but requires to compute many (e.g. 1, 000) alignments
• Computational complexity of a single alignment: O(nm) ≈ O(n2 )
• Actual distribution of alignment scores is known to follow an Extreme
Value Distribution
0.02
0.04
0.06
0.08
0.10
Histogram of scores
0.00
p−val = 0.02
40
50
60
70
80
Extreme Value Distribution
41 / 43
EVD
• Probability of a score larger than S
(
–Kmn e
)
–λS
P(x > S) = 1 – e
for some constants K and λ
• Alignment softwares include fitted values
of K and λ for a wide range of
substitution scoring matrices
• The lack of normality is due to
non-independence between possible
starting points of matches
Gaussian
Concluding remarks
42 / 43
• All the above algorithms maximizing a score can be easily adapted when
minimizing an edit distance or cost
• O(nm) ≈ O(n2 ) is too much when aligning a query sequence to a large database
of possibly homologous sequences
• BLAST (Basic Local Alignment Search Tool) is a popular heuristic algorithm to
speed-up such computation
BLAST uses a fast search method to find approximately equal small segment
pairs and greedily extends the local alignments formed by such pairs
▶ Many BLAST variants exist:
▶
• BLASTp to search a protein database with a protein query
• BLASTn to search a nucleotide database with a nucleotide sequence query
• BLASTx to search a protein database with a translated nucleotide query
blast.ncbi.nlm.nih.gov/Blast.cgi
Further Reading
43 / 43
Chapter 2: Pairwise alignment
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Hidden Markov Models
Vincent Branders
November, 2024
Substituting Pierre Dupont
Outline
2 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Outline
3 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Ab initio gene finding
4 / 40
One strand of a DNA sequence
…ACGCGTATAAT GC…CATATGACGCGT…AATCCGTAACGGTCGAAAAA …
Gene finding problem
• Find coding versus non-coding fragments
• Identify splicing between exons and introns (eukaryotes)
Illustrations from Molecular Biology of the Cell (© Garland Science 2008)
Baseline approach
5 / 40
…ACGCGTATAAT GC…CATATGACGCGT…AATCCGTAACGGTCGAAAAA …
1
Find all ORFs in the original sequence
2
Find all ORFs in random permutation(s) of the original sequence
3
Accept as significant ORFs, any ORF in the original sequence longer than a
prescribed length: e.g. the 99% percentile of random ORF lengths (p–value =
1%)
Limitations
• Coding vs non-coding fragments are not only characterized by their length
• Intron-exon boundaries also need to be identified for eukaryotes
An alternative approach
6 / 40
Learning
Given some labeled data
• estimate a statistical model M+ for coding fragments (or exons)
• estimate a statistical model M– for non-coding fragments (or introns)
Segmentation
For any new sequence to analyze
• slide a window of size L along the sequence to extract a subsequence x
• compute the log-odds ratio, to determine whether x is coding or non-coding
P(x|M )
∏L
log P(x|M+) = log ∏i=1
L
–
i=1
▶
P(xi |M+ )
P(xi |M– )
=
∑L
P(xi |M+ )
i=1 log P(xi |M– )
with P(x|Mi ), the probability of observing (generating) the subsequence x
given (from) the model Mi
• If the log-odds ratio is greater than zero, x is likely to be coding; otherwise, it is
non-coding
Questions to address
7 / 40
1
Which statistical model to represent specific fragments?
Multinomial model
Markov chain
▶ Hidden Markov model
▶
▶
2
Sliding window size L?
▶
No need to use such a fixed window size
Multinomial model
8 / 40
Emission on transitions vs. emission on the states
An equivalent representation
• each symbol is generated on a specific state
C
G
A
T
• all transition probabilities pointing to the same
state are equal
• one such model for each segment type (e.g. coding
vs non-coding)
Markov chain
9 / 40
M+
M–
C
G
C
G
A
T
A
T
• Each transition probability P(xi |xi–1 , M) is now specific to a dimer
P̂(xi |xi–1 , M) =
f(xi–1 xi )
from segments modeled by M
f(xi–1 )
• Log-odds computation
P(x|M+ ) ∑
P̂(xi |xi–1 , M+ )
log
=
log
P(x|M– )
P̂(xi |xi–1 , M– )
L
i=1
Second-order Markov chain
10 / 40
• Each transition probability P(xi |xi–1 , xi–2 , M) is specific to a 3-mer
P̂(xi |xi–1 , xi–2 , M) =
f(xi–2 xi–1 xi )
from segments modeled by M
f(xi–2 xi–1 )
• Log-odds computation
P̂(xi |xi–1 , xi–2 , M+ )
P(x|M+ ) ∑
=
log
P(x|M– )
P̂(xi |xi–1 , xi–2 , M– )
L
log
i=1
Limitations of the Markov chain approach
11 / 40
• Estimate each MC from well annotated segments
▶
the problem needs to be solved “manually” on a sufficiently large set of
segments
• Gene length variability
sliding window length is arbitrary
computation with many possible window lengths is expensive
▶ need to decide which window length is more relevant
▶
▶
• Eukaryotes
▶
▶
at least 3 models needed: out-of-genes, introns, exons
with even more segment length variability
We need to change our approach
A combined model
12 / 40
A
C
T
G
A
C
T
G
…ACGCGTATAAT GC…CATATGACGCGT…AATCCGTAACGGTCGAAAAA …
Learning
• Define a single model for all possible segments
Note: not all transitions are depicted here
• Estimate all transition probabilities simultaneously
Segmentation
Find the most likely state sequence to generate the whole sequence
Hidden Markov Model
13 / 40
A
C
T
G
A
C
T
G
A more compact representation
• transition probabilities between states
• discrete emission probabilities on states
• one state for each segment type:
(e.g. coding vs non-coding)
• states are hidden but their emissions are observed
A T C G
A T C G
Outline
14 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Hidden Markov Model
15 / 40
0.4
0.1
0.6
1
[a 0.2]
[b 0.8]
0.9
0.7
2
0.3
[a 0.9]
[b 0.1]
Definition
A discrete HMM (with state emission)
• Σ is a finite alphabet
• Q is a set of states
∑
• A a |Q| × |Q| transition probability matrix ( q′ ∈Q Aqq′ = 1)
∑
• B a |Q| × |Σ| emission probability matrix ( a∈Σ Bqa = 1)
∑
• π an initial probability distribution ( q∈Q πq = 1)
An HMM example
16 / 40
0.4
0.1
0.6
1
0.9
[a 0.2]
[b 0.8]
[
A=
0.1 0.9
0.7 0.3
]
0.7
Σ = {a, b}
[
]
0.2 0.8
B=
0.9 0.1
Note
Σ and Q need not have the same size
DNA: Σ = {A, T, C, G}
Protein: Σ = {the 20 amino acids}
0.3
2
[a 0.9]
[b 0.1]
Q = {1, 2}
[
]
π = 0.4 0.6
Path likelihood
17 / 40
The likelihood P(s, ν|M) of a sequence s = s1 . . . s|s| along a path or state sequence ν = q1 . . . q|s| in a HMM M
P(s, ν|M) =
|s|
∏
P(si , qi |M) = πq1 Bq1 s1
i=1
Aqi–1 qi Bqi si
i=2
0.4
0.1
|s|
∏
0.6
1
[a 0.2]
[b 0.8]
0.9
0.7
2
0.3
[a 0.9]
[b 0.1]
P(abb, 122|M) = 0.4 × 0.2 × 0.9 × 0.1 × 0.3 × 0.1
a
b
b
Probability of generating a sequence from an HMM
18 / 40
The likelihood P(s|M) of a sequence s = s1 . . . s|s| in an HMM M
P(s|M) =
∑
P(s, ν|M)
|s|
ν∈Q
0.4
0.1
0.6
1
[a 0.2]
[b 0.8]
P(abb|M)
=
0.9
0.7
2
0.3
[a 0.9]
[b 0.1]
P(abb, 111|M) + P(abb, 112|M)+
P(abb, 121|M) + P(abb, 122|M)+
P(abb, 211|M) + P(abb, 212|M)+
P(abb, 221|M) + P(abb, 222|M)
O(|Q||s| ) possible state sequences !!
Outline
19 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Outline
20 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Viterbi algorithm
21 / 40
ν ∗ = argmaxν P(s, ν|M)
0.7
[a 0.9]
[b 0.1]
0.9
0.4
2
1
[a 0.2]
[b 0.8]
0.6
0.4
x 0.9
x 0.2
x 0.3
x 0.1
max
x 0.1 x 0.3
max
x 0.1
x 0.9
x 0.9
x 0.7
x 0.7
x 0.8
x 0.1
x 0.8
max
max
b
b
0.1
0.6
0.3
Most likely state sequence for abb = 212
a
0.027216
max
0.024192
Viterbi recurrence
22 / 40
ν ∗ = argmax
ν P(s, ν|M)
Auxiliary quantity: γ(k, t) = P(s1 . . . st , νt∗ = k|M)
The probability of a most likely path ν ∗ reaching state k at step t
0.7
[a 0.9]
[b 0.1]
0.9
0.4
2
1
[a 0.2]
[b 0.8]
0.6
0.4
x 0.9
x 0.2
x 0.3
x 0.1
max
x 0.1 x 0.3
max
x 0.1
x 0.9
x 0.9
x 0.7
x 0.7
x 0.8
x 0.1
x 0.8
max
max
b
b
0.1
0.6
0.3
Initialization: γ(k, 1) = πk Bks1
Recurrence: γ(k, t) = maxl [γ(l, t – 1)Alk ]Bkst
back(k, t) = argmaxl [γ(l, t – 1)Alk ]
Termination: P(s, ν ∗ |M) = maxl γ(l, |s|)
q∗|s| = argmaxl γ(l, |s|)
a
0.027216
max
Time complexity: Θ(m|s|)
0.024192
Viterbi alignment
23 / 40
• P(s, ν ∗ |M) gives the probability of an optimal path ν ∗
• Computations are done usually with log’s:
– log γ(k, t) = minl [– log γ(l, t – 1) – log Alk ] – log Bkst
• The actual path ν ∗ can be recovered by backtracking from q∗
|s|
• Time complexity is Θ(m|s|) with m the number of HMM transitions
• The path ν ∗ defines an alignment between states and symbols
• This alignment defines a segmentation of the sequence : a | b | b
0.7
[a 0.9]
[b 0.1]
0.9
0.4
2
1
[a 0.2]
[b 0.8]
0.6
x 0.9
x 0.3
max
x 0.1 x 0.3
x 0.9
0.4
x 0.2
x 0.8
x 0.1
x 0.9
x 0.7
x 0.1
max
a
0.027216
max
x 0.7
x 0.1
x 0.8
max
max
b
b
0.1
0.6
0.3
2
0.024192
1
2
Outline
24 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Forward recurrence
25 / 40
P(s|M) =
∑
ν P(s, ν|M)
Auxiliary quantity: α(k, t) = P(s1 . . . st , νt = k | M)
The likelihood of emitting the first t symbols and reaching state k at time t
0.7
[a 0.9]
[b 0.1]
0.9
0.4
2
1
[a 0.2]
[b 0.8]
0.6
x 0.9
x 0.3
Σ
x 0.1 x 0.3
x 0.9
0.4
x 0.2
x 0.8
Σ
x 0.1
x 0.9
x 0.7
x 0.1
Σ
x 0.7
x 0.1
x 0.8
Σ
0.1
0.6
0.3
Initialization: α(k, 1) =∑πk Bks1
Recurrence: α(k, t) = ∑
l [α(l, t – 1)Alk ]Bkst
Termination: P(s|M) = l α(l, |s|)
a
b
b
Time complexity: Θ(m|s|)
Outline
26 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
The learning problem
27 / 40
Given an HMM structure and one or several sequence(s) to model
A T C G
A T C G
estimate the HMM parameters: A, B, π
Supervised learning
Assumption
28 / 40
The learning sequences are annotated with their respective segments
A T C G
…GCCATATGACGCGT…AATCCGTAACGGT…
A T C G
• Bki = f(k,i)
f(k)
f(k, i) = number of times symbol i is observed on state k
▶ f(k) = number of times state k is used
▶
• Akl = f(k,l)
f(k)
▶
f(k, l) = number of times a transition from state k to state l is used
1 (k)
• πk = ∑f|Q|
j=1
▶
f1 (j)
f1 (k) = number of times state k is used as first state
△
1
)
(or f1 (k) = 1 ⇒ πk = |Q|
Viterbi training
Unsupervised learning
29 / 40
Fix initial parameter values A0 , B0 , π 0
2 repeat
1
1
2
3
compute a most likely path through a Viterbi alignment for each learning
sequence given the parameters Ai , Bi , π i
estimate the emission and transition relative frequencies from such path(s)
recompute the parameter values Ai+1 , Bi+1 , π i+1 from those frequencies
untill some stopping criterion is met (e.g. max number of iterations)
Forward-Backward or Baum-Welch Algorithm
Unsupervised learning
30 / 40
• Viterbi training is an approximation as it considers that each learning
sequence is generated along a single path
(a most likely one)
• A more accurate estimation is obtained if one considers all possible paths
to generate each sequence
▶
▶
actual frequencies are replaced by expected frequencies
special case of expectation-maximization (EM) procedure
Outline
31 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Outline
32 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
GC content
33 / 40
Haemophilus influenzae (NC_000907)
Haemophilus influenzae (GC content anomaly)
A−T
G−C
0.3
0.3
0.4
0.4
0.5
0.5
0.6
0.6
0.7
0.7
A−T
G−C
0.8
0.8
Haemophilus influenzae (window=20,000 bp)
0
500000
1000000
1500000
1500000
1550000
1600000
1650000
1700000
CpG islands
34 / 40
0.8
Haemophilus influenzae (window=20,000 bp)
0.3
0.4
0.5
0.6
0.7
A−T
G−C
0
500000
1000000
1500000
• GC-rich regions are also called CpG islands (̸= GC or CG dimers)
• Baseline analysis use a sliding window of a prescribed length
short length: tends to be noisy
large length: may miss some short islands
▶ actual length varies
▶
▶
2-state HMM to model CpG islands
35 / 40
A T C G
A T C G
• GC-rich state: higher probability to emit G or C
• alternative state: lower probability to emit G or C
• transition probabilities reflect that is much more likely (e.g. 0.998) to stay
in a given state than switching regime (e.g. 0.002)
• initial probabilities reflect the chance to start or not with a CpG island (by
default, π1 = π2 = 0.5)
Those probabilities can either be a priori fixed or estimated through supervised
learning, Viterbi or Baum-Welch
Segmentation into CpG islands
36 / 40
Outline
37 / 40
1. Motivating example: gene finding
2. HMM definition
3. The 3 fundamental questions
3.1 How to compute the most likely state sequence?
3.2 How to compute a sequence likelihood?
3.3 How to estimate HMMs parameters?
4. Back to concrete examples
4.1 Segmentation into CpG islands
4.2 Profile HMMs
Profile HMMs
38 / 40
• The topology of a profile HMM reflects the structure of a multiple
alignment between proteins belonging to the same family
• A new sequence can be aligned to the model to segment it in conserved
versus less conserved parts
Further reading
39 / 40
Chapter 3: Markov chains and hidden Markov models
Software tools
40 / 40
• R packages
cran.r-project.org/web/packages/HMM/
cran.r-project.org/web/packages/hmm.discnp/
• HMMER: biosequence analysis with profile HMMs
hmmer.org/
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Multiple alignment - Profile HMMs
Vincent Branders
November, 2024
Substituting Pierre Dupont
Outline
2 / 32
1. Multiple alignment
2. Profile HMMs
Outline
3 / 32
1. Multiple alignment
2. Profile HMMs
The Multiple Alignment Problem
4 / 32
• Arrange sequences into rectangular arrays with the goal that residues in a
given column are
homologuous: derived from a common ancestor;
superposable: in a 3D structure alignment;
▶ or play a common functional role: catalytic site, localisation signal,
protein-proetin interaction site, …
▶
▶
• Looking for conserved segments (either globally or locally)
Why doing multiple alignment ?
5 / 32
1
Detecting similarities, conserved regions or motifs, between sequences
(closely or distantly related)
▶
2
Detecting structural patterns (hydrophobicity, hydrophilicity, …)
▶
3
4
Basis for secondary and tertiary structure prediction
Computing consensus sequence
Describing patterns or profiles that can be used to assign new sequences
into families
▶
▶
5
Which may have structural or functional importance
e.g., the G-Protein Coupled Receptor
Also allow to remove distant sequences (outliers) from protein families
Inferring evolutionary trees
…
Multiple sequence alignment as an optimization problem
6 / 32
• How to optimally align sequences ?
All sequences show some similarity (even random sequences)
Similarity levels might vary throughout the sequences
▶ Sequences might show important length variation and presence or absence
of various domains
▶
▶
Idea: define the alignment as a combinatorial optimization problem
Questions
• How to define the scoring system?
• How to align the sequences to optimize the score?
Scoring a multiple alignment
7 / 32
Classical approach
• Assume statistical independence between columns
• A multiple alignment m with L columns can then be scored as
S(m) =
▶
S(mi ) = score for column i
∑L
i=1 S(mi )
Sum-of-Pairs score
8 / 32
Sum of pairs
• Column score:
S(mi ) =
▶
▶
∑
k
l
k<l s(mi , mi )
where mki is the residue in sequence k in column i
and s(a, b) is given by a substitution scoring matrix
(e.g. PAM or BLOSUM)
• Gap penalty
▶
▶
linear: s(a, –) = s(–, b) = –d ; s(–, –) = 0
affine: all gaps are scored separately
Is PAM or BLOSUM relevant?
Optimal alignment through dynamic programming
From 2 to 3 sequences
9 / 32
2-D edit lattice
Illustrations from www.bioalgorithms.info
3-D edit lattice
Optimal alignment through dynamic programming
From 2 to 3 sequences
10 / 32
22 – 1 = 3 possible moves
23 – 1 = 7 possible moves
Illustrations from www.bioalgorithms.info
2k – 1 : include or not a residue for each sequence,
exclude the combination where no sequence include a residue
Dynamic programming
11 / 32
• Optimal alignment between k sequences can be computed through dynamic
programming
• The time complexity for k sequences of average length n̄ is in O(2k n̄k ) and
the space complexity in O(n̄k ) (for storing the hyper-cube)
▶
in practice, computation must be limited to very few sequences due to the
exponential growth with k
MSA algorithm
12 / 32
• MSA (multiple sequence alignment) is an optimized dynamic programming
algorithm which first computes all pairwise alignments and then limits the
exploration of the (hyper-)cube to regions consistent with those alignments
▶
▶
time complexity in O(k2 n̄2 ) but somewhat complex to program
MSA can optimally align ≈ 10 sequences of up to 200-300 residues in
reasonable time
• a parallel extension G-MSA is reported to align up to 500 sequences of 236
residues on average within 10 seconds on a Linux machine including 2 cores with
GPUs [J. Blazewicz et al., 13]
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
Heuristic algorithms
13 / 32
• sub-optimal heuristic algorithms have been designed to reduce the
computational cost of computing a multiple alignment
• progressive alignment methods greedily aggregates pairs of aligned
sequences into groups
• CLUSTALW is a popular heuristic algorithm
▶
organize the set of sequences in a phylogenetic tree using the
neighbor-joining algorithm
▶
use it as a guide tree to align closest sequences first and progressively align
sequences of decreasing similarity
Outline
14 / 32
1. Multiple alignment
2. Profile HMMs
Beyond multiple alignments
15 / 32
Motivations
• multiple alignments are most often based on pairwise alignments
• a new sequence x may be only distantly and locally related to each
sequence in a known family (biological question)
▶
▶
all pairwise alignments between x and each family members may look poor
need to model statistical features shared by the family members
• computing the alignment between x and a probabilistic model of the family
may be much more efficient computationally
Probabilistic models of a known family
16 / 32
Questions
1
given a multiple alignment between sequences how to build a global/local
model M from it?
2
how to compute the matching score between a query sequence x and the
model M?
3
how to get rid of the initial alignment?
Position-specific scoring matrices (PSSM)
17 / 32
• Local model for a window length L and ungapped score matrix from N
sequences
∏
∏
f(x )
P(x|M) = Li=1 P(xi |M) = Li=1 Ni
• Log-odds score
∑
P(x |M)
S = Li=1 log qi
xi
▶
qxi = the background model (e.g. multinomial model)
• One evaluates the score S between x and M for all positions xi and a sliding
window of size L
Illustration from http://sites.google.com/site/iiserbioinformatics/tutorials
PSSMs are very simple HMMs
18 / 32
P(x|M) =
∏L
i=1 P(xi |M)
• P(xi |M) are emission probabilities on match states
• transition probabilities are all equal to 1 (linear structure)
• need to account for possible gaps
• better to avoid a prescribed window length L
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
Adding insert states
19 / 32
• to account for portions of x that do not match the model
• emission probabilities on insert states are typically defined through the
background model
▶
no contribution to the log-odds score log
q
P(xi |Ij )
= log qxi = 0
q xi
xi
• transition probabilities to insert states and back are equivalent to affine gap
penalties log AMj Ij + log AIj Mj+1 + (k – 1) log AIj Ij
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
Adding delete states
20 / 32
• portions of the model M that are not matched by any residue xi could be
modeled by skipping transitions
• to allow arbitrary long gaps it is more convenient to introduce delete states
which are silent states
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
A full profile HMM
21 / 32
Deriving a pHMM from a multiple alignment
22 / 32
• a match state for each conserved position (e.g. at least 50% non-gaps)
• insert states: e.g. columns with at least 50% gaps
• delete states: gaps on match positions
• emission (for match states) and transition probabilities are estimated from the
counts
▶
f(k,i)
Bki = f(k)
• f(k, i) = number of times symbol i is observed on state k
• f(k) = number of times state k is used
▶
f(k,l)
Akl = f(k)
• f(k, l) = number of times a transition from state k to state l is used
Illustrations from Biological Sequence Analysis (© Cambridge University Press 1998)
Smoothing probability estimates
23 / 32
Whenever the initial multiple alignment is limited to a few sequences, some
emission/transition probabilities may be null
Additive smoothing with pseudo-counts
∑ with e.g. 10–6 ≤ ε ≤ 1
• Bki = f(k,i)+ε
f(k)+
i
ε
f(k, i) = number of times symbol i is observed on state k
▶ f(k) = number of times state k is used
▶
′
∑ ′ with e.g. 10–6 ≤ ε′ ≤ 1
• Akl = f(k,l)+ε
f(k)+
▶
l
ε
f(k, l) = number of times a transition from state k to state l is used
Unsupervised learning
24 / 32
Objective: no need for an initial multiple alignment but just a collection of
unaligned sequences
Procedure
1
choose a general pHMM structure
2
choose the number of match states:
3
e.g. half the average sequence length
estimate the pHMM parameters through Viterbi training or Baum-Welch
▶
aligning you family of sequences
Matching a sequence to a pHMM
Viterbi recurrence
25 / 32
• Computations are done usually with log’s:
– log γ(k, t) = minl [– log γ(l, t – 1) – log Alk ] – log Bkxt
• Including a background model to produce a log-odds score:
B
log γ(k, t) = maxl [log γ(l, t – 1) + log Alk ] + log qkxt
xt
0.7
[a 0.9]
[b 0.1]
0.9
0.4
2
1
[a 0.2]
[b 0.8]
0.6
x 0.9
x 0.3
max
x 0.1 x 0.3
x 0.9
x 0.2
x 0.1
x 0.8
x 0.1
x 0.9
x 0.7
0.4
max
a
0.027216
max
x 0.7
x 0.1
x 0.8
max
max
b
b
0.1
0.6
0.3
• A background model can similarly be included into the forward recurrence
0.024192
Silent states
26 / 32
Begin, End and D states are silent = non-emitting
Computing with silent states
27 / 32
1
All to emitting
2
Emitting to silent
3
Silent to silent
pHMM for non-global alignments
28 / 32
• non-conserved fragments are modeled through flanking insert states using
the background emission probabilities
• flanking delete states allow for starting or ending the profile at any point
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
pHMM for non-global alignments
29 / 32
• forcing the match of the complete profile (or own delete states)
• no flanking delete states
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
pHMM for non-global alignments
30 / 32
• allowing repeated matches to subsections of the profile
Illustration from Biological Sequence Analysis (© Cambridge University Press 1998)
Further reading
31 / 32
▶
▶
Chapter 5: Profile HMMs for sequence families
Chapter 6: Multiple sequence alignment methods
Blazewicz, J., Frohmberg, W., Kierzynka, M. and Wojciechowski, P.
G-MSA - A GPU-based, fast and accurate algorithm for multiple sequence
alignment
Journal of Parallel Distributed Computing, Vol. 73, p. 32–41, (2013).
http://dx.doi.org/10.1016/j.jpdc.2012.04.004
Database and software tools
32 / 32
• Multiple Sequence Alignment by CLUSTALW
www.genome.jp/tools/clustalw/
• PFAM: database a protein families represented as MSA and HMMs
pfam.xfam.org/
• HMMER: biosequence analysis with profile HMMs
hmmer.org/
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Large scale gene expression analysis
Vincent Branders
December, 2024
Substituting Pierre Dupont
Outline
2 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Outline
3 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Gene expression
4 / 65
Illustration from Molecular Cell Biology, 5e (© WHFreeman 2004).
DNA Microarrays
5 / 65
DNA Microarrays measure the level of expression of
all genes in a single experiment
1
Data measurements
2
Preprocessing and sample
normalization
3
Gene selection and sample
classification
4
Diagnosis, prognosis or prediction
of the response to a treatment
Affymetrix® technology
6 / 65
Alternative measurement technologies
7 / 65
• Other companies sell DNA chips (Agilent®, …)
• Multiplex qPCR (Applied Biosystems®, …)
▶
▶
more sensitive to fluctuation of gene expression values
limited to ≈ 100 genes
• RNAseq (Illumina®, Ion Torrent®, …)
▶
▶
fastly evolving
scaling effects influence per sample cost
From gene expression measurement to marker discovery
8 / 65
Objective
Find a subset of genes whose expression is correlated with
• a diagnosis
• a prognosis
• a response to treatment
Gene selection
9 / 65
Objectives
• Insight into the data and the predictive model
• Link between data analysis and medical expert
• Biological validation on a few genes rather than thousand ones
• Reduction of the financial cost of a diagnosis/prognosis kit (technological
constraints)
Difficulties
• Measurements are noisy
• Gene expression varies due to many factors (gender, cell type, growth of
the organism, chemical environment of the cell, …) often not related to the
response to be predicted
• Financial cost: 500 . . . 1, 000 €/experiment
• Small n (e.g. 50), large p (e.g. 50, 000) problems
Supervised selection
10 / 65
sample 1
...
sample n
test sample
gene 1
x1,1
...
xn,1
x1
gene 2
x1,2
...
xn,2
x2
...
...
...
...
...
gene p
x1,p
...
xn,p
xp
response
y1
...
yn
?
• The number p of input dimensions (probes, probesets or genes) may be
very large (104 . . . 106 )
• The number n of samples is typically much smaller (≈ 50 . . . 100)
• Each sample is characterized by a vector x of p measurements
• Each training sample has a known response: class label y (y ∈ {–1, 1} or
y ∈ N) or y ∈ R
Gene selection
Find a small subset of genes, (a.k.a features, attributes or input variables), to
predict the response or class y of new samples
Unsupervised selection
11 / 65
sample 1
...
sample n
cluster
gene 1
x1,1
...
xn,1
?
gene 2
x1,2
...
xn,2
?
Objective
Find clusters of genes and/or samples that
share a similar profile: up or down regulated
genes across the same samples
Heatmap produced by agglomerative
clustering respectively on rows (samples) or
columns (genes)
...
...
...
...
...
gene p
x1,p
...
xn,p
?
cluster
?
...
?
Outline
12 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
DNA Microarray Summarization
13 / 65
• Define a single probeset expression level from the various probe intensities
• Popular techniques: MAS 5.0, RMA, GC-RMA
1
2
3
background adjustment: optical noise correction, probe affinity adjustment
(influenced by the GC content), RMA ignores the MM probes
sample normalization: quantiles should be stable across samples, after
conversion to log intensities for (GC-)RMA
summarization: median polish
Feature normalization
14 / 65
sample 1
...
sample n
gene 1
x1,1
...
xn,1
gene 2
x1,2
...
xn,2
...
...
...
...
gene p
x1,p
...
xn,p
• Make sure that each gene (probeset) has roughly the same expression range
across all samples
• Z-score normalization
x –µ
Replace xi,j by i,jsj j with µj the mean level of expression of probeset j over
the training samples and sj its standard deviation
Feature normalization example
15 / 65
5
After
5
Before
●
●
●
●
●
● ●
●●
x2
●● ●
●
●
●
●
●
●●
●
0
0
●
●
●
●
●
x2
●●
●
●
●
●
●
● ●
●
●● ●
●● ●
● ●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
−5
−10
−10
−5
●
−10
−5
0
x1
5
−10
−5
0
x1
5
Distance between expression values
16 / 65
sample 1
...
sample n
gene 1
x1,1
...
xn,1
gene 2
x1,2
...
xn,2
Euclidean distance
...
...
...
...
gene p
x1,p
...
xn,p
v
u n
uX
d(x1 , x2 ) = ∥x1 – x2 ∥ = t (xi,1 – xi,2 )2
i=1
Outline
17 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Unsupervised selection
18 / 65
sample 1
...
sample n
cluster
gene 1
x1,1
...
xn,1
?
gene 2
x1,2
...
xn,2
?
Objective
Find clusters of genes and/or samples that
share a similar profile: up or down regulated
genes across the same samples
Heatmap produced by agglomerative
clustering respectively on rows (samples) or
columns (genes)
...
...
...
...
...
gene p
x1,p
...
xn,p
?
cluster
?
...
?
Agglomerative clustering example
19 / 65
Euclidean distance
This is the same algorithm as UPGMA used
in phylogeny but
• the distance matrix is computed from
e.g. euclidean distances between genes
(across samples) or between samples
(across genes) instead of being derived
from alignment scores from taxa
• most commonly, the single link rule
(dkl = min(dil , djl )) or complete link
rule (dkl = max(dil , djl ))
is used to update the distance matrix
when merging 2 clusters Ck = Ci ∪ Cj
d(x1 , x2 ) =
A
B
qP
n
i=1 (xi,1 – xi,2 )
C
D
E
A
B
8
C
6
2
D
5
8
6
E
7
6
4
3
F
8
6
5
4
1
F
2
Agglomerative clustering example
20 / 65
5
4
3
2
1
A
B
A
C
D
B
C
E
F
D
E
A
B
8
C
6
2
D
5
8
6
E
7
6
4
3
F
8
6
5
4
1
F
Agglomerative clustering example
21 / 65
5
4
3
2
1
A
B
A
C
D
B
C
E
F
D
E EF
EF F
A
Single-link rule
B
8
C
6
2
D
5
8
6
7
6
4
3
8
6
5
4
E
F
1
Agglomerative clustering example
22 / 65
5
4
3
2
1
A
B
C
A
D
E
B
C
F
D
EF
A
B
8
C
6
2
D
5
8
6
EF
7
6
4
3
8
6
5
4
1
Agglomerative clustering example
23 / 65
5
4
3
2
1
A
B
C
D
A
E
BC
F
D
EF
A
Single-link rule
BC
8
6
2
D
5
8
6
EF
7
6
4
3
8
6
5
4
1
Agglomerative clustering example
24 / 65
5
4
3
2
1
A
B
C
D
A
BC
E
F
D
A
BC
6
D
5
6
EF
7
4
3
EF
Agglomerative clustering example
25 / 65
5
4
3
2
1
A
B
C
D
A
BC
E
F
DEF
A
BC
Single-link rule
DEF
6
5
6
7
4
3
Agglomerative clustering example
26 / 65
5
4
3
2
1
A
B
C
D
A
E
BC
A
BC
6
DEF
5
4
F
DEF
Agglomerative clustering example
27 / 65
5
4
3
2
1
A
B
C
D
A
E
BCDEF
A
Single-link rule
BC
DEF
6
5
F
4
Agglomerative clustering example
28 / 65
5
4
3
2
1
A
B
C
D
E
A
BCDEF
A
BC
DEF
5
F
Outline
29 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Supervised gene selection
30 / 65
sample 1
sample 2
...
sample n-1
sample n
test sample
gene 1
x1,1
...
...
...
xn,1
x1
gene 2
x1,2
...
...
...
xn,2
x2
...
...
...
...
...
...
...
gene p
x1,p
...
...
...
xn,p
xp
class
+
+
...
?
• we discuss binary classification first:
e.g. responders (+ or class 1) vs non-responders (- or class 2)
• samples can be indexed by their class label
▶
means and variances can be computed on samples of a given class
• find a subset of most discriminating genes for the prediction of the class of
any new sample
Feature selection methods
31 / 65
Supervised feature selection methods are categorized as follows:
• Filters: Model-independent methods
• Wrappers: Use a predictive model to evaluate subsets of features
• Embedded: Integrate feature selection directly into the model training
process
Feature selection: filters
32 / 65
• Use only the training data + class labels during the feature selection step
• Standard techniques: fold changes, t-Test, mutual information, . . .
• Train a single classifier taking the selected features as inputs
• The simplest and less computing intensive approach
Feature selection: wrappers
33 / 65
• Train a classifier on several subsets of all possible features
Exhaustive evaluation of all possible subsets is unfeasible
Note: there are O(2p ) subsets with p ≥ 10, 000
▶ Typical solutions: use feature ranking or forward/backward selection
▶
• Select the feature set that optimizes the performance of the trained
classifier
Feature selection: embedded approaches
34 / 65
• Define the feature selection and the classifier estimation as a combined
optimization process
• Include classifier optimization in the feature selection process
• More elegant/relevant but also more computing intensive than a filter
Outline
35 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Non-specific filtering
36 / 65
gene 1
x1,1
...
...
...
xn,1
gene 2
x1,2
...
...
...
xn,2
...
...
...
...
...
...
gene p
x1,p
...
...
...
xn,p
class
+
+
...
-
▶
before normalization to unit variance!
0.50
0.20
0.05
0.02
• genes with a small variance across all
training samples are unlikely to be
discriminating between classes
• keep only those genes (e.g. 25 %) with
the larger variances
2.00
5.00
sample 1
sample 2
...
sample n-1
sample n
0
2000
4000
6000
8000
10000
12000
Fold changes
37 / 65
sample 1
sample 2
...
sample n-1
sample n
gene 1
x1,1
...
...
...
xn,1
gene 2
x1,2
...
...
...
xn,2
...
...
...
...
...
...
gene p
x1,p
...
...
...
xn,p
class
+
+
...
-
Select genes with the larger fold changes between both conditions
x̄+
x̄+
x̄– or log x̄– = log x̄+ – log x̄– or x̄+ – x̄–
Comments on fold changes
38 / 65
• whenever x̄+ < x̄– , one considers a small value as important
▶
log2 x̄x̄+– should be ≥ 1 or ≤ –1
• is a two-fold change significant?
▶
▶
dependence on the measurement technology
dependence on the class conditional variance
Log_2 FC = 1.468
Log_2 FC = 0.25
0
1.1
1
2
1.2
3
1.3
4
●
1.0
−1
●
−2
●
●
●
●
t-Test relevance index
39 / 65
• A feature relevance J(x) can be defined according to the distance between
the average feature value in each class
• The larger the distance the better, relatively to standard deviations
t-Test statistic (actually Welch t-statistics)
x̄+ – x̄–
J(x) = q
S2+ /n+ + S2– /n–
with n+ (resp. n– ) the number of examples labeled as + (resp. –) and the estimated
Pni
variances in each class S2i = ni1–1 j=1
(xij – x̄i )2
Confidence measure
40 / 65
• The Welch statistics follows a t-distribution with a number of degrees of
freedom equal to:
(S2+ /n+ +S2– /n– )2
(S2+ /n+ )š(n+ –1)+(S2– /n– )š(n– –1)
0.1
0.2
0.3
0.4
• p-values assess the significance of the difference between the two class
means
0.0
2.5 %
2.5 %
−2.228
−3
−2
2.228
−1
0
1
2
3
• A feature is selected if its associated p-value is below a prescribed threshold
(e.g. 5% ⇒ |J(x)| ≥ 2.228 when d.f. = 10 )
The R Project for Statistical Computing
An efficient way of computing p-values, and many other useful things…
41 / 65
http://www.r-project.org/
> t.test(x1,x2)
Welch Two Sample t-test
data: x1 and x2
t = 0.9183, df = 7.002, p-value = 0.389
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.908216 2.061549
sample estimates:
mean of x mean of y
1.866667 1.290000
where x1 (resp. x2) is the vector of expression values of a given gene from samples
labeled as class 1 (resp. class 2)
p–value > 0.05 ⇒ the difference between the 2 class means is not considered
significant for this feature ⇒ discard the feature
Alternatives to a simple t-Test
42 / 65
• Mann-Whitney rank test is an alternative non-parametric test
• ANOVA offers a generalization of the t-Test in a multi-class (> 2) setting
• Pairwise t-tests between one class and the others is a common alternative
• Kruskal-Wallis is a generalization of Mann-Whitney to multi-class
The multiple test problem
43 / 65
A microarray experiment to distinguish between patients with a positive or a negative
diagnosis
Among 50,000 gene expression values measured in each experiment, only those genes that are
differently expressed, with a p-value ≤ 0.05 = α, are selected
The probability of type I error of a statistical test
Conclude that the mean expression values among the 2 classes are significantly
different for a given gene while they are not ⇒ the feature is falsely selected with
probability α
Test multiplicity
• The test will be performed for each gene ⇒ 50,000 times from the same
experiment
• If α = 0.05, we are expecting to select wrongly 50, 000 × .05 = 2, 500 genes !!
Multiple test correction
44 / 65
Bonferroni correction
Divide the critical value (e.g. α = .05) by the number of tests nt performed
.05 = 10–6
Example: nαt = 50,000
⇒ only genes with associated p-values ≤ 10–6 are selected
Very conservative, often leads to select no feature
False Discovery Rate correction
Benjamini-Hochberg correction
45 / 65
1
Select a confidence level α (e.g. 0.05)
2
Rank the p-values (one for each feature) in increasing order
p1 ≤ p2 ≤ · · · ≤ pnt
3
Iterate over the nt features
Find the maximal index i such that
▶
4
pi × nt
<α
i
Keep all features up to index imax
Notes:
• If pn < 0.05 FDR correction leads to select all features
t
• FDR correction is equivalent to Bonferonni correction whenever a single feature is
selected: p1 × nt < α ⇔ p1 < nα
t
• Those corrections do not change the relative ranking of features, just the selection
threshold
• See R function p.adjust
Feature ranking with mutual information
46 / 65
I(X; Y) = –
X
P(xi , yj ) log2
P(xi , yj )
P(xi )P(yj )
P(xi , yj ) log2
P(yj |xi )
P(yj )
ij
=–
X
ij
• A feature X is more relevant if its mutual information with the class value
is higher
• If X tends to bring no information to predict Y then
P(yj |xi ) ≈ P(yj ) and I(X; Y) → 0
• I(X; Y) = 0 if and only if X and Y are independent
• I(X; Y) is invariant under rescaling of the variables X and Y
(often rescaled to unit variance)
Univariate versus multivariate filters
47 / 65
• Correlation measures, t-Test (ANOVA), and I(X; Y) are univariate filters
• Mutual information can be used to select several variables at a time
I(X1 , . . . , Xk ; Y) but MI depends on the distributions P(X1 , . . . , Xk , Y),
P(X1 , . . . , Xk ) and P(Y), which need to be reliably estimated
▶
replace the joint problem by an approximation with a greedy selection of
features
Maximum relevance minimum redundancy
[Peng et al., 05]
48 / 65
1
Select the feature with maximum mutual information with the response
X̂ = argmaxX I(X; Y)
Φ = {X̂}
▶ F = {X , . . . , X } \ {X̂}
p
1
▶
▶
2
Repeat
// Initialize the set of selected features
// The remaining set of features
1 X
X̂ = argmaxX∈F
I(X; Y)
–
I(X; Xj )
| {z }
|Φ|
Xj ∈Φ
maximize relevance |
{z
}
minimize redundancy
Φ ← Φ ∪ {X̂} ; F ← F \ {X̂}
until an appropriate number of features are selected
Filters in a nutshell
49 / 65
• Use only the training data + class labels during selection
• Filters offer interesting baselines which are fast to compute
• Popular univariate filters are based on a t-Test (with multiple test
correction) or mutual information
▶
they ignore the interactions between genes!
• Maximum relevance minimum redundancy is a popular multivariate
extension
Are filters independent from a predictive model?
50 / 65
• The two step approach is sometimes considered as a benefit since the
features are claimed to be selected independently from the subsequent
classifier/regression model
▶
▶
Is it really better? (see embedded methods)
Is it really the case?
t-Test revisited
51 / 65
t-Test statistic
A feature x is selected whenever the difference between the class means is
significant (after correction for multiplicity)
x̄+ – x̄–
J(x) = q
2
S+ /n+ + S2– /n–
0.0 0.1 0.2 0.3 0.4
0.0 0.1 0.2 0.3 0.4
Equivalently, one easily discriminates between the classes using 2
uni-dimensional Gaussian classifiers (with a common variance)
−5
0
5
10
−4
−2
0
2
4
6
Outline
52 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Wrapper principle
53 / 65
• Estimate a classifier from a given subset of all possible features
• Select the feature subset that optimizes the performance of the classifier
(usually on an independent validation set)
▶
▶
Feature selection depends on the evaluation protocol of the classifier
There are O(2p ) possible subsets
Univariate feature ranking
54 / 65
Multivariate Forward/Backward selection
55 / 65
• Forward selection goes bottom-up
• Backward selection goes top-down
Search order matters
56 / 65
• x3 alone is better than x1 or x2 alone, but
x1 together with x2 offer the best
discrimination
• Single best feature:
Univariate feature ranking selects x3
Forward selection selects x3
▶ Backward selection selects x
2
▶
▶
• 2 best features:
Univariate feature ranking selects (x3 , x2 )
Forward selection selects (x3 , x1 )
▶ Backward selection selects (x , x )
1 2
▶
▶
Wrappers in a nutshell
57 / 65
• A wrapper with univariate feature ranking offers a good baseline
The t-Test statistics can be used to rank features only (no need for multiple
test correction nor fixing a confidence measure)
▶ Classifier performance is used to decide how many features to keep
▶ This can outperform a pure filter approach while not increasing much the
computational cost
▶
• A backward selection may be preferable over a forward selection, but
should not be used to select just a few features (what “a few” means
depends on the data. . . )
• More sophisticated search strategies are possible (backward + forward,
randomized search, …)
• If one can afford the computational cost of a multivariate selection, one
should probably consider an embedded approach
Outline
58 / 65
1. Introduction
2. Preprocessing
3. Unsupervised gene selection
4. Supervised gene selection
4.1 Filters
4.1.1 Non-specific filtering
4.1.2Fold changes
4.1.3t-Test
4.1.4Mutual information
4.1.5Multivariate filters
4.2 Wrappers
4.3 Embedded Methods
Embedded Methods
59 / 65
• Define the feature selection and the classifier estimation as a combined
optimization process
• The features are selected as a by-product of the estimated classifier and its
parameters
Linear Discriminants
60 / 65
• The actual number of dimensions may be ≈ 10, 000 for microarray
classification
• The linear discriminant is a hyperplane in R≥10,000
P
• Decision rule: sign ( pj=1 wj xj + w0 ) = sign (w⊤ x) ( with x0 △
=1)
⇒ |wj | is a measure of the importance of the jth feature
Linear Separability
61 / 65
Facts
• The data is linearly separable if the two classes can be perfectly separated
by a hyperplane
• A hyperplane in R10,000 can separate perfectly at least 10, 001 (unaligned)
points, given any possible 2 class labeling
• There is no problem to find a perfect linear separator of less than 100
points in R≥10,000 (e.g. for microarray data)
• The problem is that there are many apparently perfect models
Linear Support Vector Machines in a nutshell
62 / 65
• When the data is linearly separable the separating hyperplane is not unique
but the maximal margin hyperplane separates the data with the largest
margin
• For each separating hyperplane, there is an associated set of support vectors
Recursive Feature Elimination [Guyon et al. , 02]
63 / 65
Embedded Backward Selection
1
Estimate a SVM on a given set of dimensions
▶
Decision rule: sign (
Pp
2
Consider |wj | as the relevance of the jth dimension
3
Remove the least relevant dimension(s)
Iterate 1 to 3 on a reduced feature set
4
▶
(initially p dimensions)
j=1 wj xj + w0 )
until a fixed number of features is selected or according to accuracy on
validation set
General Summary
64 / 65
• Feature selection aims at reducing the dimensionality of the data while
preserving the interpretation of the original features
• Filters methods use only the data + class labels:
▶
simple, fast, generally univariate (often an implicit use of a classifier)
• Wrappers take the performance of the classifier into account
▶
▶
Multivariate as soon as the classifier is multivariate
Often computing intensive
• Embedded methods take the structure of the classifier into account
More elegant and often faster than wrappers, not always better in terms of
performance
▶ A way to get an insight into a black-box classifier
▶
Further Reading
65 / 65
Guyon, I., Gunn, S., Nikarvesh, M. and Zadeh, L.A. (editors) (2006).
Feature Extraction: Foundations and Applications.
Springer.
Bolstad, B.M., Irizarry, R.A., Astrand, M. and Speed, T.P. (2003)
A comparison of normalization methods for high density oligonucleotide array data based
on variance and bias
Bioinformatics, Vol. 19 (2), pp. 185-193.
Guyon, I., Weston, J., Barnhill, S., and Vapnik, V. (2002).
Gene selection for cancer classification using support vector machines.
Machine learning, 46, 389–422.
Peng, H., Long, F., and Ding, C. (2005).
Feature selection based on mutual information criteria of max-dependency,
max-relevance, and min-redundancy.
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, N° 8, pp.
1226-1238
UCLouvain
Introduction to Bioinformatics
LGBIO2010: Predictive Modeling
Vincent Branders
December, 2024
Substituting Pierre Dupont
Outline
2 / 25
1. Linear models
2. Tree-based models
3. Final exam
From biomarker measurement to prediction
3 / 25
Objective
Find a model that is predictive of an outcome: a diagnosis, a prognosis, a
response to treatment
• such a model is a function mapping the input (= measured markers) to an output
(= the outcome)
• the model is estimated (= learned) on former patient data to predict the
outcome for new patients
• wrappers and embedded methods use predictive models to select markers (e.g.
genes), we detail here how to learn such models
sample 1
...
sample n
test sample
marker 1
x1,1
...
xn,1
x1
marker 2
x1,2
...
xn,2
x2
...
...
...
...
...
marker p
x1,p
...
xn,p
xp
outcome
y1
...
yn
?
Outline
4 / 25
1. Linear models
2. Tree-based models
3. Final exam
Linear discriminants
Binary classification
5 / 25
• Binary classifier: yi = ±1 (responders vs non-responders)
Linear Decision Boundary: g(x) = 0
6 / 25
g(x) =
∑p
j=1 wj xj + w0 = ⟨w, x⟩ + w0
• Any point x is on the decision surface if and only if
g(x) = 0. Any x in R1 (resp. R2 ) is on the positive side:
g(x) > 0 (resp. negative side, g(x) < 0)
Binary classifier:
(
)
f(x) = sign g(x) = sign (⟨w, x⟩ + w0 )
• The parameters of this decision boundary have to be
estimated from learning data:
the weight vector w controlling the orientation of the
hyperplane
▶ the intercept w , controlling the distance to the origin
0
w0
w0
√
=
∥w∥
▶
⟨w,w⟩
Illustration from Pattern Classification, Duda et al., Wiley-Interscience, 2007.
Linear Support Vector Machines
7 / 25
• When the data is linearly separable, which is guaranteed whenever
n ≤ p + 1, the separating hyperplane is not unique but the maximal margin
hyperplane separates the data with the largest margin
• For each separating hyperplane, there is an associated set of support vectors
Outline
8 / 25
1. Linear models
2. Tree-based models
3. Final exam
Decision Tree Representation
9 / 25
A (simplified) example: high risk (+) vs low risk (-) of breast cancer
BRCA1
not mutated
• each internal node tests a feature
mutated
Smoker
• each branch corresponds to a feature value
Age
no
yes
< 55
>=55
−
+
−
+
• each leaf node predicts a class label
• each path from the root to a leaf
represents a conjunction (= logical AND)
of feature values
IF BRCA1 gene is mutated AND Age < 55
THEN patient is predicted at low risk (-)
CART Learning Algorithm (basics)
10 / 25
Algorithm SimpleCART
Input: A labeled training dataset
S = {(x1 , y1 ), . . . , (xn , yn )}
Input: The root node of the current (sub-)tree
Output: A classification tree T
if S contains several distinct class labels then
Assign best feature xj to node
foreach value v of xj do
Add new nodev as a child of node in T
Attach Sjv = {x ∈ S|xj = v} to nodev
SimpleCART(Sjv , nodev )
// yi is the class of xi
// Initially, T is a single node tree
// Current node is not pure
// Feature selection at current node
// Split S according to xj values
else
Assign unique class label to node
return T
• there can be more than 2 distinct class labels
Notes:
Ex: yi = low, medium, or high (risk/response)
• discrete features can have more than 2 values
Ex: xj a gene with 4 possible levels of expression
Evaluate the level of impurity of a tree node
11 / 25
…
...
...
...
xp
x1,p
...
xn,p
class
y1
...
yn
0.2
x2
x1,2
...
xn,2
0.1
x1
x1,1
...
xn,1
0.0
example 1
...
example n
0.3
0.4
0.5
Gini function (2 classes)
0.0
0.2
0.4
0.6
0.8
p+
• A tree node is pure if all examples assigned to it have the same class label
• The Gini function measures the impurity of
∑aCsample S2 having C different classes
Gini(S) = 1 –
p (y)
y=1 S
where pS (y) denotes the proportion of examples labeled y in S
• When only 2 classes:
2 + p2 ) = 2p (1 – p )
Gini(S) = 1 – (p⊕
⊕
⊕
⊖
Example: 100 patients in S (root node), 25% high risk (+), 75% at low risk (–)
Gini(S) = 2 × .25 × .75 = 0.375
1.0
Best feature selected at a tree node
12 / 25
Maximize the drop of impurity
Splitting sample S with respect to attribute xj results in a drop of impurity:
Drop(S, xj ) = Gini(S) –
Drop(S,(BRCA1) =
)
76
24
0.375 – 100
× 0.123 + 100
× 0.278 = 0.2148
∑
|Sjv |
v∈Values(xj ) |S| Gini(Sv )
Drop(S,(Smoker) =
)
45
55
0.375 – 100
× 0.345 + 100
× 0.397 = 0.0014
Select best feature (here BRCA1) at root node, split data accordingly and apply the
same strategy recursively to the new nodes
Binarize features
13 / 25
• CART learns binary decision trees
Discrete k–ary features can be tested
multiple times
• Continuous features are binarized with respect to a threshold θ
xj ≤ θ ? ⇒ {TRUE,FALSE}
decision thresholds are fixed according to prior knowledge
Example: BodyTemperature ≤ 38°
▶ or candidate thresholds are defined midway between examples with distinct
class labels and the threshold maximizing the drop of impurity is kept
▶
BodyTemperature:
y:
35.9
+
36.1
–
36.7
–
θ1 = 36.0, θ2 = 38.3
37.3
–
38.1
–
38.5
+
39.2
+
CART Learning Algorithm (revisited)
14 / 25
Algorithm CART
Input: A labeled training dataset
S = {(x1 , y1 ), . . . , (xn , yn )}
// yi is the class of xi
Input: The root node of the current (sub-)tree
// Initially, T is a single node tree
Input: n0
// A number of examples below which splitting is stopped
Input: imp0
// An impurity level below which splitting is stopped
Output: A classification tree T
if |S| ≥ n0 AND Gini(S) ≥ imp0 then
// Feature selection at current node
Assign binarized feature xj maximizing Drop(S, xj ) to node
// Only 2 possible values
foreach value v of xj do
Add new nodev as a child of node in T
Attach Sjv = {x ∈ S|xj = v} to nodev
// Split S according to xj values
CART(Sjv , nodev )
else
Assign majority class label to node
return T
PROs and CONs of Decision Trees
15 / 25
• Decision (or classification) trees are/were highly popular in the medical domain
because of their easy logical interpretation
• CART can also learn regression trees for which a level of response is predicted
(instead of a class label)
select features which maximize the drop of response variance (instead of the
drop of impurity)
▶ assign average response value to leaf nodes (instead of majority class)
▶
• Despite their success, decision trees should be taken with care because of their
strong tendency to overfit
the model captures patterns which are highly specific to the training set
the generalization to new examples (e.g. novel patients) tends to be poor
▶ recursive partitioning leads to overfitting since fewer and fewer data is used
to refine the model
▶
▶
Strategies to limit overfitting
16 / 25
• Post-pruning: simplify the tree once it is learned
• Pre-pruning: n0 , imp0
▶
easy but criterion local to each node
• Tree ensemble or Forest
▶
▶
with better predictive results on new examples
a reduced interpretability: majority vote among e.g. 500 trees instead of 1
Bagging
17 / 25
Algorithm Bagging
Input: A labeled training dataset
S = {(x1 , y1 ), . . . , (xn , yn )}
// yi = ±1 is the class of xi
Input: A total number B of bagging rounds
Input: A learning algorithm of Algo returning a binary classifier X → {–1, +1}
Output: A combined classifier
for b ← 1 to B do
Sb = Resample (S)
// Randomly samples S with replacement
fb (x) = Algo(Sb )
// Build a classifier on Sb using learning algorithm Algo
return sign
(∑
B
f (x)
b=1 b
)
• The final result is a combined classifier by voting with uniform weights
• Bagging reduces the overfitting to the dataset S and generally outperforms
significantly the learning algorithm Algo applied once on S
Random Forests
18 / 25
Bagging + decision trees + (partially) random feature selection
• The general approach is like bagging: build models on successive
resampling (with replacement) of the training data and make a majority
vote to form the combined classifier
• Decision trees are built with no post-pruning but splitting of a node is
stopped when it contains too few examples (typically less than n0 = 5
examples)
• While growing the tree, the feature selected at each node maximizes the
drop of impurity (Gini index) among a random pre-selection of υ features
out of the p possible features
when υ = 1 only one feature is randomly selected at each node and there is
no need to compute the impurity
▶ when υ = p standard decision trees are built as the best among all
attributes is selected
⌈√ ⌉
▶ Typical values are υ = ⌈log p⌉ or υ =
p
2
▶
Further information about learning predictive models
19 / 25
• LINGI2262 Machine Learning: classification and evaluation
• LELEC2870 Machine Learning: regression, deep networks and
dimensionality reduction
• LINGI2369 Artificial Intelligence and Machine Learning Seminar
• LINGI2263 Computational Linguistics
Outline
20 / 25
1. Linear models
2. Tree-based models
3. Final exam
Some general recommendations
22 / 25
• the final exam is closed book
• a pocket calculator may be useful
▶
smartphones, tablets or laptops are not allowed
• the primary objective of the evaluation is to check your understanding of
the presented concepts
▶
answering why a property holds (or not) is more important than stating that
it holds (or not)
• if you are asked to give a mathematical formulation, some math is expected
in your answer!
whenever you use a mathematical notation, the easiest approach is to be
consistent with the one used in the lecture slides
▶ be ready to explain in words, concisely but accurately, the math you write
▶
• one way to test your understanding of an algorithm is to make you analyze
a variant, possibly including an (obvious) bug
Question example I
23 / 25
Define mathematically the Bonferroni correction, explain in words the mathematical notation
you use and how such a correction is used to assess which candidate marker genes are significant
for diagnosing whether a patient suffers from cancer or not
Possible answer:
If α denotes a threshold value (e.g. α = 0.05, also known as the significance level) to
estimate when a statistical test is deemed significant, the Bonferroni correction replaces
such a threshold by nα , where nt stands for the number of tests conducted (from the same
t
sample). For any p-value < nα , the test is considered significant.
t
When assessing candidate marker genes to diagnose cancer, the statistical test under study
is evaluating whether the mean expression values between the 2 conditions (true cancer
versus not) actually differ. The number nt of tests conducted, i.e. the multiplicity of the
test, is the number of gene expression values measured since one test is conducted for each
gene.
Interact with a computer to get prepared
24 / 25
When you want to revisit some theoretical concept (e.g. a pairwise alignment
algorithm) it could be extremely useful to
1
try it on paper first on a toy example (e.g. align 2 short sequences)
check the results on a computer (⇒ use some R function you learned
through the projects)
3 think about the effect of some variant on paper
2
what happens if the gap open penalty is increased to a very large value?
what if the substitution score between a pair of letters becomes strongly
positive/negative?
▶ is there a link between the result of a global versus local alignment?
▶
▶
4
check your understanding by running it on a computer
Your best strategy is to think first and try in practice next!
Conclusion
25 / 25
Thank you for your participation in this course
Good luck for your exams!