A RANDOM POLYGONS MODEL OF GLANDULAR STRUCTURES
IN COLON HISTOLOGY IMAGES
Korsuk Sirinukunwattana?†
David R.J. Snead‡
Nasir M. Rajpoot?
Department of Computer Science and Engineering, Qatar University, Doha, Qatar
†
Department of Computer Science, University of Warwick, Coventry, UK
‡
Department of Histopathology, University Hospitals Coventry and Warwickshire, Coventry, UK
?
ABSTRACT
In this paper, we present a stochastic model for glandular
structures in Hematoxylin and Eosin stained histology images, choosing colon tissue as an example. The proposed
Random Polygons Model (RPM) treats each glandular structure in an image as a polygon made of a random number of
vertices, where the vertices represent approximate locations
of epithelial nuclei. We formulate the RPM as a Bayesian
inference problem by defining a prior for spatial connectivity
and arrangement of neighboring epithelial nuclei and likelihood about the presence of glandular structure. The inference
is made via a Reversible-Jump Markov Chain Monte Carlo
simulation. Our experimental results show that the RPM
yields favorable results, both quantitatively and qualitatively,
for extraction of glandular regions in histology images of
human colon tissue.
Index Terms— Histology image analysis, Gland modeling, Random polygons, Bayesian inference, Reversible-Jump
Markov chain Monte Carlo.
1. INTRODUCTION
Motivation: In this paper, we address the problem of modeling glandular structures in colon histology images. Glandular structures are important for diagnosis of several epithelial
cancers, such as colon and prostate. Glands in epithelial tissue normally have a tubular structure and consist of lumen
forming interior of the tubular structure surrounded by epithelial nuclei at the boundary, as shown in Figure 1a. While
there are other tissue constituents, such as stroma nuclei and
cytoplasm, arrangement of epithelial nuclei around the lumen
can be used as a strong cue for the extraction of glandular
structures whose morphology has been shown to have a high
diagnostic value for colon and prostate cancer [1, 2].
Related Work: Existing methods for glandular structure
segmentation can be categorized into two main classes: (a)
texture based approaches and (b) structure based approaches.
For textural methods, Farjam et al. [3] employed variance
and Gaussian filters to extract texture features from glandular
structures. In structural methods, Naik et al. [4] used a level
set method to segment lumen areas in a gland. A Bayesian
classifier is used to detect potential lumen areas and a level
set curve is automatically initialized at the boundary of the
detected lumen areas. Nguyen et al. [5] employed the prior
knowledge about glandular constituents in order to extract
glandular regions. Their algorithm first jointly segments nuclei and cytoplasm to form a rough glandular boundary and
then using a region growing algorithm to expand the lumen
area. Gunduz-Demir et al. [6] proposed object graphs for
segmentation of glandular structures. Their algorithm first
identifies nuclei and lumen using object-based information.
Boundary of the gland is then constructed by connecting centroids of the nuclei objects.
Contributions: We propose a Random Polygons Model
(RPM) for modeling glandular structures in Hematoxylin and
Eosin (H&E) stained histology images, choosing colon tissue
as an example. The proposed approach treats each glandular
structure as a polygon made of a random number of vertices,
where the vertices represent approximate locations of epithelial nuclei. Based on the biological fact about the presence
of epithelial nuclei on the glandular boundary surrounding
the lumen, edges connecting these vertices would result in
boundaries of any glandular structures in the image being
represented by polygons, as shown in Figure 1(d). We formulate the modeling of glandular structures using the RPM as
a Bayesian inference problem, in which a prior distribution
of polygons is related to the connectivity and arrangement of
their vertices while a likelihood term is related to the presence of glandular structure encapsulated by the polygons.
We exploit the Reversible-Jump Markov Chain Monte Carlo
(RJMCMC) method using the Metropolis-Hastings algorithm
to infer all the maximum a posteriori polygons.
2. THE RANDOM POLYGONS MODEL
Let I be an input image and Z be a set of all coordinates of
pixels for the image I. Let V ⇢ Z denote a set of all vertices
representing nuclei’s location. Let X : Z ! [0, 1] denote a
map such that 8z 2 Z, X(z) is a probability of a pixel at location z belonging to a glandular structure. We refer to X(Z)
as glandular probability map hereafter. See Section 3 for details on how nuclei vertices and the glandular probability map
are obtained. Given the evidence y = (V, X(Z)) available
SN
non-glandular structures. Let ⌦S be a set of all pixels z 2 Z
enclosed by the polygon S. The likelihood L(y|S) is written
as
Y
Y
L(y|S) =
X(z)
(1 X(z)).
(3)
G
L
z2⌦S
E
(a)
(b)
(c)
(d)
Fig. 1: (a) A sample colon histology image showing various components (epithelial cell or E, stromal nuclear or SN, lumen or L,
goblet cell or G). (b) Representative nuclear vertices are shown in
yellow dots. (c) A glandular probability map. The brighter intensity
indicates the higher probability of a pixel belonging to a glandular
structure. (4) A maximum a posteriori polygon is shown in green
boundary.
to us, we infer the most likely polygon that encloses an individual glandular structure through a Bayesian framework as
follows.
2.1. The Bayesian Formulation
Let S = {z1 , ..., zn : zj 2 V } be a simple polygon consisting
of n vertices, where the polygon edges are defined in the given
order of vertices. According to the Bayes’ rule, the posterior
distribution of polygon S given observation y is
P (S|y) / L(y|S)P (S),
(1)
where L(y|S) denotes the likelihood distribution of y given
S and P (S) denotes the prior distribution of S.
The prior probability of S is defined based on (a) the
length of polygon edges and (b) the angles at polygon corners. Assume that both factors can be independently modeled by normal distributions. The prior probability of S is
expressed as
⇣
⌘Y ⇣
⌘
Y
P (S) =
N le µ l , l
N ✓z µ✓ , ✓ , (2)
e2ES
z2S
where ES denote a set of all edges of S, le denotes length of
the edge e, ✓z denotes an acute angle at vertex z, N (·|µ, )
denotes the probability density function of a normal distribution with mean µ and standard deviation , and µl , l and
µ✓ , ✓ are mean and standard deviation for the length of edges
and the acute angle of polygon corners, respectively.
The likelihood of observation y = (V, X(Z)) given polygon S is defined in terms of how close the area inside S
matches a glandular structure, while the area outside matches
z62⌦S
2.2. Reversible-Jump Markov Chain Monte Carlo
The posterior distribution P (S|y) in Equation (1) cannot be
determined analytically. We, therefore, resort to Markov
chain Monte Carlo (MCMC) simulation to approximate the
posterior distribution. Reversible-Jump Markov chain Monte
Carlo (RJMCMC) using Metropolis-Hastings algorithm1 [7]
is employed due to its ability to accommodate changes in dimensionality of the state space. This property is vital as it allows us to sample polygons with varying number of vertices.
Let polygon Si denote the current state of the Markov chain,
and polygon Sf = g(Si , z ⇤ ) denote the polygon representing
the proposed state, proposed by an invertible deterministic
function g, where z ⇤ denotes a random variable required for
dimension matching condition. The acceptance ratio for a
transition from Si to Sf is given by
↵(Si , Sf ) = min {1, R} ,
(4)
where
@Sf
L(y|Sf ) P (Sf ) Q(Si |Sf ) P (Sf ! Si )
R=
,
L(y|Si ) P (Si ) Q(Sf |Si ) P (Si ! Sf ) @(Si , z ⇤ )
Q(Sf |Si ) Q(Si |Sf ) is a proposal distribution of Sf given
Si Si given Sf , P (Si ! Sf ) P (Sf ! Si ) is a probability of choosing the move type that allows the transition from
Si to Sf Sf to Si , and |@Sf /@(Si , z ⇤ )| is a Jacobian term
dealing with the change in dimension of variables. To fulfill
the reversibility condition of the Markov chain, we define two
move types which are the reverse of the other as follows.
Addition of a Vertex and Deletion of a Vertex: Assume
that the current state of the Markov chain is a polygon Si =
{z1 , ..., zk 1 , zk , zk+1 , ..., zn }. This move proposes a new
polygon Sf by first sampling side of Si according to Gibbs
distribution, where the side with higher deviation from µl is
more likely to be selected. Suppose that side e⇤ = zk zk+1 is
chosen. We define a score for each vertex z 2 V \ Si as
⇣
⌘
fe⇤ (z) ⌘ N lzk z + lzzk+1 lzk zk+1 , d ⇥
(5)
N (✓˜zk |µ✓ , ✓ )N (✓˜z |µ✓ , ✓ )N (✓˜zk+1 |µ✓ , ✓ ),
where d denote mean and standard deviation of the summation of the length of edges lzk z +lzzk+1 , and ✓˜zk , ✓˜z , ✓˜zk+1 denote acute angles at corners 6 zk 1 zk z, 6 zk zzk+1 , 6 zzk+1 zk+2 ,
respectively. Further, let Ue⇤ be a set of the top m vertices
z 2 V \ Si in terms of the score fe⇤ . A new vertex z ⇤
is then uniformly drawn from Ue⇤ and added into Si between vertices zk and zk+1 . Hence, Sf = g(Si , z ⇤ ) =
1 Note that the Metropolis-Hastings algorithm requires the target distribution to be known up to a constant, whereas the Gibbs sampling requires
that the full conditional distributions of parameters of the target distribution
are known. The latter, thus, is not suitable for our method where the full
conditionals are not available.
{z1 , ..., zk , z ⇤ , zk+1 , ..., zn }, |@Sf /@(Si , z ⇤ )| = 1, and
exp(|le⇤ µl |/Tl )
1
Q(Sf |Si ) = P
,
(6)
exp(|l
µ
|/T
)
|U
e
l
l
e⇤ |
e2ESi
P
where Tl = |E1S | e2Si le is a temperature parameter for
i
Gibbs distribution, and |S| denote the cardinality of set S.
This move occurs with probability P (Si ! Sf ) = radd .
The reverse move from Sf to Si can be accomplished by
sampling a vertex z ⇤ and removing it from Sf . The sampling
process is done according to a Gibbs distribution in which the
likelihood of a vertex, whose acute angle is more deviated
from the mean angle µ✓ , being selected is higher than those
with smaller deviation. Thus,
exp(|✓z⇤ µ✓ |/T✓ )
Q(Si |Sf ) = P
,
(7)
µ✓ |/T✓ )
z2Sf exp(|✓z
P
where T✓ = |S1f | z2Sf |✓z µ✓ |. Suppose that the deletion
occurs with probability P (Sf ! Si ) = rdelete .
It should be emphasized that the above description of
moves is based on a single polygon as individual polygon
will be initialized using seed area derived from the glandular
probability map (described in Sections 3 and 4). Other type
of moves such as birth and death of polygons are thus not
necessary.
3. GLANDULAR MODELING FRAMEWORK
Figure 1 summarizes the RPM framework. Given an H&E
image (Figure 1a), two evidences, i.e. vertices representing
nuclei locations (Figure 1b) and the glandular probability map
(Figure 1c), necessary to perform an inference for random
polygons are generated. A maximum a posteriori polygon
(Figure 1d) is estimated using the RJMCMC.
3.1. Glandular Probability Map
Following [8], the generation of the glandular probability map
X(Z) for a given image I is a 3-step process.
Step 1: Superpixel Segmentation. We segment image
I into superpixels using Simple Linear Iterative Clustering
(SLIC) algorithm [9].
Step 2: Feature Extraction. For each superpixel i, let Bi
denote a union of intermediate neighboring superpixels of i.
A feature vector for superpixel i,
h
i
>
> >
fi = ficolor , fitexture
,
(8)
comprising two cues, namely color and texture, from the superpixel i itself and its neighboring superpixels. The color
>
feature is calculated as ficolor = [h(i), h(Bi )] , where h(i) is
a b-dimensional normalized histograms of quantized L⇤ a⇤ b⇤
color intensity calculated over superpixel i. The texture feature is defined as fitexture = [S̄p(i), S̄p(Bi )]> , where S̄p (i) denotes a scattering SPD descriptor [8] calculated on superpixel
i.
Step 3: Glandular Probability Assignment. A glandular
probability for each superpixel is obtained from a decision
tree classifier trained on the extracted feature. This results in
a map indicating the likelihood for each pixel belonging to
the glandular region. The map is then processed by a spatial average filter, e.g. Gaussian filter, yielding the glandular
probability map X(Z).
3.2. Formation of Nuclear Vertices
We employ a color-deconvolution method [10] to extract a
Hematoxylin channel from I. By thresholding the Hematoxylin channel, we obtain a binary mask image corresponding to the approximate location of nuclei in I. We sample
vertices representing nuclei locations from a set of coordinates of perimeter pixels of the binary mask in a greedy fashion and put them in a set V with the criteria that for any
z1 , z2 2 V, kz1 z2 k d, where d is a desired minimum
distance between any two vertices.
4. EXPERIMENTAL RESULTS AND DISCUSSION
Dataset: We tested our algorithm on a benchmark dataset [6]
consisting of 72 images of H&E stained colon biopsy from
36 patients. The images were captured at 20⇥ magnification
and the expert annotation is provided. Because color is one of
the cues used by our method, we pre-process images with a
recently proposed stain-normalization method [11] to reduce
the effect from stain variation across images.
Settings: We follow the protocol described in [6] where one
third of the images are used for training and the remaining for
testing. The value of parameters for the RPM are empirically
determined on the training set (µl = 12.7, l = 4.2, µ✓ =
157.2, ✓ = 23.3, d = 10, m = 10, b = 20, d = 8, radd =
rdelete = 0.5). To reduce computational complexity of the algorithm, we start random polygons from seed areas obtained
by thresholding the probability map, where a single polygon
is generated per seed area. We assess the performance of segmentation algorithms using the Dice index. The Dice values
are calculated both at image level and object level to capture
the quality of segmentation of individual glands as well as the
whole image. Due to the stochastic nature of the RPM, we repeat an experiment for 5 times and report the average results.
As a post-processing operation for the RPM, false positive
polygons are removed using criteria related to the number of
vertices and the area of polygons. Furthermore, a relatively
smoother contour of the glandular structures is generated using cubic spline interpolation.
Table 1 and Figure 2 show the segmentation results of different algorithms. The proposed RPM approach gives supeApproaches
Gunduz-Demir et al.
Farjam et al.
Nguyen et al.
Naik et al.
RPM
Image - Level
0.889 ± 0.046
0.592 ± 0.119
0.671 ± 0.116
0.719 ± 0.123
0.901 ± 0.041
Object - Level
NA
0.518 ± 0.149
0.457 ± 0.094
0.556 ± 0.136
0.845 ± 0.085
Time
NA
3.4 ± 1.2
6.3 ± 2.6
106.6 ± 57.4
134.0 ± 152.9
Table 1: Comparative performance and execution time. Figures reported are mean ± standard deviation of Dice indices (2nd and 3rd
column) and execution time measured in second (4th column). The
best result for segmentation accuracy is shown in bold in 2nd and
3rd columns. NA = not available.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2: Example results for different gland segmentation approaches. (a) Ground truth. (b) Farjam et al. [3]. (c) Nguyen et al. [5]. (d) Naik et
al. [4]. (e) Probability map. (f) Random Polygon Model. Segmentations are represented as yellow shaded area in b,c,d, and f.
rior segmentation results both at the image level and the object
level. The following factors contribute to the better performance of the RPM. Firstly, it uses superpixel-level features
(Figure 2e) for initial detection of glandular structure, which
is proven to be more reliable than pixel-level features (Figure 2b,c,d). Secondly, it models glandular structures using
a glandular probability map and nuclear vertices which are
consistent with the biological fact about the presence of epithelial nuclei surrounding lumen. However, similar to other
stochastic modeling process, the RPM has higher computational complexity than deterministic algorithms, which results
in the algorithm taking longer time than its competing algorithms to complete the segmentation as shown in Table 1.
5. CONCLUSIONS
In this work, we have presented a novel random polygons
model for modeling glandular structures in colon histology
images. The model employs Bayesian inference and the
Reversible-Jump Markov chain Monte Carlo (RJMCMC) to
estimate polygons enclosing a glandular structures, whose
vertices are sampled from epithelial nuclei. With the flexibility of RJMCMC, a new type of move such as birth, death,
merging and splitting of polygons can be defined in the future.
6. ACKNOWLEDGMENTS
This paper was made possible by NPRP grant number
NPRP5-1345-1-228 from the Qatar National Research Fund
(a member of Qatar Foundation). The statements made herein
are solely the responsibility of the authors. Korsuk Sirinukunwattana acknowledges the financial support provided by the
Department of Computer Science, University of Warwick,
UK. The authors thank Dr. Cigdem Gunduz-Demir for kindly
providing us the dataset used in [6].
7. REFERENCES
[1] D. F. Gleason and G. T. Mellinger, “Prediction of prognosis
for prostatic adenocarcinoma by combined histological grading and clinical staging,” The Journal of Urology, vol. 167, no.
2, pp. 953–958, 2002.
[2] S. R. Hamilton, L. A. Aaltonen, International Agency for
Research on Cancer, World Health Organization, and others,
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
Pathology and genetics of tumours of the digestive system,
2000.
R. Farjam, H. Soltanian-Zadeh, K. Jafari-Khouzani, and R. A.
Zoroofi, “An image analysis approach for automatic malignancy determination of prostate pathological images,” Cytometry Part B: Clinical Cytometry, vol. 72, no. 4, pp. 227–240,
2007.
S. Naik, S. Doyle, A. Madabhushi, J. Tomaszewski, and
M. Feldman, “Gland segmentation and gleason grading of
prostate histology by integrating low-, high-level and domain
specific information,” in Workshop on Microscopic Image
Analysis with Applications in Biology, 2007.
K. Nguyen, A. K. Jain, and R. L. Allen, “Automated gland
segmentation and classification for Gleason grading of prostate
tissue images,” in Pattern Recognition (ICPR), 2010 20th International Conference on. IEEE, 2010, pp. 1497–1500.
C. Gunduz-Demir, M. Kandemir, A. B. Tosun, and C. Sokmensuer, “Automatic segmentation of colon glands using objectgraphs,” Medical Image Analysis, vol. 14, no. 1, pp. 1–12,
2010.
P. J. Green, “Reversible jump Markov chain Monte Carlo computation and Bayesian model determination,” Biometrika, vol.
82, no. 4, pp. 711–732, 1995.
K. Sirinukunwattana, D. S. Snead, and N. M. Rajpoot, “A
Novel Texture Descriptor for Detection of Glandular Structures in Colon Histology Images,” in accepted in SPIE Medical
Imaging, 2015.
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and
S. Susstrunk, “SLIC superpixels compared to state-of-the-art
superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282,
2012.
M. Macenko, M. Niethammer, J. S. Marron, D. Borland, J. T.
Woosley, X. Guan, C. Schmitt, and N. E. Thomas, “A Method
for Normalizing Histology Slides for Quantitative Analysis.,”
in ISBI, 2009, vol. 9, pp. 1107–1110.
A. M. Khan, N. M. Rajpoot, D. Treanor, and D. Magee, “A
nonlinear mapping approach to stain normalization in digital
histopathology images using image-specific color deconvolution,” IEEE Transactions on Biomedical Engineering, vol. 61,
no. 6, pp. 1729–1738, 2014.