Histological Image Processing Features Induce a Quantitative Characterization of Chronic Tumor Hypoxia

Andrew Sundstrom; Elda Grabocka; Dafna Bar-Sagi; Bud Mishra

doi:10.1371/journal.pone.0153623

Abstract

Hypoxia in tumors signifies resistance to therapy. Despite a wealth of tumor histology data, including anti-pimonidazole staining, no current methods use these data to induce a quantitative characterization of chronic tumor hypoxia in time and space. We use image-processing algorithms to develop a set of candidate image features that can formulate just such a quantitative description of xenographed colorectal chronic tumor hypoxia. Two features in particular give low-variance measures of chronic hypoxia near a vessel: intensity sampling that extends radially away from approximated blood vessel centroids, and multithresholding to segment tumor tissue into normal, hypoxic, and necrotic regions. From these features we derive a spatiotemporal logical expression whose truth value depends on its predicate clauses that are grounded in this histological evidence. As an alternative to the spatiotemporal logical formulation, we also propose a way to formulate a linear regression function that uses all of the image features to learn what chronic hypoxia looks like, and then gives a quantitative similarity score once it is trained on a set of histology images.

Citation: Sundstrom A, Grabocka E, Bar-Sagi D, Mishra B (2016) Histological Image Processing Features Induce a Quantitative Characterization of Chronic Tumor Hypoxia. PLoS ONE 11(4): e0153623. https://doi.org/10.1371/journal.pone.0153623

Editor: Roger Chammas, Universidade de São Paulo, BRAZIL

Received: November 11, 2015; Accepted: April 2, 2016; Published: April 19, 2016

Copyright: © 2016 Sundstrom et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files. Histology image data are deposited into Harvard Dataverse: http://dx.doi.org/10.7910/DVN/SI32FV.

Funding: This work was supported by National Science Foundation - DGE-0333389 - IGERT: Program in Computation (http://www.nsf.gov/) (AS), and National Science Foundation - CNS-0926166 - Collaborative Research: Next-Generation Model Checking and Abstract Interpretation with a Focus on Embedded Control and Systems Biology (http://www.nsf.gov/) (BM). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

As a tumor grows, it rapidly outstrips its blood supply. High proliferation causes high cell density that overtaxes local oxygen supply. This leaves portions of the tumor with an oxygen concentration significantly lower than in healthy tissues. This stress condition is tumor hypoxia. Hypoxia is strongly correlated with poor prognosis as it renders tumors less responsive to chemotherapy and radiotherapy [1–3].

Hypoxia-inducible factors (HIFs) are transcription factors that respond to changes in available oxygen in the cellular environment, specifically to hypoxia. When activated, HIF-1 upregulates several genes to promote survival in low-oxygen conditions. These include glycolysis enzymes that allow cells to synthesize ATP in an oxygen-independent manner, and vascular endothelial growth factor (VEGF) that cells release to promote angiogenesis. So hypoxia is directly instrumental in tumor progression.

Prolonged or extreme hypoxia can lead to necrosis, and tumors often have central regions called necrotic cores [4]. Necrosis in turn activates inflammatory responses that produce cytokines that stimulate tumor growth [3]. Recent research has investigated the interactions between hypoxic tumor cells and immune cells (tumor-associated macrophages [5]) and cells that synthesize extracellular matrix (tumor-associated fibroblasts [6, 7]). Both are involved with inflammatory processes tied to tumor progression. In the context of the tumor microenvironment, these interactions regulate tumor properties like spatial patterns of cell localization, angiogenesis, and collective invasion and migration [8, 9].

Thus it is of theoretical and clinical significance to understand how, and under what conditions, hypoxia arises in tumors. More fundamentally, we need to better characterize tumor hypoxia from available evidence, so it can be reliably detected in its various states and contexts.

Tumor hypoxia exhibits two major forms—intermittent and chronic. Intermittent hypoxia derives from the pervasive presence of fluctuating oxygenation in whole tumors, and operates in a length scale that exceeds the locality of specific vessels [10]. Chronic hypoxia derives from a vessel-dominant oxygenation dynamics whose parameters correspond to vessel and tissue properties, and the radial distance from the vessel.

In this study we choose to investigate chronic tumor hypoxia situations where there is a presumed steady-state gradient of oxygen near a source vessel, diminishing in magnitude as a function of radial distance away from that vessel. This phenomenon has been investigated since the clinical study of Thomlinson and Gray [4] first characterized “tumor cords”. Red blood cells release oxygen by diffusion into the tumor tissue regions in need. The oxygen is metabolized by respiring cells near the blood vessel, and consequently oxygen tension diminishes as a radial distance away from the vessel. At radial distances in excess of ∼ 100 μm, there is insufficient oxygen to maintain cell viability. Between the bands of viable and necrotic cells, one typically finds a region 1–2 cell layers thick where oxygen tension is hypoxic—this is consistent with our tumor image data, described below. Moreover, in a solid tumor mass, mitotic index and cell viability decreases as a function of radial distance away from the nearest blood vessel [4, 11].

We develop a method to induce a quantitative characterization of these chronic tumor hypoxia situations from histological evidence, namely image data taken from H&E and anti-pimonidazole stained slices of tumors. In this way, we take a reductionist approach, which we understand does not integrate the full complexity of tumor vascularity and hypoxia. Rather, we choose to focus on our simplified biological system to better characterize it by way of our computational modeling techniques—assembling a logical description comprising quantitative image features, and assembling a linear function whose terms comprise quantitative image features and are weighted according to a linear regression. We show how one can use the image features we develop (and an unbounded set of other image features) to produce an automated, scalable, and unbiased spatiotemporal characterization of chronic tumor hypoxia.

The quantitative study of chronic hypoxia near blood vessels is part of a well established literature that seeks to improve our understanding of microvascular oxygen transport in tissues by building theoretical models.

Krogh (1919) was one of the first to systematically investigate the architectural relationship between blood capillaries and muscle cells, and the conditions under which oxygen flows from blood cells to muscle cells [12–14]. In particular, the Krogh cylinder model [13] gives a quantitative, predictive description of oxygen tension within an idealized system of a single capillary. It defines two concentric cylinders, one of muscle tissue (having radius R) surrounding another of vessel (having radius r); it describes the oxygen tension at distance x into the muscle tissue (T_x) as a function of: the oxygen tension in the capillary (pO₂), the diffusion constant for oxygen in muscle [12], the rate of oxygen consumption, and the the radii R and r. When x = R, the model gives the maximum tension difference (T₀ − T_R) necessary to supply the muscle with oxygen at any point along the capillary. If T₀ − T_R is greater than the oxygen tension of venous blood, then the same portion of muscle (near the venous end of the capillaries) will be hypoxic; if T₀ − T_R is less than the oxygen tension of venous blood, then oxygen tension is positive everywhere within the muscle tissue.

Krogh approached the complex problem of microvascular oxygen transport in tissues by parsing it into three aspects: “(1) The physical problem of the rate at which oxygen diffuses into and through the tissues; (2) the anatomical problem of the number and distribution of capillaries with respect to the cells; and (3) the physiological problem of regulating the supply of blood and by that the availability of oxygen under the conditions of rest and in exercise.” [15]. Since Krogh’s groundbreaking work, many researchers have developed theoretical models to address the biochemical, structural, geometric, and hemodynamic complexity involved in the problem. In the past two decades, multi-vessel models have been developed to consider microvascular arrays and networks. These models have shown the physiological significance of heterogeneities in vessel spacing, oxygen supply, flow path of red blood cells, and interactions between capillaries and arterioles [16].

Some of these models consider tumor tissue in various respects. Kang, et al [17] models oxygen transport during tumor hyperthermia. Kavanagh, et al [18] models tumor oxygenation under varying hemoglobin-oxygen affinities. Kirkpatrick, et al [19] explores the influence of kinetic and physical factors on substrate metabolism in a Krogh tumor model. Secomb, et al [20] presents theoretical simulations of oxygen delivery to tumor tissues by networks of microvessels, based on in vivo observations of vascular geometry and blood flow in the microcirculation in mammary adenocarcinoma tumors. This is a rich area of active research.

In our study, we are not concerned with modeling any particular aspect of microvascular oxygen transport in tissues, let alone the full dimensionality of this complex phenomenon. While we acknowledge the Krogh and multi-vessel models could provide their own set of quantitative features for characterizing chronic tumor hypoxia near a vessel, we have taken a simpler approach, to empirically measure the anti-pimonidazole gradients by image analysis. Such measured gradients are but one of a potentially large set of image features to be combined logically and functionally in a later phase of processing, discussed below.

Helmlinger, et al [21] experimentally measured interstitial pO₂ in vivo in a number of xenographed tumors. Profiles of pO₂, where the interstitial regime is delineated by the centroids of two adjacent blood vessels, show expected gradients whose slope is negative moving away from the first vessel, then eventually become positive moving toward the second vessel. The slope property in these experiments qualitatively matches the slope property in our data involving single vessels (negative slope moving away from the vessel). It is not clear to us, however, whether their pO₂ profiles would provide a meaningful quantitative comparison with the analogous anti-pimonidazole intensity gradients we measure in our image data. More analysis is required to establish the compatibility of these two lines of evidence before in vivo studies like Helmlinger, et al could provide an empirical validation of our histological results, or vice-versa. Moreover, the authors make three findings of interest to our study: (1) they found no correlation between pO₂ and blood flow rate; (2) they found no correlation between intravascular pO₂ and blood flow rate; and (3) pO₂ did not correlate with the two measured parameters used to compute blood flow rate—red blood cell velocity and vessel diameter (within respective specified ranges). Taken together, these findings highlight the admissibility of histology images as a source of data for measuring oxygen gradients. Although histology images represent single time points, and capture a range of vessel diameters conveying a range of possible blood flow rates, the evidence of oxygen gradients in these images is intact to the extent it can be measured in these images.

There has been recent progress in automated tumor segmentation on histological images, for example example by Wang, et al [22]. They developed a robust tumor segmentation technique and tested it on H&E and immunohistochemistry stain slides. Their method comprised a tissue architecture extraction approach and a tumor texture learning model. The tissue architecture extraction approach used a stain separation method and an unsupervised multistage entropy-based segmentation method, and the tumor texture learning uses a Markov random field image segmentation system. Their method allowed fine pixel based segmentation for small tissue samples. Their tissue domain was human lung tumors. For their purposes they defined three classes of tissue morphology: tumor, stroma, and a third catch-all category for lymphoid, inflammatory cells, and necrosis. Importantly, they did not try their method on anti-pimonidazole stain images, which, especially in low concentrations of anti-pimonidazole, render images that have strikingly low contrast. While their approach seems to us a promising texture-learning-based alternative to the simple intensity-based method we employ [23], it is unclear to us whether their method can perform effectively on anti-pimonidazole images and thus characterize chronic tumor hypoxia.

A number of recent computational studies [24–27] have employed statistical model checking algorithms to verify spatiotemporal logical propositions in biological systems. They used Probabilistic Bounded Linear Temporal Logic (PBLTL) to characterize phenomena of interest in: a fibroblast growth factor signaling model, circadian rhythm, yeast heterotrimeric G protein cycle control, and the HMGB1 signaling pathway in cancer. In one study, Grosu, et al [28] developed a system to tackle the problem of learning and detecting emergent behavior in networks of cardiac myocytes. They constructed a Linear Spatial-Superposition Logic (LSSL) formula that characterized spatial patterns such as spirals, whose multiscale spatial characterizations are learned through a classification process. Their system successfully detected the emergence of spiral patterns and hence the approaching state of fibrillation. In the spirit of these studies, we aim to develop a spatiotemporal logical proposition—composed of explicit image feature predicates—that captures at least some characteristics of chronic tumor hypoxia.

In addition, we construct a linear regression function that learns what hypoxia is in terms of estimated linear coefficients on the image feature terms. We adapt this method from our earlier work [29] in a different image processing domain where it showed promising results.

Materials and Methods

Experimental setup

Our study is based on experiments that demonstrate hypoxia arising in human colon cancer. In this experiment, 2 × 10⁶ human colon cancer cells were injected into both flanks of nude mice. When the tumor volume reached ∼1500 mm³ (∼4 weeks post-injection), pimonidazole was administered via intraperitoneal injection. Ninety minutes after pimonidazole administration mice were euthanized, the tumors were excised and immediately fixed in formalin. Slides were then prepared from sections 10 μm apart, alternating between H&E and anti-pimonidazole stains. Mice were euthanized by carbon dioxide-induced narcosis. All animal work was approved by New York University Langone Medical Center Institutional Animal Care and Use Committee.

H&E staining

Hematoxylin and eosin stain (or “H&E stain”) is a common staining method in histology. It colors cell nuclei blue, then counterstaining colors non-nuclear, eosinophilic structures graded shades of orange, pink, and red. In our study, we use H&E stains of the tumor tissue for the primary purpose of locating blood vessels and for discriminating collagen. In Fig 1 (top), we see blood vessels appear within the boundary of the tissue as open lumens (white) populated with several to many red blood cells (small, bright pink spheroids). Collagen deposits appear as continuous structures (light pink) that infuse the tumor lesions and usually do not extend into the necrotic tissue (lightest pink, with interstitial spacing and much smaller, unenclosed nuclei).

Download:

Fig 1. Histology stains.

H&E (top) and anti-pimonidazole (bottom) stains of one of our study’s canonical tumor sections.

https://doi.org/10.1371/journal.pone.0153623.g001

Anti-pimonidazole staining

Anti-pimonidazole staining is an immunohistochemical stain protocol used to detect and locate live cells undergoing hypoxia. In plasma, pimonidazole has a half-life of 25 minutes. It distributes to all tissues following injection, but it forms stable covalent adducts with thiol groups in proteins, peptides, and amino acids, only in those cells that have an oxygen concentration less than 14 micromolar (equivalent to a partial pressure pO₂ = 10 mm Hg at 37 C). In the immunohistochemistry, anti-pimonidazole binds to these adducts allowing their detection. In addition to hypoxic regions in tumors, normal tissues of certain organs such as liver, kidney, and skin possess cells at or below pO₂ of 10 mm Hg; these normal tissues, and only these, will bind pimonidazole. In Fig 1 (bottom) we see an anti-pimonidazole stain of one of our study’s canonical tumor sections. Hypoxic cells stain brown by degree of hypoxia. Notice the blood vessels are much more difficult to locate, though it is still possible. In most cases our procedure to locate vessels is to first manually register the H&E and anti-pimonidazole images (see note below), where the sections of the tumor are taken 10 μm apart; second, locate the vessels on the H&E stain; then finally use this position on the anti-pimonidazole to approximate the vessel position, or to simply guide a more detailed examination of the anti-pimonidazole image until the vessel can be positively identified. Collagen complicates our study structurally and colorimetrically, which can be seen in the figure: collagen is difficult to distinguish from the necrotic tissue that surrounds the lesions.

Aligning Z-stack images

To align two images, we match points in one image to corresponding points in the other to determine the displacement. In our H&E and anti-pimonidazole histology images we use blood vessel locations (the ones used above as centers of circular gradients) as our respective point sets, since these structures are easy to identify and match in both images. In S7 Fig we see the canonical anti-pimonidazole image with a vector field overlay. The three blue vectors denote the displacements of the three gradient centers. Each blue vector is labeled with P_i at the head (center position i in the anti-pimonidazole image) and H_i at the tail (the corresponding center position i in the H&E image). The vector lengths (in pixels) are labeled, as are the vector angles (in radians), measured relative to their respective dotted blue horizontal lines.

Notice the vector lengths and angles vary. If this were a straightforward image registration between identical, but translated, images, then we would expect the vectors to have identical lengths and angles. But several factors complicate the simple displacement alignment process. First, each image is a slice of an asymmetrical three-dimensional object undergoing morphological transformation. Second, the structures we chose are blood vessels, which presumably grow in directionally independent ways. Third, the microtome used to slice the tumor sample exerts nonuniform directional force upon the tissue. These and other, lesser, factors contribute to the complex transformation between the two images that involves translation, rotation, and scaling—yet even these taken together cannot account for the tissue’s morphological change between slices. If we assume rotation as a function of x and magnitude as a function of y, then we can fit two respective second-order polynomials to these three positional data with low error, and thereby create the vector field shown in red.

Image analysis

Our approach consists in extracting qualitative and quantitative features from the histology images, namely the anti-pimonidazole stains. We classify these as: (1) features that derive from segmenting the image into the three tissue types depicted: viable tumor cells, hypoxic tumor cells, and necrotic tumor cells; (2) features related to the intra-lesion hypoxia gradient, as measured from radial distance away from the nearest vessel; (3) features that derive from multiscale analysis; and (4) features that relate to qualitative generalities about bounded and nested structure.

Once we have a set of image features, we proceed in two separate but related directions. First, we attempt to construct a logical proposition to describe hypoxia in space and time using an extension of Bounded Linear Temporal Logic (BLTL), whose primitives are image feature predicates. This is a human-driven process, following from human learning and generalization. Second, we attempt to construct a linear regression function that learns what hypoxia is in terms of estimated linear coefficients on the image feature terms. This is a machine-driven process, kept on the rails by a combination of false-positive and false-negative control, and feature dimensionality reduction where possible.

Stratifying image data

For our initial examination of anti-pimonidazole images, the only selection criterion we applied was to keep to the interior of the tumor, away from its extremities. Since these are xenographed tumors, there are potentially many confounding factors at work near the interface between human tumor and mouse stroma. This was a baseline criterion, applied to all of the images we investigated, regardless of any further stratification. This gave us a set of 20 high-concentration anti-pimonidazole images, taken at 20× magnification, of various regions of the tumor interior. But as we became interested in the role vessels play in oxygenation of the tissue, we decided to further stratify the data, and select just those images whose 10× fields of view are ≥90% filled with non-necrotic cancer cells, and contain at least one blood vessel. This stratification gave us 8 such high-concentration anti-pimonidazole images, each taken at 10× and 20× magnification, having corresponding registered H&E images from a section 10 μm away.

Image preprocessing

We used Fig 1 as our canonical image for running examples. We did this for presentational convenience; our intuitions were developed examining many images, and our methods are applied to all specified images. The first step in our image preprocessing algorithm was to convert the RGB histology image into an 8-bit grayscale image. See S1 Fig (top).

Then we applied Gaussian smoothing (using a 5 × 5 mask and standard deviation of 5.0) iteratively until the high frequency structural information was averaged away (stopping at 100 iterations). See S1 Fig (bottom). We used no formal criteria for establishing these parameters, assuming that a consistent protocol for smoothing all images prior to downstream processing was more important than the degree of smoothness. We will address this issue in future work.

Segmenting by histogram multithresholding

To get a qualitative feel for how we might identify tissue type by intensity level, we performed a preliminary investigation using two types of plot on our canonical image. When we viewed image intensity as a mesh plot (S2 Fig (top)), we observed three distinct planes of intensity in the image: necrotic tissue above, hypoxia tissue in the deepest recesses along the outer contour of the lesion, and viable (non-hypoxic) tissue rising up from that, but not to the height of the necrotic tissue. We also observed the backbone of collagen that runs along the middle of the lesion, and we were unable to distinguish collagen intensity levels from those of the necrotic tissue. We decided more information was given in the contour plot (S2 Fig (bottom)), where the proximity of equipotential curves conveys the steepness of the gradients in intensity.

To get a quantitative feel for how we might identify tissue type by intensity level, we examined image intensity histograms (S3 Fig). The histogram of the whole image showed a clear bimodal distribution, but selected sub-images showed a trimodal distribution. Using this distribution as a guideline, we segmented our canonical image into three non-overlapping intensity intervals: [0, 156] for hypoxic, [157, 175] for viable, and [176–255] for necrotic tissue, depicted as red-colored pixels in the top, middle, and bottom of S4 Fig, respectively. Naturally, because sharp thresholds truncate neighboring distributions, false-positive and false-negative cases are bound to emerge from this coarse approach. In the viable interval we saw false-positive outer contours around the hypoxic tissue, and the false-negative inner backbone areas where there are collagen deposits; and in the necrotic interval we saw false positive areas where collagen forms an inner backbone that partitions the viable tissue.

Since our canonical image is taken from a set of high-concentration anti-pimonidazole images, where the viable-hypoxic distinction is visually and numerically easier to make, we expected this intensity interval partition approach to perform worse on the low concentration anti-pimonidazole images, which we found (data not shown).

Given the cross-image variation we observed in the average intensity level for each tissue type, we became convinced that the manually-derived, fixed values we used for the intensity level partitions above could not be applied to all of our images. Thus we sought to use an adaptive approach, deciding on Otsu’s method [23] for automatic multiple thresholding, implemented in the Matlab Image Processing Toolkit as multithresh.

Despite the obvious Type I and Type II errors discussed above, we believed intensity-level-based segmentation could still be used to compare gross measures of viable-like and hypoxic-like cell areas within a whole image, and then provide characteristic ratios that could become image features.

Measuring image intensity gradients

One of the most salient and consistent features of the anti-pimonidazole images under investigation is the presence of a gradient in the brown stain for hypoxia. In any given lesion, stain density is maximal at the outermost contour of the lesion, abutting necrotic tissue that surrounds it, and then diminishes steadily as a function of distance away from the extremity, toward the center (or central 1D spine) of the lesion. Equivalently, stain density decreases steadily as a function of radial distance away from the center (or orthogonally from the central 1D spine). The central area of a lesion is usually marked by a vessel.

The Intensity-Sample-Ray-Bundles algorithm.

For our gradient measurement analysis, we designed an algorithm to perform radial intensity level sampling, along rays that extend from a given lesion center. One specifies three parameters: a center, (x_c, y_c), usually in the centroid of a blood vessel; n, the number of equal-angle-spaced rays that will sample the circle’s area; and m, the number of equal-angle-defined “bundles” (sectors) into which the rays will be considered for statistical analysis. For example, if n = 80, then a sample ray will be extended every radians, and if m = 1, then the rays that fall within 2π radians (all of the rays) will be considered for that bundle’s statistical analysis.

The image is first smoothed, as before. For a given ray, intensity level is sampled radially, from the inside out, until it encounters the edge of the image. One may specify (as optional parameters) the distance between samples along the ray, d_s in pixels (1 by default), and the square neighborhood radius, r_n in pixels, over which to average for that sample (0 by default since the image is already smoothed).

Once the samples have been taken along all of the rays, the rays are “stacked” and “sliced” in the following way. Each ray is an array or integers, whose index value (in the case of default value of d_s) corresponds 1:1 to pixel distance away form the center. So if we “stack” all of the rays, aligning their array representations by their start index, we will have a measurement matrix, M, that has m rows and c columns, where , and , the length of the hypotenuse of the triangle whose right angle sides are the x and y dimensions of the image being sampled. If d_s = 1 (by default), then c = l_max. To see why c takes this value, consider the following extreme case we must be prepared to handle. If we place a center in one corner of the image, then a ray may extend to the opposite corner, requiring l_max array locations for its measurements.

Given M, we now compute mean, median, and standard deviation along column “slices” of M. This results in , , and vectors, whose array representation indices correspond to radial pixel distance away from the center. Since rays have different lengths—they each encounter the edge of the image in a different place, at a different distance from the center from the other rays—they each populate a row of M to a different extent, up to a certain column index; the remaining columns are populated with ∞ so that the part of our algorithm computing , , and knows when to drop this ray from the computation.

Now we compute the radius of the measurement area, r_m, in the following way. One may specify (as an optional parameter) a threshold length, l_t (defaults to 1000 pixels), over which to locate the global minimum (darkest point) in . That is, .

Our algorithm now creates three plots of the data, where the x-axis denotes distance from the center, and the y-axis denotes intensity level. The first shows every ray measurement (various colors), upon which (blue) and (red) are overlaid; its title gives r_m. The second shows (blue) ± (gray), overlaid with segmented least squares fits to (black); its title gives the length (l), slope (s), and least squares error (e) for each fitted segment. The third shows (red) ± (gray), overlaid with segmented least squares fits to (black); its title gives the length (l), slope (s), and least squares error (e) for each fitted segment. The segmented least square fits are given by a dynamic programming algorithm [30], using a cost parameter C = 200. We should note now that this entire process is bounded by, and repeated for, each bundle. So for example, if m = 4, then , , , and r_m are computed, and plots are created, for those rays that fall within each successive of the circle.

Fig 2 shows the circles (red) defined by the r_m found for each of the three centers specified in our canonical image (n = 80, m = 1), corresponding to vessel locations in the registered H&E image. The intensity analysis for the three circles’ areas is given in Fig 3. S6 Fig shows the sectors (red) defined by the r_m found for each bundle of each of the three centers specified in our canonical image (n = 80, m = 8), corresponding to vessel locations in the registered H&E image. We do not show the corresponding 24 intensity analysis figures.

Download:

Fig 2. Loci of single-bundle hypoxia gradients.

Circles (red) defined by the r_m found by the Intensity-Sample-Ray-Bundles algorithm for each of the three centers we specified, corresponding to vessel locations in the registered H&E image. Here we show m = 1 sector (2π radians per sector) for each center. Sectors are labeled with red numbers, counterclockwise, just outside of the red sector contour.

https://doi.org/10.1371/journal.pone.0153623.g002

Download:

Fig 3. Hypoxia gradient analysis.

Intensity level analysis produced by the Intensity-Sample-Ray-Bundles algorithm for centers 1 (left 3 panels), 2 (middle 3 panels), and 3 (right 3 panels). Intensity-Sample-Ray-Bundles creates three plots of the data, where the horizontal axis denotes distance from the center (pixels), and the vertical axis denotes intensity level. The first panel shows every ray measurement (light gray), upon which (blue) and (red) are overlaid; its title gives r_m (pixels). The second panel shows (blue) ± (gray), overlaid with segmented least squares fits to (black); its title gives the length (l, pixels), slope (s), and least squares error (e, pixels) for each fitted segment. The third panel shows (red) ± (gray), overlaid with segmented least squares fits to (black); its title gives the length (l, pixels), slope (s), and least squares error (e, pixels) for each fitted segment. The segmented least square fits are given by a dynamic programming algorithm using a cost parameter C = 200.

https://doi.org/10.1371/journal.pone.0153623.g003

Stratifying image data with respect to gradients.

Our first examination of high-concentration anti-pimonidazole images using this method was inconclusive. While it provided evidence for the presence of a gradient following the description above, the slopes of the relevant segments in the linear fit to the mean and median intensity measurements contained too much variation for a meaningful measurement of gradient steepness. It is common practice in many biology experiments to stain tissues using at least two concentration levels. The higher (or highest) concentration functions as a binary test for effectiveness of the stain. It answers: Is the phenomenon captured? Did it stain correctly? Provided that it did, follow up staining is conducted at lower concentrations. In the case of our data set, two concentrations, high and low, were used. Since the high-concentration images might contain excessive contrast, saturating the regions of hypoxia—beneficial for intensity-level-based image segmentation—this may swamp the more subtle gradient signal. We realized that we should attempt the same analysis on a corpus of low-concentration anti-pimonidazole images. For the purposes of measuring gradients, we sought to stratify the data differently than before, and select low-concentration anti-pimonidazole images, taken at 10× magnification, that contain one or more complete lesions, each containing one or more blood vessels. This gave us 23 such anti-pimonidazole images, each taken at 10× magnification, having corresponding registered H&E images from a section 10 μm away.

Measuring Quad-Tree statistics

To examine the property of intensity variance at different scales in the image, we employed the Quad-Tree algorithm, adapting it to work with any aspect ratio, not just square images. This works in the following way. For the given rectangle R, consider the set of pixels, P, within it, and the corresponding set of intensity values, I_P. If the then decompose R into four equal-size rectangles, R₁, R₂, R₃, R₄, and perform the quad-tree algorithm on R₁, R₂, R₃, R₄. This method quickly locates those regions of the image that contain a sufficiently high noise-to-signal ratio. S5 Fig shows the quad-tree decomposition of our canonical image.

We implemented a version of Quad-Tree, that we call Ply-Stats-Quad-Tree, that reports statistics related to the search tree for the image that it processes. These include the count, sum, mean, median, standard deviation, and coefficient of variation (CV) for the number of leaves at each ply, and a histogram of the counts of leaves at each ply. We use CV in intensity value of the current frame’s pixels as our splitting property, where CV exceeding a given threshold, τ, generates a split. The algorithm reports search tree statistics for the Quad-Tree dissection at a given value of τ.

Deriving canonical EPC signatures

The Euler-Poincaré characteristic (EPC), one of the Minkowski functionals [31–33], is a measure of structural connectedness (or alternatively, porousness), and it has been used recently in two applications. The first concerns measuring bone density. Rath, et al [34] used the EPC to visualize and assess local trabecular bone structure; and Roque, et al [35] used the EPC to identify low bone density from vertebral tomographic images. The second application is in classifying tumors. Hutterer, et al [36] used the EPC to assign a characteristic signature curve to each AFM image of different tumor types, then used that curve as the basis of a classification method. We were intrigued by the use of characteristic EPC curves as an image feature by which to logically characterize, or functionally classify, chronic tumor hypoxia, and so apply this algorithm in our analysis.

We implemented an algorithm that follows directly from the approach taken by Hutterer, et al [36] to construct an Euler-Poincaré signature curve for an image. First, it converts the RGB image to an 8-bit gray level image , but does not smooth. Then for each gray level i = 1, …, 255, it produces a binary intensity-thresholded image and records for each i. This method gives a signature EPC curve for each that could serve as an image feature.

Spatiotemporal logical characterization of hypoxia in tumor histology

Next we consider spatial partitioning, where continuous boundaries that separate tissue types are introduced into the image. This requires some degree of familiarity to manually parse these histology images, and so lacks the scalability in the number of images we require for statistical analysis. In Fig 4 we have another canonical anti-pimonidazole image, its manual partitioning, and its labeled partitions. Segmentation by partitioning reveals containment properties of the different regions and leads us to infer which tissue structures are nestable.

Download:

Fig 4. Tissue types by manual spatial partitioning.

Another (unsmoothed) canonical anti-pimonidazole image (top), its manual partitioning (middle), and its labeled partitions (bottom). Key: V = viable, N = necrotic; unlabeled, brown regions are hypoxic.

https://doi.org/10.1371/journal.pone.0153623.g004

We first make some observations about the histological data that we can formulate as grammatical transformations. Then, in the results and discussion section, we use these transformations to globally constrain spatiotemporal logic predicates comprising the specific image processing features discussed above.

Using a grammar to describe how tissue regions transform.

In Fig 4 (bottom), we segment and unambiguously identify viable (V), hypoxic (H), and necrotic (N) tissue regions in our anti-pimonidazole images. After identifying these regions on our full set of images, we observe the following qualitative patterns. Temporally: N always expands. Spatially: at any given time, in any given image, selecting a point and proceeding in a single direction away from the point will traverse either the V → H (ascending gradient) → N → H (descending gradient) → V cycle, or the V → H (ascending gradient → descending gradient) → V cycle; and the variation in the width of H, measured in the V → N direction, is much less than the variation in the width of the N or V regions measured in any direction—where N and V are blobs, H tends to be a well-defined band about V.

From these observations, we formulate the following two axioms for the tissue regions.

A1 (spatial) V and N are invalid neighbors; H must separate them.
A2 (temporal) There is a temporal monotonicity in how a region develops: V becomes H, and H becomes N, where N is the absorbing state.

From axioms A1 and A2, we can derive both context-free and context-sensitive grammar production rules for the spatiotemporal transformation of hypoxia. The context-free production rules correspond to origination of a new tissue type, to nesting, to diversification. The context-sensitive production rules correspond to elimination of an existing tissue type, to collapsing, to homogenization. Axioms A1 and A2 lead us to derive valid production rules and restrict us from deriving invalid production rules.

Here are the four valid production rules:

V → V H V (H origination in V) by A2
H → H N H (N origination in H) by A2
H V H → H (V elimination in H) by A2
N H N → N (H elimination in N) by A2

Here are the eight invalid production rules:

H → H V H (V origination in H) by A2
N → N H N (H origination in N) by A2
N → N V N (V origination in N) by A2
V → V N V (N origination in V) by A1
H N H → H (N elimination in H) by A2
V H V → V (H elimination in V) by A2
N V N → N (V elimination in N) by A1
V N V → V (N elimination in V) by A1

Using a logic to describe hypoxia.

We have defined above some quantitative and qualitative image features we now wish to incorporate into a logical proposition that describes what hypoxia is like in space and time. We will apply thresholds to the quantitative features to render them as predicates, and thus build up our final proposition out of these predicates.

Extending Probabilistic Bounded Linear Temporal Logic.

The logic we develop here is an adaptation of Probabilistic Bounded Linear Temporal Logic (PBLTL) [24] that accommodates the three dimensions of space as well as time.

For a stochastic model simulation S, let the set of state variables SV be a finite set of real-valued variables. A Boolean predicate over SV is a constraint of the form u ∼ v, where u ∈ SV, ∼ ∈ { ≥, ≤, = }, and . A BLTL property is built on a finite set of Boolean predicates over SV using Boolean connectives and spatiotemporal operators. The syntax of the logic is given by the following grammar: , where , and . We can define additional spatiotemporal operators such as and in terms of the bounded until . A PBLTL formula is a one of the form P_≥θ(ϕ), where ϕ is a BLTL formula and θ ∈ (0, 1). We say that S satisfies PBLTL property P_≥θ(ϕ), denoted by S ⊨ P_≥θ(ϕ), if and only if the probability that an execution of S satisfies BLTL property ϕ is greater than or equal to θ.

Let x_d denote the spatial dimension x₁, x₂, or x₃ we wish to specify, and let x_lim and t_lim denote the limits in spatial dimension x_d and time dimension t, respectively, we wish to specify. The spatiotemporal operators can be interpreted as follows:

means within x_lim spatial units in x_d, ϕ₁ holds until ϕ₂ holds.
means within t_lim time units in t, ϕ₁ holds until ϕ₂ holds.
means within x_lim spatial units in x_d, ϕ holds.
means within t_lim time units in t, ϕ holds.
means for x_lim spatial units in x_d, ϕ holds.
means for t_lim time units in t, ϕ holds.

Continuing to follow Jha, et al [24], we define the semantics of our extended BLTL with respect to executions of S. Let σ ⊨ ϕ denote that an execution trace σ of S satisfies ϕ. Let σ = (s₀, t₀), (s₁, t₁), … be an execution of the simulator along states s₀, s₁, … with durations . We denote the execution trace starting with state i by σⁱ. The value of the state variable x in σ at state i is denoted by V(σ, i, x). The semantics of our extended BLTL for a trace σ^k starting at the k^th state () is defined as follows:

σ^k ⊨ x ∼ v iff V(σ, k, x)∼v
σ^k ⊨ ϕ₁ ∨ ϕ₂ iff σ^k ⊨ ϕ₁ or σ^k ⊨ ϕ₂
σ^k ⊨ ϕ₁ ∧ ϕ₂ iff σ^k ⊨ ϕ₁ and σ^k ⊨ ϕ₂
σ^k ⊨ ¬ϕ iff σ^k ⊨ ϕ does not hold
iff such that (1) ∑_{0<l ≤ i}(x_{d, k+l} − x_{1, k+l−1}) ≤ x_lim, (2) σ^k+i ⊨ ϕ₂, and (3) for each 0 ≤ j<i, σ^k+j ⊨ ϕ₁.
iff such that (1) ∑_{0 ≤ l < i} t_k+l ≤ t_lim, (2) σ^k+i ⊨ ϕ₂, and (3) for each 0 ≤ j < i, σ^k+j ⊨ ϕ₁.

Each of the last two semantic statements has three necessary conditions, which we clarify as follows. (1) In the case of spatial units, the sum of the spatial intervals in x_d along the state sequence k, k+1, …, k+i should be less than or equal to the limit value of x_lim specified—this implements “within x_lim spatial units in x_d.” In the case of time units, the sum of durations along the state sequence k, k+1, …, k+i should be less than or equal to the limit value of t_lim specified—this implements “within t_lim time units.” (2) This implements “at some state i beyond state k, ϕ₂ holds.” (3) This implements “For each state from k up to but not including state i, ϕ₁ holds.”

Linear regression functional characterization of hypoxia in tumor histology

We now propose a second way to characterize hypoxia in tumor histology, using a simple machine learning approach to adaptively weigh the contributions of each and every image processing feature to score candidate histology images (or simulation results) for their similarity to ones containing stable local regions of hypoxia.

Ergodic assumption.

We assume the chronic tumor hypoxia process that generates our image data to be ergodic: since we see so many instances of lesions, we are likely seeing every temporal state of a typical lesion, and so, in the limit of static images, we observe the temporal and spatial phenomenon of hypoxia.

Linear regression learning.

We would now like to incorporate the quantitative image features defined above into a linear functional form, whose weights are learned by regression [37], for a lesion hypoxia similarity metric. Our approach here is adapted from earlier work in a different domain [29]. This entails solving an overdetermined system of equations, given by a₁ f_{1, j}+…+a_n f_{n, j} = 1, where the a_i, i = 1, …, n are the n feature coefficients to be learned and the f_{i, j}, i = 1, …, n, j = 1, …, m are the corresponding n feature values over m ≥ n observations forming the feature matrix, F. We train a linear regression model on m calibrating lesions, having known similarity score 1, using values from the n features, giving . The model has the analytic solution . This gives a trained similarity estimator, .

This formulation of assumes all lesions, i.e. their associated feature values, have equal weight, owing to their equivalent validity as observations. However, such an assumption may be challenged on the grounds that upon taking into consideration the difference between the empirically measured null distribution and the actual shape of the distribution in feature measurements, certain observations appear to be false positives, and others false negatives—a notion formally addressed by robust regression, namely, the Beaton-Tukey formulation.

Weighting training data to address Type I and Type II errors.

Normally, false positive examples appear as ones that deviate significantly from the null-distribution, and if not discarded, can affect the statistical estimators adversely. However, instead of discarding such outliers using sharp-thresholds, and using the filtered examples in the estimator, one may assign to each data point a positive weight that signifies how likely it is that a particular example is an outlier. Such a weighting scheme could be based on the ideas underlying robust M-estimators—a class of central tendency measures that make them resistant to local misbehavior caused by outliers (e.g., false positives). We adapted the Beaton-Tukey biweight [38]—an iteratively reweighted measure—for this purpose of central tendency. We note that other schemes, such as Huber’s M-estimator, could have been used with similar performance. Both the biweight and the Huber weight functions are available in standard statistical packages. Here we use Matlab’s robustfit command with default parameters (weight function “bisquare,” using a tuning constant of 4.685).

In the context of our system, the x_i, i = 1, …, m are the feature values of the m calibrating lesions in the training set. Each lesion is assigned a weight, w_i. If its weight is zero, then the corresponding lesion is discarded from the training set. Of the m training molecules, m′ remain. This gives a weighted-trained similarity estimator, .

In our modeling of estimation error above, one or more features in training may introduce too much variance (systematic error) or dependence (model error). We would like our model to have an extensible and adaptive structure, where any number of features may be used, and proceed with confidence, knowing that noisy or dependent features will have a contribution to the estimate that shrinks to zero. We now apply one of the following patterns of shrinkage to the feature coefficients, .

Shrinking feature coefficients to reduce feature space dimensionality.

In 1961, James and Stein published their seminal paper [39] describing a method to improve estimating a multivariate normal mean, , under expected sum of squares error loss, provided the degree of freedom k ≥ 3, and the μ_i are close to the point to which the improved estimator shrinks.

When extreme μ_i are likely, then spherical shrinkage may give little improvement. This may occur, for instance, when the μ_i arise from a prior distribution with a long tail. A property of spherical shrinkage is that its performance is guaranteed only in a small subspace of parameter space, requiring that one select an estimator designed with some notion of where is likely to be, such that the estimator shrinks toward it. An extreme μ_i will likely be outside of any small selected subspace, implying a large denominator and so little, if any, shrinkage in , thereby giving no improvement. To address this problem, Stein proposed a coordinate-based (or truncated) shrinkage method.

Applying the metric.

Once the weighted-trained model feature coefficients, a_i, have undergone shrinkage, , we have our final hypoxia similarity estimator, that can measure out-of-sample lesions for their similarity to hypoxic lesions. gives a [0, 1] numerical score instead of a {0, 1} outcome. A simulator that implements this scoring function can then feed a branch-and-bound (optimization) process that can explore the simulator’s configuration parameter space.

Code availability

All code described in this paper is written in Matlab and available in the GitHub repository: https://github.com/aesundstrom/tumor-hypoxia-image-processing

Histology image availability

All histology images of chronic tumor hypoxia used in this paper are available in the Harvard Dataverse: http://dx.doi.org/10.7910/DVN/SI32FV

Results and Discussion

First, we discuss four experiments corresponding to the four image processing features we develop here—segmenting by histogram multithresholding, measuring image intensity gradients, measuring quad-tree statistics, and deriving canonical EPC signatures. Next, we use these image processing features to develop a spatiotemporal logical characterization of chronic tumor hypoxia in histology images. Finally, we propose another way to characterize chronic tumor hypoxia in histology images, using a simple machine leaning approach.

Segmenting by histogram multi-thresholding

Setup.

We applied Otsu’s method to multithreshold a set of n_T = 66 images across stratification criteria, magnification, and high and low concentrations of anti-pimonidazole. To distinguish between results for the high- and low-concentration images, we place, alongside the results for the total set of images, those results for n_H = 36 high-concentration images and n_L = 30 low-concentration images, computed separately. See Table 1. The table organization also reflects the distinction between unsmoothed and smoothed gray images. We illustrate this distinction in Fig 5.

Download:

Table 1. Otsu’s multithreshold segmentation of unsmoothed versus smoothed images over the total set of images (n_T = 66), high anti-pimonidazole images (n_H = 36), and low anti-pimonidazole images (n_L = 30).

We report pixel areas as proportions of the entire set of pixels in the image (I); hence H:I, V:I, and N:I. We also report another proportion of interest, namely that of hypxic to viable cells in the image, H:V.

https://doi.org/10.1371/journal.pone.0153623.t001

Download:

Fig 5. Otsu segmentation and smoothing.

How Otsu’s multithreshold segmentation differs between unsmoothed gray (upper left) and smoothed gray (lower left) images. Corresponding images on the right show dark blue regions that denote hypoxic cells, light blue regions that denote viable cells, and yellow regions that denote necrotic cells.

https://doi.org/10.1371/journal.pone.0153623.g005

Results.

In Table 1 we observed the following for unsmoothed and smoothed images. Otsu’s method found intensity level partitions whose means are remarkably stable (CV = {0.09, 0.09}, {0.09, 0.09}) across such a variable total set of images. As expected, the stability of these partitions increased as we stratified the images into high-concentration (CV = {0.07, 0.07}, {0.09, 0.08}) and low-concentration (CV = {0.06, 0.04}, {0.05, 0.05}) subsets. Of the pixel proportions, the most stable mean value was always V:I, for the total set and both strata. The mean H:V ratio was also stable across strata (CV = {0.24, 0.20, 0.22}) for unsmoothed images, but not as much (CV = {0.45, 0.37, 0.33}) for smoothed images. In unsmoothed images, across strata the H:V ratio had a similar mean value (σ = {0.36, 0.32, 0.39}); in unsmoothed images, across strata, the mean values varied significantly (σ = {0.52, 0.39, 0.67}); between unsmoothed and smoothed images the corresponding mean H:V ratio values seemed to have no relationship ({0.36, 0.52}, {0.32, 0.39}, {0.39, 0.67}), though the smoothed, high-concentration mean value (0.39) did seem to fit with the cross-strata values in the unsmoothed images.

Discussion.

The mean partition values, and the mean H:V ratio values for unsmoothed images, were stable. They could serve as image features.