Overview of GBJ

The Generalized Berk-Jones statistic provides a powerful parametric approach for testing the association between a set of SNPs and a phenotype using marginal SNP summary statistics. Consider a gene set that contains d SNPs. The d summary statistics for these SNPs, Z = (Z₁, …, Z_d)^T, follow a joint multivariate normal distribution, GBJ aims to robustly test H₀: μ_Z = 0_d×1 against H₁: μ_Z ≠ 0_d×1 when μ_Z contains a subset of zeros and while accounting for the correlation between test statistics. Thus common GSA features such as LD between SNPs and multiple neutral variants are automatically incorporated into the statistical framework. GHC and other tests in the goodness-of-fit class operate similarly, but for reasons of space, we will limit comparisons to simulation results.

The null hypothesis corresponds to the situation where no SNPs in the entire set are associated with the outcome after correction for confounders. When performing GSA with genotype-level data for each subject it is necessary to both calculate Z and estimate Σ, and when summary statistics are available, it is only necessary to estimate Σ.

Calculation of Z and Σ with genotype-level data

Suppose we have genotype-level data for i = 1, 2, …, n subjects at a set of j = 1, 2, …, d SNPs, so that the genotype vector for subject i is G_i = (G_i1, …, G_id)^T. Let G = [G₁, …, G_n]^T be the n × d genotype matrix. Suppose also that we have a set of q additional covariates contained in X = [X₁, …, X_n]^T, which is an n × q matrix with X_i = (X_i1, …, X_iq)^T for i = 1, …, n. Denote the outcome by Y_i and let μ_i be the mean of Y_i conditional on G_i and X_i. Consider the generalized linear model [35] (GLM) for μ_i given by where g(⋅) is a canonical link function, for instance, g(μ_i) = μ_i for normally distributed phenotypes and g(μ_i) = logit(μ_i) for binary phenotypes. We are interested in testing the null hypothesis of no gene set effect H₀: β = 0_d×1 against the alternative H₁: β ≠ 0_d×1. As some SNPs in a gene set are likely neutral variants, we allow elements of β to equal 0 under H₁. The marginal score test statistic under the null is (1) for any SNP j = 1, …, d in the gene set. Here Y = (Y₁, ‥, Y_n)^T, , and is the MLE of α under H₀: β = 0_d×1. Also we define the single variant vector G_·j = (G_1j, …, G_nj)^T, the projection matrix P = W − WX(X^T WX)⁻¹ X^T W, and the GLM weight matrix . The standard GLM dispersion parameter is given by , and the standard GLM variance function is , where for a normally distributed phenotype and for a binary phenotype. The Z_j are asymptotically equivalent to the individual SNP test statistics calculated by many popular tools, such as the Wald statistics produced by PLINK [36]. A consistent estimate of the correlation between Z_j and Z_k is then given by (2)

Note that H₀: β = 0_d×1 in the regression model corresponds to the null hypothesis H₀: μ_Z = 0_d×1 from above. This relationship can be derived directly by calculating the expectation of the score statistic in Eq (1) as a function of β. The multivariate normality and covariance matrix follow from a Taylor expansion [37] of Eq (1). Thus testing for an effect in the gene set corresponds directly to testing for a non-zero mean among the multivariate normal test statistics.

Estimation of Σ with precalculated summary statistics

Precalculated marginal summary statistics are much more available than subject-level data. When using these summary statistics, we need to estimate their correlation structure from a reference LD panel containing individuals of a similar ethnicity, e.g. data from the 1000 Genomes Project [38]. Specifically, let and denote the genotype of each subject in the reference panel at SNPs j and k. Also let X^(r) = (1, PC₁, …, PC_m) denote a modified design matrix, where m is the same number of principal components used in the original analysis, and PC₁, ‥, PC_m are principal components calculated from the reference data. Using X^(r) instead of X, instead of (G_.j, G_.k), and any constant in place of in Eq (2) provides a good approximation to .

The motivation for this approximation arises from the observation that Eq (2) is exactly the correlation structure of the genotypes in the set if X includes only an intercept term. The correlation structure of the genotypes can be well-approximated by a reference panel. When X does include PCs or non-genetic terms, the PCs can still be well-approximated from a reference panel, and non-genetic terms often demonstrate negligible correlation with the genotypes, so they will possess minimal impact on the estimation of Eq (2). Using a constant for is appropriate because the fitted mean generally does not vary much with the PCs, and since our approximation utilizes a design matrix with only the PCs, will be close to the same value for each subject i.

The Generalized Berk-Jones statistic

Next assume we have calculated or been supplied a vector of test statistics . Denote by the survival function of a standard normal random variable, with Φ⁻¹(t) denoting its inverse. Further designate |Z|_(j) as the order statistics of the vector that arises from applying the absolute value operator to each element of Z, so that |Z|₍₁₎ is the smallest element of Z in absolute value. Finally define the significance thresholding function where t > 0 is the threshold.

It is helpful to think of S(t) as the “number of significant SNPs at threshold t.” For example, in GWAS, some researchers set t = 5.45131 so that S(t) counts the number of SNPs with p-values less than 5 × 10⁻⁸, the commonly-used cutoff for declaring genome-wide significance [39]. Other set-based methods implicitly set t equal to |Z|_(d) and carry forward |Z|_(d) as the representative test statistic for the entire set [23]. However in both of these examples, the choice of t is rather arbitrary and relies on a one-size-fits-all-sets approach. In particular, neither choice of t makes full use of the data, ignoring factors such as the size of the set and the LD pattern among SNPs. More importantly, moderately significant SNPs that do not reach a GWAS threshold or demonstrate the lowest p-value in a gene can cumulatively produce a major contribution to the phenotype. A key concept behind GBJ is that it can adaptively find the threshold t that best maximizes power for any given set while adjusting for the size of the set and the correlation structure of the SNPs.

Consider first the case of no LD, Σ = I, where I is the identity matrix. Then for a fixed t under the null hypothesis, S(t) has the binomial distribution S(t)∼Bin(d, π) with . This observation motivates the Berk-Jones (BJ) statistic [40], which can be written as [41], where , and solves the equation In other words, Berk-Jones is the maximum of a set of likelihood ratio tests performed on S(t) at all observed test statistic magnitudes greater than or equal to the median observed magnitude. This interpretation relies on the characterization of S(t) as a binomial random variable, which we further emphasize by defining so that the indicator functions 1(|Z_j| ≥ t) may be viewed as the identically distributed binary variables that constitute a binomial variable. While the BJ statistic can be written without , we find that this notation helps simplify the interpretation and also provides a natural transition to the GBJ. By taking the maximum of the likelihood ratio test over many possible thresholds, Berk-Jones allows the data to set the threshold that provides the most power in the presence of an appreciable subset of neutral variants.

When SNPs in a gene set are in LD and Σ ≠ I, then S(t) no longer has a binomial distribution, and the Berk-Jones statistic can lose much of its power in finite samples. However the Generalized Berk-Jones incorporates the additional correlation information by explicitly conditioning on Σ, GBJ is still the maximum of a set of likelihood ratio type tests, but it gains notable power over BJ by accounting for the correlation between test statistics. When Σ = I, GBJ reduces to the standard Berk-Jones. The p-value of the GBJ statistic can be calculated analytically.

In integrating the test statistic for each SNP in the gene set, GBJ incorporates much more information than methods that only keep the most significant p-value in each gene [11, 14, 16] and discard the rest in an ad-hoc fashion. Other tests may not discard individual test statistics but instead summarize the values in mean [12] or count-based procedures [14, 15] so that the individual magnitudes are lost. For example, test statistics of Z₁ = −2.5 and Z₂ = 5 make equal contributions to a procedure that counts the number of p-values less than 0.05. However, the variant with test statistic Z₂ = 5 clearly conveys more information about genotype-phenotype association than the variant with Z₁ = −2.5. In contrast, GBJ does not discard or summarize information and utilizes each marginal test statistic, as well as the joint correlation structure. When Σ = I, GBJ enjoys certain asymptotic power properties that other tests may not demonstrate [41].

GBJ step-down inference

As an extension of GBJ, we propose a step-down inference procedure to filter out gene sets that are driven to significance based on the signal from only a very small proportion of genes, such as the earlier example involving Ear Morphogenesis and FGFR2. The procedure begins by performing gene-level association analysis. First, create a list of the unique genes over all gene sets under consideration, then define each gene as its own set and apply GBJ over each single gene to find single gene p-values. Sort the single genes in increasing order of p-value, which can be interpreted as ranking the genes by their level of association with the outcome.

For any given gene set, obtain a measure of how much its association signal is dependent on a few highly associated genes by applying GBJ to the set after removing all SNPs that belong to the top k genes. Setting k = 1 will identify gene sets where the signal is driven by a single gene. If a gene set remains significant even after removing k > 1 genes, then the set possesses signals dispersed through many different genes. In this work we will use k = 1 and k = 3. As we show below, FGFR2 drives the significance of many gene sets in a breast cancer analysis, but the step-down procedure allows us to uncover pathways that show no association outside of FGFR2 and may be less suitable for follow-up.

GWAS summary statistics datasets

Three large, publicly available summary statistic datasets are analyzed in this study. We first obtained summary statistics from the largest breast cancer GWAS cohort to date [30], with 122,977 cases and 105,974 controls of European ancestry. Most subjects were genotyped on the OncoArray, a custom-designed array for cancer studies that also has genome-wide coverage of over 570,000 SNPs. Data for these subjects were imputed using the full 1000 Genomes Project Phase 3 reference panel, resulting in estimated genotypes for approximately 21 million variants. Other subjects were included from various smaller studies, including the iCOGS project [42] and 11 smaller GWAS. Results across studies were then meta-analyzed, and after quality control, approximately 12 million SNPs produced a final test statistic for association with breast cancer.

For height, we downloaded summary statistics from the Genetic Investigation of Anthropometric Traits (GIANT) GWAS [43]. In this study, 253,288 individuals of European ancestry were genotyped on multiple Affymetrix, Illumina, and Perlegen arrays. All individuals were then imputed to the Phase II CEU HapMap release. After meta-analysis and quality control, there were over 2.5 million SNPs with a final summary statistic for association with height.

The last dataset used was downloaded from the Psychiatric Genomics Consortium schizophrenia mega-analysis [44]. In the primary GWAS of this study, 34,241 cases and 45,604 controls were genotyped across 49 cohorts. The vast majority of samples were obtained from subjects of European descent, but three cohorts did contain individuals of East Asian ancestry. All subjects were imputed using 1000 Genomes Project data as a reference panel, and test statistics were meta-analyzed across cohorts. For this study, summary statistics were made available for approximately 9.5 million variants.

Pathway definition database

To provide a fair comparison between GBJ and GSEA in re-analysis of the breast cancer data, we conduct all pathway analysis using the same gene set database (Gary Bader Lab, Human GOBP all pathways, no GO IEA; April 1, 2017) used in the original breast cancer pathway analysis [30]. This file compiles gene sets from a number of databases including Gene Ontology (GO) Biological Process [31], Reactome [45], Panther [46], and others. In all, the file contains 16,528 gene sets. While we are unaware of a comprehensive method to assess the quality of gene set databases, advantages of this list include the incorporation of multiple different sources, public availability, and monthly updates.

As the selected pathway database is a direct aggregation of multiple sources possessing varying levels of curation, some preprocessing of the list is necessary before beginning analysis. We first truncate the database to remove all pathways with more than 200 genes or less than three genes. Removing pathways with a large number of genes is common practice [47, 48], as extremely large pathways are difficult to interpret, and this step was performed in the original GSEA analysis as well. The original GSEA analysis further removed all pathways with fewer than ten genes, which is also a relatively common strategy to reduce false positives and lower the multiple testing burden [49], however it has been noted that this threshold may exclude certain specific and informative functional sets such as protein complexes [26]. We choose to set the lower limit for pathways at three genes because there may be interesting insights to be gleaned from smaller gene sets and because we believe GBJ is powerful enough to overcome the increased multiplicity burden. In total, there are 10,742 pathways with between three and 200 genes.

SNP-gene mapping

For each set of summary statistics, we map individual SNP test statistics to gene sets if they lie within 5 kb of a gene in the set, with coordinates provided by Ensembl 90 gene annotations [50]. Estimates of the correlation between summary statistics are calculated using unrelated subjects from European cohorts (TSI, FIN, GBR, IBS, CEU) in the 1000 Genomes Phase 3 data release. Summary statistics belonging to SNPs that have minor allele frequency less than 3% in the reference panel are removed as their data would be unstable for estimation. The original GSEA analysis maps SNPs to genes in a slightly different manner and also performs some additional manual curation. These additional steps are unique to the breast cancer dataset, and we do not include them here to preserve generality and facilitate comparison of results across traits.

To further reduce the computational burden of a GBJ analysis, we additionally trim SNPs that are in the same gene and are correlated at r² > 0.5. This pruning is performed in PLINK and occurs at successive multiples of 0.5 for very large sets, so that all tested sets are less than 1,500 SNPs in size. Even after the pruning procedure, GBJ still incorporates a very large amount of information, as the median number of SNPs in our breast cancer analysis is 470. In contrast, the median number of genes in the GSEA analysis is 26; a test on 26 genes with GSEA incorporates only the 26 minimum p-values from those genes.

The data-intensive nature of GBJ does pose problems for a small number of extremely large pathways. Some gene sets are larger than 1,500 SNPs even after pruning all pairs of SNPs in the same gene with r² > 0.0625. These sets are not tested to remain consistent in our analysis protocol across all three phenotypes. However for typical GSA focusing on a single outcome, it would be straightforward to perform additional pruning or manual curation to accommodate testing the largest pathways.

Simulation

We use simulation to compare the performance of GBJ against the self-contained version of GSEA (as described by Wang et al. [11]), self-contained MAGMA, GHC, and SKAT [51]. GSEA and MAGMA are two of the best-performing methods in the comprehensive simulation study of a recent GSA review [28]. SKAT is not often mentioned in the GSA literature, possibly because it was developed as a test requiring individual-level genotype data [51, 52], but it is known to demonstrate excellent set-based testing power and can be implemented with summary statistics if the same correlation matrix approximation we have introduced for GBJ is applied. To match the settings of our real data analysis, we use genotype data from the reference panel of n = 350 unrelated Europeans in the 1000 Genomes Project. These genotypes have been pruned as described above, and we use all SNPs located in 10,000 genes chosen at random. Each of the 10,000 genes contains between 7 and 25 SNPs, which corresponds to the middle 50 percentile of pruned gene size over the entire gene database.

For each iteration of the power simulation, we choose 10 genes at random to be the tested pathway, and a of the 10 genes are given b causal SNPs each. The true disease model is then where G_ij is the genotype of subject i at causal SNP j. We study various sparsity levels, causal SNP configurations, and causal effect sizes, demonstrating how the relative performance of each test can vary widely depending on the number and location of signals in a set. We also consider random placement of causal SNPs throughout the pathway. Testing is performed at at α = 0.01 with 100 runs performed at each parameter setting except for a Type I error simulation with 4,000 runs performed at β = 0.

Marginal summary statistics for each SNP in the pathway are calculated according to Eq (1). As the SNPs used in simulation are correlated due to linkage disequilibrium, the test statistics calculated in these simulations will also be correlated. The joint covariance matrix of the test statistics is estimated according to Eq (2), and this estimate is used for the GBJ and GHC statistics. A simulation demonstrating the accuracy of our proposed approximation to Eq (2) for precalculated summary statistics uses a separate breast cancer dataset [53, 54] and is described in S1 Appendix. MAGMA is applied with default self-contained settings, and 500 permutations are used for GSEA inference. SKAT is implemented with default settings as well.

We also perform a simulation to assess the validity of the step-down inference procedure. Data is generated as in the power simulation, but in each iteration we remove the most significant gene—chosen separately for each test—from the pathway before generating a pathway p-value. For GBJ, GHC, MAGMA, and SKAT, we use the default gene-level analysis to determine the significance of each gene, and for GSEA we remove the gene with the most significant SNP. In the simulations with only one causal gene (a = 1) we are benchmarking the discriminatory ability of this procedure, as the step-down procedure should remove the only causal SNPs, resulting in power equal to the Type I error rate regardless of effect size. In settings with a > 1, we can assess power to identify gene sets with dispersed signals.

Classification of biologically important systems

To summarize our results from applying GBJ across multiple phenotypes, we search for the biological systems that demonstrate the largest degrees of significance across each phenotype. Pathways are categorized into different systems by exploiting the directed acyclic graph structure of Gene Ontology (GO) Biological Process pathways. Starting with the top-level Biological Process category, GO defines successively smaller groups of pathways so that each child term is more specialized than its parent term. Using this natural structure, it is possible to group categories of pathways at different levels of granularity.

We create categories from the first level immediately following the Biological Process root. Specifically, we use the 11 top-level sets Biological Adhesion, Cellular Component Organization or Biogenesis, Developmental Process, Growth, Immune System Process, Localization, Locomotion, Metabolic Process, Reproduction, Response to Stimulus, and Signaling. For a given phenotype and category, we first calculate the expected number of significant pathways in the category conditional on the total number of significant pathways for the outcome. If pathways in each category truly have the same chance of reaching significance, then the expectation is simply equivalent to the percentage of all tested pathways that belong to the category multiplied by the total number of pathways significantly associated with the outcome. For each phenotype, we calculate the difference between the observed and expected number of significant pathways arising from each category, taken as a percentage of the expected number, to determine which categories harbor more significant pathways than expected. Pathways are deemed significant using the Bonferroni-corrected significance level of α = 0.05/10, 742 = 4.65 ⋅ 10⁻⁶.

Simulation

We first compared the power of GBJ to other gene set methods through simulations carried out with genotypes from the 1000 Genomes Project. One clear trend from these results was the impact of signal configuration on the relative power of each test. Because non-GSEA tests utilized data from multiple SNPs in each gene, we expected such tests to perform better at detecting signal configurations where multiple signals were placed into a single gene. However, even when each gene held only one signal (Fig 1B and 1D), a situation that would appear very favorable for GSEA, we found that GSEA rarely achieved power close to the best test. GBJ, GHC, and SKAT generally performed well in these settings, and MAGMA often lagged behind GSEA. When signals were more densely packed into a smaller number of genes (Fig 1A and 1C), GBJ, GHC, and SKAT increased their advantage over GSEA and MAGMA significantly, demonstrating the ability of these tests to account for grouped signal configurations. GBJ appeared to show the most robust performance across different signal configurations, never falling too far behind the best-performing test, unlike GHC (Fig 1D) or SKAT (Fig 1A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. GSA power simulation over four different moderately sparse configurations of gene signal density.

Simulated power of MAGMA, GSEA, SKAT, GHC, and GBJ (all self-contained versions) with random sets of ten genes selected from 10,000 total genes. From the ten genes in the set, a genes are selected to hold b causal SNPs each. The four subfigures correspond to (A) a = 1, b = 4, (B) a = 4, b = 1, (C) a = 4, b = 2, (D) a = 8, b = 1. The effect size is given on the x-axis. We perform 100 simulations at each parameter setting and test at α = 0.01. The relative power of each test is affected by both the signal configuration and signal sparsity.

https://doi.org/10.1371/journal.pgen.1007530.g001

Another clear trend was the difference in relative power as signal sparsity was modified. GHC, GBJ, and SKAT were previously observed [33, 34] to demonstrate their best performance in very sparse (number of signals less than ), moderately sparse (number of signals between and ), and dense (number of signals greater than ) settings, respectively, and we noted similar trends in our simulations. The number of SNPs in each simulated gene set was determined randomly but generally fell in the low hundreds, and so the very sparse regime included scenarios with four or less causal SNPs. The moderately sparse regime spanned from approximately five to 14 causal SNPs, and about 15 or more signals constituted a dense setting.

GHC frequently demonstrated the most power in the very sparse settings (S1 Fig), although GBJ followed closely and even overtook GHC in certain scenarios. SKAT, GSEA, and MAGMA performed much worse, an expected development given that these three tests were not specifically developed to detect sparse signals. Under the moderately sparse settings of Fig 1, GBJ possessed the most power more often that any other test. With more abundant signals (S2 Fig), SKAT and GBJ generally showed the most power, and GHC notably lagged behind in certain configurations (S2C and S2D Fig). The effect of grouped signals was still observed as MAGMA and GSEA again appeared to possess relatively less power than the other three tests in dense settings when more signals were placed in the same gene (S2A and S2B Fig).

Simulations with random placements of causal SNPs further emphasized the large impact of sparsity on relative performance. With only two or five causal SNPs (Fig 2A and 2B), GHC and GBJ generally produced the most power. As the number of SNPs increased, GBJ continued to show either the best or second best power, with SKAT overtaking GHC in the settings with many causal SNPs (Fig 2E and 2F). The differences between tests diminished as both the number of causal SNPs and the effect size rose, with all methods performing reasonably well at the largest effect sizes in dense settings. This behavior illustrated that many different tests excel at detecting strong and abundant signals, but the rare and weak settings offer more challenges.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. GSA power simulation with unstructured signal configurations over different sparsity settings.

Simulated power with random sets of ten genes selected from 10,000 total genes. Causal SNPs are chosen at random from all SNPs in the set. The subfigures correspond to (A) 2, (B) 5, (C) 8, (D) 9, (E) 12, and (F) 15 signals. We perform 100 simulations at each parameter setting and test at α = 0.01. GBJ offers robust performance through very sparse, moderately sparse, and dense settings, while other tests show decreased power in certain scenarios.

https://doi.org/10.1371/journal.pgen.1007530.g002

As researchers never know the signal sparsity prior to testing, utilizing GBJ or GHC provides some protection against signals that are difficult to detect and can offer more advantages than selecting a test for the dense setting, where most methods will perform well. GBJ in particular shows exceptional robustness across different settings and offers comparable power to SKAT even when there are many signals. Further, we would usually expect GBJ to outperform GHC in GSA because gene set analysis often involves many genes and because the very sparse regime becomes much smaller in size relative to the other settings as the number of SNPs in the set grows. For example, when there are only 100 SNPs in a set, the very sparse setting encompasses 1-3 signals, while the moderately sparse setting encompasses 4-10 signals. When there are 1,000 SNPs, the very sparse setting only stretches from 1-6 signals while the moderately sparse setting stretches from 7-32 signals. GBJ also generally runs at a faster speed than GHC (S1 Table). GHC would be a better choice under extreme sparsity, as in only one or a few signals in the entire set.

All methods appeared to control the Type I error rate fairly well at α = 0.01 (S3 Fig), as their power remained approximately equal to the nominal size of the test when there was no true effect. This observation suggested that our inference was valid for the situations considered in simulation. Assessments of the correlation matrix approximation for summary statistics demonstrated the accuracy of the approach for GSA settings (S4 Fig and S2 Table).

Simulations for the step-down procedure (Fig 3) showed that the proposed method was able to recognize pathways containing signal in only one gene, regardless of the choice of GSA test statistic. When only one gene contained causal SNPs (Fig 3A), the power for all tests remained close to zero regardless of effect size, as the causal SNPs were removed before pathway level inference. GBJ again showed good power when multiple causal SNPs were located in each gene, and GSEA performed well with only one causal SNP per gene (Fig 3D). As the method demonstrated excellent segregation of single-gene effects, we suggest the step-down procedure as an important complementary tool in GSA.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Step-down inference power simulation over four different configurations of gene signal density.

Simulated power using step-down inference procedure with MAGMA, GSEA, GHC, and GBJ (all self-contained versions) with random sets of ten genes selected from 10,000 total genes. From the ten genes in the set, a genes are selected to hold b causal SNPs each. The four subfigures correspond to (A) a = 1, b = 4, (B) a = 4, b = 1, (C) a = 4, b = 2, (D) a = 8, b = 1. The effect size is given on the x-axis. For each method and each iteration, we first determine a most significant gene. Then that gene is removed and we perform inference on the remaining nine genes in the set. We perform 100 simulations at each parameter setting and test at α = 0.01.

https://doi.org/10.1371/journal.pgen.1007530.g003

Re-analysis of breast cancer GWAS

We next investigated whether the same trends seen in our simulation could be found in re-analysis of the breast cancer summary statistic dataset. Self-contained GSEA was originally used to analyze a total of 4,507 pathways containing more than ten genes, finding 448 to be significant when using permutation to control the false discovery rate (FDR) at q = 0.05. GBJ was applied to 10,742 pathways (with more than three genes) from the same master list and found 2,703 to be significant at the Bonferroni-corrected family wise error rate of 4.65 ⋅ 10⁻⁶. When we restricted comparisons to the 3,952 pathways tested by both approaches, GSEA found 352 significant while GBJ found 2,095 significant at their respective error rates.

From the raw significance numbers alone, GBJ appeared to offer far more power in a real GWAS summary statistic dataset. GBJ found more than twice as many significant pathways as GSEA on a percentage basis, even when controlling a more stringent error rate and using a conservative Bonferroni correction. The increased power could not be attributed to smaller pathways alone, as GBJ declared a higher percentage of pathways significant across various pathway sizes (S5 Fig). To further investigate, we compared the GSEA and GBJ significance ranking (Fig 4) of the 3,952 pathways tested by both methods (p-value or q-value rank out of 3,952, lower is more significant). To emphasize the role of moderately strong associations, points were colored according to their density of suggestive signals, which we defined as a pathway’s proportion of SNPs with p < 10⁻⁵. SNPs demonstrating such a level of association would generally not stand out as the strongest signal in their region, but a large proportion of suggestive signals in any single gene or pathway could still indicate biologically relevant gene sets. For the sake of presentation, we only plotted pathways ranked in the top ten percentile by GSEA, in the top ten percentile by GBJ, in the 30^th-40^th percentile by GBJ, and in the 60^th-70^th percentile by GBJ (see S6 Fig for full data).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Significance rank of breast cancer pathways tested by both GBJ and GSEA.

The significance ranking of pathways according to GBJ and GSEA, colored according to density of SNPs with p < 10⁻⁵. Pathways are ordered according to p-value (GBJ) or q-value (GSEA), smaller rankings indicate more significance. For the sake of presentation, we only show pathways in the top ten percentiles of either GSEA or GBJ as well as pathways in the 30^th-40^th and 60^th-70^th percentiles of GBJ. Pathways ranked by GBJ as more significant generally have a higher proportion of SNPs with p < 10⁻⁵. In contrast, there is no discernible relationship between a pathway’s GSEA ranking and its density of suggestive signals.

https://doi.org/10.1371/journal.pgen.1007530.g004

The frequency of blue pathways—indicating higher density of suggestive signals—clearly increased for pathways that were ranked as very significant according to GBJ. Such a pattern was desirable, as pathways with a higher density of small p-values should be more significant. However, scanning horizontally across the plot, there did not appear to be a strong relationship between GSEA rank and frequency of blue pathways. Approximately the same number of blue pathways could be found near the GSEA 25^th, 50^th, and 75^th percentiles. This pattern suggested that GSEA was not very sensitive to the density of medium-strength signals.

The proportion of SNPs with p < 1 ⋅ 10⁻⁵ is not a perfect measure for evaluating the significance of pathways, as other factors including linkage disequilibrium and strength of the largest signal do also play an important role. However, while GBJ takes into account all of the above factors, GSEA ignores signal density and disregards all p-values that are not the smallest in a gene. This difference was likely a major contributing factor to the large discrepancy between GSEA and GBJ rankings for points in the top left and bottom right hand corners of Fig 4. Other methods employing the same strategy of choosing a minimum p-value to represent each gene, for example, non-default versions of MAGMA, may possibly experience similar drawbacks.

Single gene vs. set-based effects

Application of Generalized Berk-Jones to height and schizophrenia resulted in even more significant pathway findings than the breast cancer analysis (S3 Table). To uncover the gene sets that were driven to significance by only one or a few genes, we applied the GBJ step-down inference procedure (see Materials and Methods) for all pathways in each phenotype possessing an initial p-value of p < 1 ⋅ 10⁻¹² (Fig 5). In a typical GSA, these pathways would likely receive the most attention for their extremely high levels of association.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. P-values of initially top-ranked pathways after removal of significant genes.

(A) P-value after removal of most significant gene for pathways demonstrating p < 10⁻¹² across each of three phenotypes. We only show the original top 500 pathways in each phenotype for sake of presentation. (B) Percentage of pathways with original significance p < 10⁻¹² passing Bonferroni-corrected significance threshold after top significant gene is removed, stratified by quantile of gene set size. (C) P-value after removal of three most significant genes for top 500 pathways across each of three phenotypes. (D) P-value of Gene Ontology Ear Morphogenesis and Nature Pathway Interaction Database TRAIL Signaling pathways as most significant genes are removed one by one (for association with breast cancer only). P-values less than 10⁻¹² are truncated at this value. It appears that many of the most significant height pathways are driven to significance by multiple highly associated genes, while the opposite is true for breast cancer.

https://doi.org/10.1371/journal.pgen.1007530.g005

Many pathways from both breast cancer and schizophrenia dropped below the Bonferroni-corrected significance level after their most significant gene was removed, but breast cancer pathways appeared to show more drastic changes in p-value (Fig 5A), although some remained highly significant (S4 and S5 Tables and S7 Fig). Thus a single gene boosted evidence of association by many orders of magnitude for a large number of breast cancer pathways. In contrast, pathways associated with height generally remained significant even after removal of their most highly associated gene (Fig 5B). This trend persisted when removing the top three most highly associated genes from each pathway (Fig 5C). Only about 12% of breast cancer pathways survived the Bonferroni-corrected significance level after three genes were removed, while approximately 38% of schizophrenia pathways and 69% of height pathways still passed this threshold. A possible interpretation of these results could be that height was much more driven by pathway-level effects of many genes working together, while breast cancer risk factors were more localized to a few key genes. It is also possible that breast cancer signal was attenuated because the summary statistics included patients with multiple different subtypes, so signal may have been diluted compared to an ER-positive only or ER-negative only analysis.

Ear Morphogenesis was one example of a breast cancer pathway where the signal was almost entirely confined to a single gene, with an initial GBJ ranking of 100th most significant gene set (p < 1 ⋅ 10⁻¹²) before the step-down procedure. When FGFR2 was removed from this pathway, the p-value of the modified gene set increased drastically to p = 0.041, far from the corrected significance level. As expected, the other genes were not very relevant to breast cancer; it is not recommended to further pursue replication of the set’s association, despite initially promising results from GBJ and GSEA. On the other hand, a gene set such as the Nature Pathway Interaction Database TRAIL Signaling pathway demonstrated more robustness to removal of its top gene, MAP3K1. TRAIL Signaling retained some set-level signal even as more and more top genes were removed from the gene set (Fig 5D). Along with MAP3K1, the pathway contained the significant genes CASP8, CFLAR, DAP3, and TNFSF10. All of these genes possessed single gene p-values less than 1 ⋅ 10⁻⁵ for association with breast cancer, while Ear Morphogenesis contained no such genes other than FGFR2. TRAIL Signaling as a mechanism has been studied extensively for its role in breast cancer [55], supporting our finding of a pathway-wide effect that extends past the most significant genes.

Over the entire pathway database, 172 pathways containing FGFR2 were tested for association with breast cancer, and 172 were significant at the Bonferroni-corrected threshold according to GBJ. Additionally, FGFR2 was the most significant gene in 169 of these pathways. After removal of FGFR2 from the pathway, only 71 of the 169 were still significant at the same threshold (S6 Table). Clearly, the composition of significant gene sets in any breast cancer pathway analysis will depend on the number of times FGFR2 and select other genes (S7 and S8 Tables) appear in the pathway definition database.

Significant biological systems

To summarize the results of our GBJ-based pathway analysis across three different phenotypes, we identified the biological processes where significant pathways in breast cancer, schizophrenia, and height were most likely to congregate (see Materials and methods). In two reassuring results, we found that the percentages of significant pathways from the Growth category were higher than expected in breast cancer and height (Fig 6). Growth mechanisms have previously been found to play important roles in studies of breast cancer [30] and height [3]. Another theme that has often been corroborated in the literature is the importance of the immune system in schizophrenia [44]. Immune-related pathways have been studied in connection with many psychiatric diseases, and our analysis underscored the reasons for such an approach, as we found a high density of significant schizophrenia gene sets arising from immune processes. Seeing that GBJ could identify outcome-category pairs known to be associated with each other offered further validation that our approach was selecting relevant pathways.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Difference between observed and expected number of significant pathways arising from Gene Ontology Biological Process categories.

For a given phenotype, expected number is equal to the percentage of all tested pathways belonging to a category multiplied by the total number of significant pathways. The difference between observed and expected counts is expressed as a percentage of the expected number. A value greater than 0 indicates there are more significant pathways in a category than expected. A value less than 0 indicates there are less significant pathways than expected. A number of familiar themes are present, including the high number of significant height pathways related to growth and the high number of significant schizophrenia pathways related to the immune system.

https://doi.org/10.1371/journal.pgen.1007530.g006

On the other hand, GBJ also illuminated some outcome-category relationships that were not as widely familiar. For instance, we saw that there was a dearth of significant pathways related to schizophrenia in the Reproduction category. Thus there may be less benefit to searching for common drivers of risk between schizophrenia and breast cancer, which showed many more significant Reproduction pathways than expected. Similarly, all three phenotypes showed fewer than expected significant pathways in Locomotion, indicating that it may be more useful to prioritize other types of pathways when studying these outcomes. While negative findings are reported less often than their positive counterparts, these results still have the potential to inform researchers of the mechanisms that may not generate as many fruitful results.