Sparsity Research Papers - Academia.edu (original) (raw)

In high-dimensional linear models, the sparsity assumption is typically made, stating that most of the model parameters have value equal to zero. Under the sparsity assumption, estimation and, recently, inference as well as the fundamental limits of detection have been well studied. However, in certain cases, sparsity assumption may be violated, and a large number of covariates can be expected to be associated with the response, indicating that possibly all, rather just a few, model parameters are different from zero. A natural example is a genome-wide gene expression profiling, where all genes are believed to affect a common disease marker. We show that the current inferential methods are sensitive to the sparsity assumption, and may in turn result in severe bias: lack of control of Type-I error is apparent once the model is not sparse. In this article, we propose a new inferential method, named CorrT, which is robust and adaptive to the sparsity assumption. CorrT is shown to have Type I error approaching the nominal level, regardless of how sparse or dense the model is. Specifically, the developed test is based on a moment condition induced by the hypothesis and the covariate structure of the model design. Such a construction circumvents the fundamental difficulty of accurately estimating non-sparse high-dimensional models. As a result, the proposed test guards against large estimation errors caused by potential absence of sparsity, and at the same time, adapts to the model sparsity. In fact, CorrT is also shown to be optimal whenever sparsity holds. Numerical experiments show favorable performance of CorrT compared to existing methods. We also apply CorrT to a real dataset and confirm some known discoveries related to HER2+ cancer patients and the gene-to-gene interaction.