Overcoming the effects of false positives and threshold bias in graph theoretical analyses of neuroimaging data - PubMed (original) (raw)

Overcoming the effects of false positives and threshold bias in graph theoretical analyses of neuroimaging data

M Drakesmith et al. Neuroimage. 2015 Sep.

Abstract

Graph theory (GT) is a powerful framework for quantifying topological features of neuroimaging-derived functional and structural networks. However, false positive (FP) connections arise frequently and influence the inferred topology of networks. Thresholding is often used to overcome this problem, but an appropriate threshold often relies on a priori assumptions, which will alter inferred network topologies. Four common network metrics (global efficiency, mean clustering coefficient, mean betweenness and smallworldness) were tested using a model tractography dataset. It was found that all four network metrics were significantly affected even by just one FP. Results also show that thresholding effectively dampens the impact of FPs, but at the expense of adding significant bias to network metrics. In a larger number (n=248) of tractography datasets, statistics were computed across random group permutations for a range of thresholds, revealing that statistics for network metrics varied significantly more than for non-network metrics (i.e., number of streamlines and number of edges). Varying degrees of network atrophy were introduced artificially to half the datasets, to test sensitivity to genuine group differences. For some network metrics, this atrophy was detected as significant (p<0.05, determined using permutation testing) only across a limited range of thresholds. We propose a multi-threshold permutation correction (MTPC) method, based on the cluster-enhanced permutation correction approach, to identify sustained significant effects across clusters of thresholds. This approach minimises requirements to determine a single threshold a priori. We demonstrate improved sensitivity of MTPC-corrected metrics to genuine group effects compared to an existing approach and demonstrate the use of MTPC on a previously published network analysis of tractography data derived from a clinical population. In conclusion, we show that there are large biases and instability induced by thresholding, making statistical comparisons of network metrics difficult. However, by testing for effects across multiple thresholds using MTPC, true group differences can be robustly identified.

Copyright © 2015. Published by Elsevier Inc.

PubMed Disclaimer

Figures

Fig. 15

Fig. 15

A toy illustration of the fluctuations of variance. Two groups of networks (red and blue) with different topologies are shown around a core transition point, _φ_pC with subject-specific and group specific offsets Δ_φ_pE and Δ_φ_pH, respectively. Bold lines indicate the variance of f(τ) for each group. The variance of f(τ) is inflated around _φ_pC leading to low SH. As τ moves away from _φ_pC, variance reduces and SH is larger, allowing the group differences to be identified.

Fig. 1

Fig. 1

Flowchart of derivation of connectivity matrices from tractography data.

Fig. 2

Fig. 2

Selected bundles for construction of ‘ground truth’ model network. Arcuate fasiculus (AF) superior longitudinal fasiculus (SLF) inferior longitudinal fasciculus (ILF) uncinate fasciculus (UF) cingulum (Cing) temproparietal pathway (TPP) corpus calosum (CC) short-range U-fibres.

Fig. 3

Fig. 3

Visualisation of networks used in simulations. (a) is the original tractography dataset without any additional processing; (b) is the model network derived using manual segmentation of cortico-cortical pathways; (c) and (d) are the sets of FP-NEs and FP-EEs to be sampled, respectively. These were derived from the streamlines present in the original tractography dataset but not in the ground truth network.

Fig. 4

Fig. 4

Impact of FP-NEs and FP-EEs on network metrics, both measured through proportional change and standard z-scores. Error bars on proportional error plots indicate standard error across iterations. Dotted lines on z-score plots indicate critical values of z for p < 0.01.

Fig. 5

Fig. 5

Impact of edge displacements on network metrics, both measured through proportional change and standard z-scores. Error bars on proportional error plots indicate standard error across iterations. Dotted lines on z-score plots indicate critical values of z for p < 0.01.

Fig. 6

Fig. 6

Effect of FPs when thresholds were applied. Thresholds are coded by colour with black indicating results for the unthresholded network. (a) Proportional change relative to the original unthresholded ground truth. (b) Proportional change relative to the thresholded ground truth network.

Fig. 7

Fig. 7

Overall effect of thresholding on network metrics, measured by proportional change. Error bars indicate standard error across iterations and FP counts.

Fig. 8

Fig. 8

Results of experiment 2a. Stability measures (see main text) of _U_-tests performed on network and non-network control metrics across thresholds. Error bars show standard error across 100 permutation.

Fig. 9

Fig. 9

_U_-statistics obtained from comparing network and non-network metrics from healthy and atrophied groups. Each level of atrophy (ξ) is coded by colour. U-statistics for group permutations are shown in grey.

Fig. 10

Fig. 10

_U_-statistics obtained from comparing network metrics from healthy and atrophied groups with 10 reassignments of group prior to inducing atrophy (ξ = 0.25).

Fig. 11

Fig. 11

Flowchart of the MTPC pipeline, demonstrated on comparisons of clustering coefficient between the healthy and atrophied groups in experiment 2b (ξ = 0.2). Steps of the pipeline are explained in the text. (1) Network metrics are computed for each subject and threshold, up to m = 30 streamlines. (2) _U_-Statistics were computed between groups for each threshold. The initial uncorrected unthresholded test yields a non-significant result. (3) Groups were permuted n = 1000 times and -statistics computed for each permutation. (4) The maximum statistic across thresholds was taken for each permutation, generating null distribution of _U_-statistics (note we use the term maximum with respect to the direction of the observed effect). (5) The upper 95th percentile of this distribution is taken as the critical value for U which is _U_crit = 1354. (6). One super-critical cluster was identified with a peak of _U_MTPC = 1267 at τ = 4 with a super-critical AUC of A_MTP_C = 2488. (7). The mean super-critical AUC was _A_crit = 306.9. (8) Both _U_MTPC and _A_MTPC exceed their respective critical values. Therefore, the null hypothesis is rejected.

Fig. 12

Fig. 12

Results of the AUC and MTPC methods comparing network metrics from healthy and atrophied groups from experiment 2b across levels of atrophy (ξ). Dotted lines indicated the respective critical values. (a) _U_-statistics (_U_AUC) obtained from comparisons of AUC of network metrics computed at subject-level. (b). Maximum _U_-statistic (_U_MTPC) computed using the MTPC method (c) Super-critical AUCs of _U_-statistics (_A_MTPC) using the MTPC method. (d). The minimum detectable ξ for each method, obtained from the intercept of the relevant statistic with its respective critical value.

Fig. 13

Fig. 13

MTPC applied to the statistical tests using the data of (Caeyenberghs et al., 2014).

Fig. 14

Fig. 14

A toy illustration of a scenario where the AUC method will fail to identify a difference between graph metrics with very different profiles along τ. The two networks (red and blue) show distinct behaviour across τ, yet computing the AUC will reveal no differences between the two networks.

References

    1. Achard S., Bullmore E. Efficiency and cost of economical brain functional networks. PLoS Comput. Biol. 2007;3:e17. -PMC -PubMed
    1. Alexander D.C. An introduction to computational diffusion MRI: the diffusion tensor and beyond. In: Weichert J., Hagen H., editors. Visualization and Processing of Tensor Fields. Springer; Berlin / Heidelberg: 2006. pp. 83–106.
    1. Andreotti J., Jann K., Melie-Garcia L., Giezendanner S., Dierks T., Federspiel A. Repeatability analysis of global and local metrics of brain structural networks. Brain Connect. 2014;4:203–220. -PubMed
    1. Baccalá L.A., Sameshima K. Partial directed coherence: a new concept in neural structure determination. Biol. Cybern. 2001;84:463–474. -PubMed
    1. Bartolomei F., Bosma I., Klein M., Baayen J.C., Reijneveld J.C., Postma T.J., Heimans J.J., van Dijk B.W., de Munck J.C., de Jongh A., Cover K.S., Stam C.J. Disturbed functional connectivity in brain tumour patients: evaluation by graph analysis of synchronization matrices. Clin. Neurophysiol. 2006;117:2039–2049. -PubMed

Publication types

MeSH terms

Grants and funding

LinkOut - more resources