Nonparametric estimation of Fisher information from real data (original) (raw)

h i g h l i g h t s • Two estimation methods (discretization and kernel-based approach) are applied to FIM and SE. • FIM (SE) estimated by discrete approach is nearly constant with σ. • FIM (SE) estimated by discrete approach decreases (increases) with the bin number. • FIM (SE) estimated by kernel-based approach is close to the theory value for any σ. a b s t r a c t The performance of two estimators of Fisher Information Measure (FIM) and Shannon entropy (SE), one based on the discretization of the FIM and SE formulae (discrete-based approach) and the other based on the kernel-based estimation of the probability density function (pdf) (kernel-based approach) is investigated. The two approaches are employed to estimate the FIM and SE of Gaussian processes (with different values of σ and size N), whose theoretic FIM and SE depend on the standard deviation σ. The FIM (SE) estimated by using the discrete-based approach is approximately constant with σ , but decreases (increases) with the bin number L; in particular, the discrete-based approach furnishes a rather correct estimation of FIM (SE) for L ∝ σ. Furthermore, for small values of σ , the larger the size N of the series, the smaller the mean relative error; while for large values of σ , the larger the size N of the series, the larger the mean relative error. The FIM (SE) estimated by using the kernel-based approach is very close to the theoretic value for any σ , and the mean relative error decreases with the increase of the length of the series. Comparing the results obtained using the discrete-based and kernel-based approaches, the estimates of FIM and SE by using the kernel-based approach are much closer to the theoretic values for any σ and any N and have to be preferred to the discrete-based estimates.

We study phase transitions and relevant order parameters via statistical estimation theory using the Fisher information matrix. The assumptions that we make limit our analysis to order parameters representable as a negative derivative of thermodynamic potential over some thermodynamic variable. Nevertheless, the resulting representation is sufficiently general and explicitly relates elements of the Fisher information matrix to the rate of change in the corresponding order parameters. The obtained relationships allow us to identify, in particular, second-order phase transitions via divergences of individual elements of the Fisher information matrix. A computational study of random Boolean networks (RBNs) supports the derived relationships, illustrating that Fisher information of the magnetization bias (that is, activity level) is peaked in finite-size networks at the critical points, and the maxima increase with the network size. The framework presented here reveals the basic thermodynamic reasons behind similar empirical observations reported previously. The study highlights the generality of Fisher information as a measure that can be applied to a broad range of systems, particularly those where the determination of order parameters is cumbersome.

In this paper, we consider parametric density estimation based on minimizing the Havrda-Charvat-Tsallis nonextensive entropy. The resulting estimator, called the Maximum Lq-Likelihood estimator (MLqE), is indexed by a single distortion parameter q, which controls the trade-off between bias and variance. The method has two notable special cases. If q tends to 1, the MLqE is the Maximum Likelihood Estimator (MLE). When q = 1=2, the MLqE is a minimum Hellinger distance type of estimator with the perk of avoiding nonparametric techniques and the difficulties of bandwith selection. The MLqE is studied using asymptotic analysis, simulations and real-world data, showing that it conciliates two apparently contrasting needs: effciency and robustness, conditional to a proper choice of q. When the sample size is small or moderate, the MLqE trades bias for variance, resulting in a reduced mean squared error compared to the MLE. At the same time, the MLqE exhibits strong robustness at expense of...