Activation Function Modulation in Generative Triangular Recurrent Neural Networks (original) (raw)
Related papers
Marginally Stable Triangular Recurrent Neural Network Architecture for Time Series Prediction
IEEE Transactions on Cybernetics
This paper introduces a discrete-time recurrent neural network architecture using triangular feedback weight matrices that allows a simplified approach to ensuring network and training stability. The triangular structure of the weight matrices is exploited to readily ensure that the eigenvalues of the feedback weight matrix represented by the block diagonal elements lie on the unit circle in the complex z-plane by updating these weights based on the differential of the angular error variable. Such placement of the eigenvalues together with the extended close interaction between state variables facilitated by the nondiagonal triangular elements, enhances the learning ability of the proposed architecture. Simulation results show that the proposed architecture is highly effective in time-series prediction tasks associated with nonlinear and chaotic dynamic systems with underlying oscillatory modes. This modular architecture with dual upper and lower triangular feedback weight matrices mimics fully recurrent network architectures, while maintaining learning stability with a simplified training process. While training, the block-diagonal weights (hence the eigenvalues) of the dual triangular matrices are constrained to the same values during weight updates aimed at minimizing the possibility of overfitting. The dual triangular architecture also exploits the benefit of parsing the input and selectively applying the parsed inputs to the two subnetworks to facilitate enhanced learning performance.
A neural network scheme for long-term forecasting of chaotic time series
Neural Processing …, 2011
The accuracy of a model to forecast a time series diminishes as the prediction horizon increases, in particular when the prediction is carried out recursively. Such decay is faster when the model is built using data generated by highly dynamic or chaotic systems. This paper presents a topology and training scheme for a novel artificial neural network, named "Hybrid-connected Complex Neural Network" (HCNN), which is able to capture the dynamics embedded in chaotic time series and to predict long horizons of such series. HCNN is composed of small recurrent neural networks, inserted in a structure made of feed-forward and recurrent connections and trained in several stages using the algorithm back-propagation through time (BPTT). In experiments using a Mackey-Glass time series and an electrocardiogram (ECG) as training signals, HCNN was able to output stable chaotic signals, oscillating for periods as long as four times the size of the training signals. The largest local Lyapunov Exponent (LE) of predicted signals was positive (an evidence of chaos), and similar to the LE calculated over the training signals. The magnitudes of peaks in the ECG signal were not accurately predicted, but the predicted signal was similar to the ECG in the rest of its structure.
Generation of unpredictable time series by a neural network
Physical review. E, Statistical, nonlinear, and soft matter physics, 2001
A perceptron that "learns" the opposite of its own output is used to generate a time series. We analyze properties of the weight vector and the generated sequence, such as the cycle length and the probability distribution of generated sequences. A remarkable suppression of the autocorrelation function is explained, and connections to the Bernasconi model are discussed. If a continuous transfer function is used, the system displays chaotic and intermittent behavior, with the product of the learning rate and amplification as a control parameter.
Prediction of chaotic time series with NARX recurrent dynamic neural networks
Proceedings of the 9th WSEAS International …, 2008
The problem of chaotic time series prediction is studied in various disciplines now including engineering, medical and econometric applications. Chaotic time series are the output of a deterministic system with positive Liapunov exponent. A time series prediction is a suitable application for a neuronal network predictor. The NN approach to time series prediction is non-parametric, in the sense that it is not necessary to know any information regarding the process that generates the signal. It is shown that the recurrent NN (RNN) with a sufficiently large number of neurons is a realization of the nonlinear ARMA (NARMA) process. In this paper we present the nonlinear autoregressive network with exogenous inputs (NARX), the architecture, the training method, the input data to network, the simulation results.
Echo State Networks (ESN) present a novel approach to analysing and training recurrent neural networks (RNNs). It leads to a fast, simple and constructive algorithm for supervised training of RNNs. A very powerful blackbox modeling tool to build models to simulate, predict, filter, classify, or control nonlinear dynamical systems, what makes ESNs excel over traditional techniques is that it can efficiently encode and retain massive information in an ESN "echo" network state about a long previous history. This makes ESNs excellent approximators of, among other nonlinear dynamical systems, chaotic time series -with obvious applications in prediction of such tasks as currency exchange rates. This research proposal aims to further improve upon the best empirical result of predicting a chaotic time series, which was obtained by an ESN, by replacing the linear readout mechanism employed by ESNs with a multi layer perceptron (MLP). This shall allow the resulting network to harness the dynamical memory of an ESN with the approximating powers of the gradient descent algorithm of a MLP, resulting in a powerful approximator, smaller in size than using just an ESN and allowing for more feasible practical implementations in telecommunications.
A Study of Recurrent neural networks (RNNs) in univariate and multivariate time series
Recurrent neural networks (RNNs) are powerful sequence models. Using their internal memory they are able to model temporal dependencies of unspeci ed duration between the inputs and the desired outputs. This property makes them good candidates for time series modeling. The aim of this thesis is to apply RNNs to univariate and multivariate time series. Initially, we examine if the order of the time series should be taken into account in the structure of the network for one step ahead time series prediction. Then, we explore the use of RNNs in the setting of multivariate time series, proposing also a new way of training, called Multiple training. Finally, we present a new approach to recover the Granger-causal relations in multivariate time series using RNNs with Group lasso. Experimental results are presented for generated linear and non-linear time series, demonstrating the performance of recurrent networks on time series prediction and Granger-causality structure recovery.
Novel FTLRNN with Gamma Memory for Short-Term and Long-Term Predictions of Chaotic Time Series
Applied Computational Intelligence and Soft Computing, 2009
Recommended by Junbin Gao Multistep ahead prediction of a chaotic time series is a difficult task that has attracted increasing interest in the recent years. The interest in this work is the development of nonlinear neural network models for the purpose of building multistep chaotic time series prediction. In the literature there is a wide range of different approaches but their success depends on the predicting performance of the individual methods. Also the most popular neural models are based on the statistical and traditional feed forward neural networks. But it is seen that this kind of neural model may present some disadvantages when long-term prediction is required. In this paper focused time-lagged recurrent neural network (FTLRNN) model with gamma memory is developed for different prediction horizons. It is observed that this predictor performs remarkably well for short-term predictions as well as medium-term predictions. For coupled partial differential equations generated chaotic time series such as Mackey Glass and Duffing, FTLRNN-based predictor performs consistently well for different depths of predictions ranging from short term to long term, with only slight deterioration after k is increased beyond 50. For real-world highly complex and nonstationary time series like Sunspots and Laser, though the proposed predictor does perform reasonably for short term and medium-term predictions, its prediction ability drops for long term ahead prediction. However, still this is the best possible prediction results considering the facts that these are nonstationary time series. As a matter of fact, no other NN configuration can match the performance of FTLRNN model. The authors experimented the performance of this FTLRNN model on predicting the dynamic behavior of typical Chaotic Mackey-Glass time series, Duffing time series, and two real-time chaotic time series such as monthly sunspots and laser. Static multi layer perceptron (MLP) model is also attempted and compared against the proposed model on the performance measures like mean squared error (MSE), Normalized mean squared error (NMSE), and Correlation Coefficient (r). The standard back-propagation algorithm with momentum term has been used for both the models.
The Power of Linear Recurrent Neural Networks
2018
Recurrent neural networks are a powerful means to cope with time series. We show how a type of linearly activated recurrent neural networks, which we call predictive neural networks, can approximate any time-dependent function f(t) given by a number of function values. The approximation can effectively be learned by simply solving a linear equation system; no backpropagation or similar methods are needed. Furthermore, the network size can be reduced by taking only most relevant components. Thus, in contrast to others, our approach not only learns network weights but also the network architecture. The networks have interesting properties: They end up in ellipse trajectories in the long run and allow the prediction of further values and compact representations of functions. We demonstrate this by several experiments, among them multiple superimposed oscillators (MSO), robotic soccer, and predicting stock prices. Predictive neural networks outperform the previous state-of-the-art for t...
Chaotic Time Series Prediction with Neural Networks - Comparison of Several Architectures
This paper presents experimental comparison between selected neural architectures for chaotic time series prediction problem. Several feed-forward architectures (Multilayer Perceptrons) are compared with partially recurrent nets (Elman, extended Elman, and Jordan) based on convergence rate, prediction accuracy, training time requirements and stability of results.
The use of NARX neural networks to predict chaotic time series
WSEAS Transactions on Computer Research, 2008
The prediction of chaotic time series with neural networks is a traditional practical problem of dynamic systems. This paper is not intended for proposing a new model or a new methodology, but to study carefully and thoroughly several aspects of a model on which there are no enough communicated experimental data, as well as to derive conclusions that would be of interest. The recurrent neural networks (RNN) models are not only important for the forecasting of time series but also generally for the control of the dynamical system. A RNN with a sufficiently large number of neurons is a nonlinear autoregressive and moving average (NARMA) model, with "moving average" referring to the inputs. The prediction can be assimilated to identification of dynamic process. An architectural approach of RNN with embedded memory, "Nonlinear Autoregressive model process with eXogenous input" (NARX), showing promising qualities for dynamic system applications, is analyzed in this paper. The performances of the NARX model are verified for several types of chaotic or fractal time series applied as input for neural network, in relation with the number of neurons, the training algorithms and the dimensions of his embedded memory. In addition, this work has attempted to identify a way to use the classic statistical methodologies (R/S Rescaled Range analysis and Hurst exponent) to obtain new methods of improving the process efficiency of the prediction chaotic time series with NARX.