Neural Networks for Conditional Probability Estimation: Forecasting Beyond Point Predictions

'1. Introduction.- 1.1 Conventional forecasting and Takens' embedding theorem.- 1.2 Implications of observational noise.- 1.3 Implications of dynamic noise.- 1.4 Example.- 1.5 Conclusion.- 1.6 Objective of this book.- 2. A Universal Approximator Network for Predicting Conditional Probability Densities.- 2.1 Introduction.- 2.2 A single-hidden-layer network.- 2.3 An additional hidden layer.- 2.4 Regaining the conditional probability density.- 2.5 Moments of the conditional probability density.- 2.6 Interpretation of the network parameters.- 2.7 Gaussian mixture model.- 2.8 Derivative-of-sigmoid versus Gaussian mixture model.- 2.9 Comparison with other approaches.- 2.9.1 Predicting local error bars.- 2.9.2 Indirect method.- 2.9.3 Complete kernel expansion: Conditional Density Estimation Network (CDEN) and Mixture Density Network (MDN).- 2.9.4 Distorted Probability Mixture Network (DPMN).- 2.9.5 Mixture of Experts (ME) and Hierarchical Mixture of Experts (HME).- 2.9.6 Soft histogram.- 2.10 Summary.- 2.11 Appendix: The moment generating function for the DSM network.- 3. A Maximum Likelihood Training Scheme.- 3.1 The cost function.- 3.2 A gradient-descent training scheme.- 3.2.1 Output weights.- 3.2.2 Kernel widths.- 3.2.3 Remaining weights.- 3.2.4 Interpretation of the parameter adaptation rules.- 3.2.5 Deficiencies of gradient descent and their remedy.- 3.3 Summary.- 3.4 Appendix.- 4. Benchmark Problems.- 4.1 Logistic map with intrinsic noise.- 4.2 Stochastic combination of two stochastic dynamical systems.- 4.3 Brownian motion in a double-well potential.- 4.4 Summary.- 5. Demonstration of the Model Performance on the Benchmark Problems.- 5.1 Introduction.- 5.2 Logistic map with intrinsic noise.- 5.2.1 Method.- 5.2.2 Results.- 5.3 Stochastic coupling between two stochastic dynamical systems.- 5.3.1 Method.- 5.3.2 Results.- 5.3.3 Auto-pruning.- 5.4 Brownian motion in a double-well potential.- 5.4.1 Method.- 5.4.2 Results.- 5.4.3 Comparison with other approaches.- 5.5 Conclusions.- 5.6 Discussion.- 6. Random Vector Functional Link (RVFL) Networks.- 6.1 The RVFL theorem.- 6.2 Proof of the RVFL theorem.- 6.3 Comparison with the multilayer perceptron.- 6.4 A simple illustration.- 6.5 Summary.- 7. Improved Training Scheme Combining the Expectation Maximisation (EM) Algorithm with the RVFL Approach.- 7.1 Review of the Expectation Maximisation (EM) algorithm.- 7.2 Simulation: Application of the GM network trained with the EM algorithm.- 7.2.1 Method.- 7.2.2 Results.- 7.2.3 Discussion.- 7.3 Combining EM and RVFL.- 7.4 Preventing numerical instability.- 7.5 Regularisation.- 7.6 Summary.- 7.7 Appendix.- 8. Empirical Demonstration: Combining EM and RVFL.- 8.1 Method.- 8.2 Application of the GM-RVFL network to predicting the stochastic logistic-kappa map.- 8.2.1 Training a single model.- 8.2.2 Training an ensemble of models.- 8.3 Application of the GM-RVFL network to the double-well problem.- 8.3.1 Committee selection.- 8.3.2 Prediction.- 8.3.3 Comparison with other approaches.- 8.4 Discussion.- 9. A simple Bayesian regularisation scheme.- 9.1 A Bayesian approach to regularisation.- 9.2 A simple example: repeated coin flips.- 9.3 A conjugate prior.- 9.4 EM algorithm with regularisation.- 9.5 The posterior mode.- 9.6 Discussion.- 10. The Bayesian Evidence Scheme for Regularisation.- 10.1 Introduction.- 10.2 A simple illustration of the evidence idea.- 10.3 Overview of the evidence scheme.- 10.3.1 First step: Gaussian approximation to the probability in parameter space.- 10.3.2 Second step: Optimising the hyperparameters.- 10.3.3 A self-consistent iteration scheme.- 10.4 Implementation of the evidence scheme.- 10.4.1 First step: Gaussian approximation to the probability in parameter space.- 10.4.2 Second step: Optimising the hyperparameters.- 10.4.3 Algorithm.- 10.5 Discussion.- 10.5.1 Improvement over the maximum likelihood estimate.- 10.5.2 Justi