Oil Price Prediction using Artificial Neural Networks
Author: Siddhant Jain, 2010B3A7506P Birla Institute of Technology and Science, Pilani

Abstract: Oil is an important commodity for every industrialised nation in the modern economy. The upward or downward trends in Oil prices have crucially influenced economies over the years and a priori knowledge of such a trend would be deemed useful to all concernd - be it a firm or the whole country itself. Through this paper, I intend to use the power of Artificial Neural Networks (ANNs) to develop a model which can be used to predict oil prices. ANNs are widely used for modelling a multitude of financial and economic variables and have proven themselves to be a very powerful tool to handle volumes of data effectively and analysing it to perform meaningful calculations. MATLAB has been employed as the medium for developing the neural network and for efficiently handling the volume of calculations involved. Following sections shall deal with the theoretical and practical intricacies of the aforementioned model. The appendix includes snapshots of the generated results and other code snippets.

Artificial Neural Networks: Understanding To understand any of the ensuing topics and the details discussed thereof, it is imperative to understand what actually we mean by Neural Networks. So, I first dwell into this topic: In simplest terms a Neural Network can be defined as a computer system modelled on the human brain and nervous system. Wikipedia elaborates on this definition as follows: “ An Artificial Neural Network, often just called a neural network, is a mathematical model inspired by biological neural networks… …In most cases a neural network is an adaptive system that changes its structure during a learning phase.” Both of these definitions stand correct in their own place and the context. In my understanding the simplest of the neural networks boil down to being an alternative statistical approach for the least square regression problem. However, advanced neural networks have the powerful capabilities of identifying influential patterns in a given set of data which can’t be done easily given the human computational limitations. This paper is understandably concerned with the applications of neural networks in forecasting models and without deviating any more we dive into the topic from the next section.

Artificial Neural Networks: Developing a Financial Model Neural Networks have become increasingly popular in finance and economics and this sector has been the second largest sponsor of research in neural networks. The popularity can be attributed to the fact that neural networks can tolerate noise and chaotic components better than any other statistical technique used for forecasting. Other obvious advantage of neural networks is that they have a greater portability in the sense that they can be ‘trained’ to learn new patterns. That said, neural networks can be excessively expensive in terms of the time consumed for computations especially if the amount of data is high (which is usually the case). Moreover there are very few control parameters in a neural network and for a researcher there is a lot which depends on trial and error. Although rules of thumb do exist but more often than not, one needs to customise their neural network on the case to case basis, given the variable to be modelled along with the data used.

Artificial Neural Networks: Forecasting In case of time series forecasting, the most commonly used neural network is backpropogation. Backpropogation neural networks consist of a collection of inputs and processing units known as neurons. The neurons in each layer are fully connected through connection strengths called weights. ‘Weights’ essentially store the knowledge of a trained network. In addition to the processing neuron there is a biasing neuron connected to each processing unit in the hidden and output layers. The bias neuron has a value of positive one and serves a similar purpose as the intercept in regression models. There can be number of hidden layers (and effectively, hidden neurons) for each model depending on the size and nature of the data set. Of course there are input and output neurons. Input neurons are the (assumed to be) independent variables and output neurons represent our target variable for prediction. Methodology To develop the model I have adapted the model developed by Boyd and Kaastra in their paper regarding the topic1. The paradigm is shown as follows:

All theoretical and practical details of the term paper will now be explained using this model as the template.
1: Designing a Neural Network for forecasting financial and economic time series, Icebling Kaastra, Milton Boyd

Variable Selection Selecting the variable is very important for any neural network, since it is one of the few things that we can control directly in a neural network. It is of course crucial to have an economic understanding of which indicators are the most likely economic predictors.

For time series prediction, it is an accepted practice and of course logical to keep the input and the output variable to be the same. The neural network predicts future values based on the present values, that is time series prediction by definition. However, logic also dictates that there can be other factors that influence a given variable. In my adaptation, I have considered this extension and have used one exogenous variable as an input variable. The chosen exogenous variable is : Supply of Oil. It is a basic economic theory that the price of any commodity is decided by its supply and demand and so it’s choice. (I didn’t use demand as my input neuron, because given the nature of the commodity, the demand of oil won’t alter much with price in the short run. There is so much dependent on oil that there will always be enough takers of oil regardless of its price).

Data Collection It makes sense, especially now, to further detail the choice of the variables. To be precise, the model is used to predict the WTI Spot Index. WTI spot index is the oil price index for USA and is one of the three major indices used for quoting international oil prices. This being the case, I have considered the oil supply for U.S.A. The data for both the variables could be obtained from the website for the U.S energy information administration (www.eia.gov). Although a researcher might be tempted in taking daily data for oil prices considering the high size of the sample available for training but taking a data with this high frequency is not advised because for daily data, there are various inconsistencies and missing points for the market has been closed or halted due to weekend or unexpected events. So, I have used monthly data for my model. Again, to be precise:  My data ranges from Jan 1986 to August 2012 and uses the average price for all days in the given month as the monthly price. EIA – the agency which maintains the data is an American governmental agency and is reputed to be consistent and reliable. Both consistency and reliability of the data are important considerations that should be kept in mind by any researcher interested in designing a neural network model. Values for both the variables were stored in Excel sheets for future use by the MATLAB code.

Data Pre-processing Data pre-processing refers to analysing and transforming the input and output variables to minimise noise, highlight important relationships, detect trends, and flatten the distribution of the variable to assist the neural network in learning the relevant patterns. Naturally, neural network training can be made more efficient if you perform certain pre-processing steps on the network inputs and targets. Generally in neural networks sigmoid transfer functions are used in the hidden layers. These functions become essentially saturated when the net input is greater than three (exp (−3) ≅ 0.05). If this happens at the beginning of the training process, the gradients will be very small, and the network training will be very slow. In the first layer of the network, the net input is a product of the input times the weight plus the bias. If the input is very large, then the weight must be very small in order to prevent the transfer function from becoming saturated. It is standard practice to normalize the inputs before applying them to the network.

Generally, the normalization step is applied to both the input vectors and the target vectors in the data set. In this way, the network output always falls into a normalized range. The network output can then be reverse transformed back into the units of the original target data when the network is put to use in the field. Apart from this sampling and filtering of data is also done. Filtering essentially removes observations from the dataset so that all biases which do not really effect the target value are removed. For MATLAB, this step of data pre-processing is done by default when a neural network is created. One can of course define their own functions and methods for the pre-processing. The most commonly used pre-processing routines is normalization and removed of unknown values.

Training, Testing and Validation sets When training multilayer networks, the general practice is to first divide the data into the following three subsets: 1. Training set, which is used for computing the gradient and updating the network weights and biases. 2. Validation set, the error on the validation set is monitored during the training process. The validation error normally decreases during the initial phase of training, as does the training set error. However, when the network begins to overfit the data, the error on the validation set typically begins to rise. The network weights and biases are saved at the minimum of the validation set error. 3. Test Set, is the set on which the model is tested. Additionally, there are four different approaches in dividing the data sets in the aforementioned categories, which are: 1. 2. 3. 4. Divide the data randomly Divide the data into continuous blocks Divide the data using an index Divide the data using through and interleaved selection

For this paper I have divided the data randomly. The logic behind doing so is that in case of economic modelling, there can be data sets which have a continuous trend. Now if such a set is used for training it will give erroneous results for future sets. Therefore it becomes important to include data from everywhere so that the model is appropriately represented in the neural network. Another parameter to be decided is the proportion of data which for each set. In accordance to the theoretical studies I have used the following proportions: Training: 65%, Testing: 20%, Validation: 15%

Neural Network Architecture In this step we decide on different attributes pertaining to the neural network architecture. These attributes namely: the number of hidden layers, the number of hidden neurons, transfer functions, though once decided to train the neural network for the first time, can be (and in most cases it is) changed if the results of the training are not satisfactory. I will discuss the theoretical aspects of the attributes. Clubbed with is attribute is the corresponding value used in my model.

Number of Hidden Layers: The hidden layers give the network its ability to generalise. In theory, it has been proved that any neural network with one hidden layer with a sufficient number of hidden neurons can approximate any continuous function. Increasing the number of hidden layer will not only increasing the computational time but will also lead to over-fitting leading to poor performance in the forecasting. In most cases one hidden layer or at most two hidden layers are sufficient in predicting a variable. For this term paper, I use two hidden layers. Number of Hidden Neurons: There is no set formula to decide the number of hidden neurons. It is to an extent based on experimentation with the given data. However theoretically, a particular heuristic is famous and goes by the name of the pyramid rule. The pyramid rule states that if there are n input neurons and m output neuron, the total number of hidden neurons should be sqrt(mxn). Such a thumb rule provided a great starting point for deciding the number of neurons, but certainly shouldn’t be accepted as the only option. After going through similar models developed by other researchers, and based on my own experimentation, I conclude on using 8 hidden neurons for my model. Transfer Function: Transfer functions are mathematical functions that decide determine the output of a processing neuron. Most commonly used transfer function is the sigmoid function (and is also used by my code). The sigmoid function, often called as the S-shaped function can be logistic, arctangent, hyperbolic tangent and sometimes even algebraic in nature. An example of a sigmoid function and the corresponding plot are given below:

Following is a representation of the neural network MATLAB models generated:

The NARX model used for initial training

The NARX(Closed loop) model used for prediction

Evaluation (performance) Criteria: To test the validity of the model, we need an evaluation criteria which can decide wether the model is reasonable or now. For this project I used the sum of Mean Squared Error (MSE), for every value in data. A lower MSE value is always preferred over a higher one.

Training Training a neural network involved iteratively presenting the network with examples of the correct known answers. The objective of training is to find the set of weights between the neurons that determine the global minimum of the error function. The BP network uses the gradient descent algorithm with adjusts the weights to move down the steepest slope of the error surface. What is happening behind thee screens is that the computers for each successive iteration takes a randomised sets for training testing and validation and assigns weight for an optimum. The iterations continue till the MSE reaches its minimum value. After that a few more iterations are done to confirm that this is the global minima and the solution is not trapped into a local minimum. The weights for each iterations are stored and the weights for the iteration with minimum value of the error are used for the model. I plot the graph of error versus iteration for emphasizing on this point (see appendix). Algorithm used for training: In this section I would like to stress upon the intricacies of the neural network model used by me and algorithms used therein. I use the NARX network for my model. The nonlinear autoregressive network with exogenous inputs (NARX) is a recurrent dynamic network, with feedback connections enclosing several layers of the network. A model for the same is given in the MATLAB documentation and is given by:

Although it is not extremely necessary to go into the minute theoretical details of this algorithm, I discuss a few key points. For prediction using NARX algorithm, a series parallel system is followed in which the input for prediction is purely feedforward in nature, but backpropogation is followed for training. This produces more accurate results and it also takes lesser time. For determination of the weights the Levenberg-Marquardt algorithm is used. The Levenberg–Marquardt algorithm (LMA), also known as the damped least-squares (DLS) method, provides a numerical solution to the problem of minimizing a function, generally nonlinear, over a space of parameters of the function. These minimization problems arise especially in least squares curve fitting and nonlinear programming. Like other numeric minimization algorithms, the Levenberg–Marquardt algorithm is an iterative procedure. To start a minimization, the user has to provide an initial guess for the parameter vector, β. In cases with only one minimum, an uninformed standard guess like βT=(1,1,...,1) will work fine; in cases with multiple minima, the algorithm converges only if the initial guess is already somewhat close to the final solution. In each iteration step, the parameter vector, β, is replaced by a new estimate, β + δ. To determine δ, the functions are approximated by their linearizations

where

is the gradient (row-vector in this case) of f with respect to β. At the minimum of the sum of squares, zero. The above first-order approximation of , the gradient of gives with respect to δ will be

. Or in vector notation,
.

.

Taking the derivative with respect to δ and setting the result to zero gives:

where

is the Jacobian matrix whose ith row equals , and where and ,

and

are vectors with

ith component solved for δ.

respectively. This is a set of linear equations which can be

Levenberg's contribution is to replace this equation by a "damped version",

where I is the identity matrix, giving as the increment, δ, to the estimated parameter vector, β. The (non-negative) damping factor, λ, is adjusted at each iteration. If reduction of S is rapid, a smaller value can be used, bringing the algorithm closer to the Gauss–Newton algorithm, whereas if an iteration gives insufficient reduction in the residual, λ can be increased, giving a step closer to the gradient descent direction. Note that the gradient of S with respect to β equals . Therefore, for large values of λ, the step will be taken approximately in the direction of the gradient. If either the length of the calculated step, δ, or the reduction of sum of squares from the latest parameter vector, β + δ, fall below predefined limits, iteration stops and the last parameter vector, β, is considered to be the solution. Levenberg's algorithm has the disadvantage that if the value of damping factor, λ, is large, inverting JTJ + λI is not used at all. Marquardt provided the insight that we can scale each component of the gradient according to the curvature so that there is larger movement along the directions where the gradient is smaller. This avoids slow convergence in the direction of small gradient. Therefore, Marquardt replaced the identity matrix, I, with the diagonal matrix consisting of the diagonal elements of JTJ, resulting in the Levenberg–Marquardt algorithm:
..

All information regarding Levenberg–Marquardt algorithm is taken through Wikipedia.

Different selection of data sets, hidden neurons etc. will lead to different number of iterations required. For one such test, 14 iterations were required and the maximum occurred at the 8 th iteration. The graph for the same is given in the appendix.

Implementation The neural network model was then used to predict values and a plot was made of the predicted values against the given values. One of the most startling feature of this step was that no matter how many changes were made in the number of iteration, hidden neurons, hidden layers etc. it was almost next to impossible to predict the prices exactly or even close to that. A little research on this topic revealed that it is indeed difficult to derive the exact value of any output variable, specially when it’s an economic or financial variable, given the number of factors controlling them. Accepting this as a limitation, the neural network shouldn’t be completely abandoned, because even though it couldn’t predict the exact values, the trends in the WTI spot prices were predicted correctly. For any policy decision, though exact prices will be helpful, knowing the trend in advance is of utmost importance and so the model is reasonably useful. A graph of predicted value versus actual value is given in the appendix and can be used for further reference.

Future Enhancements: With this my neural network model was completed. Although there is a lot of scope for future enhancements, a few of which I list below: 1. More number of exogenous input variables can be used 2. Different training algorithms can be compare and the best one based on that comparison should be used 3. Right now, only mathematical pre-processing is done. Pre-processing of the data based on the historical background should be done, to remove outlier which spoil the training data.

Conclusion Neural networks are an efficient medium for modelling financial and economic variables. Oil prices, though an extremely volatile and unpredictable variable can be predicted to some extent using the power of neural networks. Although the exact values can’t be predicted but trends are easily recognised and depicted by the neural network.

Appendix 1

Number of iterations versus Mean Squared Error

Appendix 2

Regression Model developed using the ANN

Appendix 3

Actual Prices versus predicted prices