Data Science Projects: NSE Real-Time Stocks Analysis and Predictions Using Python LSTM Model
What is NSE
National Stock Exchange is India’s largest stock exchange in Mumbai, which was first established in November 1992 as the first fully automated, nationwide electronic exchange. First Data Science Projects
I’ll show you in this article :
how to fetch data of any stock(NSE) in realtime
How you can evaluate the stock price using basics visualizations:
Predicting future stock price using machine learning algorithms
This is my first data science machine learning project
Note: It’s very difficult to predict future costs, even for machine learning models. Particularly with regard to the stock market, stock price itself is a derivative of the underlying company and its capital structure, i.e. both economic and control factors, as well as human emotions and intentions.
There are several external factors to consider for knowing whether a stock’s price will rise or fall in the future.
Same as stock price, the only input the algorithm takes is the stock price, there are many other factors which can affect the stock price. Only note that your key purpose should be learning new techniques and instruments.
So Lets gets started
Which machine learning algorithm to use?
There is no such model that is always best. You can use either Regression or Forecasting algorithms to predict stock prices, but the data needs to be in a specific format for an algorithm to function that the machine understands when processing an ML algorithm. LSTM is one such prediction algorithm.
Long Short Term Memory, commonly called “LSTMs,” is a special form of neural recurrent network that can learn large-term dependencies. Unlike traditional feedforward networks, LSTM has feedback links, i.e. it is essentially their natural behavior to remember knowledge for long periods of time, not something they strive to know!
LSTMs also have this chain-like structure just like the RNN network has the shape of a chain of neural network repeating modules but the LSTM repeating module has a different structure. Rather than having a single layer of the neural network, there are four, which communicate in a very unique way.
How can LSTM Predict the Future Price of the stock?
In sequence prediction problems, LSTMs are very efficient, since they can store past information. This is critical in our case as a stock’s preceding price is crucial in predicting its future price.
In this project, I have used the last 30days ‘s stock close price to predict the closing price for the next day
Steps to develop LTSM Data Science Project Stock Price Prediction Model
We begin by importing Numpy for scientific computations, Matplotlib for plotting graphs, Pandas for loading and manipulating our dataset, sklearn for pre-processing our data, Keras for training/fitting/predictions on our dataset and nsepy for extracting historical and real-time data from the NSE website.
First install the nsepy library using the command pip install nsepy
Next, you’ll be asked to enter the stock symbol in order to fetch the stock information such as Hindustan Unilever in my case . We use the DateTime module of python to extract the to date information, end = date.today).
Df.tail() is used to fetch the last 5 data from the data frame
Our data frame has 2568 rows and 15 columns.
Visualize the closing price history from last 30th day’s information
Next Step is to Split the data into training and Testing Data
Build a new data frame, which stores the value of the close price column.
Coverting data into numpy array
Divide 75% data as training rest 25% as testing data
We use the MinMaxScaler from Scikit-learn to scale our dataset to have values from 0 to 1.
We scale our dataset because it is a good idea to scale the data to maximum output before feeding the data into a neural network.
Modify the training dataset with the required timestamp
train_datawill contain all the values of rows from index 0 to training_data_len with all the columns.
Append the values of the past 30 days to x train i.e. it will contain all values from index 0 to 29 for the first run and values from index 1 to 30 for the second run, etc.
Append the 31st day value to y train, i.e. the index 30th value for its first run and then the index 31st value for the second run and so on.
Then we use Numpy to convert x train, y train to an array, then we reshape the array into a 3-dimensional array with x train samples, 30 days timestamp and one function per phase. It’s the shape which our LSTM architecture accepts.
Building the LSTM Architecture
Model is an object of sequential () for initializing a layer of neural networks.
add LSTM layer to our network with parameters, source: Keras
units=50 , can be a positive integer, the dimensionality of the output space
return_sequence , can be True or False, determines whether to return the last output in the output sequence or the full sequence.
input_shape , which is the shape of our training set.
add densely connected neural network layer
next we compile the model using adam optimizer and for the loss we use mean_squared_error.
Then we fit the model with a batch size of 50 on x train and y train to run over 100 epochs.
Finally, we saved the trained model to the system
Create the testing dataset
Test datawill contain all the values of the training data len-30 index rows to the end of our dataset with all the columns
Y test must include all rows from the training data len index to the end of the dataset with all columns (in this case it is just ‘Close’ column).
x test will contain all values from index 0 to 29 for the first run of test data
and values from index 1 to 30 for the second run of test data and so on from test data.
We then transform x test to a Numpy array, then reshape the array to a 3-dimensional array, i.e. the form agreed by our LSTM architecture (similar to our x train and y train).
Predicting Future Stock using the Test Set
First we’ll make x test data predictions and undo the scaling.
Then we calculate the root mean squared error (rmse), which is used to measure the efficiency of our model. The lower the value our model’s achieved the better.
Rmse of hul is 36.68
Finally we use Matplotlib after projections to represent the results of the expected stock price and the real stock price.
To know about what is Stock Market read my share market article