In our last post, we discussed the importance of developing a strong forecasting engine to predict future energy consumption based on data from past energy consumption. We then discussed the suitability of using a Deep Convolutional Neural Network (DCNN) to form accurate predictions and examined how to use this model by converting a series of load data into images.In today’s blog, we explain the codes we’re using for data processing.
Before utilizing our DCNN, we need to convert a time series into gray-scale images by developing a class with some methods. We’ll begin this process by importing the required libraries as follows:
## import required libraries
from sklearn.model_selection import train_test_split
import math
import pandas as pd
import numpy as np
import tensorflow as tf
import random
In this application, we will use the libraries listed above to facilitate coding. We’ll use Pandas to help process the data and NumPy to convert the data into a format that we can later input into Tensorflow. To facilitate programming, we will create a class (“MakeImage”) that has some methods.
### building a class to convert time series to images
# defining variables of the class
class MakeImage:
def __init__(self):
self._df = None
self._size_width = None
self._index_of_columns_2_be_batched = None
Our “size_width” variable is used to specify image width. “index_of_columns_2_be_batched” indicates which columns of our Pandas dataframe (“df”) we will use for further operations like creating batches of data.
# defining setters and getters
@property
def df(self):
return self._df
@property
def size_width(self):
return self._size_width
@property
def index_of_columns_2_be_batched(self):
return self._index_of_columns_2_be_batched
@df.setter
Our data must be saved in the first column of an excel file in a zero index sheet. In this class we have named this specific data column “load_data.” You may rename this as follows:
def df(self, value):
self._df = pd.read_excel(value, 0, names = ["load_data"])
@size_width.setter
def size_width(self, value):
self._size_width = value
@index_of_columns_2_be_batched.setter
def index_of_columns_2_be_batched(self, value):
self._index_of_columns_2_be_batched = value
The following function can be used to append another column (“label”) to our defined “df” (Pandas dataframe). As discussed in Part 1 of this blog series, these labels indicate whether a given instance of energy consumption has increased (1) or decreased (0) compared to the last instance:
def add_label(self):
df = self.df
df.ix[0,'label'] = 0.0
df.ix[1:,'label'] = [1.0 if df.ix[i,'load_data'] > df.ix[i-1,'load_data'] else 0.0 for i in range(1,len(df.index))]
Our next method creates chunked data from features and labels with fixed sizes that will later be used to make images. The output of this method is two NumPy arrays, namely “nparray_features” and “nparray_label” for chunked features and labels, respectively.
def chunk_features_and_label(self, col_name, col_label):
df = self.df
image_width = self.size_width
nparray_features = np.zeros(shape = (len(df.index) - image_width, image_width))
nparray_label = np.zeros(shape = (len(df.index) - image_width, 1))
for i in range(len(df.index) - image_width):
nparray_features[i, :] = df.ix[i: i + image_width - 1, col_name].as_matrix().reshape(image_width)
nparray_label[i, :] = df.ix[i + image_width, col_label].reshape(-1,1)
return nparray_features, nparray_label
The following function receives a feature set NumPy and readies it for image conversion:
def features_2_images(self, x):
size_width = self.size_width
x_image = np.zeros(shape = (x.shape[0], x.shape[1]*x.shape[1]))
print(x.shape[0])
for i in range(x.shape[0]):
x_temp = x[i, :].copy()
x_copy = np.copy(x[i,:])
x_int = np.zeros(shape = (x_copy.shape), dtype = int)
x_copy2 = np.copy(x[i,:])
x_copy.sort()
for j in range(np.size(x_copy)):
for k in range(np.size(x_copy)):
if x_copy2[j] == x_copy[k]:
x[i, j] = int(k)
n_values = np.max(x[i,:].astype(int)) + 1
squared = np.eye(n_values)[x[i,:].astype(int)]
flatten = squared.flatten()
x_image[i, :] = flatten
return x_image
To convert labels into an appropriate format for our proposed methodology, we need to turn them into NumPy arrays with the same dimension as the number of classes (in our case, 2). The following function uses a one hot encoding technique to convert them:
def convert_2_onehot_encoding(self, vector_of_one_dim):
nparray_lab_onehot = np.zeros(shape = (vector_of_one_dim.shape[0], 2))
for i in range(vector_of_one_dim.shape[0]):
n_values = 2
nparray_lab_onehot[i, :] = np.eye(n_values)[vector_of_one_dim[i,:].astype(int)]
return nparray_lab_onehot
In our next method, we simply receive “batch_size” as an input argument and calculate the number of batches based on input.
def number_of_batches(self, x, batch_size):
return int(x.shape[0]/batch_size)
Using the input images and their corresponding labels, the following function will randomly select a batch of combinations in each call, e.g. “x_batch” and “y_batch” with “batch_size.”
def next_batch(self, x, y, batch_size):
index_for_batch = self.index_of_columns_2_be_batched
if len(index_for_batch) >= batch_size:
selected_index = random.sample(index_for_batch, batch_size)
else:
selected_index = random.sample(index_for_batch, len(index_for_batch))
x_batch = [x[i] for i in selected_index]
y_batch = [y[i] for i in selected_index]
self.index_of_columns_2_be_batched = [i for i in index_for_batch if i not in selected_index]
return x_batch, y_batch
The function above concludes our set-up process. Now, we must create a class and an instance of it. Then, by calling the methods of this object, the required input data for our proposed DCNN will be prepared as follows:
### create a class and initialize an instance of it
makeImage = MakeImage()
## Setter instance variables
# put the path of your time series data in your system as shown below
path_2_excel_file = 'path to data/load_data.xlsx'
makeImage.df = path_2_excel_file # a string that shows path of data
# set the image size (we determined for this application that 32 is a good choice)
makeImage.size_width = 32
makeImage.columns_2_pad = ['load_data', 'label']
makeImage.add_label()
# process load data to be proper for the model
X_not_ready_yet, Y_not_ready_yet = makeImage.chunk_features_and_label('load_data', 'label')
X_ready = makeImage.features_2_images(X_not_ready_yet)
Y_ready = makeImage.convert_2_onehot_encoding(Y_not_ready_yet)
# select 70% of data for training the model and 30% for testing
X_train, X_test, y_train, y_test = train_test_split(X_ready, Y_ready, test_size = 0.3)
# the batch size is selected to be 100; it is possible to adjust it based on the problem’s requirements
batch_size = 100
# we need to calculate number of batches
number_of_batches = makeImage.number_of_batches(X_train, batch_size)
In this blog, we described how to develop some codes to facilitate the data processing component of our proposed methodology. These codes are purposefully developed so that minor changes can make them reusable for solving most time series problems. In our final blog, we’ll discuss our proposed DCNN topology and establish a computation graph, using Tensorflow to process the output data from today’s codes.