CS615 Deep Learning Assignment 1 - Forward Propagation Spring 2024
Introduction
CS 615 - Deep Learning
Assignment 1 - Forward Propagation Spring 2024
In this first assignment we’ll begin designing and implementing the modules we’ll use in our deep learning architectures. In addition, we’ll start getting used to importing datasets.
Allowable Libraries/Functions
Recall that you cannot use any ML functions to do the training or evaluation for you. Using basic statistical and linear algebra function like mean, std, cov etc.. is fine, but using ones like train are not. Using any ML-related functions, may result in a zero for the programming component. In general, use the “spirit of the assignment” (where we’re implementing things from scratch) as your guide, but if you want clarification on if can use a particular function, DM the professor on discord.
Grading
Theory
Testing layers independently
Test with dataset on connected layer set
TOTAL
Table 1: Grading Rubric
20pts 50pts 30pts 100pts
1
Datasets
Kid Creative We will use this dataset for binary classification. This dataset consists of data for
673 people in a CSV file. This data for each person includes:
1. Observation Number (we’ll want to omit this)
2. Buy (binary target value, Y )
3. Income
4. Is Female
5. Is Married
6. Has College
7. Is Professional
8. Is Retired
9. Unemployed
10. Residence Length
11. Dual Income
12. Minors
13. Own
14. House
15. White
16. English
17. Prev Child Mag
18. Prev Parent Mag
We’ll omit the first column and use the second column for our binary target Y . The remaining 16 columns provide our feature data for our observation matrix X.
2
1 Theory
1. Acknowledge teach of the following by typing your name after each of the following statements:
-
(a) While you can use online resources, you may not copy them. (1pt) Yixu Xia
-
(b) You cannot place any photos in your report (1pt) Yixu Xia
-
(c) You cannot place any code snippets in your report (1pt). Yixu Xia
-
(d) You cannot use any ML frameworks in your code, accept for some purposes explicitly mentioned (1pt).
Yixu Xia -
(e) While you may work in Jupyter notebook, or the like, you must, export your code as a python file for submission (and check that it works). (1pt)
Yixu Xia
2. Given a single input observation x = 1 2 3 and a fully connected layer with weights of 1 2
W = 3 4 as biases b = −1 2, what are the output of the fully connected layer given x 56
as its input (5pts)?
3. Given an input, h = [10, −1], what would be the output(s) if this data was processed by the following activation functions/layers (10pts)?
(a) Linear (b) ReLU
(c) Sigmoid
(d) Hyperbolic Tangent
(e) Softmax
1.1 answer
2.y = [21, 30]
3.(a).y = ax, y = [10, 1], whena = 1
(b)y = max(0, h), y = [10, 0]
(c)y = 1 , y = [0.999, 0.269] 1+e−h
(d)y = tanh h, y = [1, −0.762] (e)y = [0.999, 1.67 ∗ 10−5]
3
2 Start Implementing Your Layers
To organize our framework, we’ll keep each layer’s implementation in its own file and import them as needed in our main script.
You have been provided the abstract base class Layer in the file Layer.py. In addition, there is a lecture slide that shows how the importing of the modules is done. Your first task is to implement the following layers (each in their own file, with the filename being the same as the layer’s name).
We are going to have a script that looks to import each module, and run their forward method to see that it generates the proper output. We will do this with both a single observation, and a batch. Both of these should be input as MATRICES (the single observation test is still a 1 × D matrix). We suggest that you do this testing yourself as well. We’re essentially doing unit testing.
InputLayer
Implement a class called InputLayer that inherits from your abstract base class. Here’s the class’ public interface:
class InputLayer(Layer ):
#Input: dataIn, an NxD matrix
#Output : None
def init (self ,dataIn): #TODO
#Input: dataIn, an NxD matrix #Output : An NxD matrix
def forward(self ,dataIn): #TODO
#We’ll worry about these later...
def gradient(self): pass
def backward( self , gradIn ): pass
This class’ constructor should take as a parameter an entire training dataset (as an N × D matrix), and initialize two attributes, meanX and stdX to be row vectors of the mean and standard devia- tion, respectively, of the features of your training dataset. For numeric stability, set any standard deviations that are zero to 1 (this will avoid divide-by-zero issue when zscoring).
In addition, you must implement the abstract method forward such that it takes a data matrix, X, as a parameter, computes the zscored version of this data using the meanX and stdX attributes, stores the input and output in attributes (for later use), and returns the zscored data.
4
Activation Layers
Next implement classes for the following activation functions:
• Linear
• ReLU
• LogisticSigmoid
• Softmax
• Tanh (Hyperbolic Tangent Function)
The public interface for each should be (where XXX is the name of the activation function, according to the bullets above):
class XXXLayer( Layer ) : #Input : None #Output : None
def init (self): #TODO
#Input: dataIn, an NxK matrix #Output: An NxK matrix
def forward(self ,dataIn): #TODO
#We’ll worry about these later...
def gradient(self): pass
def backward( self , gradIn ): pass
5
FullyConnectedLayer
Finally, let’s create a class for a fully connected layer, aptly called FullyConnectedLayer. This too should inherit from Layer. Following the material in the lecture slides, this class should have two attributes, a weight matrix and a bias vector. The constructor should take in two explicit parameters, the number of features coming in, and the number of features coming out of this layer, and use these to initialize the weights (and biases, which are technically weights as well) to be random values in the range of ±10−4. Its forward method once again takes in data X, storing it in its parent’s previous input attribute, and computes the output (storing it with the parent class, and returning it) as:
Y = XW + b
where W is the weight matrix and b is the bias vector. For now, the gradient method may return nothing. In addition provide getter and setter methods for the weight and bias attributes. Here’s the public interface:
class FullyConnectedLayer(Layer ):
#Input : #Input : #Output :
of data
the data
sizeIn , the number of features sizeOut , the number of features for
coming in coming
out .
None
def init (self , sizeIn , sizeOut): #TODO
#Input : None
#Output : The sizeIn x
def getWeights( self ): #TODO
#Input : The sizeIn x #Output : None
sizeOut
sizeOut
weight
weight
matrix .
matrix .
def setWeights(self , weights): #TODO
#Input : None
#Output: The 1 x sizeOut bias vector
def getBiases(self): #TODO
#Input: The 1 x sizeOut bias vector #Output : None
def setBiases(self , biases): #TODO
#Input : dataIn , an NxD data matrix #Output: An NxK data matrix
def forward(self ,dataIn): #TODO
#We’ll worry about these later...
6
def gradient(self): pass
def backward( self , gradIn ): pass
7
3 Connecting Layers and Forward Propagate
Now let’s assemble a simple network and forward propagate data through it! With real data! Our architecture will be:
Input→FC (1 output)→Logistic Sigmoid
We’ll use the dataset mentioned earlier in the assignment. Read in the dataset as your input data X and pass it through the architecture mentioned above. From an implementation standpoint, you’ll likely want to create instances of your classes and organize them in some sort of an ordered structure such that the output of one layer is the input of the next. See the example code at the end of the Building Blocks slides.
In your report, just provide the output of the last layer pertaining to the first observation.
3.1 output
[0.50006194]
8
Submission
For your submission, upload to Blackboard a single zip file containing:
1. PDF Writeup 2. Source Code 3. readme.txt file
The readme.txt file should contain information on how to run your code to reproduce results for each part of the assignment.
The PDF document should contain the following:
-
Part 1: Your solutions to the theory question. DO NOT put screen shots or code in this.
You should be typesetting your numeric results.
-
Part 2: Nothing
-
Part 3: The output pertaining to the first observation from the final layer, when given the augemented medical cost dataset as its input.
9
