K6312 Take-home Assignment I

In this task, you need to develop linear regression model to predict students grade. The data is from the first week lab. Through the task, you will be asked to try different features sets and different training size and compare their prediction performances. CourseNana.COM

Import Libraries

In [ ]:

# Pandas and numpy for data manipulation
import pandas as pd
import numpy as np
np.random.seed(123)

In [ ]:

# Standard ML Models for comparison
from sklearn.linear_model import LinearRegression

# Splitting data into training/testing
from sklearn.model_selection import train_test_split

# Metrics
from sklearn.metrics import mean_squared_error, mean_absolute_error, median_absolute_error

Load data and examine their main characteristic CourseNana.COM

In [ ]:

import pandas as pd
file_name = "student-mat.csv"
df_data = pd.read_csv(file_name, sep=';')
print(df_data.shape)

(395, 33)

In [ ]:

df_data.head(2).append(df_data.tail(2))

/var/folders/l5/w9vxyb4s46b7rmcp_txnn9fw0000gn/T/ipykernel_76773/768104290.py:1: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  df_data.head(2).append(df_data.tail(2))

Out[ ]:

	school	sex	age	address	famsize	Pstatus	Medu	Fedu	Mjob	Fjob	...	famrel	freetime	goout	Dalc	Walc	health	absences	G1	G2	G3
0	GP	F	18	U	GT3	A	4	4	at_home	teacher	...	4	3	4	1	1	3	6	5	6	6
1	GP	F	17	U	GT3	T	1	1	at_home	other	...	5	3	3	1	1	3	4	5	5	6
393	MS	M	18	R	LE3	T	3	2	services	other	...	4	4	1	3	4	5	0	11	12	10
394	MS	M	19	U	LE3	T	1	1	other	at_home	...	3	2	3	3	3	5	5	8	9	9

4 rows × 33 columns CourseNana.COM

Here, we will focus on prediction of G3. Filter out grades that were 0 CourseNana.COM

In [ ]:

df_data.rename(columns={'G3': 'Grade'}, inplace=True)
df_data = df_data[~(df_data['Grade']==0)] # delete the rows with Grade 0
print(df_data.shape)
df_data.head(2).append(df_data.tail(2))

(357, 33)

/var/folders/l5/w9vxyb4s46b7rmcp_txnn9fw0000gn/T/ipykernel_76773/1705270932.py:4: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  df_data.head(2).append(df_data.tail(2))

Out[ ]:

	school	sex	age	address	famsize	Pstatus	Medu	Fedu	Mjob	Fjob	...	famrel	freetime	goout	Dalc	Walc	health	absences	G1	G2	Grade
0	GP	F	18	U	GT3	A	4	4	at_home	teacher	...	4	3	4	1	1	3	6	5	6	6
1	GP	F	17	U	GT3	T	1	1	at_home	other	...	5	3	3	1	1	3	4	5	5	6
393	MS	M	18	R	LE3	T	3	2	services	other	...	4	4	1	3	4	5	0	11	12	10
394	MS	M	19	U	LE3	T	1	1	other	at_home	...	3	2	3	3	3	5	5	8	9	9

4 rows × 33 columns CourseNana.COM

In [ ]:

df_data.columns

Out[ ]:

Index(['school', 'sex', 'age', 'address', 'famsize', 'Pstatus', 'Medu', 'Fedu',
       'Mjob', 'Fjob', 'reason', 'guardian', 'traveltime', 'studytime',
       'failures', 'schoolsup', 'famsup', 'paid', 'activities', 'nursery',
       'higher', 'internet', 'romantic', 'famrel', 'freetime', 'goout', 'Dalc',
       'Walc', 'health', 'absences', 'G1', 'G2', 'Grade'],
      dtype='object')

In [ ]:

# one-hot vectors for df_data
df_used = pd.get_dummies(df_data)
df_used.columns

Out[ ]:

Index(['age', 'Medu', 'Fedu', 'traveltime', 'studytime', 'failures', 'famrel',
       'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences', 'G1', 'G2',
       'Grade', 'school_GP', 'school_MS', 'sex_F', 'sex_M', 'address_R',
       'address_U', 'famsize_GT3', 'famsize_LE3', 'Pstatus_A', 'Pstatus_T',
       'Mjob_at_home', 'Mjob_health', 'Mjob_other', 'Mjob_services',
       'Mjob_teacher', 'Fjob_at_home', 'Fjob_health', 'Fjob_other',
       'Fjob_services', 'Fjob_teacher', 'reason_course', 'reason_home',
       'reason_other', 'reason_reputation', 'guardian_father',
       'guardian_mother', 'guardian_other', 'schoolsup_no', 'schoolsup_yes',
       'famsup_no', 'famsup_yes', 'paid_no', 'paid_yes', 'activities_no',
       'activities_yes', 'nursery_no', 'nursery_yes', 'higher_no',
       'higher_yes', 'internet_no', 'internet_yes', 'romantic_no',
       'romantic_yes'],
      dtype='object')

In [ ]:

df_used.head(2).append(df_used.tail(2))

/var/folders/l5/w9vxyb4s46b7rmcp_txnn9fw0000gn/T/ipykernel_76773/3661735764.py:1: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  df_used.head(2).append(df_used.tail(2))

Out[ ]:

	age	Medu	Fedu	traveltime	studytime	failures	famrel	freetime	goout	Dalc	...	activities_no	activities_yes	nursery_no	nursery_yes	higher_no	higher_yes	internet_no	internet_yes	romantic_no	romantic_yes
0	18	4	4	2	2	0	4	3	4	1	...	1	0	0	1	0	1	1	0	1	0
1	17	1	1	1	2	0	5	3	3	1	...	1	0	1	0	0	1	0	1	1	0
393	18	3	2	3	1	0	4	4	1	3	...	1	0	1	0	0	1	0	1	1	0
394	19	1	1	1	1	0	3	2	3	3	...	1	0	0	1	0	1	0	1	1	0

4 rows × 59 columns CourseNana.COM

Split dataframe into features dataframe and target dataframe

In [ ]:

df_X = df_used.drop(columns=['Grade'])
df_y = df_used[['Grade']]

In [ ]:

df_X.columns

Out[ ]:

Index(['age', 'Medu', 'Fedu', 'traveltime', 'studytime', 'failures', 'famrel',
       'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences', 'G1', 'G2',
       'school_GP', 'school_MS', 'sex_F', 'sex_M', 'address_R', 'address_U',
       'famsize_GT3', 'famsize_LE3', 'Pstatus_A', 'Pstatus_T', 'Mjob_at_home',
       'Mjob_health', 'Mjob_other', 'Mjob_services', 'Mjob_teacher',
       'Fjob_at_home', 'Fjob_health', 'Fjob_other', 'Fjob_services',
       'Fjob_teacher', 'reason_course', 'reason_home', 'reason_other',
       'reason_reputation', 'guardian_father', 'guardian_mother',
       'guardian_other', 'schoolsup_no', 'schoolsup_yes', 'famsup_no',
       'famsup_yes', 'paid_no', 'paid_yes', 'activities_no', 'activities_yes',
       'nursery_no', 'nursery_yes', 'higher_no', 'higher_yes', 'internet_no',
       'internet_yes', 'romantic_no', 'romantic_yes'],
      dtype='object')

In [ ]:

df_y.columns

Out[ ]:

Index(['Grade'], dtype='object')

Split data into training/testing sets

In [ ]:

# 75% is for training and 25% is for testing
X_train, X_test, y_train, y_test = train_test_split(df_X, df_y, 
                                                    test_size = 0.25,
                                                    random_state=123)

Baseline for Students Grade Prediction

In [ ]:

# a very simple baseline: use the median value as the prediction results
baseline = np.median(y_train)
baseline_mae = np.mean(abs(baseline - y_test))
baseline_rmse = np.sqrt(np.mean((baseline - y_test) ** 2))
print('Baseline on test data, MAE is %0.2f' %  baseline_mae)  
print('Baseline on test data, RMSE is %0.2f' %  baseline_rmse)   

Baseline on test data, MAE is 2.83
Baseline on test data, RMSE is 3.51

/opt/anaconda3/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3438: FutureWarning: In a future version, DataFrame.mean(axis=None) will return a scalar mean over the entire DataFrame. To retain the old behavior, use 'frame.mean(axis=0)' or just 'frame.mean()'
  return mean(axis=axis, dtype=dtype, out=out, **kwargs)

Let's build a linear regression model to predict the student grade using three features: studytime, traveltime and higher_yes

In [ ]:

feature_train = X_train[['studytime', 'higher_yes','traveltime']]

In [ ]:

feature_train.head()

Out[ ]:

	studytime	higher_yes	traveltime
375	3	1	4
121	4	1	1
339	2	1	1
199	2	1	1
282	4	1	2

In [ ]:

lr = LinearRegression()
lr.fit(feature_train, y_train)

Out[ ]:

LinearRegression()

In [ ]:

# Metrics
feature_test = X_test[['studytime', 'higher_yes','traveltime']]
predictions = lr.predict(feature_test)
mae = np.mean(abs(predictions - y_test))
rmse = np.sqrt(np.mean((predictions - y_test) ** 2))
print('Using Linear Regression, MAE is %0.2f' %  mae)  
print('Using Linear Regression, RMSE is %0.2f' %  rmse)    

Using Linear Regression, MAE is 2.96
Using Linear Regression, RMSE is 3.54

/opt/anaconda3/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3438: FutureWarning: In a future version, DataFrame.mean(axis=None) will return a scalar mean over the entire DataFrame. To retain the old behavior, use 'frame.mean(axis=0)' or just 'frame.mean()'
  return mean(axis=axis, dtype=dtype, out=out, **kwargs)

In [ ]:

print('w0:',lr.intercept_)
print('w1,w2,w3:',lr.coef_)

w0: [8.62325546]
w1,w2,w3: [[ 0.62150473  2.10847672 -0.33782372]]

Grade = 8.62 + 0.62 * studytime + 2.11 * higher_yes - 0.34 * traveltime CourseNana.COM

The above linear regression model shows no improvement compared with the baseline model. Let's try to improve the performance by using different feature settings.

Task 1 (3 points)

Pls use the following three features: CourseNana.COM

1. failures
2. studytime
3. absences

to predict students' grade. CourseNana.COM

Report the MAE and RMSE on both training and testing data. CourseNana.COM

In [ ]:

Task 2 (3 points)

Pls use the following six features: CourseNana.COM

1. failures
2. studytime
3. absences
4. Medu
5. goout
6. higher

to predict students' grade. CourseNana.COM

Attention: higher is categoricial features. You need to use its one-hot transformations as inputs. CourseNana.COM

Report the MAE and RMSE on both training and testing data. CourseNana.COM

Task 3 (4 points)

In this task, let us check the effects of training data size. Firstly, we will use the same feature setting as in Task 1 (i.e., only three features) and the same testing dataset. Then, let's try to reduce the training data size: (1) using only 10% of the original training dataset, (2) using only 40% of the original training dataset. CourseNana.COM

Report the MAE and RMSE on the testing dataset under the two different settings. CourseNana.COM

Attention: Please use the same X_test, y_test as in Task 1 for testing. CourseNana.COM

CourseNana.COM

Task 4 (Bonus Question) (1 point)

Upon finishing those above tasks, you may found using different features and different training data sizes, the model will arrive at different perofmrances (Here is RMSE and MAE). Could you try to describe their relationship and explain why it is? CourseNana.COM