Multivariate Polynomial Regression using gradient descent. In this assignment, polynomial regression models of degrees 1,2,3,4,5,6 have been developed for the 3D Road Network [North Jutland, Denmark] Data Set using gradient
descent method. R squared, RMSE and Squared Error values have been calculated and compared for each model to find the models which best fit the data, as well as ones which overfit the data. L1 and L2 regularisation has been implemented to explore the effect of regularisation on testing loss and overfitting. The first attribute[OSM_ID] has been dropped. LONGITUDE and LATITUDE values have been used to predict the target variable, ALTITUDE. The code is divided into two files, generate_polynomials.py and polynomial_regression.py.Multivariate-Polynomial-Regression
Dataset
Attributes:
Structure
The generate_polynomials.py file will calculate the terms
- polynomial_regression.py implements gradient descent for the 6 models which minimises the loss function:
Gradient Descent
For each model, the training error was plotted for each iteration. It is clear that the error drops with each iteration. The following figure shows the plot of training error for degree 3 model
R Squared, RMSE and Squared-error
R Squared and RMSE was computed for each model.
It follows that up till degree 3, the testing error drops with increasing degree, but increasing degree there after results in an increase in error. This suggests that the degree 3 model best fits the data, where as models of degree 4, 5 and 6 are overfitting the data. The increasing average absolute values of weights with increasing degree also suggests that the weights are assuming arbitrarily large values to fit the data.
Regularisation
To address the problem of overfitting, L1 and L2 regularisation has been implemented for the degree 6 model. The following figures show the effect of regularisation on testing error.
Regularisation results in a sharp decrease in testing error. In fact, the loss for degree 6 polynomial model with regularisation is comparable with the loss for degree 1,2,3 and 4 polynomial models without regularisation.
Average absolute weight decreases sharply for the models with regularisation. Once regularised, the ws aren’t assuming large values to cause the model to oscillate wildly and overfit the data.
Instructions for executing:
Run python polynomial_regression.py
to build models for degrees 1 through 6,generate comparative graphs for R Squared, RMSE and Sqaured Error, using gradient descent with and without regularisation.
#
multivariate-polynomial-regression
Here are 2 public repositories matching this topic...
- Code
- Issues
- Pull requests
Multivariate Polynomial Regression using gradient descent with regularisation
- Updated May 9, 2020
- Python
- Code
- Issues
- Pull requests
Evaluation of multivariate Bernstein polynomials
- Updated Jul 2, 2022
- Fortran
Improve this page
Add a description, image, and links to the multivariate-polynomial-regression topic page so that developers can more easily learn about it.
Curate this topic
Add this topic to your repo
To associate your repository with the multivariate-polynomial-regression topic, visit your repo's landing page and select "manage topics."
Learn more