Linear Models in Practice
The term linear model implies the model is specified as a linear combination of features. Based on training data, the learning process computes one weight for each feature to form a model that can estimate or predict the target value. For example, the target (dependent variable) is the amount of insurance a customer will purchase and the independent variables are age and income, a simple linear model would be the following:
>>> Estimated target = 0.2 + 5age + 0.0003income
Suppose the data consists of n observations { xi, yi }n. Each data point is represented as { xi, yi }. Linear models can be used both for regression and classification problems. Various types of linear models exist for various problems. Let’s go a bit more into this.
Linear Regression
Linear regression is a linear model, e.g. a model that assumes a linear relationship between the input variables (x) and the single output variable (y). More specifically, that y can be calculated from a linear combination of the input variables (x). When there is a single input variable (x), the method is referred to as simple linear regression. When there are multiple input variables, literature from statistics often refers to the method as multiple linear regression.
The prediction function for a regression is:
y_pred = w[0] * x[0] + w[1] * x[1] + … + w[p] * x[p] + b
where x is a features vector with a length p of a single point, w and b are parameters of the model that are learned and y_pred is a prediction.
Different techniques can be used to prepare or train the linear regression equation from data;
Simple Linear Regression (SLR)
Simple linear regression is a type of regression analysis where the number of independent variables is one and there is a linear relationship between the independent(x) and dependent(y) variable.
The linear model tries to estimate the coefficient of the linear graph and use it for future prediction of new points. Based on the given data points, we try to plot a line that models (fits) the points the best. The motive of the linear regression algorithm is to find the best values for a_0 and a_1. The line can be modeled based on the linear equation shown below.
>>> y = a_0 + a_1 * x
Linear Regression (Ordinary Least Squares)
The model finds parameters that minimize Mean Squared Error (called the cost function) between prediction and the true target. The Ordinary Least Squares procedure seeks to minimize the sum of the squared residuals. This means that given a regression line through the data we calculate the distance from each data point to the regression line, square it, and sum all of the squared errors together. This is the quantity that ordinary least squares seeks to minimize. It has the same mathematical equation with the SLR.
When a coefficient becomes zero, it effectively removes the influence of the input variable on the model and therefore from the prediction made from the model (0 * x = 0). This is a form of regularization that change the learning algorithm to reduce the complexity of regression models by putting pressure on the absolute size of the coefficients, driving some to zero.
Multiple Linear Regression (MLR)
The model is often used for predictive analysis since it defines the relationship between two or more variables. In other terms, MLR examines how multiple independent variables are related to one dependent variable. Once each of the independent factors has been determined to predict the dependent variable, the information on the multiple variables can be used to create an accurate prediction on the level of effect they have on the outcome variable. This is done after the model has calculated the values of coefficients that best fits the training data. And also to note: the number of features used, the number of coefficients to be calculated by the model. The model creates a relationship in the form of a straight line (linear) that best approximates all the individual data points.
For example: calculating the net income of people in a population. The factors/independent variables involved may be the salary of the person, taxes, savings etc.
Logistic Regression
The logistic regression model applies the same principle with the linear regression. However, in this case, a cut off value is set for binary classification as positive or negative, 0.5 in the plot shown above. The logistic regression model can also be used for the estimation of values within a particular range.
Worthy to note are the other types of regression models like the Ridge Regression and Lasio Regression. Regularization seek to both minimize the sum of the squared error of the model on the training data (using ordinary least squares) but also to reduce the complexity of the model (like the number or absolute size of the sum of all coefficients in the model).
Practical tutorial for this lesson can be found in the git repo below
https://github.com/aiplus-futa/Machine-Learning-Class/tree/master/Linear%20Models
Thanks for reading!
Additional links for more knowledge can be found below:
