6. Types of Regression Lines

This section explores various types of regression lines used in statistical modeling and machine learning to understand and predict relationships between variables.

6.1 Simple Linear Regression

Definition: Models the linear relationship between a single independent variable (predictor) and a dependent variable (response).

Formula: $$ Y = a + bX + \varepsilon $$ Where:

• $Y$ is the dependent variable.
• $X$ is the independent variable.
• $a$ is the y-intercept (the value of $Y$ when $X$ is 0).
• $b$ is the slope of the line (the change in $Y$ for a unit change in $X$).
• $\varepsilon$ is the error term, representing the unexplained variation in $Y$.

Use Case: Predicting an individual's salary based on their years of work experience. For instance, if $a=30,000$ and $b=2,000$, a person with 5 years of experience might be predicted to earn $30,000 + 2,000 \times 5 = $40,000$.

6.2 Multiple Linear Regression

Definition: Extends simple linear regression by modeling the relationship between a dependent variable and two or more independent variables.

Formula: $$ Y = a + b_1X_1 + b_2X_2 + \dots + b_nX_n + \varepsilon $$ Where:

• $Y$ is the dependent variable.
• $X_1, X_2, \dots, X_n$ are the independent variables.
• $a$ is the y-intercept.
• $b_1, b_2, \dots, b_n$ are the coefficients for each respective independent variable, indicating the change in $Y$ for a unit change in that variable, holding others constant.
• $\varepsilon$ is the error term.

Use Case: Estimating house prices ($Y$) using features like the area of the house ($X_1$), its location ($X_2$), and the number of bedrooms ($X_3$).

6.3 Polynomial Regression

Definition: Models the relationship between variables as an nth-degree polynomial. This is useful when the relationship is not strictly linear and can be captured by curves.

Formula (for a 2nd-degree polynomial): $$ Y = a + b_1X + b_2X^2 + \varepsilon $$ General Formula: $$ Y = a + b_1X + b_2X^2 + \dots + b_nX^n + \varepsilon $$ Where:

• $Y$ is the dependent variable.
• $X$ is the independent variable.
• $a$, $b_1, b_2, \dots, b_n$ are the coefficients.
• $n$ is the degree of the polynomial.
• $\varepsilon$ is the error term.

Use Case: Predicting population growth over time, which often follows a curved pattern rather than a straight line.

6.4 Ridge Regression

Definition: A type of regularized linear regression that is used to reduce model complexity and mitigate multicollinearity (high correlation between independent variables). It adds a penalty term to the ordinary least squares cost function, which shrinks the coefficients.

Objective Function (to minimize): $$ \sum_{i=1}^{m}(Y_i - \hat{Y}i)^2 + \lambda \sum{j=1}^{n} b_j^2 $$ Where:

• $Y_i$ is the actual value of the dependent variable for the $i$-th observation.
• $\hat{Y}_i$ is the predicted value.
• $m$ is the number of observations.
• $b_j$ are the coefficients of the independent variables.
• $\lambda$ (lambda) is the regularization parameter, controlling the strength of the penalty. The $\sum b_j^2$ term penalizes large coefficients.

Use Case: Predicting outcomes in situations where features are highly correlated, such as predicting stock prices using various economic indicators.

6.5 Lasso Regression

Definition: Similar to Ridge regression, Lasso (Least Absolute Shrinkage and Selection Operator) regression is another form of regularized linear regression. Its key difference is that it uses the L1 norm for regularization, which can shrink some coefficients exactly to zero. This property allows Lasso to perform automatic feature selection, effectively removing less important predictors from the model.

Objective Function (to minimize): $$ \sum_{i=1}^{m}(Y_i - \hat{Y}i)^2 + \lambda \sum{j=1}^{n} |b_j| $$ Where:

• $Y_i$ is the actual value of the dependent variable for the $i$-th observation.
• $\hat{Y}_i$ is the predicted value.
• $m$ is the number of observations.
• $b_j$ are the coefficients of the independent variables.
• $\lambda$ (lambda) is the regularization parameter. The $\sum |b_j|$ term penalizes the absolute values of the coefficients.

Use Case: Particularly useful when dealing with high-dimensional datasets where only a subset of predictors are truly important for prediction. For example, predicting disease risk using a large number of genetic markers.

6.6 Logistic Regression

Definition: Primarily used for binary classification problems, where the dependent variable can take only two values (e.g., Yes/No, 0/1, True/False). It models the probability of an event occurring.

Formula: $$ P(Y=1) = \frac{1}{1 + e^{-(a + bX)}} $$ Where:

• $P(Y=1)$ is the probability that the dependent variable $Y$ is 1 (or the event occurs).
• $X$ is the independent variable.
• $a$ is the intercept.
• $b$ is the coefficient.
• $e$ is the base of the natural logarithm. The expression $(a + bX)$ is often passed through a sigmoid (or logistic) function to constrain the output probability between 0 and 1.

Use Case: Predicting whether a customer will purchase a product (Yes/No) based on their demographic information or past behavior.

---

Related Concepts (Mentioned in Navigation):

• 6.10 Power Regression Line: Models relationships where the dependent variable changes by a constant factor for a constant multiplicative change in the independent variable. Formula: $Y = aX^b$.
• 6.11 Applications of Regression Line: Broadly covers the use of regression lines in various fields like economics, finance, biology, and social sciences for forecasting, understanding trends, and identifying relationships.
• 6.12 Importance of Regression Line: Crucial for understanding relationships between variables, making predictions, identifying trends, and informing decision-making.
• 6.13 Statistical Significance of Regression Line: Involves assessing whether the relationship between variables is likely real or due to random chance, often using p-values and confidence intervals for coefficients.
• 6.14 Applications of Regression Line: (Duplicate, see 6.11)
• 6.15 Practice Questions on Regression Line: Exercises designed to test understanding of regression concepts and calculations.
• 6.2 Logistic Regression Line: (See 6.6)
• 6.3 Polynomial Regression Line: (See 6.3)
• 6.4 Ridge and Lasso Regression: (See 6.4 and 6.5)
• 6.5 Non-Linear Regression Line: A general term for regression models that do not assume a linear relationship, including polynomial, exponential, and logarithmic regressions.
• 6.6 Multiple Regression Line: (See 6.2)
• 6.7 Exponential Regression Line: Models relationships where the rate of change of the dependent variable is proportional to its current value. Formula: $Y = ae^{bX}$.
• 6.8 Pricewise Regression Line: (Likely referring to piecewise regression) Models relationships using different linear equations for different segments or ranges of the independent variable.
• 6.9 Time Series Regression Line: Regression models specifically designed for analyzing data collected over time, often incorporating temporal dependencies and trends.