Penalty Functions
- Penalty functions describe the irritation with a residual of a certain value.
- The goal is to minimize total irritation by minimizing the sum of these penalties.
- L1 (Lasso) penalty functions are sparsifying, often resulting in many zero entries in the solution to an optimization problem.
- L1 penalties are relatively relaxed for large residuals and upset with small residuals, while quadratic penalties are the opposite.
- Huber penalties blend quadratic and L1 penalties, matching least squares for small residuals and resembling an L1 penalty for large residuals.
Regularized Approximation
- Regularized approximation is a criterion problem with multiple objectives, often involving tracking a desired trajectory, minimizing input size, and minimizing input variation.
- Weights (Delta and Ada) shape the solution and can be adjusted to achieve desired results.
- Changing the penalty function to the sum of absolute values (L1) would likely result in a sparse solution.
- Sparse first difference means that most of the time the value is equal to the previous value, encouraging piece-wise constant inputs.
Signal Reconstruction
- Signal reconstruction aims to form an approximation of a corrupted signal by minimizing a regularization function or smoothing objective.
- The trade-off in signal reconstruction is between deviating from the given corrupted signal and the size of the smoothing cost.
- Cross-validation can be used to select the amount of smoothing by randomly removing a portion of the data and pretending it is missing.
- When the penalty is an L1 Norm on the first difference, it is called the total variation, and the solution is expected to have a sparse difference corresponding to a piece-wise constant approximation.
- Total variation smoothing preserves sharp boundaries in images.
- Excessive regularization of total variation denoising results in a cartoonish effect with constant grayscale values in different regions.
Robust Approximation
- Robust approximation addresses uncertainty in the model by ignoring it or taking an average of possible values.
- The most common method for handling uncertainty in practice is to ignore it or use a mean and variance as a probability distribution.
- A better approach is to simulate multiple scenarios or use an educated guess to account for uncertainty.
- Regularization in machine learning and statistics addresses uncertainty in data.
Handling Uncertainty
- Different approaches to handling uncertainty include stochastic methods, worst-case methods, and hybrids of these.
- Stochastic methods assume that the uncertain parameter comes from a probability distribution.
- Worst-case methods assume that the uncertain parameter satisfies certain constraints.
- Robust stochastic methods combine elements of both stochastic and worst-case methods.
- A simple example illustrates the different methods and their effects on the resulting model.
- A practical trick for handling uncertainty is to obtain a small number of plausible models and use a min-max approach.
- The speaker discusses how to generate multiple plausible models when faced with uncertainty in data.
- One approach is to use bootstrapping, which is a valid method in this context.
- Another principled approach is to use maximum likelihood estimation and then generate models with parameters that are close to the maximum likelihood estimate.
- The speaker introduces the concept of stochastic robustly squares, which is a regularized least squares method that accounts for uncertainty in the data.
- Regularization, such as Ridge regression, can be interpreted as a way of acknowledging uncertainty in the features of the data.
- The speaker provides an example of a uniform distribution on a unit disc in Matrix space to illustrate the concept of uncertainty in model parameters.
- The speaker emphasizes the importance of acknowledging uncertainty in data and models, and suggests that simply admitting uncertainty can provide significant benefits.
Maximum Likelihood Estimation
- The video discusses statistical estimation, specifically maximum likelihood estimation.
- Maximum likelihood estimation aims to find the parameter values that maximize the likelihood of observing the data.
- The log likelihood function is often used instead of the likelihood function since maximizing both functions is equivalent.
- Regularization can be added to the maximum likelihood function to prevent overfitting.
- Maximum likelihood estimation is a convex problem when the log likelihood function is concave.
- An example of a linear measurement model is given, where the goal is to estimate the unknown parameter X from a set of measurements.
- The log likelihood function for this model is derived and shown to be convex when the noise is Gaussian.
- The maximum likelihood solution for this model is the least squares solution.
- The noise distribution can also be Laplacian, which has heavier tails than a Gaussian distribution.
- Maximum likelihood estimation with Laplacian noise is equivalent to L1 estimation.
- L1 estimation is more robust to outliers compared to least squares estimation because it is less sensitive to large residuals.
- The difference between L2 fitting and L1 approximation can be explained by the difference in the assumed noise density.
- Huber fitting can be interpreted as maximum likelihood estimation with a mixture of Gaussian and exponential distributions.
Logistic Regression
- Logistic regression is a model for binary classification where the probability of an outcome is modeled using a logistic function.
- The logistic map corresponding to the maximum likelihood is infinitely sharp because making it less sharp would drop the probability and lose log likelihood.
- When the log-likelihood is unbounded above, it means the data are linearly separable, and this issue can be fixed by adding regularization.
Hypothesis Testing
- In basic hypothesis testing, a randomized detector is a 2xN matrix that determines the probability of guessing the distribution from which an outcome originated.
- The confusion matrix, or detection probability matrix, is obtained by multiplying the randomized detector matrix by the probability distributions P and Q.
- The goal is to have the detection matrix be an identity matrix, indicating perfect accuracy in guessing the distribution.
- The choice of the randomized detector involves a multi-criterion problem, balancing the probabilities of false negatives and false positives.
- In some cases, an analytical solution exists for the optimal randomized detector, resulting in a deterministic detector.
- The likelihood ratio test is a simple method for determining the distribution from which an outcome originated based on the ratio of P over Q.
- In infinite dimensions, determining the density of two continuous distributions can be done by comparing their ratio to a threshold.
- Deterministic detectors can be used to classify data points based on their density ratio.
- Other approaches, such as minimizing the probability of being wrong, can also be used for classification.
- These approaches often result in non-deterministic detectors.
