Generative Adversarial Networks (GANs)
- GANs use an adversarial process to train a generator and a discriminator network.
- The generator network aims to produce realistic samples that can fool the discriminator, while the discriminator network tries to distinguish between real data and samples generated by the generator.
- GANs offer more flexibility in defining the generative model family since the training objective is not based solely on maximum likelihood.
- GANs have the advantage of not requiring likelihood evaluations, making them flexible in choosing the generator architecture.
- Training GANs is challenging due to the Mini-Max optimization problem and convergence issues.
Two-Sample Test
- Two-sample tests are used to determine whether two groups of samples come from the same probability distribution.
- A test statistic is used to compare the two groups of samples, and a threshold is set to determine whether to reject the null hypothesis.
- Choosing a good test statistic is important to minimize type one and type two errors.
- Instead of handcrafting a test statistic to compare two probability distributions, we can train a classifier (discriminator) to distinguish between samples from the data distribution and samples from the model distribution.
- The loss of the classifier can be used as a test statistic, where a high loss indicates that the two distributions are different and a low loss indicates that they are similar.
GANs Training
- The generator is trained to fool the discriminator by minimizing the objective function V.
- The optimal discriminator is the conditional probability of a point x belonging to the positive class (real samples).
- The K Divergence is minimized when P(Theta) = P, but in practice, we cannot reach the global optimum.
- Among suboptimal models, it may be preferable to have one that cannot fool a discriminator over one with high compression.
- A likelihood-based model can be used as a discriminator, but it defeats the purpose of not needing access to the likelihood.
