Overviewing on Meta Learning: What is Few/Zero Shot Learning?
Meta learning, also known as "learning to learn", is an approach to enhance machine learning through machine learning itself. It consists of two kinds of algorithms: the basic learner and meta learner, which help models effectively adapt to new tasks or rapidly changing environments.
While this structure may seem similar to a GANs, there are significant differences in its purpose and operational mechanism. In meta learning, the two algorithms do not compete with each other, but rather focus on cooperating to optimize the learning process and improve adaptability to new tasks.
In meta learning, two types of algorithms work together to enhance learning efficiency. This process operates through two distinct learning levels: basic-level learning and meta-level learning:
In basic-level learning, the algorithm works similarly to standard machine learning and focuses on solving specific tasks or problems.
In meta-level learning, the algorithm learns abstract features and strategies that enhance basic-level learning.
While meta learning encompasses a broad range of research directions and methodologies, we can identify several key topics that form the foundation of this field. These interconnected areas of study have emerged as particularly significant in advancing our understanding of meta learning,
Episode-based training refers to randomly constructing subsets of classes from the dataset during the learning process. Each episode simulates a new task, and the trained model must be able to perform generalization through the query set.
In Embedding Generalization, the model learns how to design points that represent distributions or data clusters. For example, in metric-based meta-learning, the model includes the process of calculating distances between each point in the query set and their means.
Task-Agnostic Knowledge is meta-learning aimed at acquiring knowledge independent of specific tasks. This includes data comparison, the ability to adapt to new tasks with minimal updates, and structures for applying previous experiences to new tasks.
In Generalization Evaluation, the model is assessed on its ability to handle and generalize to unseen classes. This is the stage where performance is measured against a given support set.
Also, meta learning algorithms can be categorized into the following types:
Model-based meta learning uses neural networks or recurrent neural networks with expanded memory to store previous experiences.
Metric-based meta learning facilitates rapid learning for new tasks by utilizing similarity or distance measurements.
Optimization-based meta learning improves learning efficiency for new tasks by learning optimization algorithms.
Task-Agnostic Meta Learning (TAML) focuses on establishing learning strategies related to specific task distributions. This enables effective categorization of various tasks.
Meta regularization, in certain contexts, is considered a meta learning approach. It prevents overfitting and enhances the model's generalization performance, contributing to its robustness across all inputs.
Few-shot Learning and Zero-shot Learning in n-way k-shot Framework and Unseen Classes
The n-way k-shot framework defines specific meta learning scenarios. In this framework, denotes the number of classes/tasks, and represents the number of examples per class. Researchers use this framework to simulate both few-shot learning and zero-shot learning, which are:
Few-shot Learning ( or FSL): The base learner can be effectively trained with only a small amount of data.
Zero-Shot Learning ( or ZSL): The base learner can perform new tasks without any training data.
However, the terms ZSL and FSL appear in contexts beyond meta learning. In fact, many researchers consider meta learning to be a subset of FSL.
Query/Support Set and Episode
The support set, query set, and episode are key concepts in the $n$-way $k$-shot framework that define meta-learning scenarios. Each component plays a specific role in training and evaluation.
An episode is a training iteration where the model encounters new tasks and learns abstract features across the training set.
Query set is a dataset used to evaluate the model's performance. It does not contain class information and is used to measure the model's performance on specific tasks. It serves as test data to evaluate the model's generalization ability in each episode.
Support set is a small dataset that contains class information. It represents specific tasks and provides examples to be used in the corresponding episode.
Unseen Classes and Seen Classes
Unseen classes appear only during evaluation, not during training. This is a core concept in meta learning, where the model must learn regularized features from the training set. For example, a disease recognition model should be able to classify rare diseases even with limited training data. Traditional machine learning struggles with handling unseen classes because it focuses on learning specific training set examples rather than broader patterns.
Unseen Classes refers to categories or classes that are specifically reserved for the training phase and are intentionally excluded from the validation process. These classes serve as novel examples that help evaluate the model's ability to generalize to previously un-encountered data patterns.
Seen Classes encompasses the set of categories that are exclusively utilized during the validation phase of the learning process. These classes help assess how well the model performs on familiar data patterns while maintaining its generalization capabilities.
In some n-way k-shot scenarios, the support set contains unseen classes while the query set contains seen classes. However, this is not a strict rule, as many algorithms use the support set during training.
Last updated