====== Incremental Learning ======

===== Introduction =====

Incremental Learning(IL) is also referred to as: Continual Learning(CL), Life Long Learning(LLL), Online Learning(OL), Never Ending Learning(NEL).

Incremental learning aims to develop artificially intelligent systems that can continuously learn to address new tasks from new data while preserving knowledge learned from previously learned tasks.(([[https://arxiv.org/abs/2010.15277|Class-incremental learning: survey and performance evaluation on image classification]]))

===== Key Challenge =====

Catastrophic Forgetting(CF): Learning multiple tasks in sequence, however, remains a substantial challenge for deep learning. When trained on a new task, standard neural networks forget most of the information related to previously learned tasks, a phenomenon referred to as “catastrophic forgetting”(([[https://arxiv.org/abs/1904.07734v1|Three scenarios for continual learning]])).

===== Scenarios =====

There are 3 scenarios on Incremental Learning(([[https://arxiv.org/abs/1904.07734v1|Three scenarios for continual learning]])), including:

  * Task Incremental Learning
  * Domain Incremental Learning
  * Class Incremental Learning

===== Methods =====

All methods of incremental learning can be concluded into 4 categories(([[https://arxiv.org/abs/1904.07734v1|Three scenarios for continual learning]])).

  * Task specific components(sub-network per task): **XDG**(Context-dependent Gating)
  * regularized optimization(differently regularizing parameters): **[[deep_learning:incremental_learning:EWC]]**(Elastic Weight Consolidation), **SI**(Synaptic Intelligence)
  * Modifying Training Data(pseudo-data, generate samples): **[[deep_learning:incremental_learning:LwF]]**(Learning without Forgetting), **DGR**(Deep Generative Replay)
  * Using Exemplars(store data from previous tasks): **[[deep_learning:incremental_learning:iCaRL]]**
===== Resources =====

==== Course ====

[[https://youtu.be/rWF9sg5w6Zk|Catastrophic Forgetting by Hung-yi Lee]]

[[https://youtu.be/Y9Jay_vxOsM|Mitigating Catastrophic Forgetting by Hung-yi Lee]]

==== Papers ====

[[https://github.com/xialeiliu/Awesome-Incremental-Learning | Awesome Incremental Learning Papers]]

[[.incremental_learning:incremental_learning_papers | Incremental Learning Papers]]