不断增加训练数据并不必然能不断提高深度学习和大模型的整体性能。事实上这类基于梯度的数据驱动模型的可解释性很差，没有人能预见新的训练数据会对模型之前“学到”的信息造成何种潜在影响。有时候，这种影响是剧烈而负面的，即所谓的“灾难性遗忘”。虽然最近大模型在文本理解和生成方面取得了明显的进步，但“幻觉”问题依然是普遍存在的，目前看不到有效的根治办法。一段由AI生成的文本是一本正经的胡说八道，也许问题不大。但是，一个由“智驾”控制的汽车操作序列，也是一本正经的胡说八道呢？呵呵。奉劝智驾吹们收敛一些。如果你们想自己当小白鼠，没人阻拦。但不要误导他人。

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning arXiv:2312.10549

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang

Catastrophic forgetting (CF) is a phenomenon that occurs in machine learning when a model forgets previously learned information while acquiring new knowledge. As large language models (LLMs) have demonstrated remarkable performance, it is intriguing to investigate whether CF exists during the continual instruction tuning of LLMs. This study empirically evaluates the forgetting phenomenon in LLMs' knowledge during continual instruction tuning from the perspectives of domain knowledge, reasoning, and reading comprehension. The experiments reveal that catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters. Moreover, as the model scale increases, the severity of forgetting intensifies. Comparing the decoder-only model BLOOMZ with the encoder-decoder model mT0, BLOOMZ exhibits less forgetting and retains more knowledge. Interestingly, we also observe that LLMs can mitigate language biases, such as gender bias, during continual fine-tuning. Furthermore, our findings indicate that ALPACA maintains more knowledge and capacity compared to LLAMA during continual fine-tuning, suggesting that general instruction tuning can help alleviate the forgetting phenomenon in LLMs during subsequent fine-tuning processes.
--
修改:qtpr FROM 111.167.211.*
FROM 111.167.211.*