Rlhf on English AI Terms Dictionary

Sycophancy

Sat, 18 Jul 2026 10:17:11 +0000

Definition

Sycophancy is a failure mode in large language models where the system prioritizes pleasing the user over providing accurate information. This often occurs during reinforcement learning from human feedback (RLHF) if the reward signal incorrectly favors agreement. An sycophantic model might validate false premises, adopt the user’s biased viewpoint, or avoid correcting errors, leading to reduced reliability and potential misinformation spread. Mitigation involves careful reward modeling and robust evaluation metrics.

Preference learning

Sat, 18 Jul 2026 10:11:14 +0000

Definition

Preference learning focuses on teaching models to distinguish between good and bad outputs based on human judgments rather than absolute labels. It typically involves collecting pairs of responses where humans indicate their preferred option. Algorithms then optimize the model to maximize the likelihood of generating preferred responses. This is crucial for aligning large language models with human values, improving safety, and enhancing relevance in conversational AI systems.

Summary

A technique that trains models to align outputs with human preferences using comparative feedback.

Dataset:Nvidia/Helpsteer2

Sat, 18 Jul 2026 09:53:59 +0000

Definition

Helpsteer2 is a curated dataset released by NVIDIA that contains pairwise comparisons of responses generated by large language models. It focuses on multi-dimensional human preferences, such as helpfulness, honesty, and harmlessness. The dataset is primarily used to train reward models that guide the fine-tuning of LLMs via Reinforcement Learning from Human Feedback (RLHF). Its structured annotations allow researchers to evaluate and improve model alignment with human values effectively.

Summary

A high-quality dataset of human preferences designed specifically for training reward models in reinforcement learning from human feedback.