Logo
Picture

GIN @HongIngee

Joined June 2022
  • Tweets
    33
  • Followers
    9
  • Following
    124
  • Likes
    76
GIN retweeted
_akhaliq Profile Picture

AK @_akhaliq

3 years ago

Secrets of RLHF in Large Language Models Part I: PPO paper page: huggingface.co/papers/2307.04… Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, honest, and harmless) assistant. Alignment with humans assumes paramount significance, and reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit. Current technical routes usually include reward models to measure human preferences, Proximal Policy Optimization (PPO) to optimize policy model outputs, and process supervision to improve step-by-step reasoning capabilities. However, due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In the first report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training. We identify policy constraints being the key factor for the effective implementation of the PPO algorithm. Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model. Based on our main results, we perform a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT. The absence of open-source implementations has posed significant challenges to the investigation of LLMs alignment. Therefore, we are eager to release technical reports, reward models and PPO codes

Trends for United States

Fauci
X Money
Tractor Supply
George Kirby
Appleton
T-Mobile
Cracker Barrel
Menasha
Cruz Montana
Mike Santana
Bryan Kohberger
Fairhope
Joc Pederson
SLY COOPER
Big T
Emily Wilson
Henderson
Rural King
NO RECESS
Welbeck

You might like

Elon Musk
@elonmusk
241.0M Followers
Dad, husband, President, citizen.
Barack Obama
@BarackObama
119.2M Followers
Welcome to the official account of Cristiano Ronaldo.
Cristiano Ronaldo
@Cristiano
112.2M Followers
45th & 47th President of the United States of America🇺🇸
Donald J. Trump
@realDonaldTrump
111.8M Followers
Rihanna
@rihanna
98.1M Followers
Making the seemingly impossible, possible. ✨
NASA
@NASA
92.2M Followers
@skylrk
Justin Bieber
@justinbieber
91.3M Followers
Original song
Taylor Swift
@taylorswift13
82.4M Followers
MAYHEM, INCLUDING “THE DEAD DANCE”, OUT NOW
Lady Gaga
@ladygaga
73.9M Followers
A proud husband and father ❤️
Virat Kohli
@imVkohli
71.3M Followers
@SKIMS
Kim Kardashian
@KimKardashian
70.2M Followers
like & subscribe
YouTube
@YouTube
68.7M Followers
Sharing things I'm learning through my foundation work and other interests.
Bill Gates
@BillGates
64.4M Followers
Filho de Deus, Pai, Feliz e Ousado !
Neymar Jr
@neymarjr
64.2M Followers
It’s our job to #GoThere and tell the most difficult stories. For breaking news, follow @CNNBRK and download the CNN app ➡️ https://t.co/7PQD7o6fLw
CNN
@CNN
61.8M Followers
“In The Dark” & “I Said I Love You First...And You Said It Back” Out Now
Selena Gomez
@selenagomez
61.6M Followers
what's happening?!
X
@X
60.8M Followers
Breaking news from CNN Digital. Check @cnn for all things CNN, breaking and more.
CNN Breaking News
@cnnbrk
60.0M Followers
Office of the Prime Minister of India
PMO India
@PMOIndia
58.4M Followers
News tips? Share them here: https://t.co/ghL9OoYKMM
The New York Times
@nytimes
53.7M Followers