Meet StableVicuna, The First Large-Scale Open-Source RLHF Chatbot by Stability AI

The development and release of chatbots have been significant in recent months. Open-source alternatives have further fueled interest in tuning large language models for a chat. However, there is a lack of open-source models that have applied both instruction finetuning and reinforcement learning through human feedback (RLHF) training.

In a blog post, Stability AI introduced StableVicuna, the first large-scale open-source chatbot trained via reinforcement learning through human feedback or RLHF. It is a further instruction fine-tuned and RLHF-trained version of Vicuna v0 13b, which is an instruction fine-tuned LLaMA 13b model. The chatbot has been benchmarked against other similarly sized open-source chatbots and has shown strong performance.

To achieve StableVicuna’s performance, a three-stage RLHF pipeline has been utilized. The pipeline involves training the base Vicuna model with supervised finetuning using a mixture of three datasets. A reward model is then trained, followed by proximal policy optimization reinforcement learning to perform RLHF training of the SFT model.

StableVicuna is available on the HuggingFace Hub as a weight delta against the original LLaMA model. Users must have access to the original LLaMA model, which requires them to apply for LLaMA weights separately. A script provided in the GitHub repo can be used to combine them and obtain StableVicuna-13B.

StableVicuna will be deployed as a Discord bot to the Stable Foundation server. Users can try the model on a HuggingFace space by visiting this link. Due to the nature of StableVicuna, feedback is encouraged to improve the user experience and expand bot performance.

Alongside the chatbot, an upcoming chat interface is also in the final stages of development. The development of StableVicuna was made possible, according to Stability AI, by Duy Phung, open-source contributors, and datasets made available by OpenAssistant, Anthropic, and Stanford. They also acknowledged OpenAssistant’s team for providing them with early access to the RLHF data set.

Originally posted on OpenDataScience.com

Read more data science articles on OpenDataScience.com, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Subscribe to our fast-growing Medium Publication too, the ODSC Journal, and inquire about becoming a writer.