Skip to main content



The LAION-AI/Open-Assistant github repository aims to provide a diverse and accessible collection of datasets that can be used to train OpenAssistant models.
Our goal is to cover a wide range of topics, languages and tasks.

To simplify the training process, all data must be UTF-8 encoded.

Current Progress

To see the datasets people are currently working on, please refer to the spreadsheet.

Repository Structure

  • Each dataset is organized into its own folder, which may include notebooks, processing scripts, markdown files and other materials that explain the dataset creation process
  • The dataset files themselves are stored on Hugging Face
  • The root lists the dataset names and corresponding Hugging Face datasets
  • The final version of each dataset is pushed to the OpenAssisstant Hugging Face

Dataset Formats

To simplify the training process, all datasets must be stored as Parquet files with the option row_group_size=100 and index=False.
There are two types of datasets accepted: instruction and text-only.

Instruction format

Instruction datasets are designed to align language models with human interactions. These can take the form of question-answer, request-response, task-solution pairs, and so on. The instruction dataset must include the following columns:

  1. INSTRUCTION (string): Instruction text
  2. RESPONSE (string): Expected response to the instruction
  3. SOURCE (string): Original data source short name, e.g. "wikipedia"
  4. METADATA (JSON string, optional): Any other useful information stored in JSON
    For example, NSFW content can be marked as {"nsfw": true}

Text-only format

For datasets that do not fit into the instruction format, text-only format is proposed. The text-only dataset must include the following columns:

  1. TEXT (string)
  2. SOURCE (string)
  3. METADATA (JSON string, optional)

Dataset Requirements

The dataset must adhere to the following requirements:

  • Must have a permissive license
  • Must not contain child sexual abuse materials
  • Must not contain materials with private individual's personal information (e.g. name, address, phone number, government ID, or medical information)

How to Contribute

To add a new dataset to OpenAssistant, follow these steps:

  1. Create an issue: Create a new issue and describe your proposal for the new dataset.

  2. Create a dataset on Hugging Face: Create a dataset on HuggingFace. See below for more details.

  3. Make a pull request: Add a new dataset loading script to this folder and link the issue in the pull request description. For more information, see below.

Creating a Dataset on Hugging Face

To create a new dataset on Hugging Face, follow these steps:

1. Convert your dataset file(s) to the Parquet format using pandas and pyarrow libraries:

import pandas as pd

# Create a pandas dataframe from your dataset file(s)
df = pd.read_json(...) # or any other way

# Save the file in the Parquet format
df.to_parquet("dataset.parquet", row_group_size=100, engine="pyarrow", index=False)

Make sure the text data in the dataframe is properly encoded as UTF-8!

2. Install Hugging Face Hub

pip install huggingface_hub
pip install 'huggingface_hub[cli]'

3. Log in to Hugging Face

Use your access token to login:

  • Via terminal
huggingface-cli login
from huggingface_hub import notebook_login

4. Push the Parquet file to Hugging Face using the following code:

from datasets import Dataset
ds = Dataset.from_parquet("dataset.parquet")

5. Update the Hugging Face file

Update the file of your dataset by visiting this link: (paste your HuggingFace name and dataset)

Making a Pull Request

1. Fork this repository

2. Create a new branch in your fork

3. Add your dataset to the repository

  • Create a folder with the name of your dataset.
  • Add files that describe your dataset and its creation, such as a README, notebooks, scrapers, etc.
  • Add your dataset to the parent
"dataset_name": "your_huggingface_name/dataset_name"

4. Stage your changes and run the pre-commit hook

pre-commit run

5. Submit a pull request

  • Submit a pull request and include a link to the issue it resolves in the description, for example: Resolves #123