GPT is something really wide

In the OpenAI available API (https://lnkd.in/gxbtuBk3), GPT-3 represents a fleet of different models that distinguish themselves by size, the data used and the training strategy. The core GPT-3 model (https://lnkd.in/g8MEAuUi) is the source of all those derived models. When it comes to size, OpenAI offers different models that balance the quality of Natural Language Generation and inference speed. The models are labeled with the names of scientists or inventors: - Davinci: 175B parameters - Curie: 6.7B parameters - Babbage: 1B parameters - Cushman: 12B parameters I am missing "Ada", which is supposed to be faster (so I guess smaller) but they don't document its size. The models can be fine-tuned in a supervised learning manner with different datasets. The Codex family is specifically designed to generate code by fine-tuning on public Github repos' data (https://lnkd.in/g2HF42gi). Most of the text generation models (text-davinci-001, text-davinci-002, text-curie-001, text-babbage-001) are actually GPT-3 models fine-tuned with human labeled data as well as with the distillation of the best completions from all of their models. OpenAI actually described those models to be InstructGPT models (https://lnkd.in/gnt9K9pu), although the training process is slightly different from the one described in the paper. Text-davinci-002 is specifically described by OpenAI as being fine-tuned with text data from the Codex model code-davinci-002, presumably performing well on both code and text data. Text-davinci-003 is a full InstructGPT model as it is the text-davinci-002 model further refined with a Proximal Policy Optimization algorithm (PPO) (https://lnkd.in/gsDTWtga), a Reinforcement Learning algorithm. The "GPT-3.5" label refers to models that have been trained on a blend of text and code from before Q4 2021 as opposed to October 2019 for the other models. OpenAI has been using GPT-3 for many specific applications. For example, they trained text and code alignment models (text-similarity-davinci-001, text-similarity-curie-001) to learn embedding representations of those data (https://lnkd.in/gCwdbUdd) in a similar manner than the CLIP model (https://lnkd.in/eHbmBb2t) powering DALL-E 2 and Stable Diffusion. They developed a model to summarize text with labeled data in a very similar manner than InstructGPT has been developed (https://lnkd.in/gdrzdWu3). They also provide a way to extract the latent representation provided by GPT-like models (text-embedding-ada-002). And we know that ChatGPT is a sibling model to InstructGPT trained from GPT-3.5 so it is probably using Text-davinci-003 as a seed.