ChatGPT IS Getting Worse

The full story on what's happening with ChatGPT and how to guard against Prompt Drift; Generate short-form videos with AI

Author

The AI Exchange
July 25, 2023

Welcome to another edition of the best damn newsletter in AI. Here we’ll break down AI topics that matter, open your mind to use cases, and keep you ahead of the curve.

Our #1 goal is to be useful. So please shoot us an email 📩 if you have questions or feedback, and especially if you implement something we share!

Here's what we're covering today:

  • Is ChatGPT getting worse? We’ll cover the actual story.

  • Up your short-form video game with Munch

  • New OpenAI launches + more!

... and if someone forwarded this email to you, thank them 😉, and subscribe here!

Let’s get to it! 👇

TODAY'S PERSPECTIVE

AI Researchers showed ChatGPT “getting worse” over time. What’s the full story?

A recent AI research paper has been making waves that shows GPT-4 and GPT-3.5 have gotten worse at certain tasks since initially measured in March.

One of the main examples:

  • For math problems that use the popular “Chain of Thought” reasoning technique, GPT-4's accuracy dropped from 97.6% to 2.4%

  • At the same time, GPT-3.5's improved from 7.4% to 86.8% from March to June.

While the headlines have been quick to declare that AI models like GPT-4 and GPT-3.5 are losing their edge, the real story is a bit more nuanced.

These models aren't necessarily getting worse; they're evolving.

And sometimes, this evolution can make prompts that used to work - not work anymore.

This is something that we call 'Prompt Drift'.

If you are using AI models like ChatGPT in your work or business - especially if you're wanting to automate with them. Then Prompt Drift is something to pay attention to.

How do you make sure your prompts stay in tip top shape?

Well the first step is to organize and manage your prompts, and create something that we call a 'Prompt Library'.

What is a Prompt Library?

It's exactly what it sounds like: a collection of prompts that you, your team, and your business have vetted and use.

Prompt Libraries help you start monitoring and managing a set of prompts that your team is relying on over time - ensuring that anytime a prompt either gets improved, or if it breaks, that is communicated.

(And as a bonus - prompt libraries are also one of the best ways to begin fostering a culture of adopting AI… The CEO of Stripe recently tweeted about how a third of their company is using their internal prompt library and chatbot tool. 🤯)

But the real magic of prompt libraries isn't just in their day-to-day use. They provide a solid foundation for measuring and monitoring how the performance of prompts changes over time, helping you stay ahead of prompt drift.

Tips for Building a Great Prompt Library 

So, what makes a great prompt library? Here are a few tips that we talk about often:

  1. Tagging: It should have an easy-to-follow tagging system to identify what tasks or business objectives different prompts are designed to solve.

  2. Versioning: It needs versioning to track how prompts improve over time.

  3. Status: It should have a way to mark when a prompt is broken, so that someone on your team can fix it.

Premium members: Read the playbook chapters on Prompt Libraries and Prompt Drift

And related — We're opening up pre-sales for the 3rd cohort of the 3 Week Prompting for AI Ops Bootcamp next week. The first 2 cohorts sold out in 12 hours, so get on the waitlist to get first dibs.

FROM OUR PARTNERS

Boost Your Short Form Video Game with Munch!

Wish you were in the short-form video game but don't have the time to manually clip and optimize them? We've got the tool for you!

Meet Munch, an AI powered tool that effortlessly generates clips from your long-form videos and optimizes them for social media. No more manual work, Munch gives you engaging clips with subtitles and video editing, all done for you.

You can upload videos, or it even works its magic directly on YouTube links. And it leverages AI to find the most engaging clips.

Don't take our word for it! Check out Munch.

And Premium Members: Join our interactive workshop on Wednesday, July 26 at 10am PT and see how Munch's generative AI can turbocharge your short-form video strategy. Register Here! 🔥 

TRENDING MEMBER RESOURCES 🔥 

Premium Members, here’s what dropped and is hot this week!

LINKS

For your reading list 📚

Leading AI products and companies keep pushing the envelope…

OpenAI is shipping too…

And if you're really nerdy...  

That's all!

We'll see you again on Thursday. Thoughts, feedback and questions are much appreciated - respond here or shoot us a note at [email protected].

... and if someone forwarded this email to you, thank them 😉, and subscribe here!

Cheers,

🪄 The AI Exchange Team