ChatGPT that can handle 1M+ tokens?

New research shows context limits could be a thing of the past; What's the most AI-ready department?

Author

The AI Exchange
April 25, 2023

Welcome to another edition of what we’re determined to make the best damn newsletter in AI. Here we’ll break down AI topics that matter, open your mind to use cases, and keep you ahead of the curve.

Our #1 goal is to be useful. So please shoot us an email 📩 if you have questions or feedback, and especially if you implement something we share!

Here's what we're covering today:

  • An end to word count limits in ChatGPT?

  • Dive into a use case that's primed for generative AI: customer support

  • Checkout what we're reading in the AI space today

... and if someone forwarded this email to you, thank them 😉, and subscribe here!

Let’s get to it! 👇

TODAY'S PERSPECTIVE

Researchers are designing ways for ChatGPT to do 1M+ tokens

Ever gotten an error when inputting too long of a transcript into ChatGPT? Or been frustrated when ChatGPT forgets the earlier part of your chat conversation?

This is because ChatGPT only remembers a few thousand tokens (or word chunks) at a time. AKA, it has small short term memory.

Short term memory is an important problem for the future of our favorite large language models. Researchers have been working hard on finding cost-effective ways to increase it. And a new AI research paper was just released on a possible approach to scale memory to 1-2M+ tokens (yes millions! 🤯) by basically letting the model learn the meaning of groups of tokens instead of only tokens.

It's still in research phase, but with how quickly this space is moving - we think it’s worth thinking ahead to a future where you could send a TON of information to these models at a time. For example:

  1. Marketers could send raw transcripts of customer interviews and extract the best positioning for a new campaign

  2. Business owners could ask ChatGPT to analyze and redline entire legal documents

  3. Software engineers could feed it an entire code base and suggest vulnerabilities or improvements

  4. Film producers could have ChatGPT analyze a script and draft movie reviews or critic takes, then iterate until they got a script they liked

  5. … or heck, even write the entire script in the first place

While this isn’t a reality yet today — don’t write off your potential AI applications if short term memory is what’s holding you back. Progress is happening.

USE CASE DEEP DIVE

Customer support, the most generative AI-ready profession?

We’ve spoken with 200+ businesses in the last few months on their AI adoption plans and successes. Across the board - one of the most mature use cases is customer support.

What is AI doing in customer support? To name a few ...

  • Drafting responses based on support docs and previous data

  • Helping customers find their answer with a “ChatGPT” for support docs

  • Even going a step further and resolving the issue by giving GPT access to tools that can issue refunds, handle reset password requests, and more

We enjoyed this recent paper which studied the impact of AI on customer service, and found a 14% productivity increase in just the initial rollout.

p.s. if you’re looking to use AI for customer support — feel free to respond and we’re happy to make a product recommendation.

LINKS

For your reading list 📚

Happening in data ownership ...

Happening with our friends at Google ...

And if you're really nerdy...

That's all!

We'll see you again on Thursday. Thoughts, feedback and questions are much appreciated - respond here or shoot us a note at [email protected]. 

... and if someone forwarded this email to you, thank them 😉, and subscribe here!

Cheers,

🪄 The AI Exchange Team