Published on

Analyzing My ChatGPT Pro Usage

Authors
  • avatar
    Name
    Max Song
    Twitter

TL;DR: I wrote a Jupyter notebook to analyze my ChatGPT conversation data and answer the all-important question: Is my pro subscription worth it?

This tweet from my favorite AI auhor Chip Huyen (her AI Engineering book is excellent!) made me think about how essential ChatGPT has become in personal and professional lives. As a developer, I've spent more time than I care to admit staring at my Github heatmap - there might even be times when I merged several small fixes as separate commits just to boost my stats 😆.

Whereas Github heatmaps measure my productivity as a developer, a ChatGPT heatmap would actually be a pretty accurate measure of my overall productivity - this is kind of crazy to think about. So, with a mixture of curiosity and trepidation, and spare time on my hands this summer, I decided to run this analysis on myself.

I followed this help article to export my ChatGPT conversation data.

Chip generously shared her heatmap-generating script on her aie-book Github repo. Building on her work, I developed an even more in-depth analysis. You can find my notebook here if you want to follow along. Here are my results:

Conversation Heatmaps

Haha, my ChatGPT conversation indeed have interesting patterns! First of all, I am happy to see that my usage went up a lot in Late August 2024, when I began my master's program. Before that, I was an occasional user. There is an interesting day 2023-11-11 when I sent 60 messages. This is probably me working on my graduate school application lol.

In the back of my mind, I knew I had become a heavy AI user during grad school, espcially this year. However, from this chart, my increase in usage isn't as drastic as I expected. I even made sure the color coding is on the same scale acoss years but the color difference isn't as much as I thougt it would be.

This made me slightly concerned - because I am on the $200 Pro subscription, I really need to use ChatGPT A LOT to get my money back. So, the question is - Did I?

Deep Research Usage

One of the perks of the Pro plan is a much higher Deep Research quota: 125 per month vs 10 of the plus plan. Deep research is a feature I absolutely love and I thought I use it a fair amount so I was curious to analyzed my Deep Research usage as a Litmus test. You can probably imagine my disappointment lol, as you can see in this graph - my usage is nowhere near the maxmium quota. Time to use deep research a lot more haha!

Now, thinking about it, this result makes a lot of sense, because 125/month is 4 requests a day, which is quite a lot for "deep research", and a deep research report can easily take 20mins to digest. I think it would make more sense for me to mentally frame this feature as "very powerful search" and skim through the output or feed it into another AI to scale up my usage.

Token Usage

To calculate the value of my ChatGPT usage based on API pricing I need to count the number of Input and Output Tokens as if I were chatting through the API. To do this, I can't simply add up all the tokens in my messages and count that as input tokens and add up all the tokens in the assistant messages and count that as output tokens. This is because each assistant message is generated using all previous messages in the thread as context. Counting tokens is further complicated by the fact that each ChatGPT conversation is represented as a tree of messages because you can regenerate a response so the conversation can have multiple branches.

Luckily, Claude 4 Opus came up with the following algorithm and code to accurately count all the tokens:

In conversational AI:

  • Each assistant response uses the entire conversation history as input context
  • User messages don't incur API costs (they're just added to context)
  • Input tokens = all previous messages in the conversation
  • Output tokens = only the current assistant response

So, here are my token usage stats, broken down by model:

These are quite interesting charts, not only does the token count better reflect the fact that I started giving ChatGPT more and more diffcult tasks in late 2024 - 2025, it also revealed the dramatic shifts in my model choice and when that started to occur. Notably, the proliferation of model options is a relatively recent phenomenon - which first started with the release of o1 in November 2024. Prior to that, nearly 100% of my usage went to a single model - namely GPT 4 and later GPT 4o. Also, it's intriguing to see once a new generation of model is release, usage of previous-gen models completely stops - models are fast depreciating assets.

Was ChatGPT Pro Worth It?

With token count data calculated and a table of API prices per million tokens for each model, I can calculate the value I got out of my ChatGPT subscription. To my relief - I actually came out quite a bit ahead. This is largely drive by my heavy usage of o1-pro via Repo Prompt and its extremely high pricing in the API 150permilliontokeninputand150 per million token input and 600 per million token output.

With this kind of numbers, it's perhaps not surprising why Sam Altman tweeted they're losing money on the pro subscription (which mostly just access to the o1 pro model at that time), which also explains why the new o3 model is significantly worse at taking in the entire repo and writing ~1000 lines of new code in one shot.

Based on this analysis, it's entirely possible to get amazing value from a ChatGPT subscription if you're a heavy user. Even the numbers I got are definitely an undercount because they don't take into account tool calls - things like web search and code execution. These are ChatGPT's "secret sauce" which are not exposed over the OpenAI API.

However, the value you get depends entirely on how you use your subscription. Deep Research usage is quite valuable. Tasks with large context inputs are valuable. Large models such as GPT 4.5, o1-pro, and o3 are respectively about an order of magniture more valuable than lower tier models.

Is API-cost maxxing a reasonable ChatGPT usage strategy? Probably not, but it sure feels great! 😆

Resources

SHARE