ChatGPT 4o vs. o1 vs. o1 Pro testing results

Published on Saturday, December 7, 2024 By Brad Wardell In Artificial Intelligence

I went ahead and sprung for the $200 a month for ChatGPT Pro as I do a lot of data analysis on very large and complicated data sets.  

The first thing I through at it was last month's full Perforce logs with diffs and and all so that I could get a nice friendly report of the various developers, how many CLs they did and an evaluation on the type of work they did and how much work they did.

Here are my results:

  • ChatGPT 4o.  Basically useless.  It didn't even get the number of CLs right and refused to even try to give a scope of work.
  • o1. Much better, it got the number of CLs right but it couldn't really figure out how much work each did.  But it still gave me a decent overview. 
  • o1Pro.  Very impressive.  The big difference here is that it went and did a relative analysis of each so that the work rating of each was relative to each other.  

So is o1Pro worth it? Not unless you're doing a lot of data analysis and even then... I'm not sure.  

For me specifically it is probably worth it.  There are plenty of tools out there that will do analysis on this kind of thing but the thing is, I don't have to know or care about them.  This is a generalized tool.