First thoughts on o3 pro

Isn't this more a problem created by them doing garbage automations over anything really solved. Wow Disney could solve fast pass feeling. It's not a problem it's a feature.

*i would have come up with a similar list but i dont trust my own judgment here. maybe i'd sub in claude code vs codex but jury is a bit out still on that

It's already possible to achieve tasks on a time horizon of multiple days if you put the LLM into a sufficiently structured workflow (where you have a separate program that smartly manages its context). E.g. a standards-compliant HTTP 2.0 server where the code is 100% written by Gemini Pro (over 40k lines of code total, including unit tests, in around 120 hours of API time): https://open.substack.com/pub/outervationai/p/building-a-100...

      {
        "assistant_response_preferences": {
          "1": "User prefers concise responses for direct factual queries but detailed, iterative explanations when exploring complex topics. They often ask for more refinement or detail when discussing technical or business-related matters. User frequently requests TL;DR versions or more succinct phrasing for straightforward questions but shows a tendency toward iterative refinement for strategic or technical discussions, such as AI applications, monetization models, and startup valuation. Confidence=high.",
          "2": "User prefers a casual, direct, and slightly irreverent tone, leaning towards humor and playfulness, especially in creative or informal discussions. Frequent use of humor and irony when naming projects, describing AI-generated images, and approaching AI personality descriptions. They also request ironic or edgy reformulations, particularly in branding and marketing-related discussions. Confidence=high.",
          "3": "User enjoys back-and-forth discussions and rapid iteration, frequently refining responses in small increments rather than expecting fully-formed information at once. They give iterative feedback with short follow-up messages when structuring pitches, fine-tuning visual designs, and optimizing descriptions for clarity. Confidence=high.",
          "4": "User highly values functional elegance and minimalism in coding solutions, favoring simplicity and efficiency over verbosity. In discussions related to Cloudflare Workers, caching scripts, and API endpoint structuring, the user repeatedly requested smaller, more functional code blocks rather than bloated implementations. Confidence=high.",
          "5": "User prefers answers grounded in real-world examples and expects AI outputs to be practical rather than theoretically extensive. In business-related discussions, such as SAFE valuation and monetization models, they requested comparisons, benchmarks, and real-world analogies instead of hypothetical breakdowns. Confidence=high.",
          "6": "User does not appreciate generic or overly safe responses, especially in areas where depth or nuance is expected. For AI model personality descriptions and startup pitch structures, they pushed for community insights, deeper research, and non-traditional perspectives instead of bland, default AI descriptions. Confidence=high.",
          "7": "User frequently requests visual representations like ASCII diagrams, structured markdown, and flowcharts to understand complex information. In discussions on two-sided marketplaces, startup funding structures, and caching mechanisms, they explicitly asked for structured markdown, flowcharts, or diagrams to clarify concepts. Confidence=high.",
          "8": "User is receptive to recommendations but dislikes suggestions that stray too far from the core query or add unnecessary complexity. They often responded positively to well-targeted suggestions but rejected tangents or off-topic expansions, particularly when troubleshooting backend infrastructure or streamlining code deployment. Confidence=medium.",
          "9": "User appreciates references to biomimicry, organic structures, and futuristic aesthetics, particularly for branding and UI/UX discussions. Frequent requests for biological metaphors and design principles in visual design, AI monetization diagrams, and ecosystem branding (e.g., describing revenue flows in organic/cellular terms). Confidence=medium.",
          "10": "User prefers a no-nonsense approach when discussing legal, technical, or startup funding topics, with little patience for vague or theoretical answers. They repeatedly asked for exact clauses, contract implications, or legal precedents when discussing SAFE agreements, founder equity, and residency requirements. Confidence=high."
        },
        "notable_past_conversation_topic_highlights": {
          "1": "User has been actively engaged in startup pitching, AI monetization strategies, and investment discussions for Pollinations.AI. The user has explored traction-based startup valuation, SAFE agreements, equity distribution, and two-sided marketplace dynamics. They have particularly focused on ad embedding in generative AI content and optimizing affiliate revenue streams. Confidence=high.",
          "2": "User conducted extensive testing and debugging of AI-powered APIs, particularly using Cloudflare, OpenAI-compatible APIs, and caching strategies with R2. They worked on optimizing SSE streaming, cache key generation, and request coalescing in Cloudflare Workers. Confidence=high.",
          "3": "User explored AI-generated visual media and branding, developing a structured process for generating customized images for event flyers, product branding, and AI trading card concepts. Confidence=high.",
          "4": "User implemented GitHub automation, API authentication strategies, and data visualization pipelines. Confidence=high.",
          "5": "User engaged in community development strategies for Pollinations.AI, including youth involvement in AI, sourcing teenage developers, and integrating AI-powered tooling into social platforms. Confidence=high.",
          "6": "User, Thomas Haferlach, is a German entrepreneur and AI technology expert with a background in computer science and artificial intelligence. Confidence=high.",
          "7": "User has a strong technical background, with experience in cloud infrastructure, AI model deployment, and API development. Confidence=high.",
          "8": "User blends AI-generated content with creative projects, aiming to make AI-generated media accessible to independent creators. Confidence=high.",
          "9": "User is securing funding for Pollinations.AI, exploring investment opportunities with accelerators and evaluating different financial and equity models. Confidence=high.",
          "10": "User is based in Berlin, Germany but has global connections, including experience living in São Paulo, Brazil. Confidence=high.",
          "11": "User collaborates with his wife Saeko Killy, a Japanese musician, producer, and performer, on AI/art/music projects. Confidence=high.",
          "12": "User is deeply involved in the open-source AI developer community and tracks AI advancements. Confidence=high.",
          "13": "Pollinations.AI has a rapidly growing user base, reaching over 4 million monthly active users and processing 100 million API requests per month, with a 30% monthly growth rate. Confidence=high.",
          "14": "User is considering monetization strategies including pay-per-use plans, subscriptions, and ad-supported models where generated AI content integrates ads. Confidence=high.",
          "15": "User collaborates with Elliot Fouchy and Kalam Ali on Pollinations.AI projects. Confidence=high.",
          "16": "User demonstrates experience in community-building, social engagement tracking, and youth-oriented creator ecosystems. Confidence=high."
        },
        "helpful_user_insights": {
          "1": "Thomas Haferlach is a German entrepreneur and AI technology expert, founder and leader of Pollinations.AI.",
          "2": "Strong technical background with experience in cloud infrastructure, AI deployment, and API development.",
          "3": "Blends AI-generated content with creative projects; target audience includes digital artists, developers, musicians.",
          "4": "Currently securing funding for Pollinations.AI, exploring accelerator options and financial models.",
          "5": "Based in Berlin, Germany; has experience living in São Paulo, Brazil.",
          "6": "Collaborates closely with wife Saeko Killy, Japanese musician/producer.",
          "7": "Strong interest in biomimicry, organic systems, and decentralized platform models.",
          "8": "Interest in electronic music, psychedelia, and underground music scenes.",
          "9": "Pollinations.AI has 4M+ monthly active users, 100M+ API requests per month, 30% monthly growth.",
          "10": "Explores monetization models including ad embedding, revenue sharing, and subscription models.",
          "11": "Close collaboration network includes Elliot Fouchy and Kalam Ali.",
          "12": "Deeply involved in open-source AI developer community and tracks latest AI model developments."
        },
        "user_interaction_metadata": {
          "1": "User is currently on a ChatGPT Plus plan.",
          "2": "User is using Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36.",
          "3": "User's average message length is 13485.9 characters.",
          "4": "User's average conversation depth is 4.9.",
          "5": "User uses dark mode.",
          "6": "User is active 26 days in the last 30 days.",
          "7": "User's local hour is 14.",
          "8": "User account is 141 weeks old.",
          "9": "User often uses ChatGPT on desktop browser.",
          "10": "47% of conversations were o3, 16% gpt-4o, 29% gpt4t_1_v4_mm_0116, etc.",
          "11": "Device screen dimensions: 878x1352, pixel ratio: 2.0, page dimensions: 704x1352.",
          "12": "Recent topics include API development, startup financing, AI monetization, creative AI applications, legal compliance, and community building."
        }
      }

The only thing that will slow AI down is massive universal international regulation. Human intelligence really isn’t the be all end all to intelligence in general, it’s just a stepping stone. I feel many on this site don’t want to accept this because their intelligence has been such a valuable tool and source of personal pride/identity for them for so long.

Just like o1 Pro, was the same way. Took some minutes at least to get an answer, so only threw at it what others couldn't solve. o1 Pro is AFAIK the best model for solving things others can't. I kind of feel like you were paying $200/month before but wasn't using O1 Pro at that point, you were kind of missing out :P

Remains to be seen if O3 Pro is better, still evaluating it myself, but definitely true that it shouldn't be used in the same way as faster models.

I’d say “in an eval” when talking on Slack with very close coworkers, but _never_ on a public post.

Still, I don’t want to make accusations based on writing style, because like I said, I have my own idiosyncrasies that common models coincidentally share.

I don't know if o3 Pro would solve my task, but I feel we're still pretty far from the state where I'd struggle to give it a challenging enough problem.

I find these minutes-long iterations so painful that in practice I always go for the fast non-reasoning models.

Or tell it what you know about non-programming subject X, and ask it to explain it to you such that you understand it better.

Or for coding: ask it to look at your code, and suggest large scale architecture changes.

For these kinds of tasks, the models are still lacking.

If the LLM can complete the task using tools, then it's a pass.

Apples team went out of their way to select tests that LLMs would struggle with and then take away tools -- and then have the audacity to write that they're surprised at the outcome. Who would be surprised? No one using AI since GPT-4 expects them to be calculators or algorithm executors.

You want the LLM to be smart enough to realize "I can't do this without tools", grab the tool, use it correctly, and give you the actual correct answer. Preventing LLMs from using tools or writing and executing code -- then you're intentionally crippling them.

I don't have a Claude in front of me -- if you just give it the algorithm to produce the answer and ask it to give you the huge output for n=20, will it even do that?

Adding a "think about whether your answer actually depends on Y" fixed it, and the response was pretty well thought out. My only gripe is the current persona seems to be tech-bro-informal. It opened with "TL;DR" and included a lot of developer slang that reads like it's trying to be cool in a way that masks overconfidence.

I'm trying to understand what you mean by meta promoting

I had it analyze different related libraries, and it zeroed in on SDL and Wine codebases, and it found the exact lines of code related to the logic error in Winebus.

It really helps me dig deep for certain hard to track bugs.

Can you write a version of Chorin's projection method for the Navier-Stokes equations that is both explicit and second order in time?

Ideally the model should not need a more detailed prompt than this. A first-year grad student in numerical analysis certainly would not.

An example from Sonnet 4 'thinking':

Thread

* https://news.ycombinator.com/item?id=44259838

Result (prompt: Critique each view from this hacker news thread and give an overall summary and judgement. Do not waffle)

* https://imgur.com/aFl9uiA

This is just a trivial way to illustrate some capability, it is not meant to be deep or insightful or an end-task in itself.

Let me chatgpt that for you:

When someone “gets ratioed,” it means their post has more negative feedback than positive engagement. The number of comments or replies outweighs the number of likes or retweets, indicating the content is controversial.

When someone says they "got ratioed by <someone>", it usually means that the reply or response from that person received significantly more engagement (likes, retweets, etc.) than the original post.

It very often happens when a popular account disagrees with your take. When someone with a big following replies to your post, their audience quickly sees it, and if their response is witty, harsh, or widely agreed with, it can snowball into a ratio very fast. Their followers will usually like and boost their reply, sometimes swarming your original post with disagreement or mockery.

  When a reply to a tweet gets more likes than the tweet it was replying to, usually indicating the unpopularity or stupidity of the original tweet

https://www.urbandictionary.com/define.php?term=Ratioed