Fantasy Baseball AI GM

None of the failures worth mentioning were “the AI is dumb.” They were cases where I’d asked a language model to do something a few lines of deterministic code should have handled instead: check an identity, rank a list, compare two numbers. The model usually got these approximately right, and that turned out to be more dangerous than being obviously wrong, because a wrong answer that looks reasonable is the one nobody catches. The fix, whenever it held, was the same: find what’s actually checkable, check it in code, and let the LLM’s job shrink down to writing good sentences about facts it’s handed.

The clearest example: early in a scoring week, the daily report confidently declared my team “cruising to a likely sweep.” My actual pitching line at that point was one inning pitched, a 27.00 ERA, and a 5.00 WHIP. A single bad outing, read as a commanding lead.

The bug was that the category-comparison logic trusted Yahoo’s own “who’s currently ahead” flag at face value. That flag credits a leader the instant either side posts any value, even a terrible one, if the other side hasn’t posted anything yet. Early in a week, that’s most categories, most days. It wasn’t just a bad headline. That same flag fed directly into the logic deciding whether a category was worth spending a scarce weekly roster move on — so a category that was actually still wide open was getting marked settled, and the system would never flag it as worth acting on. The fix was to stop trusting a “leader” flag when either side hasn’t posted a real value yet. Absent, not zero. A category with no real data on either side reads as unresolved, full stop, no matter what the raw numbers say.

A second failure took three tries before I found the real fix. The model would occasionally leave visible self-correction in a final report — a first recommendation, a “wait, actually,” and a revised one, both left in the email. I tried a single sentence forbidding visible reconsideration. Reduced it, didn’t eliminate it. I tried a more structural instruction: work everything out silently, then write once, straight through. Fixed the specific case that prompted it, then recurred on a different one days later. I widened that instruction to cover the whole report. Still recurred.

Three attempts at solving this through prompt wording had failed. The problem was structural: asking a model to keep its own reasoning invisible inside one continuous stream of text is an unnatural constraint, because there’s no real boundary between “thinking” and “the answer” unless something gives it one. Claude’s extended-thinking mode does exactly that: reasoning happens in a genuinely separate channel that never becomes part of the visible response. Turning it on is what actually stopped the leaking.

It came with its own complication. My first attempt used an adaptive thinking-effort setting at its highest level, and it timed out — a hard platform-level request ceiling, given how much context this prompt carries (every candidate player’s recent stats, projections, and matchup data). A second attempt at a lower effort level hit the exact same wall, at nearly the same elapsed time, which was the tell that the problem wasn’t the effort level at all. “Adaptive” doesn’t bound generation time the way a fixed budget does. Switching to a small, hard-capped thinking budget fixed it: the model still gets a genuine separate reasoning channel, generation finishes reliably in about two minutes, and the leaking hasn’t come back since.

A third bug flipped on the exact day it mattered most. The system’s sense of “how late is it in the scoring week” assumed a Sunday-through-Saturday week; my league’s actual week runs Monday-through-Sunday. That meant Sunday, the real last day of the week, was being read as the first day of a fresh one — and the system would happily suggest spending a move on a category that had already locked in. I fixed the week-boundary assumption. The same symptom came back a few hours later, on the correct day this time. The date math was reading the day of the week in server time instead of my actual US time zone, so near UTC midnight, which falls in the early evening here during baseball season, the calculation would quietly roll over to “tomorrow” before it actually was tomorrow. The fix was to anchor every date calculation to the same already-correct, timezone-aware date string used elsewhere in the system, instead of asking a raw date object what day it thinks it is. A fix that’s correct in isolation can still fail in production because of an entirely separate, adjacent assumption. The fastest way I found the second bug was a real person, at the actual moment it mattered, telling me it was still Sunday evening where they were, and me taking them at their word over what the code confidently calculated.

The last one: a “which available pitcher should I stream” ranking favored volume, two starts beating one, far more strongly than it should have. A pitcher on a real losing streak with two starts that week kept outranking a healthier pitcher with better recent form and only one start. The ranking logic was mathematically fine on quality; it just weighted start-count as an unconditional override instead of a tiebreaker. I re-ordered it to lead with actual recent performance, and let two-start volume only break a near-tie. Even after that fix, the language layer could still pick the lower-ranked option and justify it with a cherry-picked good outing instead of the more reliable larger sample the ranking used — which is exactly what a separate ranking-deviation guard exists to catch: not to stop the model from ever disagreeing with the ranking, but to make sure that disagreement is visible instead of silently trusted.