We Tested Kimi K3 on Real Investment Research Questions. It Refused Most of Them.

We Tested Kimi K3 on Real Investment Research Questions. It Refused Most of Them.

Kimi K3 launched this week — Moonshot AI's 2.8-trillion-parameter mixture-of-experts model with a 1-million-token context window, the largest open-weights model ever announced. The benchmarks are frontier-level. The pricing undercuts everyone. The hype is deserved.

So we did the obvious thing: we pulled eight real questions our users asked Atlantis this month — not cherry-picked prompts, actual production queries in three languages — and ran them through K3.

The result surprised us twice. Once because of what K3 did. And once because of what we did.

The setup

• 8 real user prompts from Atlantis production logs: a DCF valuation, an earnings deep-dive, a technical analysis in Chinese, a Brazilian equity comparison in Portuguese, a micro-cap health check, an index question, a valuation-theory question, and a crypto comparison.
• Same day, both sides. Atlantis answered in the production app; K3 answered via the Moonshot API (`kimi-k3`), out of the box, the way a user experiences it.
• Full disclosure: K3 ran without tools. That's not a rigged test — that is the comparison. A chat model versus a research product built on live financial data. The question is what grounding is worth.

Where the 2.8T model tapped out

K3, in 11 seconds:

> "I don't have access to real-time market data, so I can't tell you where the S&P 500 closed yesterday. My knowledge has a cutoff date, and I don't know today's date."

Honest. Correct behavior, even. Also completely useless if you're an investor asking the question.

K3 produced a genuinely well-structured analysis — of Q3 FY2025, reported November 20, 2024. It even flagged this itself: "my knowledge ends in late January 2025." That's six quarters behind. NVIDIA's actual latest quarter (Q1 FY2027, reported May 20, 2026) came in at $81.6B revenue — more than double the $35.1B quarter K3 analyzed. An investor acting on K3's "latest earnings" would be trading on a company that no longer exists.

Atlantis's same-day answer: Q1 FY2027, $81.62B revenue (+85% YoY), $75.2B data center, 75% gross margin, ~$91B Q2 guidance versus $86B consensus — each figure sourced, and each one checks out against NVIDIA's own 8-K.

A ~$14M OTC micro-cap. K3 correctly identified the company, then declined to give numbers and offered a how-to-research-it-yourself framework. Reasonable — and again, not an answer. Atlantis pulled what actually exists (OTC disclosures, press releases), rated the company's financial health 2.5/10, and wrapped the whole thing in an explicit uncertainty warning because the underlying data is thin. That's the job: the best honest answer available, with the confidence level attached.

Where K3 genuinely shines

Fairness matters: on the valuation-theory question ("what is PEG, what's a reasonable level?" — asked in Chinese), K3 was excellent. Clear formula, Peter Lynch's original logic, the variants, the failure modes. Timeless knowledge is exactly what a giant model is good at, and pretending otherwise would be dishonest.

If your question has no expiry date, a frontier chat model is a fine tutor. Almost nothing in markets has no expiry date.

The part where we flunked our own test

Here's the chapter we didn't plan.

When we first ran "where did the S&P close yesterday" through Atlantis, it answered 5,659.91 — confidently, with a polite hedge. The real close was 7,533.77. Our answer was a mid-2024 index level, off by roughly 25%.

Root cause: our symbol resolver had no concept of market indices. It fetched nothing, no guard fired, and the underlying model filled the gap from its training memory — the exact failure we exist to prevent, one asset class over from where we'd built the guardrails.

K3 refused the question. We fabricated politely. On that query, the refusal was the better answer.

We shipped the fix the same day: index questions now resolve to live tracked-ETF data (SPY, QQQ, DIA, IWM), memorized index levels are banned at the prompt level, and a regression test locks the behavior in. The re-run, in production:

> "The S&P 500 closed at 7,533.77, down 38.63 points (−0.51%) on the day" — cited, correct to the decimal, and up front that the index level is web-sourced while its own tools track SPY (which closed −0.51%, consistent with the index move).

What we took away

• The model is not the product. The data layer is the product. A 2.8-trillion-parameter model without grounding loses to a much smaller model with live data on almost every question investors actually ask.
• Honest refusal beats confident staleness. K3's transparency about its limits is the right behavior — and it's also a ceiling. A research tool has to clear that ceiling with real data, not vibes.
• Benchmark yourself before you benchmark others. We went looking for content and found a bug. The test paid for itself before we published a word.

Every claim in this post is reproducible: ask K3 and Atlantis the same eight questions and compare. That's the whole point.

---