CASE STUDIES & RESEARCH

We Test Before We Recommend

ABOUT US

We test before we recommend. This is where we publish the experiments behind our advice: what we tested, how we tested it, what we found, and where the results stop.

How we run these studies

  • Data we can show you. We use fictional or anonymized company records, so every number and question can be published.
  • Answers checked in advance. Each question has a verified correct answer before any AI sees it.
  • Repeated, not cherry-picked. We run tests more than once and report what we got, including the results that did not flatter anyone.
  • Limits stated up front. Every study says what it did not test, so you can judge how far the results apply to your business.

Research: AI and company data (a two-part study)

We gave four AI setups the same fictional coffee-company records, saved four ways: plain text, Markdown, JSON and CSV. Two of the setups ran on open-weight models on our own server; two used Claude Sonnet 5 in the cloud. Then we asked 40 questions with known answers.

Tested November 2026. Results describe these model versions at that time.

PART 1 · TESTED NOVEMBER 2026

Same data, four formats. Does file format change the AI’s answers?

No format consistently won. The best format changed from one AI setup to the next. Format did change file size and reading time: JSON was the largest file and the slowest to read.

Read Part 1 →

PART 2 · TESTED NOVEMBER 2026

Which AI setup made the difference?

Turning on thinking took Claude Sonnet 5 from about 70% to 100%. On our own server, the open-weight Qwen3.6 27B reached about 93%. Every setup declined all 10 questions the records could not answer.

Read Part 2 →

Each AI was given the whole file at once; no embeddings, search or RAG were used. These results do not measure the accuracy of a retrieval (RAG) system.

Client case studies

Client case studies are coming to this page. Until then, see Our Clients and Our Work.

Want a question tested on your own data?

We run the same kind of test on your records, privately, before you commit to an AI tool or vendor.

LET'S TALK