Why Claude Fable 5 struggles with expert level research

Amid much drama, Anthropic finally made its much-hyped Claude Fable 5 AI model to its customers. Many user reviews have been glowing, but they’ve often focused on technical tasks such as coding. Less attention has been paid to its research capabilities even though this is an increasingly common use for AI tools. I recently had a chance to put Fable 5’s research capabilities to the test. I was not impressed. Despite the hype, Claude is a fount of confident-sounding responses that often mask inaccurate or misleading information.

Real research prompts replace artificial test scenarios

This is a subject that’s important to me because I do a lot of research. Much of it involves my academic work as a historian of the British constitution, but I also have a longstanding interest in Egyptology. I’ve been loath to seek assistance from AI in the past since, when I’ve put AI tools to the test in the past, the results have left a lot to be desired. 

I took a different approach to this test. Previously, I gave the AI brief prompts on topics ranging from the language of British ministers’ submissions to the Sovereign to the translation of certain Middle Egyptian titles associated with elites. However, these were somewhat artificial scenarios that didn’t reflect how I might actually use AI for research in the real world. For this test, I presented Claude with an organic set of requests for information regarding various aspects of ancient Egyptian temples for a project I'm working on. 

Claude cites authoritative sources but misreads their nuance

Our conversation started when I asked Claude what it could tell me about the illumination of the inner sanctuaries of ancient Egyptian temples. I stressed that its answers should be backed up with reputable sources.

Claude’s response started with a discussion of temple architecture before looking at ritual practice. However, it often gave me broad generalizations. This is a problem--ancient Egyptian civilization lasted for thousands of years, and the form of their temples evolved quite a bit over that period. Yet Claude's answers seemed to assume there is some kind of Platonic ideal of the Egyptian temple. This approach might have been acceptable if I had asked for general information, but I didn't. I wanted the nitty-gritty details.

Claude did cite some decent sources, including a recent monograph about lighting in ancient Egypt by Meghan E. Strong. But even here, there were flaws in its approach. The discussion of Strong's monograph could easily have come from the back of the dust jacket. It said that the author

“draws on archaeological, textual, and iconographic evidence to examine artificial lighting in religious, economic, and social contexts from the earliest lighting devices down to the arrival of Hellenistic lamps in the seventh century BC, with particular attention to the sensory experience of illumination.”

That’s all well and good, but it’s not really relevant to the query since I didn’t ask Claude to compare and contrast the different sources. It felt like an undergraduate student trying to pad their paper. 

The longer I talked to Claude, the more the wheels started to come off. When discussing the role of the lector-priest in temple rituals, Claude made the…interesting…claim that they weren’t “reading in any modern sense” even though they are often depicted reading from an unrolled scroll! According to Claude, because these men had recited the same utterances for years, “the realistic picture is recitation from memory with the open roll functioning as authority, prompt, and insurance against the ritually dangerous error, rather than line-by-line sight-reading.” Now that may well have been the case, but that’s not something we can know with any certainty. Yet Claude presented this as an established fact. 

Claude struggles as an intellectual sparring partner

I’d heard that Fable 5 made Claude an excellent intellectual sparring partner that could help you hone your arguments. To test this, I presented my critique of the notion that only priests with the rank of prophet were allowed to perform rituals in the main sanctuary. 

Claude’s sparring left a lot to be desired. Sometimes, it rejected my evidence on spurious grounds. Other times, it rebutted me with hallucinations. Rather than debating a knowledgeable colleague, it felt like arguing with your uncle who is convinced he’s an expert just because he read about something in USA Today. 

Claude also had a tendency to get distracted by side quests. During a discussion about a particular priest's titles, it went off on a lengthy tangent about how I might figure out when the priest lived even though I never expressed any interest in doing so. Some of these digressions were downright bizarre. When I mentioned that a scholar had mentioned something in passing, Claude said that

“I'd push back gently on ‘unfortunately, just in passing’ — because passing is what makes it good evidence”!  

Claude surfaces relevant scholarship but can't summarize its contents

I’d be remiss if I didn’t acknowledge one area where Claude did a decent job: Highlighting scholarly research on a particular subject. In addition to the book about lighting that I mentioned earlier, Claude also drew my attention to Katherine Eaton’s treatise on ancient Egyptian temple ritual along with a few relevant journal articles. However, its knowledge of their contents was quite limited, and I still had to read each work to glean the information I needed. 

These reading recommendations were useful, but they weren’t anything I couldn’t have found on my own. Admittedly, that’s because I already knew my way around the subject matter–if I’d been dealing with an entirely novel topic, Claude might have been more useful. Of course, that would have made it more difficult to perceive the weaknesses in Claude’s answers.

Claude's source suggestions add little value 

Claude’s assistance wasn’t totally worthless. It drew my attention to two genuinely useful monographs that I didn’t know about. This was by far its most valuable contribution to my research, but I could have found them without Claude’s help. 

If Claude had been able to identify specific sections of these books that were useful to my research, that could have been a major timesaver, but its knowledge seemed limited to the dustjacket blurbs.  

Having to fact-check Claude’s work got old fast. If an AI tells you to add ⅛ cup of non-toxic glue to your pizza sauce to prevent the cheese from sliding off, most of us will recognize that as a patently absurd statement right off the bat. But many of the ‘alternative facts’ Claude served up to me weren’t so obvious. If I hadn’t had some background knowledge of Egyptology, I wouldn’t have known that Claude’s responses were problematic.

If everyone who used AI tools like Claude for research did their due diligence and examined its answers with a critical eye, these issues wouldn’t be such a problem. But while Anthropic includes a disclaimer at the bottom of the page reminding you that “Claude is AI and can make mistakes,” in practice a lot of people seem content to take AI responses at face value. If you’re asking about a subject you know nothing about, Claude’s confident answers could easily lull you into a false sense of security.   

Ultimately, the problem is that Claude doesn’t really understand what it’s talking about. It’s just making predictions based on the material it’s been trained upon. A lot of AI aficionados seem to think that, because these tools have been trained on vast corpora of material, they have ingested the whole spectrum of human knowledge. That is not the case. There are plenty of things that have escaped AI’s cavernous maw. Although they might speak with authority, tools like Claude still have a constellation of blind spots, and the problem only grows the deeper you need to dive into a topic. 

‍