In February 2026, a user asked Carnegie Mellon University Libraries’ AI reference agent a seemingly straightforward question: Did the Libraries offer an audiobook streaming service?
ScottAI said yes.
It was wrong.
The agent had found a 2019 Libraries news article and treated the outdated information as current. Even after the user questioned the response, ScottAI repeated it.
For STEM Librarian Haoyong Lan, the exchange illustrated one of the central challenges for libraries as artificial intelligence reshapes how people find information. Giving an AI system access to authoritative sources is not enough. Those sources need to be current. The system needs to know which one to consult. And when the answer is uncertain, it needs to know what to do next.
Those are familiar problems for librarians. Lan believes that gives his profession an important role not just in using AI, but in designing the systems through which people increasingly discover information.
This idea is driving the development of ScottAI, an AI reference agent built in-house at Carnegie Mellon University Libraries. Developed by Lan with coding contributions from Zeyang Zhang, a master’s student in CMU’s Artificial Intelligence Engineering program, and input from colleagues, ScottAI lets users ask questions conversationally while searching a growing network of Libraries sources.
The project places the Libraries within an early wave of academic libraries developing their own AI-powered reference tools.
STEM Librarian Haoyong Lan demonstrates how ScottAI responds to a research data management question by connecting the user with specialized Libraries expertise rather than attempting to handle the consultation itself.
From menus to conversation
ScottAI grew out of an earlier experiment with conversational technology.
In 2023, the Libraries introduced Ask Andy, a rule-based chatbot that answered common questions through predetermined menus. Lan contributed to the project but saw limitations in the approach.
“The rule-based chatbot was too rigid,” he said. “It did not have the flexibility for users to ask questions in a natural-language style.”
Lan began exploring what that model could mean for library reference.
Rather than purchasing a finished AI reference product from an outside vendor, he led the development of one built around the Libraries’ own information environment. ScottAI launched in July 2025 as a natural-language interface that could answer questions about the Libraries around the clock. In its first year, users logged 1,414 sessions, providing Lan with a growing record of how people encounter and interact with the agent.
The first version drew primarily from Libraries webpages and frequently asked questions. It could converse more naturally than Ask Andy, but it could not search the Libraries Catalog or other dynamic resources.
The audiobook question exposed the shortcomings of that approach. ScottAI had found information, but it had no reliable way to determine whether that information still reflected the Libraries’ services.
Using Google Vertex AI Agent Builder, Lan connected ScottAI directly to live Libraries systems through application programming interfaces, or APIs. The agent can now search several Libraries sources, including the Libraries Catalog, KiltHub, Scholars at CMU, databases and e-resources, and University Archives collections.
When a user asks a question, ScottAI must interpret what the person is seeking, determine which source is most likely to contain the answer, search that source, and translate the results into a conversational response.
“The purpose is to connect users with all of our knowledge infrastructure and help them uncover resources they might never have discovered themselves,” Lan said.
This infrastructure distinguishes ScottAI from a general-purpose chatbot.
Building reference expertise into AI
Every time a new source is connected to ScottAI, the agent needs context about what it contains and under what circumstances it should be searched. Librarians help provide that context, but they are also helping determine which sources ScottAI should be able to search in the first place.
Associate Dean for Academic Engagement Nicky Agate sees those connections as part of an approach she describes as “libraries everywhere.”
“Part of openness is meeting our community where they are,” Agate said. “We've connected ScottAI with KiltHub so students and researchers can ask it for scholarly resources in plain language.”
Agate also advocated for connections to the Libraries’ open access agreements tool, helping researchers explore publishing options, and its YouTube channel, making recorded workshops easier to discover on demand.
Librarians also develop instructions governing how ScottAI responds, including when it should ask a follow-up question instead of assuming it understands what a user means. For Lan, the work builds on his broader efforts to help researchers understand and work critically with AI.
One conversation demonstrated why that matters.
A user asked ScottAI to find research by Charles M. Eastman related to building information modeling. The agent’s initial search returned information about columnist Charles M. Blow, clearly not the person the user was seeking.
ScottAI recognized the mismatch and asked for clarification. As the conversation continued, it eventually directed the user to CumInCAD, a specialized index of research on computer-aided architectural design.
The user did not need to know that CumInCAD existed before asking the question.
“That’s the real reference conversation librarians have with users — continuing to dig deeper, asking what they are looking for and going back and forth until we get the most accurate information,” Lan said.
Instead of requiring users to know which database to search, ScottAI can draw on librarians’ expertise to help connect a research question with the right specialized resource.
The AI reference agent’s name is a portmanteau of Scotty, CMU’s Scottish terrier mascot, and AI.
Putting ScottAI to the test
Building ScottAI has also required an unusual kind of quality control: asking librarians to try to break it.
Faculty librarians have tested the agent using questions drawn from their own subject areas and reference work, looking for places where ScottAI retrieves the wrong information or fails to recognize the limits of what it knows.
The testing has become part of ScottAI's development cycle. When a librarian identifies a weak response, Lan can examine the conversation, determine where the problem lies, and use the example to refine the system.
“The AI chatbot could only be as good as the data it was fed,” Lan said.
Testing has occasionally exposed outdated or incomplete information on Libraries webpages, giving staff an opportunity to correct the source for both ScottAI and people using the website directly.
Knowing when AI isn't the answer
Lan sees the agent primarily as a first point of contact for routine interactions, while librarians focus their expertise on more specialized needs.
“ScottAI can serve as a starting point for users to help find relevant information,” Lan said. “But if users have more in-depth questions — research data management, for example, or discipline-specific questions — that’s where the subject librarian can come in. AI-powered chatbots are not replacing librarians. They’re augmenting our ability to better serve users.”
When ScottAI cannot answer a question, the agent can direct users to the Libraries’ ticketing system for staff follow-up. It can also draw on information about Libraries personnel to connect users with a librarian whose expertise matches their needs.
The goal isn't for ScottAI to answer every question. It's to help users get to the right answer, whether that comes from a Libraries resource or a librarian.
By Shannon Riffe