In an era where data processing demands are skyrocketing, AT&T has redefined its approach to artificial intelligence to manage an astounding usage of 8 billion tokens per day. Chief Data Officer Andy Markus and his team recognized that relying solely on large reasoning models was neither practical nor cost-effective for such massive scale operations.
To address this, AT&T embarked on a significant overhaul of its internal AI systems, specifically developing the Ask AT&T personal assistant. This initiative led to the creation of a multi-agent stack using a framework called LangChain. In this architecture, larger language model “super agents” oversee smaller, specialized “worker” agents that perform targeted tasks. This realignment has resulted in substantial improvements in latency, speed, and response times, achieving cost savings of up to 90%.
Markus expressed his belief that the future of agentic AI lies in numerous smaller language models (SLMs), which can be just as accurate as larger models within specific domains. Recently, his team employed this newly architected stack in conjunction with Microsoft Azure to launch the Ask AT&T Workflows, a graphical drag-and-drop tool designed to help employees automate various tasks. This tool utilizes AT&T’s proprietary capabilities for document processing, natural language-to-SQL conversion, and image analysis, enhancing decision-making by leveraging AT&T’s own data.
“As the workflow is executed, it’s AT&T’s data that’s really driving the decisions,” Markus noted. The approach emphasizes asking questions specific to their data rather than general inquiries, ensuring that the focus remains sharp on relevant information. Despite the automation, human oversight is crucial; all agent actions are logged, and role-based access controls are in place throughout the process.
Smart Design Choices in AI Implementation
Markus highlighted that AT&T avoids a “build everything from scratch” mentality. Instead, the company adopts a strategy of using “interchangeable and selectable” models, which allows them to phase out custom-built solutions in favor of more efficient off-the-shelf options as they evolve. “In this space, things change every week, if we’re lucky, sometimes multiple times a week,” he stated. This adaptability is vital for integrating new functionalities swiftly.
AT&T’s rigorous evaluation of available options has led to impressive milestones, such as their Ask Data with Relational Knowledge Graph outperforming competitors on the Spider 2.0 text-to-SQL accuracy leaderboard. The company is likewise making strides with other tools scoring high on benchmarks like BERT SQL.
While the push towards advanced AI tools is evident, Markus cautioned against overengineering solutions. He advocates for a pragmatic approach—assessing whether a task genuinely requires complex AI solutions or if a simpler model would suffice. “Accuracy, cost, and tool responsiveness should be core principles,” he advised, emphasizing that even as solutions become more intricate, these foundational elements should guide development.
Employee Engagement and Productivity Gains
The Ask AT&T Workflows tool has already been implemented for more than 100,000 employees, with over half reporting daily usage. Active users have experienced productivity increases of up to 90%. Markus noted, “We’re looking at whether they utilize the system repeatedly because stickiness is a good indicator of success.”
Within this tool, employees have two engagement paths: a pro-code option for those comfortable with programming and a more accessible no-code option featuring a drag-and-drop interface. Interestingly, even highly skilled participants at tech-focused hackathons showed a preference for the no-code option, highlighting the tool’s user-friendly design.
Employees are deploying agents across various functions. For example, network engineers utilize agents to manage alerts and reconnect customers when connectivity issues arise. One agent can analyze telemetry to pinpoint a network problem, check logs for known issues, and initiate a trouble ticket. Subsequent agents can propose solutions, even generating new code for patches, while another documents the resolution process for future reference. Human oversight ensures that these automated processes align with operational expectations.
Transforming Software Development with AI
This same methodical approach extends into AT&T’s coding practices, now termed “AI-fueled coding.” Developers are leveraging agile methodologies alongside function-specific archetypes in integrated development environments (IDEs). The result is high-quality code that can reach near-production standards in a single iteration, minimizing the back-and-forth typically associated with coding.
Markus has described this AI-enhanced coding process as “tangibly redefining” the software development cycle, leading to shorter timelines and greater output of production-grade code. Non-technical teams are also benefiting, as they can use simple language prompts to create software prototypes. For instance, a recent project that would have traditionally taken six weeks to develop was completed in just 20 minutes using AI tools.
As AT&T continues to innovate and adapt in the AI landscape, the company’s forward-thinking strategies not only enhance operational efficiency but also empower employees by simplifying complex tasks. The ongoing evolution of AI capabilities will likely yield further enhancements in productivity and engagement across the organization.
For readers interested in the future of AI and its applications in the workplace, AT&T’s transformative journey offers valuable insights into how large organizations can navigate the complexities of advanced technology while enhancing workforce productivity.
