The Rise of Cloud Visibility Platforms: Why SQL is the New Secret Weapon for Enterprise Cloud Management
Over $178 billion is projected to be spent on public cloud services in 2024, yet a staggering 92% of enterprises are concerned about cloud waste. This isn’t a lack of cloud adoption; it’s a crisis of visibility. As cloud estates sprawl across multiple providers and services, traditional monitoring and governance tools are failing to deliver the unified insights needed to control costs, maintain security, and ensure compliance.
The Problem with Siloed Cloud Data
For years, cloud management has been a patchwork of specialized tools. One platform for security, another for cost optimization, and yet another for asset inventory. This fragmentation creates data silos, making even simple questions – like identifying an unused EBS volume attached to a running EC2 instance – incredibly complex and time-consuming. As Yevgeny Pats, founder of CloudQuery, puts it, “The biggest challenge with existing tools is that they’re siloed… making it hard to get a unified view across domains.” This lack of a single source of truth hinders agility and increases the risk of costly errors.
CloudQuery: A Developer-First Approach to Cloud Governance
CloudQuery is emerging as a key player in addressing this challenge. The company recently secured $15 million in funding led by Partech to scale its approach to centralized cloud visibility. Unlike many existing solutions geared towards top-down governance, CloudQuery takes a decidedly cloud governance developer-first approach. It pulls data from over 60 sources – including AWS, Azure, GCP, Okta, and Wiz – into a single, queryable data warehouse.
At its core, CloudQuery leverages the power of ClickHouse, an open-source database known for its speed and efficiency, and the Apache Arrow framework for data analytics. This architecture allows the platform to continuously sync data from various cloud providers, normalizing it into a relational structure. The real differentiator? Accessibility through standard SQL queries and built-in reports. This empowers developers – the individuals closest to the cloud infrastructure – to proactively identify and resolve issues.
SQL: The Unexpected Powerhouse in Cloud Management
While the integration of Large Language Models (LLMs) into cloud management is gaining traction, CloudQuery smartly positions LLMs as an enhancement, not a replacement, for fundamental data access. “We selectively use LLMs for natural language querying, SQL generation and recommendations, but always on top of a foundation of accurate, transparent data,” explains Pats. Because LLMs excel at understanding SQL, they can translate complex business questions into actionable queries, generating customized reports in plain English. However, the underlying accuracy and transparency rely on the robust data foundation provided by SQL and a centralized data warehouse.
Beyond Visibility: Use Cases Expanding Rapidly
Customers are discovering that CloudQuery’s value extends far beyond initial asset inventory. The platform is being leveraged for compliance monitoring, security posture management, and, crucially, cost optimization. This expansion is driven by the flexibility of the SQL-based data model, allowing teams to tailor queries to their specific needs.
Hexagon’s Serverless Data Lake Success Story
Software company Hexagon provides a compelling example. Their cloud center of excellence (CCoE) team successfully built a fully serverless data lake using CloudQuery to collect data from all their cloud accounts. By storing this data in S3 and querying it with AWS Athena and visualizing it with AWS QuickSight, they achieved a unified view of their cloud infrastructure. This implementation not only improved visibility but also allowed them to replace a costly and limited AWS IPAM solution, resulting in significant cost savings and streamlined IP address management. The team’s experience also drove key feature improvements to CloudQuery, including Parquet support and enhanced data partitioning.
The Future of Cloud Governance: Data Mesh and the Rise of the Cloud Data Platform
The success of CloudQuery and similar platforms points towards a broader shift in cloud management: the emergence of the data mesh principle applied to cloud infrastructure. Instead of centralized data teams struggling to serve diverse needs, a data mesh empowers individual teams to own and manage their cloud data as a product. This requires tools that can seamlessly integrate with diverse data sources and provide flexible access through standardized interfaces like SQL.
We can expect to see further innovation in this space, including:
- Increased Automation: AI-powered automation will proactively identify and remediate cloud misconfigurations and security vulnerabilities.
- Enhanced Cost Intelligence: More sophisticated cost analysis tools will provide granular insights into cloud spending, enabling organizations to optimize resource allocation.
- Integration with FinOps Platforms: Seamless integration with FinOps platforms will streamline cloud cost management and accountability.
- Expansion of Data Sources: Cloud visibility platforms will continue to expand their support for new cloud services and data sources, including SaaS applications and edge computing environments.
The era of fragmented cloud data is coming to an end. Organizations that embrace centralized visibility, developer-centric tools, and the power of SQL will be best positioned to unlock the full potential of the cloud and navigate the complexities of modern cloud environments. What are your biggest challenges with cloud visibility and governance? Share your thoughts in the comments below!
