Be part of the occasion trusted by enterprise leaders for almost twenty years. VB Rework brings collectively the folks constructing actual enterprise AI technique. Study extra
Gaining visibility — and, in the end, insights — into enterprise cloud property is rising ever more difficult.
Cloud estates are sprawling and fragmented, and stock capabilities in present instruments may be slender and unintuitive, separating components like value and safety information into disconnected platforms with restricted flexibility.
Cloud governance firm CloudQuery is positioning itself to deal with this downside by centralizing cloud property, safety metadata and value in a single place, and making it accessible by way of straightforward, built-in SQL queries and reviews. The corporate is taking a developer-first method to cloud governance, pulling information from 60-plus sources — together with AWS, GCP, Azure, Okta and Wiz — right into a single, queryable information warehouse.
The corporate is now asserting a $15 million funding spherical led by Partech to additional scale its method to cloud visibility.
“The largest problem with present instruments is that they’re siloed — one for safety, one for value, one for asset stock — making it arduous to get a unified view throughout domains,” CQ founder Yevgeny Pats instructed VentureBeat. “Even easy questions like ‘What EBS quantity is connected to an EC2 that’s turned off? are arduous to reply with out stitching collectively a number of instruments.”
CloudQuery beneath the hood
CloudQuery makes use of two key applied sciences beneath the hood: Information warehouse and open-source database ClickHouse and the Apache Arrow framework for growing information analytics purposes.
This high-performance plugin structure inbuilt Go connects on to APIs like AWS, Azure, Google Cloud Platform (GCP) and plenty of different platforms pulling in configuration, safety, and value metadata. The platform constantly syncs information from dozens of cloud suppliers and providers right into a normalized, centralized asset stock.
“We place a powerful emphasis on information accuracy and freshness, syncing at excessive frequency to make sure groups are working with probably the most dependable, up-to-date data,” mentioned Pats.
That information, he defined, is structured relationally to energy CloudQuery’s SQL engine and built-in reviews, in order that groups can have full flexibility with out counting on black-box instruments.
The corporate additionally “selectively” makes use of massive language fashions (LLMs) for pure language querying, SQL technology and suggestions, “however all the time on high of a basis of correct, clear information,” mentioned Pats. He identified that as a result of AI understands SQL properly, instruments like Claude and OpenAI can create personalized reviews and evaluation in plain English.
Taking a developer-first method is crucial, mentioned Pats, as a result of builders are in the end those constructing, working and securing at the moment’s cloud infrastructure. Nonetheless, many cloud visibility instruments have been constructed for top-down governance, not for the folks truly within the trenches.
“If you put builders first, with accessible information, versatile APIs and native language like SQL, you empower them to maneuver quicker, catch points earlier and construct extra securely,” he mentioned.
Clients are discovering methods to make use of CloudQuery past asset stock. “Many begin with visibility, then rapidly develop into use instances like compliance monitoring, safety posture administration, value optimization, all from the identical core platform,” mentioned Pats.
How Hexagon constructed a serverless information lake for all its cloud shops
One enterprise already seeing outcomes is Hexagon. The software program firm’s cloud middle of excellence (CCoE) workforce had a objective to construct a completely serverless information lake that would accumulate information from all of its cloud accounts and retailer it in a single information lake.
Additionally they wished the flexibility to question this information utilizing SQL and visualize it with instruments they have been aware of (equivalent to AWS QuickSight), and discover the historical past of their cloud configuration over time.
The workforce constructed a serverless information pipeline utilizing CloudQuery to gather information from all accounts and retailer it in S3. AWS Glue then ingests information into Glue DB in a format that Amazon Athena can question, which Athena then does, then visualises in QuickSight.
“Having a completely serverless answer was an necessary requirement,” Hexagon cloud governance and FinOps skilled Peter Figueiredo and CloudQuery director of engineering Herman Schaaf wrote in a weblog submit. “This determination introduced plenty of advantages since there isn’t any want for time-consuming updates and nearly zero upkeep.”
They did have to beat some challenges, notably with Amazon S3 assist plugins. The CCoE workforce was one of many first to check out CloudQuery options within the S3 vacation spot and provided insights resulting in new options. These embrace:
- Parquet assist: The CloudQuery file vacation spot initially solely supported CSV and JSON information codecs. Errors in JSON interpretations led CloudQuery so as to add Parquet assist.
- Information partitioning: A CloudQuery file vacation spot plugin now permits partitioning on preliminary write (which beforehand wasn’t obtainable, leading to additional pointless steps).
- Useful resource view for Athena: CloudQuery initially solely provided a sources view for AWS suitable with Postgres. However Athena didn’t assist this, so CloudQuery added a perform that may retrieve an inventory of all tables to construct or replace a sources view.
Figueiredo’s workforce used CloudQuery to exchange AWS’s VPC IP tackle supervisor (IPAM) — which he referred to as costly and restricted in that it doesn’t cowl different cloud suppliers.
Finally, his workforce runs CloudQuery in ‘information lake’ mode utilizing “extremely low-cost infrastructure” together with AWS S3, ECS, Glue, Athena and Lambda,” Figueiredo instructed VentureBeat. This retains prices low and permits Hexagon to merge all its IP addresses throughout completely different cloud suppliers.
“We are able to rapidly question any IP throughout the board and discover who the house owners are,” mentioned Figueiredo. “We are actually capable of accumulate all we’d like at a really low value with close to zero upkeep. That is the holy grail for our workforce.”
