.svg)
Listen to this article
We are pleased to announce the general availability of Firebolt’s next-generation Cloud Data Warehouse (CDW), which delivers low-latency analytics with drastic efficiency gains.
Today's data-intensive applications require instant analytics at scale, both for internal operations and external customer-facing use cases. Yet, existing solutions force organizations to compromise. Traditional Cloud Data Warehouses (CDW) often lack the necessary speed to support interactive query experiences, while query accelerators and OnLine Analytical Processing (OLAP) engines struggle to offer the full range of SQL features and elasticity to scale efficiently to changing workloads. This split approach increases complexity and costs, leaving data engineers and organizations stuck managing multiple systems that can’t deliver milliseconds query responses or full SQL functionality. Moreover, data freshness is often compromised - data must be moved frequently from the Cloud Data Warehouse to an accelerator, creating delays and inconsistencies that slow down decision-making and analytics.
Firebolt eliminates this compromise. Our next-generation cloud data warehouse combines the best of both worlds: ultra-low latency, high-concurrency analytics at 100TB scale, and advanced SQL support - all in a single, cost-effective platform. Engineered specifically for low-latency analytics, Firebolt’s platform enables data engineers to achieve milliseconds query performance, handle thousands of concurrent queries, and scale resources seamlessly while ensuring that data remains up-to-date without increasing the total cost of ownership (TCO).
Evolving market dynamics: The shift in data and business needs
As businesses increasingly rely on data to drive decision-making and differentiate themselves, they are also transforming the way that data is used and managed. Two significant trends are emerging that are reshaping the challenges that organizations face:
1. Data is growing exponentially in both volume and diversity
Data is expanding not only in sheer volume but also in complexity. Modern businesses must manage a diverse range of data types, including structured transactional data, semi-structured logs, and unstructured customer interactions. These data types must be ingested, processed, and analyzed instantly to derive actionable insights. By 2025, the International Data Corporation predicts that 180 zettabytes (180 billion terabytes) will be created, captured, copied, and consumed globally by 2025.
2. Business needs are evolving beyond internal optimization to external applications
Previously, data was primarily used to understand and improve internal business operations. Today’s modern organizations also use data to drive external-facing applications, such as customer dashboards, personalized recommendations, live analytics, and more. Businesses are providing data-driven experiences to customers, which means that they need information from multiple sources and interactions with sub-second latency:
These evolving needs create a unique set of requirements for the data infrastructure. First, data infrastructure needs to scale as data continues to grow. As companies look beyond internal use, they are building data-intensive applications that power new customer experiences, unlock new revenue streams, and create competitive advantages. These data-intensive applications require instant (milliseconds) query response times, high concurrency, and efficient scalability at low cost. Lastly, companies continue to prefer cloud-based deployments due to elasticity, on-demand consumption, and scale of resources.
Current solutions and limitations
Cloud Data Warehouses (CDWs) have been focused on optimizing internal business operations, providing deep insights into historical data to enhance business performance. However, existing CDWs struggle to deliver on the modern organization's needs, especially when it comes to millisecond response times and high query concurrency scenarios.
Organizations and businesses have turned to leveraging and implementing hybrid solutions to address these evolving needs, where existing CDWs are augmented with various caching layers. Many solutions are used today to serve user-facing applications, from caching systems like Redis to accelerators like Clickhouse, Druid, and Pinot, and even traditional RDBMSs like MySQL for smaller datasets.
However, while delivering some benefits, this hybrid infrastructure also forces organizations to accept compromises. Let’s review some of the pitfalls for each.
Traditional Cloud Data Warehouse (CDW) limitations
CDWs are not designed to support the new breed of interactive, data-intensive applications that require instant responses. CDWs lack:
- Milliseconds query performance: Response times often range from seconds to minutes, making them unsuitable for applications requiring milliseconds-level performance.
- Insufficient concurrency levels: These platforms need help to support thousands of concurrent queries essential for external-facing applications.
- High efficiency that comes with a high cost: Achieving low-latency responses requires significant compute resources, leading to extremely high costs.
Query accelerators (OLAP Engines) limitations
On the other hand, query accelerators come with their own set of limitations:
- Lack of advanced SQL Support and ACID Compliance: These platforms often need more depth of SQL features, making them less suitable for complex analytics.
- Limited data processing scalability: Solutions do not offer scalable data processing to query large volumes of data at scale, resulting in out-of-memory-like experiences.
- Elasticity constraints: Query accelerators are not built to scale dynamically like cloud data warehouses, leading to limited flexibility.
- Inability to isolate workloads: These solutions do not have built-in workload isolation capabilities. Multiple data copies are required to implement such, which comes with additional cost and complexity.
The compromise: Implications of the hybrid architecture
By adopting hybrid architectures that combine traditional data warehouses with query accelerators, organizations are forced to make the above-mentioned compromises. While this approach provides some level of flexibility and benefits, it results in:
- Increased complexity: Maintaining two systems with unique requirements and configurations increases complexity. Furthermore, each platform requires different skill sets for development and maintenance tasks.
- Higher Total Cost of Ownership (TCO): Running multiple platforms leads to duplicated infrastructure, licensing, and management costs.
- Performance trade-offs: The lack of integration between systems can cause latency, consistency issues, and reduced performance.
- Data staleness: When data is duplicated or transferred between different systems (e.g., from a traditional data warehouse to a query accelerator), there is typically a delay between the updated source data and the target system reflecting those updates. This lag can make data stale, particularly problematic in environments requiring analytics and insights over new and fresh data.
Introducing Firebolt’s Next-Generation Cloud Data Warehouse
To meet these demands, Firebolt is introducing a next-generation cloud data warehouse now available in the US East, US West, and EU Central regions (with additional regions becoming available in near future).
As businesses demand faster and more efficient data processing, Firebolt delivers a robust solution tailored for data-intensive applications. Whether you ingest terabytes of data daily or execute queries across hundreds of TBs of datasets, Firebolt ensures you stay ahead of the curve by providing consistent, milliseconds performance and scalable infrastructure to handle the most demanding applications with high concurrency needs. With Firebolt, you can be confident that your data infrastructure is future-proof.
Firebolt offers:
- High Efficiency
Firebolt is engineered for millisecond query performance while maintaining a low price/performance ratio. This isn’t just theory – our architecture ensures that data insights are delivered faster, even at the petabyte scale, with zero compromise on performance. This efficiency gives organizations the confidence to handle any data-intensive task with ease.
Similarweb processes 100 queries per second on 1PB production data with milliseconds response times. This level of performance is critical for powering their website analytics platform.
Firebolt immediately gave us faster performance at a much greater scale, which let our customers analyze huge datasets with sub-second performance. It also gave us the flexibility to deliver complex data features much faster.- Yoav Shmaria, VP R&D, Platform, SimilarWeb
- Concurrency at Scale
With over 4000 queries per second (QPS), Firebolt ensures that your applications handle heavy query loads with consistent performance. This level of concurrency is essential for data-intensive applications requiring high throughput and performance under heavy load.
Bigabid runs 1 million ad auctions per second and saw query performance improve by 400x after switching to Firebolt, which enabled them to optimize mobile advertising in real time.
- Flexible Infrastructure
Firebolt's fully decoupled architecture allows compute, storage, and metadata to scale independently. With multi-dimensional scaling, users can optimize resource utilization and reduce operational costs. Fine-grain control enables you to scale precisely based on workload needs, avoiding wasteful over-provisioning.
Engines add a huge amount of needed flexibility in the way we provision query power without causing a bottleneck or costing us development cycles. Our application has low latency requirements but with a highly variable number of active users throughout the day. The multi-dimensional scaling provided by Firebolt Engines will allow us to easily deal with this variable number of users, keeping the user experience smooth while minimizing the costs for both compute and development time. Win win! - Eden Trainor, Tech Lead of Machine Learning Engineering at Compass Digital Labs
- SQL Simplicity
In Firebolt, everything is expressed using SQL only. From infrastructure provisioning, through data processing, to management and observability, is all available through a SQL dialect. Best of all, Firebolt SQL support is PostgreSQL-compliant allowing your teams to harness the power of Firebolt without the need to learn a new query language. This ensures compatibility with existing workflows and a simplified support for both structured and semi-structured data analytics.
Diving into Firebolt’s architectural advancements
We made significant improvements to our platform to fully support the needs of modern data-intensive applications and deliver the previously mentioned benefits. Our new architecture provides flexible infrastructure through decoupled storage compute and metadata layers. It also provides a set of data services that can be employed depending on the workload needs. Lastly, Firebolt provides a rich set of APIs and SDKs to satisfy various data application needs.
Flexible infrastructure
With its three-way decoupled architecture, Firebolt fully decouples compute, storage, and metadata layers, allowing each to scale independently. This flexibility has various benefits.
Organizations typically have multiple and heterogeneous workloads, ranging from data loading and ELT patterns to low-latency interactive analytics. The new platform fully supports deploying workloads on different compute (Firebolt Engines), enabling full workload isolation. With complete workload isolation, each workload can be fine-tuned to provide desired and predictable performance characteristics.
As Firebolt engines are stateless, any provisioned engine is fully Read/Write enabled. Furthermore, any engine has complete access to all managed data and databases. Any data or schema modifications made through one engine are fully transactional, and changes are immediately visible through another engine endpoint, leading to both full ACID compliant transactions as well as strong and global consistency.
Each Firebolt engine comes with lots of flexibility. With built-in multi-dimensional elasticity, Firebolt engines can be scaled in three dimensions: up and down, out and in, and for concurrency. In addition, engines support granular scaling mechanisms where capacity can be added in a fine-grain fashion and in increments of 1 node at a time. These capabilities allow Firebolt users to avoid costly scaling operations (without over-provisioning) that are present in other solutions today.
In addition, engine scaling operations are done fully online, leading to impact-less experiences for client applications. This allows each deployed engine to be right-sized for the workload while delivering optimal price/performance characteristics.
Composable data services
We optimized the product for low latency and high concurrency scenarios. Scalability and efficiency are built into every layer of Firebolt’s query processing stack, from the query optimizer that is responsible for evaluating and selecting optimal query plans to our new efficient runtime. This new Firebolt’s runtime allows scalable data processing and supports beyond-main memory query execution, including streaming data shuffling. Firebolt employs various caching mechanisms ranging from solid-state drive (SSD) caches to intermediate and full-result in-memory caches to deliver performance and efficiency. Another key ingredient for efficient querying is data indexing. Firebolt supports primary and aggregating indexes, which enhance to help deliver efficiency for both selective and complex aggregate queries. Lastly, our new multi-stage and distributed query execution engine allows nearly linear scaling as new nodes are added to the system.
Having access to fresh data is an essential requirement for data applications. With rapid data ingestion speeds and low latency singleton DML (insert/update/delete) operations, Firebolt provides diverse utilities for data to be refreshed efficiently. This allows Firebolt to be synchronized with changes that occur in different systems of records so that your data remains both current and readily available.
Elasticity is something that is required and taken for granted today. However, getting insight into actual resource utilization and recognizing situations when to release unutilized resources is not readily available in various solutions. Firebolt provides rich observability functionality where Firebolt engines can be monitored in near real-time to determine current CPU and memory utilization. With rich observability, data engineers can provision the right-sized engines for the workload to be run – new nodes can be added to improve performance due to resource bottlenecks or removed to realize cost savings due to under-utilization.
Firebolt also comes with a layered security model that gives developers and administrators the ability to manage access to the data, all with the ease of SQL. Identity management and integration with external identity to provide a single sign-on experience, multi-factor authentication, network IP allowed and blocked lists, and fine-grained Role Based Access Control (RBAC) allow users with needed flexibility and control to manage access to various database objects.
.png)
Finally, to support the full development lifecycle and activities, we have also introduced the notion of Firebolt Workspaces. The Firebolt Workspace is a versatile development environment that supports collaboration among various roles in the development lifecycle. Data engineers, data analysts, and administrators have tools and features tailored to their needs. Developers can focus on building and optimizing queries, while administrators have access to advanced controls for managing security, performance, and resources. This collaborative environment streamlines workflows, ensuring that teams can work together efficiently while maintaining a focus on individual responsibilities and project goals.
Developer ecosystem
Firebolt offers rich APIs and SDKs that empower developers to build data-driven applications seamlessly. The platform's SQL-first approach makes integration simple with PostgreSQL-compatible SQL dialect, ensuring that complex analytics can be performed without requiring new query languages.
Performance benchmark
Achieve 10s milliseconds in query latency and up to 4000 QPS in concurrency runs with Firebolt's unique architecture -built for low latency analytics, optimized indexes, and data caching. Achieve faster analytics on hundreds of TBs of datasets while scaling effortlessly to support thousands of concurrent users—all with superior cost efficiency. Don’t compromise on speed or scalability. We have constructed a FireNEWT (Nexus for Efficient Workload Testing) benchmark to prove some of these claims. We have used several query patterns that we’ve observed our customers and prospects use when creating this benchmark.
The graph below shows results for one of the benchmarks that test system concurrency: 4 million distinct queries were run against Firebolt across engine configuration with ten L (large node) clusters.
SELECT sourceip, searchword
FROM uservisits
WHERE visitdate = '2024-09-06'
and destinationurl = 'https://www.amazon.com/Tommy-Hilfiger-Casual-Quartz-Watch/dp/B0CM48ZSVC/'
group by sourceip, searchword
As can be seen from the graph, system throughput increases as the new clusters are online. This can be done completely transparently to the actual data-intensive application that sends queries to the system. For more information and other performance numbers, refer to the resources section in this blog.
IQVIA runs complex queries over 1 billion patient records, maintaining milliseconds-level performance, which is critical for its analytics-driven healthcare solutions.
Ezora increased analytics speed by 30x, eliminating the need for pre-aggregations and enabling faster insights.
Conclusion: Powering the Future of Data-Intensive Applications
Firebolt’s next-gen cloud data warehouse is designed to meet the challenges of today’s data-intensive applications head-on. Whether milliseconds query performance, scaling to hundreds of TBs datasets, or handling thousands of queries per second, Firebolt delivers unmatched performance, efficiency, and cost control.
Start using Firebolt today with $200 in free credits and experience how Firebolt transforms data into real-time insights at any scale.
Resources
- Technical blog: How we built Firebolt
- Benchmark blogs
- Knowledge center: Go-to-hub for expert resources and insights
- Technical whitepaper: The Firebolt CDW whitepaper
- Case studies: How organizations are achieving low-latency, high-efficiency analytics at scale
%20logo.png){kind=icon}
