FAQ

Firebolt is engineered to execute thousands of queries concurrently without compromising speed. It offers unparalleled cost efficiency with industry-leading price-to-performance ratios, and it scales seamlessly to handle hundreds of terabytes of data with minimal performance impact.

Firebolt uses advanced query processing techniques such as granular range-level data pruning with sparse indexes, incrementally updated aggregating indexes, vectorized multi-threaded execution, and tiered caching, including sub-plan result caching. These techniques both minimize data being scanned and reduce CPU time by reusing precomputed, enabling query processing times in tens of milliseconds latency on hundreds of TBs of data.

Data pruning in Firebolt involves using sparse indexes to minimize the amount of data scanned during queries. This allows for tens of millisecond response times by reducing I/O usage, making your queries highly performant.

Firebolt scales to manage hundreds of terabytes of data without performance bottlenecks. Its distributed architecture allows it to leverage all available network bandwidth and execute queries at scale with efficient cross-node data transfer using streaming data shuffle.

Firebolt's aggregating index pre-calculates and stores aggregate function results for improved query performance, similar to a materialized view that works with Firebolt's F3 storage format. Firebolt selects the best aggregating indexes to optimize queries at runtime, avoiding full table scans. These indexes are automatically updated with new or modified data to remain consistent with the underlying table data. In multi-node engines, Firebolt shards aggregating indexes across nodes, similar to the sharding of the underlying tables.

Firebolt's engine uses vectorized execution, which processes batches of thousands of rows at a time, leveraging modern CPUs for maximum efficiency*. Combined with multi-threading, this approach allows queries to scale across all CPU cores, optimizing performance.

_{* Boncz, Peter A., Marcin Zukowski, and Niels Nes. "MonetDB/X100: Hyper-Pipelining Query Execution." CIDR. Vol. 5. 2005.}

_{* Nes, Stratos Idreos Fabian Groffen Niels, and Stefan Manegold Sjoerd Mullender Martin Kersten. "MonetDB: Two decades of research in column-oriented database architectures." Data Engineering 40 (2012).}

Sub-plan result caching allows Firebolt to reuse intermediate query artifacts, such as hash tables computed during previous requests when serving new requests, reducing query processing times significantly. It includes built-in automatic cache eviction for efficient memory utilization while maintaining real-time, fully transactional results.

The Shuffle operation is the key ingredient to executing queries at scale in distributed systems like Firebolt. Firebolt leverages close to all available network bandwidth and streams intermediate results from one execution state to the next whenever possible. By overlapping the execution of different stages, Firebolt reduces the overall query latency

Firebolt first distributes data across all nodes using shuffle and ensures that total cluster memory is used. If this is still not enough, Firebolt selectively spills portions of the data into local SSDs to manage intermediate data structures beyond main memory. This approach allows Firebolt to run queries with working set sizes that exceed main memory, allowing the system to scale even with limited hardware resources.

‍

Firebolt engines can scale up and scale out. For the most demanding high QPS workloads, Firebolt also supports concurrency scaling by allowing additional clusters to be added to the existing engine. Firebolt supports up to 10 clusters within a single engine, enabling concurrency scaling to manage heavy workloads. New clusters can be added on-demand to handle spikes in concurrent queries, ensuring optimal performance under any load.

Everything in Firebolt is done through SQL. Firebolt’s SQL dialect is compliant with Postgres’s SQL dialect and supports running SQL queries directly on structured and semi-structured data without compromising speed. Firebolt also has multiple extensions in its SQL dialect to better serve modern data applications.

Firebolt has full support for array data type, both ANSI SQL/Postgres compliant with support for correlated queries, lateral joins and UNNEST. But Firebolt also has a rich collection of SQL functions, making dealing with arrays simpler and more efficient, including array lambda functions.

Firebolt offers observability views through information_schema, allowing you to access real-time engine metrics. These insights help you correctly size your engines for optimal performance and cost efficiency. Read more here- https://docs.firebolt.io/general-reference/information-schema/views.html

Yes, Firebolt is designed for ease of use, leveraging SQL simplicity and PostgreSQL compliance. It allows data professionals to manage, process and query data effortlessly using familiar SQL commands.

For an in-depth understanding of Firebolt's capabilities, explore the Query Life Cycle Whitepaper, Firebolt Docs, and Pricing information available on our website.

Firebolt provides the following options for importing data:

• “COPY FROM” SQL command to import data from S3 buckets with support for built-in schema inference and automatic table creation.

• “Load data” wizard in the WebUI to explore, set options, infer schema, and load data into Firebolt tables.

• Direct read for CSV and Parquet files from S3 using read_csv and read_parquet table valued functions

• External tables to point to data stored in external Amazon S3 buckets, supported formats include CSV, Parquet, AVRO, ORC and JSON.

Parquet and CSV file formats are supported at the current time. For other formats such as AVRO, JSON or ORC, please use the external table option. 

Firebolt provides the ability to filter on file-level information such as name, modified time, and size using metadata fields: $source_file_timestamp, $source_file_name, $source_file_size. For more information, please refer to the “Load data using SQL” guide.

Streaming ingestion is on the roadmap for Firebolt. To address near real-time ingestion scenarios, Firebolt recommends using micro batching. Various tools such as Kinesis Firehose can be used to persist data to S3 in Parquet or Avro format.

In general, a Firebolt engine with multiple nodes (scale-out configuration). To size an engine appropriately, follow steps below:

‍

1. Start with the smallest node type and number of nodes.

CREATE ENGINE ingest_engine TYPE=S NODES=1;

‍

2. Monitor ingestion time, CPU and RAM by querying ”information_schema.engine_metrics_history”.

SELECT event_time, cpu_used, memory_used
FROM information_schema.engine_metrics_history
WHERE event_time > CURRENT_DATE - INTERVAL ‘1’ day;

‍

3.  If CPU or RAM utilization is high and to exploit parallelism during ingestion consider scaling out your engine. For example, we scale-out to 4 nodes to increase ingestion throughput.

ALTER ingest_engine SET NODES=4;

‍

Yes. Firebolt supports updates and deletes with transactional consistency. Additionally, Firebolt delivers updates and deletes with sub-second latency. For additional information on updates and deletes, please refer to our data management lifecycle blog. You can  also refer to our DML benchmark for more information.

The “COPY TO” SQL command is available to export data to S3. 

CSV, TSV, JSON and Parquet formats are supported.

Yes, Firebolt supports the ingestion and manipulation of semi-structured data types, such as JSON. The JSON data can be ingested either directly into text columns or parsed into individual columns. This flexibility allows for both schema-on-read and flattened into individual columns, depending on the nature of the input data and the desired query performance​. Additionally, array data types combined with lambda expressions can be used to process repeated data present in JSON.

Firebolt optimizes data ingestion performance through:

• Parallel data processing: Firebolt leverages multithreading to read data in parallel for data ingestion.
• Multi-node parallelism: Scaling loading processes as the engine increases in scale.
• Pipelined execution: Ensures all resources are fully utilized by streaming data across transformation stages, improving overall throughput and reducing idle times​.

Firebolt uses transactional semantics and ACID (Atomicity, Consistency, Isolation, Durability) guarantees for data ingestion operations. Each DML execution is treated as a separate transaction. Firebolt ensures that ongoing reads or other operations do not see data being ingested until the transaction is completed and committed, maintaining data consistency and integrity​. There are no partial inserts or copies to clean-up. 

Firebolt provides the ability to transform data during ingestion. You can use standard SQL functions to change data types, perform arithmetic operations, or use string functions to manipulate text data. When ingesting data from external sources, transformations can be applied directly within the INSERT INTO SELECT statement.

Firebolt is fully ACID compliant and treats every operation as a transaction. For example, a COPY FROM operation using schema inference will not present a partial view of the table while the copy is running. The entire table is presented only when the entire COPY operation is successful. In this case, the entire COPY FROM operation is a transaction.

Multistage distributed execution in Firebolt allows complex ELT queries to utilize all resources of a cluster. A stage can be split across different nodes of the cluster, allowing every node to work on a part of the data independently This approach optimizes resource utilization and speeds up data transformation by parallelizing data extraction, loading, and transformation steps.

Firebolt first distributes data across all nodes using shuffle, and makes sure to use total cluster memory. If this is still not enough, Firebolt selectively spills portions of the data into uses data spilling techniques that leverage local SSDs to manage intermediate data structures beyond main memory. This approach allows Firebolt to run queries with working set sizes that exceed main memory, allowing the system to scale even with limited hardware resources.

Yes, ELT processes in Firebolt can be automated using several tools and approaches:

• Firebolt Python SDK: Allows programmatic control over database operations, including data ingestion and transformation.
• Airflow Integration: Firebolt integrates with Apache Airflow, enabling you to schedule and automate complex ELT workflows.
• dbt (Data Build Tool): You can also use dbt with Firebolt for managing transformations and building data pipelines in a version-controlled environment​​.

With Firebolt, you can run your ELT jobs on a separate engine that is isolated from the Firebolt engine supporting customer-facing dashboards. With this isolation, customer-facing workloads are not negatively impacted.

To reduce costs, the isolated ELT engine can be run with auto_stop and auto_start configured to eliminate idle time and can be right-sized to meet the needs of ELT jobs. Firebolt engines can be dynamically scaled to meet the needs of individual workloads. For more information, please refer to the elasticity whitepaper.

‍

‍

Firebolt enables choice of engine shapes to address ingestion and analytics workload through multidimensional elasticity defined by the following attributes:

• Choice of node type (S,M,L, or XL)
• Number of nodes
• Number of clusters

The choice of engine shape for ingestion and analytics workloads comes down to the nature of the workload and the resources requirements (CPU, RAM etc) of the workloads themselves.

‍

Yes, Firebolt supports read-write from multiple engines at the same time. Firebolt leverages three-way decoupling of compute, storage and metadata. As a result, compute is stateless and any engine can read-write from any database. Additionally, the metadata layer enables ACID compliance and provides for strong consistency across engines. 

Yes. All Firebolt engines support read-write from any database. You can query and ingest data from the same engine.

Yes, you can ingest and query data simultaneously on the same Firebolt table. Firebolt engines support concurrent read and write operations on the same table while maintaining strong consistency. Any committed changes made to the data, such as inserts or updates, are immediately visible to all queries across all engines without requiring additional data or metadata synchronization. 

Yes. Engines can be configured to automatically stop after a certain period of idle time. This ensures idle engines are automatically turned off. Additionally, the engines can be configured to start automatically when a new query is addressed at the engine endpoint. These two features are handy to control compute consumption. 

Firebolt achieves workload isolation by decoupling storage from compute resources (engines). Each engine operates independently, allowing multiple workloads to run simultaneously without affecting each other’s performance. This isolation ensures that workloads can scale independently based on their unique requirements without impacting other workloads accessing the same data​.Firebolt also offers strong consistency - ensuring data changes are  immediately visible across all engines.

Workload isolation enables multiple benefits:

1. Eliminates noisy neighbor issues. For example, separating ingestion from customer facing analytics ensures that customer dashboards are not impacted by ingestion jobs.
2. The ability to right size the engine for each specific workload, eliminating potential overprovisioning.
3. Managing costs by eliminating idle time with auto-stop / auto-start feature. Engines do not need to run 24 x 7 and engines are used for the duration of the workload.
4. Ability to isolate access for users/roles from specific engines. For example, restricting internal and external users to specific engines and databases.

Firebolt provides capabilities that are optimized for different workload profiles. For example, batch ingestion might benefit from parallel processing and scale-out while customer facing workloads that rely on aggregations and lookups can benefit from Firebolt’s indexing capabilities. Similarly, Firebolt’s multi-stage distributed query engine is equipped to address long running queries that handle complex transformations. Additionally, Firebolt engines can be granularly scaled to address workload demands.

You can dynamically scale Firebolt engines in three ways:

• Scale Up/Down: Adjust the engine size by changing the node type to a larger or smaller size (e.g., from Large to X-Large or vice versa) depending on memory and CPU requirements.
• Scale Out/In: Increase or decrease the number of nodes in an engine to distribute workload processing across more or fewer nodes. This is particularly useful for handling large data loads or increasing concurrency for query processing​​.‍
• Scale for concurrency: Each engine can be composed of multiple clusters ( up to 10). This provides for linear concurrency scaling without changing engine endpoint.

Firebolt provides observability metrics that help monitor engine performance. Key metrics include:

• cpu_used: Percentage of CPU used.
• memory_used: Percentage of RAM consumed.
• disk_used: Percentage of SSD capacity utilized.
• cache_hit_ratio: Frequency of data retrieval from the local SSD cache versus S3.
• spilled_bytes: Amount of data spilled from memory to disk.
• running_queries: number of queries currently executing
• suspended_queries: number of queries waiting to be executed

These metrics, accessible through the engine_metrics_history view, guide decisions on whether to scale up, scale out, or adjust engine configurations based on current and historical engine utilization​.

• Start Small and Scale Gradually: Begin with a smaller engine configuration and scale based on workload demands.
• Scale Up first before Scale Out: Single node usually will have better performance than multiple nodes for the same FBU price.
• Monitor Engine Metrics Regularly: Use observability metrics to track engine performance and adjust resources proactively.
• Isolate Critical Workloads: Use separate engines for critical workloads to prevent performance degradation due to noisy neighbors.
• ‍Leverage Auto-Start and Auto-Stop: Configure engines to start automatically when a new query is issued and stop after a period of inactivity to optimize costs​​.

Access to engines and workload resources in Firebolt is managed using role-based access control (RBAC). Administrators can define roles and assign them specific permissions, such as the ability to start/stop engines or modify engine configurations. This ensures that only authorized users can manage critical workloads, enhancing security and operational control​.

Yes, more details in our End User License Agreement (EULA) and Data Processing Addendum (DPA).

Yes, our SOC 2 Type-2 + HIPAA report is available subject to a Non-Disclosure Agreement (NDA).

Firebolt is certified for ISO 27001 and ISO 27018. Certification reports are available upon request.

Yes. As a business associate under HIPAA, we support business associate agreements (BAAs) to ensure healthcare data protection. Our SOC 2 Type-2 + HIPAA report is available subject to a Non-Disclosure Agreement (NDA)

‍

A separate BAA with Firebolt is required since our service includes proprietary technology and other sub-processors not covered under the standard AWS HIPAA Eligible Services.

Firebolt is not PCI-DSS compliant and does not permit credit card data storage on its platform.

While Firebolt adheres to NIST SP 800-53, NIST 800-171, and NIST CSF guidelines, we are not currently FedRAMP compliant.

Firebolt processes customer data in compliance with both GDPR and CCPA regulations. We securely collect, store, and manage data according to the highest standards, ensuring that all GDPR and CCPA requirements are met.

For Data Subject Access Requests (DSARs) or any privacy-related inquiries, please reach out to us at privacy@firebolt.io

Yes, our policies, including Disaster Recovery (DR) and Business Continuity Plans (BCP), are tested regularly to ensure effectiveness.

Yes, customer access is managed via Auth0, while organizational access is controlled using Okta. All accesses are logged and monitored, and alerts are in place for any unauthorized configuration changes across our systems.

We use tools like SCA, SAST for code analysis, along with practices such as Fuzzing, scanning for pipeline weaknesses (like the use of unverified external sources), and secret scans as part of our secure software development lifecycle.

We use AWS Shield, WAF, and other logical layers to protect against DDoS. Additionally, we leverage auto-scaling to maintain availability during attacks by dynamically adjusting resources like EC2 instances, ELBs, and other global services capacity. (Though some scenarios may require manual intervention).

Firebolt employs a comprehensive security strategy that includes network security policies, encryption practices, tenant isolation, and governance controls. We are committed to safeguarding your data through state-of-the-art security systems, policies, and practices.

Yes, both IP Allow/Deny-listing is supported. More details on our Network Policy page.

Yes, this feature is supported. more details on our Identity Management page.

Yes, MFA is supported. more details on our MFA page.

Yes, RBAC is supported. more details on our RBAC page.

Yes, customers can choose the region in which they run the service to meet data sovereignty requirements. More on our Available Region page.

Yes, we support encryption for data at rest and in motion. More details are available on our Security blog.

Firebolt database itself inherently reduces the risk of SQL injection by minimizing the use of certain vulnerable constructs, customers are still encouraged to implement additional controls at their application level such as:

1. Ensure all user inputs are strictly validated before being processed. 
2. Escape potentially dangerous characters that could be used in unexpected ways.
3. ‍Include SQL injection tests in your regular security testing and code review processes.

Besides our runtime binary hardening, Firebolt leverages a runtime protection tool that provides deep visibility and protection at the process level.

Customers own their data and can delete it via commands like DROP DATABASE. Regardless, and upon contract termination, all customer data is deleted within 30 days.

Researchers can report vulnerabilities by contacting security@firebolt.io.

Customer data is stored in S3 buckets with high availability and durability. Our recovery objectives are:some text

1. RTO (Recovery Time Objective): 12 hours
2. RPO (Recovery Point Objective): 1 hour
3. SLA (Service Level Agreement): 99.9%

Yes, our insurance includes:

1. Commercial General Liability
2. Workers' Compensation and Employers' Liability
3. Crime Insurance
4. Professional & Technology Errors and Omissions
5. Cyber Security Liability

‍

Yes, we support encryption for data at rest and in motion. More on our technical best practices can be found in our Security blog: “Building Customer Trust: A CISO's Perspective on Security and Privacy at Firebolt”