You can use the AUTO_STOP feature available in Firebolt engines to make sure that your engines are automatically stopped after a certain amount of idle time. Engines in stopped state will not be charged, hence do not incur any costs. As with other engine operations, this can be done via SQL or the UI. For example, while creating an engine, you can specify the idle time, using AUTO_STOP, as below:
CREATE ENGINE IF NOT EXISTS MyEngine WITH
TYPE = “S” NODES = 2 CLUSTERS =1 AUTO_STOP = 15;The above command will ensure that MyEngine will be automatically stopped if it has been idle for 15 minutes continuously. Alternatively, you can achieve the same after an engine has been created.
ALTER ENGINE MyEngine SET AUTO_STOP = 15;
Multistage distributed execution allows complex ELT queries to utilize all cluster resources by splitting stages across nodes. This parallelization optimizes resource utilization, speeding up ELT processes.
When using service accounts in Firebolt, access errors can occur due to incorrect credentials, missing permissions, or expired tokens. To troubleshoot these errors, follow these steps:
- Check your credentials: Ensure that the service account credentials (client ID and secret) are correct and active.
- Verify permissions: Make sure the service account has the appropriate role or permissions to access the resources you are trying to interact with, such as S3 buckets or Firebolt tables.
- Refresh authentication tokens: If you're using tokens, ensure they have not expired. Tokens generated for service accounts typically have a limited lifespan.
- Logs and error details: Review the error message logs for more specific information about the access failure.
- Review documentation: Follow the Firebolt documentation on service accounts for proper setup, permissions, and token management.
If the engine has the AUTO_START option set to True, an engine in a stopped state will be automatically started when it receives a query. By default, this option is set to True. If this option is set to False, you must explicitly start the engine using the START ENGINE command.
To work with authentication tokens in Firebolt:
Generate a token via Firebolt’s authentication endpoint using your client ID and secret.
Example:
curl -X POST https://id.app.firebolt.io/oauth/token \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'client_id=YOUR_CLIENT_ID' \
--data-urlencode 'client_secret=YOUR_CLIENT_SECRET'Use the token in API requests by including it in the authorization header:
--header 'Authorization: Bearer YOUR_ACCESS_TOKEN'Refresh tokens regularly, as they expire. Keep tokens secure using environment variables or secret managers.
For more details, refer to the Firebolt API documentation.
Firebolt provides three different observability views that provide insight into the performance of your engine.
1/ engine_running_queries - This view provides Information about currently running queries. This includes whether a query is running or in the queue. For queries that are currently running, this view also provides information on how long it has been running.
2/ engine_query_history - This view provides historical information about past queries - for each query in history, this includes the execution time of the query, amount of CPU and Memory consumed and amount of time the query spent in queue, among other details.
3/ engine_metrics_history - This view provides information about the utilization of CPU, RAM and Storage for each of the engine clusters. You can use these views to understand whether your engine resources are being utilized optimally, whether your query performance is meeting your needs, what percentage of queries are waiting in the queue and for how long. Based on these insights, you can resize your engine accordingly.
Yes, ELT processes can be automated using: Firebolt Python SDK for programmatic database operations. Apache Airflow for scheduling and automating complex workflows. dbt (Data Build Tool) for managing data transformations in a version-controlled environment.
Firebolt enables running ELT jobs on a separate engine isolated from the customer-facing engine. This prevents disruptions and allows scaling ELT engines dynamically with auto-start and auto-stop features to reduce costs.
Monitoring the status of your Firebolt engine using the REST API is a key step to ensure smooth operations. Firebolt provides a way to programmatically check engine status by querying the system engine. This article explains how to authenticate, retrieve the system engine URL, and query the engine status using Firebolt's REST API.To begin, ensure that you have an active service account with the necessary permissions. You will need the service account credentials to generate an access token for API authentication.
After obtaining an access token, use the following request to retrieve the system engine URL:
curl https://api.app.firebolt.io/web/v3/account/<account_name>/engineUrl \
-H 'Accept: application/json' \
-H 'Authorization: Bearer <access_token>'Once you have the system engine URL, you can query it to check the engine's status with a simple SQL query, as shown below:
curl --location 'https://<system_engine_URL>/query' \
--header 'Authorization: Bearer <access_token>' \
--data "select status from information_schema.engines where engine_name = '<your_engine_name>'This will return the current status of your engine, helping you monitor its activity and health.
No, Firebolt doesn't support adding new columns without rebuilding the table. However, you can create a new table with the updated schema or use views to simulate schema changes.
This ensures that the schema remains optimized for performance, which is critical in high-performance analytical databases like Firebolt.
Alternatively, Firebolt offers a flexible approach where you can create views to simulate changes like renaming or restructuring tables without needing to rebuild or re-ingest data. For instance, you can create a view that selects all columns from the original table, effectively simulating the addition of new columns:
Example Usage: To simulate renaming a table or altering its structure, create a view:
CREATE VIEW IF NOT EXISTS new_games ASSELECT * FROM games;This approach allows you to redirect queries to the new view (new_games), making it function like a table with updated schema without altering the original table.
Firebolt uses Firebolt Units (FBU) to track engine consumption. For example, for an engine with Type "S", 2 nodes, and 1 cluster running for 30 minutes, it would consume 8 FBUs:
FBU per Hour = 8 * 2 * 1 = 16 FBUs
Consumption = (16 / 3600) * 1800 seconds = 8 FBUs
Firebolt database itself inherently reduces the risk of SQL injection by minimizing the use of certain vulnerable constructs. Customers are still encouraged to implement additional controls at their application level such as:
- Ensure all user inputs are strictly validated before being processed.
- Escape potentially dangerous characters that could be used in unexpected ways.
- Include SQL injection tests in your regular security testing and code review processes.
CSV file ingestion into an external table may fail with errors such as:
Cannot parse input: expected '|' but found '<CARRIAGE RETURN>' instead.This usually means the file delimiter in the CSV doesn't match the table definition or the number of columns differs.
To troubleshoot check the delimiter: Ensure FIELD_DELIMITER matches the CSV file's delimiter. Also, compare the file to the table definition column-by-column. Finally, if the file is large, create a temporary external table to view the entire row as a single string:
CREATE EXTERNAL TABLE ext_tmp (blob text) URL = 's3://some_bucket/somefolder/' TYPE = (CSV FIELD_DELIMITER=' ');Example Query:
SELECT * FROM ext_tmp LIMIT 2;This helps inspect rows and verify column consistency.
Use the NULLIF function to convert empty strings to NULL, which can then be cast to the appropriate data type.
When casting columns to data types like DATE or NUMERIC in Firebolt, empty strings in the source data can cause errors. This occurs because empty strings cannot be directly cast to other data types.
Use the NULLIF function to convert empty strings to NULL, which can then be cast to the appropriate data type without causing errors.
Example:
INSERT INTO tournaments_nullif_example_fact
SELECT
NULLIF(dt, '')::date
FROM tournaments_nullif_example;In this example, NULLIF(dt, '') converts empty strings in the dt column to NULL, allowing the data to be safely cast to a DATE type. This method ensures smooth casting of columns with empty strings in Firebolt.
Firebolt provides Role-based Access Control (RBAC) to help customers control which users can perform what operations on a given engine. For example, you can provide users with only the ability to use or operate existing engines but not allow them to create new engines. In addition, you can also prevent users from starting or stopping engines, allowing them to only run queries on engines that are already running. These fine-grained controls help ensure that customers do not end up with runaway costs resulting from multiple users in an organization creating and running new engines.
Customer data is stored in S3 buckets with high availability and durability. Our recovery objectives are:
- RTO (Recovery Time Objective): 12 hours
- RPO (Recovery Point Objective): 1 hour
- SLA (Service Level Agreement): 99.9%
To investigate query timeouts or delays, you can start by using Firebolt’s Query History and Query Profile tools, which provide detailed insights into query performance, including execution time, memory usage, and any potential bottlenecks. You can also check engine logs and metrics using Firebolt’s Engine Metrics History to identify issues like memory limitations, network latency, or resource constraints.
For troubleshooting steps, check the Firebolt documentation on query analysis.
%20logo.png){kind=icon}
