Managing cost and usage of resources and data assets, especially in an era of potential economic downturn, is becoming a necessity and part of typical responsibilities for any IT organization. This is all the more important in the cloud world, where resources are spun up and down with dynamic access and cost management needs. In this blog, we introduce a simple yet flexible Firebolt
object model that organizes and aligns your data warehouse resources to your operating model.
Today, companies are challenged with establishing a solid governance model to manage their cloud data warehouse resources. Data security, cost management, resource isolation, and observability are among some of the challenges that various organizations face that a governance model must address.
For example, development, staging and production environments require isolation from each other to limit the blast radius of any development changes spilling into production by mistake, or to limit a developer’s access to only their code base and data. Departments in a company need to isolate and provide access to only their teams while limiting access to other departments. Another requirement may be to provide consolidated billing, but with the ability to understand consumption by department or development environment.
To address these requirements, Firebolt supports concepts of organizations and accounts. These two concepts are represented as two unique objects within the object model hierarchy.
But first - let’s introduce some core principles followed by our object model.
The Firebolt object model is hierarchical and comes with strong containment properties — parent objects can contain one or more child objects. Child objects are sole children of their parent objects (and cannot be shared). Furthermore, there are two classes of objects: global and regional. As the name suggests, global objects are managed globally and can contain objects that are deployed and grouped regionally.
The picture below depicts a Firebolt supported object model:
An organization, as a top-level and a global object, groups all objects together.
It contains a few similarly global organizational entities, and can contain one or more accounts that are deployed regionally. Conceptually, the organization helps enforce global policies in the form of system level authentication and principals accessing the system - it manages Single Sign On (SSO) and network related security/protection, as well as being used for billing consolidation purposes.
An organization also simplifies management by consolidating multiple accounts under a single logical entity.
The following Firebolt functionality is tied to the organization:
On the other hand, accounts are designed to depict physical representation of your data warehouse implementation. As regional objects, Firebolt accounts can be created within any of the Firebolt supported regions (for more information about supported regions refer to our documentation). All account children objects are contained within the region that account was created in.
Accounts contain objects that help you implement access control and authorization policies within your data warehouse. They also house your data model that is managed and deployed within your data warehouse through objects such as databases, tables, views, and indexes. And finally, compute used to support your data warehouse operation is represented by engine objects. Firebolt engines are a unit of economics that help you govern the spend and price/performance characteristics of your data warehouse. Accounts are identified and referenced
by the name. They are unique within a context of one organization and enable the following functionality:
In summary, organizations and accounts provide you with flexibility and structure to fit your needs. They allow better resource allocation, fine-grained access control,
and overall easier management of your data warehouse. During the registering process, Firebolt allows creation of your organization and the first account while giving you the opportunity to create additional accounts either manually or programmatically as you go.
Sections below describe some common use cases and scenarios that organizations and accounts help address.
When data applications are being built, engineers typically develop in different environments than where the end product will be running. IT organizations typically follow best practices in creating one environment for development, one for staging, and one for production. This is done to achieve environment separation and isolation, protect production environments from unintended changes, and to orchestrate holistic Continuous Integration / Continuous Delivery (CI/CD) pipelines and streamline deployment processes.
As the development cycle progresses from the development to staging and production environments, engineering teams need to run corresponding workloads in the relevant environment to validate data applications behave as intended. Furthermore, development environments typically do not have very stringent performance needs and Service Level Agreements (SLAs), unlike production environments. Similarly, development sandboxes may come with less-restrictive permissions to allow developers to experiment with new features, unlike production environments that need to be tightly managed and protected.
Let’s explore each environment and needs.
To support all of this, each environment requires different resources and configurations to serve its needs and purpose. With organizations and accounts, it is easy to support these needs by creating a separate account for each environment. Engineering teams now have full control and can navigate the intricacies of data applications rollout to production with precision and efficiency.
It’s very common for any company, irrespective of its size, to have multiple departments – finance, IT organization, marketing and so on. For bigger organizations, this may come with different manageability functions, where dedicated IT organizations support independent department operations. Furthermore, different departments may require data to be physically segregated. As an example, sensitive financial data can only be seen and accessed by the finance department while market insights can only be seen by marketing. Or, various departments may be geographically dispersed requiring physical data separation and access for compliance reasons. EU privacy regulations, that force data locality and require complete and geographical data isolation, fall into this last category.
For situations like these where companies want to achieve physical departmental separation and manageability while enjoying a unified view over operations of all those departments, an organization and accounts can be leveraged. Given that Firebolt accounts can be deployed regionally, companies using Firebolt can enjoy deployment flexibility while satisfying stringent compliance requirements. This way, departments get needed independence while the organization enforces a global security policy, such as Single Sign On, across all departments.
With organizations and multiple accounts in Firebolt, it is easy to deliver on the above requirements and needs; simply create a different account for each department. Within each account, create users, and databases, load data, and grant privileges to the new objects so your teams can work independently and efficiently.
Quick and accurate cost management is a critical concern for businesses. As Firebolt introduces the concept of organizations, it’s important to understand how this new framework can support billing and chargeback policies.
Managing costs across multiple accounts and teams can be a complex and time-consuming endeavor. With the introduction of the organization object, Firebolt simplifies the process of monitoring consumption and allocating costs across those various accounts. A Firebolt organization is linked to AWS Marketplace for billing purposes. By linking accounts with an organization, you seamlessly achieve unified billing while preserving the ability to uniquely identify charges per each account. This brings several benefits:
.png)
The introduction of organizations opens the door to implementing flexible chargeback policies that align with your company’s structure and goals. Here are some key considerations to keep in mind as you start tackling this need:
By leveraging unified billing and implementing flexible chargeback policies, your company can establish a robust framework for accurate cost allocation, streamlined reporting, and optimized resource usage.
Getting started with organizations requires simply registering with a valid business email address. Once that’s done, an organization is created with your first account. Shown below are the basic steps to establishing the foundational elements of your organization and accounts.
Let’s say John from Acme has registered with Firebolt. During the registration process, Acme organization and his first account are automatically created. Now, John wants to create an additional account named AcmeBI. He runs the following SQL command to do so:
CREATE ACCOUNT AcmeBI;
To verify the account was created, he runs:
SELECT * FROM INFORMATION_SCHEMA.ACCOUNTS WHERE
account_name = ‘AcmeBI’
// running above statement enables John to see that an account named AcmeBI exists in the system.
Now, John wants to invite his team members to join the Acme organization and take Firebolt for a spin. Since John’s company is using Okta as an identity provider, he can simply configure his Firebolt organization to integrate with Okta so that eligible users from Okta can access Firebolt. To do so, John goes into the ‘Configure’ space in the Firebolt Workspace and sets up SSO by executing the following statement:
ALTER ORGANIZATION SET SSO = ‘{
“signOnUrl”: “https://abc.okta.com/app/okta_firebolt_app_id/sso/saml”,
“signOutUrl”: “https://myapp.acme.com/saml/logout”,
“issuer”: “Okta”,
“provider”: “Okta”,
“label”: “Okta”,
“fieldMapping”: “mapping”,
“certificate”: “XXXXXXXXXXXXXXXX” }’
Now, John wants to grant the team members access to the test account. To do so, he creates a login and a user for each team member. At this point, John’s team members can log in using SSO, and they can easily access the test account.
Here’s an example of how to create a login:
CREATE LOGIN ‘kate@acme.com’ WITH FIRST_NAME = ‘Kate’ LAST_NAME = ‘Peterson’;
To create users and link them to appropriate logins, John runs the following command:
CREATE USER kate WITH LOGIN_NAME = ‘kate@acme.com’;
Now, John creates his first database in the test account:
CREATE DATABASE my_db;
USE DATABASE my_db;
In addition to that, he creates an internal table called “rankings” and loads it with data using the COPY statement.
CREATE TABLE IF NOT EXISTS rankings (
GameID INTEGER,
PlayerID INTEGER,
MaxLevel INTEGER,
TotalScore BIGINT,
PlaceWon INTEGER,
TournamentID INTEGER
) PRIMARY INDEX GameID, TournamentID, PlayerID;
COPY
INTO rankings(gameid $1
, playerid $2
, maxlevel $3
, totalscore $4
, placewon $5
, tournamentid $6
)
FROM ‘s3://firebolt-sample-datasets-public-us-east-1/gaming/parquet/rankings/’ WITH PATTERN = ‘*’ TYPE = PARQUET;
Now, John want to enable Kate to run selects and inserts over the new table so he grants her with appropriate privileges:
GRANT SELECT ON TABLE rankings to marketing_role;
GRANT INSERT ON TABLE rankings to marketing_role;With organizations, Firebolt provides a structured framework for managing accounts, authentication, databases, engines, etc. Moreover, organizations address security, resource allocation, usage, and cost management challenges. Organizations demonstrate their versatility in catering to diverse business needs through distinct use cases, including development/staging/production environments and departmental separation. With unified billing ability and support for implementing chargeback policies, organization and accounts simplify cost allocation and enhance financial transparency. In a world where efficient data governance is paramount, organizations offer a holistic approach to streamline processes, bolster security,
and optimize resource utilization.
For more information about Firebolt
%20logo.png){kind=icon}
