As companies continue to handle the impact of an economic slowdown, we want to shed light on what's really going on in the market. The data function of any organization is critical, so understanding the priorities of key stakeholders, namely the data team, is essential. Rather than a general state of the market, we used this survey as an opportunity to speak directly to the data teams across 300 organizations and ask for their outlook for the year ahead.
With the rise of data apps, we’ve uncovered that data volumes are growing at a tremendous pace, and data practitioners continue to face price, performance and reliability challenges. The billion dollar question is: what will grow faster - the volume of data itself, the users’ expectations, or the technology that supports the usage of this data?
One thing is clear: today’s data teams are tasked with a continual uphill battle to stay two steps ahead, continuously moving and improving performance. Which technology decisions will get them across the line?
To get a true picture of the data function in today’s organizations, we commissioned a survey of 300 data stakeholders, who work in the Gaming, Internet, Software Development, and Marketing & Advertising industries in the USA. In all of these industries, there is a big shift towards data apps (e.g., data intensive, customer facing analytics). This report was administered online by Global Surveyz Research, a global research firm. The survey is based on responses from executives, analytics engineers, data engineers, and data architects, from company sizes between 150 to 3,000 employees. The respondents were recruited through a global B2B research panel, invited via email to complete the survey, with all responses collected during Q3 2022. The average amount of time spent on the survey was six minutes and 10 seconds. The answers to the non-numerical questions were randomized to prevent order bias in the answers.
Nothing is going to stop data from growing exponentially. The total volume of data managed is projected to grow at a CAGR of 41% through 2023, with even more dramatic growth in certain industries, such as Marketing and Advertising. The data shows the complexity of big data is beyond what current solutions can handle. As usage explodes, 71% of companies recognize the expectation for query response time is between 10 seconds and one minute, and yet 70% admit the real-world response time is up to five minutes.
When respondents were asked for their main data-related challenges, price and performance came out on top. However, it’s interesting to compare these with practical considerations when investing in a data warehouse. In this case, cost didn’t make the top five considerations, and top criteria for buying a platform were security (4.12/5) performance (4.10/5) and the availability of a free trial (4.03/5). While cost is a major pain, it is considered as unavoidable when working with data warehouses. As a result, during the selection process of data warehouses, cost considerations rank lower than security, performance, ’try before you buy’, architecture, integrations and more.
As data volumes grow, and user expectations around response time and data usage become more demanding, what does 2023 look like? The atmosphere in the market is that things are slowing down, as seen by the 60% who expect their budgets to stay the same or even decrease. However, decision makers are still planning for growth. Executive team members such as Vice Presidents and Heads of Data are more than twice as likely to see the need for additional budget, compared to Architects and Engineers. In addition, 80% of respondents expect their data teams to grow in 2023 at an average growth rate of 11.8%.
The most coveted analytic feature is ANSI SQL compliance. This could be tied to the hiring and upskilling challenge faced by companies who prefer to standardize their work and remove operational inefficiencies of using proprietary SQL dialects for different platforms.
All survey respondents admitted they have challenges. Clearly, in spite of all the advantages of cloud computing with its consumption-based pricing model and breadth of offerings, the top challenges are price (59%) and performance (56%). Price and performance can be seen as two sides of the same coin. In the current data warehouse world, performance in terms of low latency and high concurrency come at a high price. Unabated data growth and increasing usage will continue to compound this problem.
Price and performance are followed by reliability and scaling at 42% and 41% respectively. A quarter of the respondents cited recruiting as a challenge, which given the global shortage of skilled personnel and a rapidly evolving analytics space is not surprising.
*Question allowed more than one answer and as a result, percentages will add up to more than 100%
Within the segments surveyed, data volumes that companies manage are growing, and are expected to continue to grow from 107TB in 2021 on average, to 213TB in 2023 (a CAGR – Compound Annual Growth Rate of 41%). We then looked at the average data volumes managed by industry (figure 3). The largest growth is in Marketing & Advertising, with a projected CAGR of 54%, while the lowest is still a significant 35%, for the Internet segment.
Survey respondents were asked to evaluate their most commonly used analytics workloads. Our questions were: what query response times are expected in their company for these workloads, and what are their real-world slowest query times in practice?
We uncovered a disconnect between expectations and reality. 71% of respondents believe expectations are for a response time of between 10 seconds and one minute, and yet in reality – 70% say that the response time is five times longer, and that it can take up to five minutes to receive a response. It is clear that technology capabilities come up short against increasing performance expectations.
*Percentages do not add up to 100% due to rounding up of numbers.
We asked survey respondents to rank their top considerations for choosing data warehouse technologies, on a scale of 1 (not important) to 5 (very important).
Top considerations are security (4.12), performance (4.10), and having a free trial (4.03). This can be seen as a checklist of items that need to be met before a solution can be adopted for use, in order of importance. It is interesting to see that while price is the biggest challenge (as shown in figure 1), costs did not make the top five criteria when choosing data warehouse technologies. Companies are willing to pay the price, even if it’s painful, as long as the technology meets security and performance requirements. Simplicity of implementation and maintenance rank lower as most cloud data warehouses are managed services that inherently reduce the effort of operational tasks.
*Question allowed more than one answer and as a result, percentages will add up to more than 100%
We also asked respondents to choose the most important features for data analytics. The most coveted analytic feature is ANSI SQL compliance (42%). This could be tied to the hiring and upskilling challenge faced by companies (as shown in Figure 1) who prefer to standardize their work and remove operational inefficiencies of using proprietary SQL dialects for different platforms.
ANSI SQL is followed by auto scaling (39%), and sub-second response times (29%). These features rank higher than capabilities such as data observability, streaming and others.
*Question allowed more than one answer and as a result, percentages will add up to more than 100%
80% of companies expect their data teams to grow in 2023, at an average growth rate of 11.8%. The data team is a mission critical part of the business, and despite the recession or the impact of the pandemic, only 1% are projecting that their data teams will shrink.
Only 40% of respondents are planning to increase their budgets in 2023 (Figure 8). 42% are planning to keep it the same. When diving into this cohort by role (Figure 9), we see the more senior executives are leading the charge for greater budget, (63% for Head/VP of Data). These executives recognize the planned growth in resources shown in the previous slide (Figure 7), while the practitioners don’t have this vantage point, and aren’t familiar with plans to grow technology-wise. Despite an atmosphere in the market that things are slowing down, executives plan to grow both budgets and teams.
For more information about Firebolt
%20logo.png){kind=icon}
