Lurkit is a fast-growing Swedish gaming tech company. Since 2019 we’ve been developing an ecosystem that connects all major stakeholders in gaming for commercial and creative growth. Our mission is to provide game developers, publishers, and marketers with the insights they need to understand and target the right audience, publishing and distributing games to content creators effectively, being able to measure results and ROI.
Our platform is now used by 50,000 people, and processes 80,000 complex queries daily containing many CTEs, joins and aggregations. To keep up with user demands and achieve our business goals, we researched possible technologies and partners in the data analytics space. We're a fast-growing startup and our approach is to adopt managed, scalable solutions that will grow with us.
Consolidating 20 TBs of Gaming Data
We pull data from millions of gaming channels on Twitch, YouTube, and TikTok. This includes everything from follower counts to which games are being streamed and viewer numbers. We then process this data and present it to our customers through a BI tool for the gaming industry. Game developers can automate marketing campaigns, distribute their games to hundreds of content creators, and receive detailed performance insights directly on our platform. This helps them understand which games are trending and who the top content creators are.
Initially, we had our own in-house analytics solution built upon MongoDB. The solution became prohibitively expensive. We also ran into issues where some complex queries simply wouldn’t run, and in other cases, response times were way too slow. We needed a solution that could deliver performance efficiently, handle a high number of concurrent users, and scale cost-effectively. We evaluated several options including Snowflake, BigQuery, and Firebolt, testing each platform on several large and complex queries. Firebolt stood out by delivering the highest performance and cost-efficiency for our use case, as well as being very helpful and involved in our project to migrate all of our data and data processing to them.
Setting up Firebolt for Micro-batch Ingestion
Integrating Firebolt into our data stack was a fairly straightforward process and brought immediate benefits. We ingest new data every ten minutes using the Firebolt Node SDK, as our users rely on updated data. Content creators, for example, want to be able to report back to a publisher shortly after a stream is done. Our previous solution enabled them to do this within 24 hours, whereas now they can get these analytics more or less instantly.
Accelerating Query Performance with Indexing
Firebolt's indexing capabilities are a key element for us. We created an aggregating index which significantly sped up our dashboard performance.This optimization was essential for handling complex queries quickly, even as our data volume grew. We saw huge reductions in query times, making our users’ analytics experience super smooth.
These performance improvements have not only enhanced the user experience but also allowed us to offer more advanced analytics. Our clients now get deeper insights into their player base that were previously impossible. For example, querying contextual data for multiple games in custom time ranges was previously infeasible, and is now actually possible. This new level of insight helps game developers pinpoint exactly the right content creators to work with.
This query shows
Handling Concurrency
Our platform needs to support many users querying data simultaneously, especially during peak times. Firebolt’s architecture allows us to deploy multiple compute engines dedicated to specific workloads, ensuring smooth performance even when demand is high. We saw big improvements in our ability to manage high concurrency. Firebolt's ability to handle high concurrent user loads without slowing down has been crucial for maintaining a high level of service.
These enhancements have allowed us to maintain a consistently fast user experience, regardless of the load on our system.
Optimizing Compute Resources and Costs
Firebolt turned out to be more cost-effective than our previous solutions, with about a 40% lower monthly cost and much higher performance. We have total flexibility and control over resources, so we can adjust the number of compute engines based on demand.
We fine-tune compute resources to balance cost and performance for each workload’s specific needs.
Looking Ahead
Firebolt has been essential in delivering fast insights to our users and allowing us to scale easily as our user base grows. We add new data sources to our platform regularly, and want to move over all of our remaining datasets to Firebolt.
%20logo.png){kind=icon}
