Key trends shaping data strategy: Insights from Big Data LDN and Snowflake DCWT 2024

In recent months, I have had the opportunity to represent CACI at two major data conferences—Big Data London and Snowflake’s Data Cloud World Tour in London. CACI took a prominent role at both events, including serving as a Black Diamond Partner at the Snowflake tour. I spoke at both events, sharing insights from our latest projects and discussing the strategic shifts shaping the data landscape.

Both conferences were electric with activity, demonstrating the strength and vibrancy of our data community. The CACI booth saw a steady stream of professionals eager to discuss our perspectives on the future of data and technology, as well as how we’re tackling current trends and challenges across industries. These interactions highlighted three key themes resonating within the industry today.

• Snowflake functionality including sharing and Container Apps

One of the questions I was asked a lot at the events was how organisations can use native data sharing functionality with Snowflake more widely. Snowflake has progressively made this easier to do, by enabling direct warehouse-to-warehouse sharing of data. Moreover, the launch of Container Apps has meant that organisations can also consolidate more processing within Snowflake, rather than handing off to processes outside of where they store this data. Increasingly, Snowflake adopters are seeking to consolidate more of their data technology and data within one common platform. This obviously increases efficiency with data processing and makes your data processing estate easier to manage.

I was inundated with questions about how we can support organisations in making more from their Snowflake estate, from supporting on enabling direct sharing and private cleanrooms, to migrating workloads to native Snowflake apps. It’s clear that this will be a core strategic push for many organisations already on Snowflake, but also a driving force behind many decisions to migrate to Snowflake.

• The rise of data Self-Serve

It was quite staggering to see how many exhibitors at Big Data London were software vendors selling self service data insight platforms, many of them powered by GenAI. The desire to give non-coders access to query data held within a warehouse/lake etc is nothing new, there has forever been this requirement. Hitherto, that access has been slightly hamstrung by the capabilities of the tools aiming to give that capability, often giving a reductive or clunky user experience. However, this sector has grown momentum recently, driven by increased data volume and by the increased desire for more data by “non-technical” people.

It’s clear that GenAI, in part, has given renewed energy to this sector, and we’ll see these type of natural language tools start to really challenge the incumbent BI tools.
One of the omnipresent challenges of the data industry has been how to seamlessly give access to data, held within data platforms, and easily give access to people not proficient in writing code. It is one of the areas that has held back true democratisation of data, and limited initiatives to create thoroughbred “data-driven” organisations. If GenAI can help by removing these barriers to data accessibility, then we’ll enter a new age of data use, and organisations will really be able to start fully leveraging all the data that they have at their disposal.

• Importance of Observability

In 2024, observability—essentially, next-level monitoring—is becoming indispensable. We find ourselves in a place where our data estates are becoming ever more expansive and expensive. The volume and cardinality of data that is being collected and processed is ever increasing, contributing to a degradation of transparency about what is happening on our data platforms. Moreover, the now ubiquitous consumption pricing model has meant that there is now increasingly need for better observability, to enable widespread transparency on what processes are running, their efficiency, how much compute they are using, and how much they cost.

We’ll see an increasing push to streamline data processing operations and to create better clarity on what processes are running and how much they are costing the business. Additionally, as data operations but ever more business critical, there will be additional scrutiny their performance and health.
In summary, these three trends—expanded Snowflake capabilities, the advancement of self-service data tools, and enhanced observability—reflect a broader industry shift towards greater accessibility, efficiency, and cost transparency in data management. As we continue to innovate and adapt, CACI is positioned to support organisations in navigating these changes and unlocking the full potential of their data assets. Together, these insights underscore a vibrant future where data is not only accessible but fully optimised to drive meaningful business outcomes.

Curious to explore how these trends could shape your organisation’s data strategy? Let’s start a conversation about unlocking the full potential of your data estate with CACI. Reach out today to discover more.