Navigating the Big Opportunities and Challenges in Data and AI

The Biggest Challenge Data and Analytics Leaders Face in the Coming Year?

It was loud and clear that leading a data team isn’t just about the analytics—it’s about handling the data tidal wave that’s rolling in daily. One of the biggest challenges we are all facing is managing the data volume, the data complexity, and the cost of keeping it all secure and integrated.

As each of our daily lives become more data-driven, companies collect and rely on more data than ever. But with this comes the critical need for top-notch data quality, privacy, and governance practices. The biggest issue here are slip-ups in data quality or ethics that can create unreliable insights or worst case a major breach with catastrophic consequences.

In New Zealand, this challenge is felt by leaders across all sectors, but perhaps most keenly in healthcare and finance. Leaders in this space are all looking for ways to standardise their data handling, tackle integration issues, and establish frameworks that’ll keep everything in line.

It was described in one instance as “a bit like taming a wild beast—challenging but doable if everyone works together”.  It means that collaboration between Chief Data Officers (CDOs) and Chief Technology Officers (CTOs) is becoming even more vital as they strive to break down silos, make data accessible across departments, and keep it safe and ethical.

Onto the second question. Much of the forum centred around AI and the question asked was…

Why Data Analytics and AI Are Such Exciting Fields to Work In Today

The major enthusiasm around data and AI is because these fields are now so incredibly accessible. It’s not just for our tech giants, highly trained computer science engineers or our stats majors, but for anyone with a smartphone or computer.

AI is quickly levelling technology, making it accessible to those without a data science background. Want to create a chatbot? Done. Need insights from a mountain of data? Tick. With user-friendly tools and powerful AI, curiosity and a willingness to try new things is all you need as a starting point.

Thanks to advancements in AI and machine learning, it’s becoming easier to analyse data in real-time, uncover patterns we never thought possible, and make more informed decisions on everything from product recommendations to healthcare diagnoses.

With agriculture being such a key industry to us, one example was given of Kiwi farmers using AI-powered sensors to monitor soil conditions, so they know the best time to plant. In healthcare it was providers using data to personalize treatments.

Small businesses everywhere are getting in on the action, leveraging AI-driven tools to understand their customers better and stay competitive. AI is taking the guesswork out of things and will allow companies to focus on what matters most. And what does matter most? By general consensus, it was delivering great experiences, driving innovation, and creating value.

What are the Big Opportunities in Data and Analytics Right Now?

Short-term, there are some opportunities that Data & Analytics  leaders are diving into straight away. The top three are:

1. Data Monetisation and Sharing: More companies are realising their data is valuable, not just for internal use but also as a shared resource. Picture this—a data marketplace where companies can share and find data, sparking all kinds of collaborations. For example, government sectors in NZ are creating data-sharing initiatives to bridge gaps across departments. Imagine if the Ministry of Health could use transport data to better plan healthcare services based on where people are traveling or living. The possibilities are endless, and the benefits could be massive!

2. Cross-Functional Data Integration: Integration may take over from Transformation as the top IT buzzword in New Zealand. In a nutshell, this is about breaking down the walls between departments and systems, bringing data together for a clearer picture. By pulling data from different corners of a business, leaders can make faster, better decisions and serve the business and customers more effectively. It drives the previously stated aim, to deliver great experiences, drive innovation, and create value.

3. Generative AI and Large Language Models (LLMs): Tools like LLMs are set to change how we interact with data. These AI models make it easy to explore “what-if” scenarios, create realistic simulations, and even automate things like customer interactions. Picture using an AI model to answer customer queries instantly or help patients fill out forms accurately. This tech is making a real impact, and as it becomes more user-friendly, with more people from all areas of the business having the chance to play with it. The results are faster decisions, smoother operations, and more creativity in problem-solving.

What are the Biggest Trends Coming in the Next 12 Months?

The coming year promises a few key trends that could shape the future of Data & Analystics. Here’s what’s on the agenda:

1. AI-Powered Analytics Everywhere: AI-driven analytics are here to stay, and they’re only going to get bigger. With AI, businesses can now automate data preparation, generate insights, and visualize them with little to no human input. This trend will make real-time decision-making more accessible across industries, from banking to agriculture. The major gain here is speed; AI can sift through mountains of data faster than ever before, freeing up people to focus on interpreting those insights and putting them to good use.

2. Edge Computing for Real-Time Analytics: With more Internet of Things (IoT) devices coming online, data processing is moving closer to the source. This means faster insights and reduced latency, which is crucial for industries that need immediate data processing—think autonomous vehicles, healthcare, and manufacturing. By processing data right where it’s collected, edge computing allows faster responses to real-world conditions, cutting down on the back-and-forth time of sending data to a central server. In NZ, this could be a game-changer for industries like logistics or agriculture, where quick decisions are key.

3. Focus on Data Privacy and Ethics: As data grows, so do concerns about privacy and ethics. In NZ, as in the rest of the world, consumers are becoming more aware of data privacy. To stay on the right side of public trust, companies are investing in robust data governance, transparent practices, and compliance with privacy regulations. This isn’t just about avoiding fines—it’s about building trust with customers who expect their data to be handled with care and transparency. As AI advances, it’ll also be essential to address ethical concerns, like avoiding biases in models and making sure AI serves everyone fairly.

To wrap it up

It is clear that Data and AI are powerful tools that will be behind driving massive improvements in efficiency and customer experience for our major enterprises, public services and healthcare industry. Our Small and Medium Enterprise (SME) sector will not be far behind. For data leaders the future is exceptionally bright. However, there is a huge desire to balance this opportunity for innovation with responsibility, to use it wisely, ethically and with a strong foundation in governance and it’s with this in mind that the industry will move forward in the next 12 months.

I hope you’ve found this summary of the event of interest. I look forward to your comments, observations and opinions.