Dan Perrin

Dan Perrin's Blog

3 analytics predictions for 2019, inspired by Big Data London

The buzz at Big Data London hinted at changes to come in 2019 – from new data-driven experiences to the rise of DataOps and the evolution of big data beyond cloud and open source.

If you work with data in the UK, chances are you were at November’s Big Data London event. This was the third year the show has been held, and it was by far the biggest and most wide-ranging yet.

I was there to find out what kind of challenges companies are facing in getting big data and analytics programmes embedded in the organisation, and to get a feel for how things might evolve next year.

The conversations I had, and the presentations I saw, suggest three key developments to come in 2019.

#1 Data will drive completely new customer experiences

Lots of organisations are still getting started with big data and analytics, and the business value of their fledgling initiatives may not yet be proven. Big Data London had plenty of content to inspire them to press on with building platforms that will give their business a true competitive advantage.

One of the new tracks at the event focused on using data to create unique customer experiences. One presentation stood out for me: New Nudges by Alastair Cole, Chief Innovation Officer at Andrews Aldridge.

Alastair showed how customer data coupled with machine learning can lead to the creation of “big ideas, crafted for individuals” – enabling truly personalised products (as opposed to recommendations of existing products) to be created and offered to consumers on the fly.

Few organisations are in a position to be able to do this successfully today, but many are heading in that direction. Our own work with companies like Rank and the Restaurant Group is focused on building platforms that can ingest vast amounts of customer data in real time, to allow machine learning algorithms to be applied to up-to-date data.

That kind of platform takes time, effort and expertise to set up and maintain, but Alastair Cole’s presentation provided a glimpse of the kind of unique value it can deliver.

#2 Advanced organisations will move to a “DataOps” model

Another new track for 2018 at Big Data London was “DataOps”. Just as DevOps has made software development more agile, data ops promises to do the same in 2019 for data-driven activities.

Anyone involved in analytics today will recognise the frustration of trying to meet business demand for instant intelligence, when the processes for gathering, pooling, cleaning and interrogating data come from a previous era and are tortuously slow.

Organisations that are serious about always-on insight, and about applying AI and machine learning to data in real-time, need to completely rethink the way data is handled and delivered in the organisation. DataOps, with its emphasis on responsive, agile processes, seems to hold the answer.

It’s certainly something we’re seeing our customers ask for. We presented a case study at Big Data London about our work with the Rank Group to speed up the process of pooling data, applying and updating machine learning algorithms, and delivering the insights back to the business.

Previously, it used to take around seven months for Rank to make new data feeds available to gain business insights. By building and managing an analytics platform in the cloud, and re-organising IT operations around the delivery of the resulting insights, we’ve been able to bring that process down to hours and minutes.

As AI and machine learning initiatives increasingly emerge from their R&D ivory tower and start to be embedded in the business, the reorganisation of operations to be more data-centric feels like something we’ll see a lot more of next year.

#3 Big data will evolve beyond cloud and open source

For a long time, big data has felt synonymous with the open source movement. If your organisation has a strategic commitment to Microsoft, you might have felt that events like Big Data London were not for you, and that you were missing opportunities to harness the full value of unstructured data.

But that’s changing, and fast. As a Microsoft Gold Partner, Adatis was privileged to present to the Big Data London audience some new capabilities coming to SQL Server next year.

With purely relational databases increasingly preventing companies from unlocking the full value of their data, Microsoft has made radical architecture changes to SQL Server 2019 to address the challenge. Its new big data clusters capability will enable teams to move quicker, work with a wider array of data, handle massive datasets and augment their code with open-source libraries and projects.

That’s important for another reason, too: it will make it easier to run big data projects on-premises. With many organisations preferring to keep their data inhouse, for regulatory, policy or (perhaps surprisingly) cost control reasons, the ability to pool structured and unstructured data for real-time analysis will be game-changing.

The challenge for data scientists: staying focused on delivering business impact

If Big Data London is any indication – and as one of the UK’s biggest big data conferences, it should be – then 2019 promises to be an exciting one for data-driven organisations.

The challenge for data scientists, though, will be to stay focused on delivering business impact, and not get bogged down in the nuts and bolts of operating and evolving the analytics infrastructure.

Database technologies, analytics and visualisation tools and cloud platforms all evolve fast, and keeping up with the underlying tech may not be the best use of time for a skilled data science team.

10 questions to ask now – and a half-day to focus on the way forward

If you’re currently mulling the best way to operationalise and evolve a big data, analytics or AI program, it makes sense to consider these ten questions before making a decision.

You may also welcome an opportunity to get your data science team together to think about the best way forward. Adatis would be pleased to organise a free half-day workshop to explore your objectives and current model, in order to uncover the best solution. For more information about what that would entail, email me at dan.perrin@adatis.co.uk.

A checklist of ten areas to consider if you are thinking about establishing a Data Analytics Platform or AI solution.

In recent years building a future-proof data and analytics solution and service has been rapidly rising-up the agenda of most CxOs with good reason. Based on our experience at Adatis, here are 10 questions that I think are worth considering before committing to how you operate and evolve your own Data Analytics Platform.

1. Can we attract the talent to do this in house?

Data and analytics skills are in very high demand in the market place and certainly those individuals with the latest cloud experience command a premium. Can you offer a role that offers the interest and reward to attract the best talent and do you have the time to invest in finding them?

2. How do we ensure the on-going efficiency of the platform?

With cloud technology there may be direct savings to be made by ensuring that the Data Analytics platform is optimised and remains so. Will the team have the time and knowledge to monitor and ensure your platform remains efficient and minimise your consumption costs?

3. Is it straightforward for the team to cover the critical hours of processing and operation?

The business dependency on a Data Analytics platform is increasing and the likelihood is that data is no longer just arriving in a batch, in the early hours of the morning. Can your team provide the necessary hours of cover to ensure that as a minimum, by the start of the business day everything is processed, and the platform will be trusted?

4. Can we build the scale of team required to cover all the essential skills?

The range of technologies that form a modern Data Analytics Platform can be bewildering. Will you have the scope to build a team with the breadth of skills required to operate a platform, from cloud infrastructure to data science model retraining and be able to collaborate effectively with your end-users?

5. Do we have the budget to invest in the training to ensure the team are effective?

To operate the data analytics platform and provide down to 3rd line support, the team will need a depth to their knowledge. Can you provide the team the exposure to learning and development opportunities such that they can become experts in the application of the technology and operation of the service?

6. Can we retain the knowledge of the platform efficiently or is this a potential risk?

Data Analytic Platforms are generally complicated and evolve over time. Will there be wasted effort and potentially an impact on the service in ensuring that the knowledge is shared and maintained, perhaps as individuals leave and join the team?

7. Would we benefit from having access to experts?

The technology and specifically cloud platforms are continually evolving. Are you able to keep track of the change or will you have access to experts who can provide you regular updates and provide recommendations of how they might be of value?

8. How will we continually improve the solution and service?

The team will likely be faced with a continual list of improvements that are required to ensure the solution evolves with the changing business. Will you have the processes and safe guarded time to respond and implement the required on-going changes, and will you be able to provide an impartial perspective to evaluate the service and identify ways in which it can be improved?

9. Are we confident on what the operational costs will be, and can we control these?

There may be several factors that add an unknown element to your operational costs e.g. call-out allowances, cloud consumption, recruitment fees, training costs, ad-hoc advice and guidance. Would it be beneficial to be able to fix your operational costs, potentially for several years?

10. And so, bearing in mind all of the above. Is in-house or even your preferred out-source partner likely to provide the right service for your organisation?

Every organisation and every platform is different, and one size does not fit all. So if you’d like some advice on answering these questions for your organisation please do message me and the Adatis team will be very happy to discuss.