How data contracts and lineage can unlock AI innovation

What we explore in this data contracts guide

In this article, we explore:

• The data dilemma stalling AI innovation 
• Data contracts: defining the rules of engagement 
• Data lineage: tracing the data journey 
• The power duo: combining data contracts & lineage 
• How businesses can harness this power duo

The data dilemma stalling AI innovation 

Developing functional AI proof-of-concepts is one thing. Scaling them into value-driving, product-grade solutions is another thing altogether. This is where many AI journeys come undone. And data is often at the heart of the problem. 

Proprietary enterprise data is key to unlocking more complex AI use cases (as we explore in our article on GenAI RAG systems). But it’s often messy, siloed, inconsistent, and lacking in context, transparency, and traceability. Where this is the case, businesses struggle to systematically capture value from their data and lay the data foundations for AI innovation. 

And it’s not just AI that suffers from poor data practices. Data inefficiencies cascade into broader business impacts, impeding decision making, problem solving, compliance, and your ability to seize new growth and innovation opportunities.  

‘Many organisations are still grappling with fundamental data challenges that prevent the intelligent transformation of their business – from data silos and provisioning to managing the exponential growth of their data'. – Neelesh Parekh, Innovation & Growth Director, Financial Services & Insurance at Zühlke.

Cracking these challenges is a prerequisite for becoming a more effective, more resilient business and unlocking the transformative potential of AI. 
 
In a separate article we explore the essential role of data quality in the journey towards data-empowered business. Here we look at data contracts and lineage: the power duo you can implement today to realise the full potential of your data assets and prime your AI initiatives for success. 

Data contracts: defining the rules of engagement

At its core, a data contract is a machine-readable agreement between data producers and consumers that defines the structure, quality, and business context of data. Think of it as a formalised handshake that ensures data is well-documented, meets stakeholder needs, and supports effective governance. It serves as the blueprint for maintaining high-quality data that can be easily managed and trusted across the organisation.

Data contracts address common data challenges by facilitating:

• Business context: data contracts embed critical business context, ensuring that every data point is relevant to the broader organisational goals.

• Collaboration: by aligning expectations between data producers (for example, software engineers and data engineers) and data consumers (for example, analysts and business teams), data contracts enhance collaboration and reduce dependencies on IT interventions.

• Governance: automated documentation through data contracts facilitates better compliance with internal and external regulatory frameworks while reducing errors associated with manual data management.

For businesses operating in complex environments, the governance benefits of data contracts are particularly attractive, as Ascot's Saki Thethy explains 

‘At Ascot, we’re increasingly focused on building robust data governance foundations, and we see data contracts as central to this goal. While we haven’t implemented data contracts yet, we recognise their potential as transformative tools for managing data-related risks and are actively exploring how they could support our governance evolution’. – Saki Thethy, Head of Data, UK & Bermuda at Ascot.

Insurance is fundamentally about risk management and informed decision-making, he explains. And so establishing structured data agreements like data contracts will allow Ascot to ‘control quality and ensure all data is both trusted and reliable for our operations’.

Best practices for implementing data contracts

Implementing data contracts successfully requires more than just a technical solution. Thoughtful planning and cross-functional collaboration is essential. 

Your implementation strategy should address the following key areas:

Data lineage: tracing the data journey

Data lineage tracks the journey of data from source to destination – across the changes and transformations throughout its lifecycle. This transparency is essential for effective data governance, quality assurance, and compliance. It answers critical questions like: Where did this data come from? How was it transformed? Where is it being used?

Data lineage is particularly relevant in today’s world, where misinformation is rampant and fraud is on the rise. For example, the ability to identify the source of ‘merged data’ for single customer views is critical for fraud analytics and sensitive data representation. These data sources are becoming a cornerstone of trust and reliability in developing AI and advanced data analytics.

How data lineage supports decision making and compliance

Understanding your data’s journey is vital for building trust and ensuring compliance. 

Here are some of the key business benefits of data lineage:

Consider this real-world example: a retail insurer implemented data lineage tools, resulting in significant improvements in compliance and trust. With transparency around the origin and movement of data, the company was able to establish clear audit trails, which simplified the task of demonstrating GDPR compliance during audits. This clarity also increased confidence in the data’s accuracy, directly reducing service costs by minimising errors.

The power duo: combining data contracts and lineage

Historically, data lineage has provided technical insights into how data moves through systems and transforms over time. So when add data contracts into the equation, you can enrich those technical insights with business context and governance – aspects like ownership, classification, and rules. Together, data contracts and lineage offer a more comprehensive understanding of both technical and business contexts. The result? Enhanced decision-making and greater compliance.

‘Data contracts and lineage provide a robust framework for data quality and governance. By integrating the two, organisations can enhance their data practices and lay far stronger foundations for their AI initiatives’. – Raffaele De Piano, Principal Data Architect at Zühlke

By integrating data contracts and lineage, you unlock essential business benefits:

• Enhanced compliance: together, contracts and lineage ensure adherence to regulatory requirements, with a transparent audit trail.
• Improved quality management: by connecting data context with its lineage, organisations can manage data quality effectively across its entire lifecycle.
• Operational efficiency: coupled with formalised contracts, clear visibility of data’s journey reduces the need for ad-hoc troubleshooting and speeds up operational workflows.
• Cross-functional collaboration: by providing both technical and business teams with the context they need, this power duo facilitates better collaboration across the organisation.

How businesses can harness this power duo

Contracts aren’t just technical initiatives. Combined with data lineage, they’re ‘strategic assets’ capable of transforming how you serve clients and compete in a rapidly evolving market. That’s according to Ascot’s Saki Tethy, whose vision is to make data contracts part of the firm’s future data governance strategy:

‘Our aim is to harness contracts to provide crystal-clear guidelines around data ownership, purpose, and quality from end to end. Combined with data lineage, this level of transparency would allow us to trace the origin and flow of data across our systems with precision, creating a clear, auditable pathway that bolsters our risk assessment and pricing capabilities’.

‘Data contracts aren’t just technical initiatives; they’re strategic assets that could help transform how we serve our clients and compete in a rapidly evolving market’. – Saki Thethy, Head of Data, UK & Bermuda at Ascot.

By establishing clear governance, Ascot will be positioned to unlock even more value from the firm's data, supporting its goal of smarter, data-driven decisions across the organisation.

Its a sentiment echoed by HSBC’s Stefan Buxton, who sees the combination of contracts and lineage as an essential foundation for innovation.

'At HSBC, we've embraced data contracts and lineage as the dynamic duo powering our AI ambitions. By mapping every data point's journey and setting clear rules of engagement, we're not just meeting compliance – we're propelling innovation.’ – Stefan Buxton, Head of Data Quality at HSBC

Looking to harness the power of data contracts and lineage in your own organisation? Here are some practical steps you can take to set yourself up to succeed:

Lay the foundations for AI success 

Data contracts and lineage are not just tools for improving data governance; they’re strategic enablers for unlocking the full potential of AI.

By systematically capturing value from your data, these frameworks enhance decision-making, support compliance, and accelerate intelligent business transformation.

In this way, the contracts/lineage ‘power duo’ we’re explored here will be critical to driving early and ongoing value from you AI initiatives.

To put it simply: to empower your business, you need to empower your data.

Register for your complimentary data health assessment

Data quality is essential for capturing value from AI at scale. Our data experts can prime your AI initiatives for success with a complimentary health check of your data. Complete the short form below to register your interest. A member of our data team will be in touch shortly.