Data Contracts - It Should Be A Thing

Data engineering is the backbone of any data-driven organisation.

Data engineering involves collecting, transforming, and delivering data to facilitate informed decision-making. In this complex and dynamic process, a formal agreement between data producers, data transformers, and data consumers is required to ensure a seamless data flow and to maintain data quality.

Introducing Data Contracts. A blueprint to define the structure, semantics, and expectations of data as it moves through the various stages of the data pipeline.

A Data Contract is often defined as “A data contract is a formal agreement or specification that defines how data should be structured, formatted, transmitted, and processed between different components of a software system, services, or systems in a distributed computing environment”. However, I have some thoughts I would like to share with you.

If you ever worked in the Information Technology field, you'll have probably heard the expression ‘Garbage In, Garbage Out’ (GIGO). The idea that the quality of the output is influenced by the quality of the input. It is important to consider how evolved the company or processes are and whether a considerable part of production bugs originate on undefined rules or data. Having said that, I would ask you to consider that a Data Contract allows software components to talk to each other under a defined structure (language).

The five rules

Data contracts play a vital role in the world of data management and technology integration. They standardise data interactions, ensuring that data is structured, shared, and used consistently across various systems and organisations. Our experience in designing and developing data integration systems has enabled us to develop what I call the five rules of data contract. A data contract is well implemented if it at least guarantees these five aspects.

1. Interoperability: Enable different components and systems to communicate seamlessly by providing a common understanding of data formats and behaviour.
2. Data Integrity: Enforce rules and validations. This reduces the chances of data corruption and ensures that data remains accurate and trustworthy.
3. Collaboration: Promote collaboration among development teams, enabling them to work on different parts of a project while ensuring that data exchanges are consistent and reliable.
4. Scalability: Provide a scalable way to manage data interactions. They allow for the addition of new data fields or changes to existing ones without disrupting existing functionality.
5. Documentation: Data contracts serve as valuable documentation, making it easier for developers to understand how data is structured and should be handled. This aids in troubleshooting, debugging, and maintaining the system.

Key Elements of a Data Contract

• Data Types: Specific data types allow for individual fields or elements within the dataset. These data types can encompass structured information, such as numbers, strings, and dates, as well as unstructured data, including arrays, objects, or user-defined data structures.
• Data Format: A data contract dictates the required presentation format for the data. This format may encompass distinct file formats, such as JSON or CSV, database table format, or even a tailor-made user-defined format.
• Data Constraints: Contracts encompass restrictions or regulations that establish acceptable values or boundaries for particular data fields. For example, in a data contract that validates a date, there may be a stipulation that confines the allowable date formats as mm-dd-yyyy, ignoring any records falling outside this constraint.
• Data Documentation: Within a data contract, there is space for supplementary information in the form of documentation or metadata, offering extra insights into the data. This might contain field explanations, units of measurement, lists of acceptable values, or references to data sources.

Bridging the gap

Data contracts are not confined to the realm of bits and bytes; they are woven into the fabric of every data-driven decision, every analytical insight, and every transformational leap forward in the digital age. They provide clarity where there is ambiguity, maintain order amidst complexity, and empower organisations to harness the full potential of their data assets. They are not just technical artefacts but also a testament to an organisation's commitment to data excellence.

They bridge the gap between technology and business and link expectations from data producers to data consumers.