Home › Cybersecurity & GRC Career Guides › Data Lineage Skills

Data Lineage Skills

Get new GRC career guides by email. We publish most weeks. No spam, and we never sell or share your address with anyone.

Lineage answers two questions that sound simple and are usually unanswerable: where did this number come from, and what breaks if I change this field?

Whether the data is fit to use is a separate question, covered under data quality.

Why anyone pays for it

Four situations, and every organization eventually meets all of them.

A regulator asks how a reported figure was produced, and the answer has to be a traceable path rather than a name of someone who understands it. An engineer wants to change a schema and nobody can say what depends on it, so nothing is ever deleted and the estate accretes. An incident occurs and the question is which downstream reports consumed bad data during the window. And a privacy request arrives asking where a person's data went, which without lineage is answered by asking around.

Each of those is expensive. Together they are the business case, and being able to state them plainly is more persuasive in an interview than describing the tooling.

Three altitudes

Table level lineage tells you this table feeds that one. It is cheap, largely automatable, and enough for impact analysis.

Column level tells you this specific field derives from those specific fields. It is substantially harder and it is what regulatory questions actually require.

Transformation level tells you what was done to the values along the way. It is the most expensive and the only one that answers "why is this number different from that one".

Most programs need column level for a defined set of critical elements and table level for everything else. Attempting transformation lineage across an entire estate is a well-known way to spend two years and deliver a diagram.

Automated capture, and where it stops

Modern tooling parses SQL and pipeline definitions to build lineage without anyone drawing it. That covers the tractable part.

It stops at the same places every time. Data moved by hand. Logic inside a stored procedure or a compiled job. Anything that passes through a spreadsheet, which in most organizations is where the interesting transformations happen. And traffic between systems that do not share a metadata layer.

Those gaps are where the regulatory exposure concentrates, precisely because they are invisible to the tool that produced the confident-looking diagram.

Scope it to what matters

Full estate lineage is a project that does not finish. Critical data elements, the fields that appear in regulatory reports, feed material decisions or carry personal data, are a scope that does.

Start from a report somebody has to defend and trace backwards. You will find undocumented steps within the first hour, and that finding is itself the deliverable that funds the rest of the work.

The AI dimension

Model governance now asks the same question about training data: which datasets went into this model, under what licence, with what consent, and can we show it. The EU AI Act's technical documentation expectations for high-risk systems make training data provenance a compliance artifact rather than an engineering nicety.

People who understand lineage are finding that skill suddenly relevant to AI work they did not train for.

Where to go next

Frequently Asked Questions

What is data lineage?

A record of where data came from, how it was transformed, and where it went. It answers two questions: how a given number was produced, and what would break if a field changed.

Why do organizations invest in lineage?

Four recurring situations. A regulator asks how a figure was produced. An engineer needs to change a schema and nobody knows what depends on it. An incident requires identifying which downstream reports consumed bad data. And a privacy request asks where a person's data went.

What are the levels of lineage?

Table level, showing which tables feed which. Column level, showing which specific fields derive from which. Transformation level, showing what was done to the values. Most programs need column level for critical data elements and table level for everything else.

Can lineage be captured automatically?

Partly. Tools parse SQL and pipeline definitions and cover the tractable part. They consistently miss data moved by hand, logic inside stored procedures or compiled jobs, anything passing through a spreadsheet, and movement between systems with no shared metadata layer.

Where does automated lineage fail most dangerously?

In the gaps, because the diagram still looks complete. Spreadsheet steps in particular are where interesting transformations happen in most organizations, and they are invisible to the tooling.

How should a lineage project be scoped?

To critical data elements: fields appearing in regulatory reports, feeding material decisions or carrying personal data. Full estate lineage is a project that does not finish. Starting from a report someone must defend and tracing backwards usually surfaces undocumented steps within the first hour.

What tools are used for data lineage?

Collibra, Alation, Atlan, Informatica, OpenLineage and Apache Atlas are common, and dbt exposes lineage for transformations defined within it. The tool matters less than the scoping decision.

How does lineage apply to AI?

Model governance asks which datasets went into a model, under what licence and with what consent, and whether that can be shown. The EU AI Act's technical documentation expectations for high-risk systems make training data provenance a compliance artifact.

What jobs require data lineage skills?

Data governance analyst, data steward, data architect, regulatory reporting roles, privacy engineering, and AI governance positions documenting training data provenance.