Data integration and governance platforms

Data pipelines and connectors 10 companies

A pipeline copies new records from each of a company's systems into its warehouse, every few hours or as each change happens. A connector is the piece of code that knows how to read one particular system.

Pipelines into the warehouse

Copying data into the warehouse and reshaping it there.

FivetranFivetran, with engineers in Bengaluru, made copying data a product. It offers hundreds of ready-made connectors that pull data out of apps and databases into the warehouse.BengaluruHiring now

MatillionMatillion, with engineers in Hyderabad, sells a visual tool for building pipelines that load data and reshape it inside the cloud warehouse.HyderabadHired recently

Hevo DataHevo Data, founded in Bengaluru, sells a pipeline that is set up without writing code. Its customers are data teams in many countries.Bengaluru

ProphecyProphecy, from Palo Alto with its engineers in Bengaluru, sells a visual editor for data engineers that writes real code underneath, which they can read and change.Bengaluru

ShakudoShakudo, with engineers in Bengaluru, does not move data itself. It sells one platform on which a data team installs and runs all its data and AI tools, on the team's own systems.

Connectors

Ready-made code for reading data out of hundreds of different systems.

NexlaNexla, with engineers in Bengaluru, sells connectors that make every data source look the same to the people using it.BengaluruHired recently

CData SoftwareCData Software sells connectors as a product, so any tool can read data from hundreds of different systems.BengaluruHired recently

CobaltCobalt, with engineers in Bengaluru, sells connectors to software companies, whose products must connect to the other apps their customers use. It now sells AI agents that build those connections.

Streaming every change

Sending each change the moment it happens, so the warehouse is only seconds behind.

ConfluentPart of IBMConfluent is the company behind Apache Kafka, the free software that carries events, such as a payment or a click, between a company's systems in real time.BengaluruHiring now

StriimStriim, with engineers in Chennai, reads each change out of a database the moment it is written and sends it onward, so the warehouse is only seconds behind.

The tech, explainedETL and ELT Embedded iPaaS Event streaming Change data capture (CDC)

Data catalogues, monitoring and data quality 9 companies

A catalogue lists every table a company has, who owns it and what it means. Monitoring tools warn when data is late or wrong. Master data tools build one agreed record of each customer, product or supplier.

Catalogues and pipeline monitoring

Listing what data exists, and warning when a pipeline breaks.

AlationAlation, with engineers in Chennai, sells a data catalogue: a searchable list of every table a company has, who owns it and what it means.ChennaiHiring now

AcceldataAcceldata, with its engineering in Bengaluru, watches pipelines and tables and warns when data is late, incomplete or wrong, before a dashboard shows the wrong numbers.BengaluruHiring now

DataHubDataHub, with engineers in Bengaluru, is the company behind the free catalogue of the same name, which was first built inside LinkedIn.

RevefiRevefi uses an AI model to read a company's warehouse and work out why a pipeline failed, and why the warehouse bill is as high as it is.

Altimate AIAltimate AI sells an AI tool that writes the code that reshapes data in the warehouse, and writes the notes that explain it.

Data quality and master data

Correcting records, and building one agreed record of each customer, product or supplier.

PreciselyPrecisely, with engineers in Delhi, Bengaluru and Pune, sells data-quality tools, extra details to fill in missing parts of records, and address data.Delhi NCR · Bengaluru · PuneHiring now

InformaticaPart of SalesforceInformatica sells all of these jobs in one suite: pipelines, data quality, a catalogue and master data.

ReltioPart of SAPReltio sells master data management: one agreed record of each customer, product or supplier, built from every system that holds one.

VerdantisVerdantis builds one agreed record for a single kind of data: a plant's spare parts and suppliers.See alsoSupply chain planning software

The tech, explainedData catalog Data observability Data quality and enrichment Master data management (MDM)

Data platforms for industry 2 companies

A plant's sensor readings, maintenance records and drawings sit in many separate systems. These companies join them so engineers can see everything about one machine in one place.

CogniteCognite, with engineers in Bengaluru, joins the sensor readings, maintenance records and drawings of an oil platform or a plant, so engineers can see everything about one pump in one place.BengaluruHiring now

eQ TechnologiceQ Technologic, with engineers in Pune, connects the design, parts and test data of aerospace and defence manufacturers, which otherwise sit in dozens of separate systems.

The tech, explainedIndustrial data platform↗ Predictive maintenance software Engineering data integration↗ Predictive maintenance software

How it all worksData engineeringOne order, recorded at once and added up later.

See how it works →

What's changing

Pipelines are starting to fix themselves.

When a pipeline breaks, a data engineer usually finds out from a wrong number on a dashboard and spends hours finding the cause. Newer tools use AI models to watch pipelines, explain what went wrong and, in some cases, write the fix.