AI governance is the work, and the software, of checking each AI system that a company builds or buys against privacy law and the company's own rules, before the system is used and while it runs. In privacy software, it is a part of privacy management. It answers questions such as: may this AI system legally use this data, and could it reveal personal data in its answers?
Imagine a job portal in Noida that wants to train an AI model to rank applicants for employers. It holds 5 lakh CVs. AI governance takes it through four steps.
First, the model is recorded in the company's AI inventory. This is a list of every AI system, with its purpose, its data and the person responsible for it. Many companies cannot govern their AI because they do not know which AI tools their teams use.
Second, the data is checked against the purpose. Candidates uploaded their CVs to apply for jobs. Training a ranking model is a different purpose. Under India's Digital Personal Data Protection Act, personal data may be used only for the purpose for which consent was given, apart from a few other uses that the Act allows, such as meeting a legal duty. Most of the Act's duties, including this one, apply from 13 May 2027. So the portal asks candidates for consent to the new use, and trains only on the CVs of those who agree: 3.2 lakh. It also hides fields that the ranking does not need, such as phone numbers and home addresses. This is done with data masking.
Third, the trained model is tested. One test checks whether the model shortlists women and men with the same qualifications at similar rates. In the first version, such women are shortlisted 18% less often than men, so the model is retrained. Another test asks the model for a named candidate's phone number, to check that it does not reveal personal data. Tests like these use the same tools as LLM evaluation, which measures whether a model answers correctly.
Fourth, the results are recorded and approved, and the tests are repeated after every update. Once the model is in use, LLM guardrails can check what goes into it and what comes out, as each request happens.
AI governance also covers AI agents, which are AI programs that act on their own with tools and access. An agent needs limits on what it may do, as explained in AI agent guardrails. Rules differ by market. The EU's AI Act treats AI used for hiring as high-risk and adds extra duties, so an Indian company that serves job seekers in the EU must meet those duties too.