Trust and safety is the work of protecting the users of an app from each other: catching fake accounts and scammers, removing harmful posts and detecting media faked with AI. It is also called T&S, and one large part of it is called content moderation. It grew up inside the large social platforms, and it is now also sold as software to smaller apps.
Every app in which strangers meet needs it. A social app must remove posts that break the law or its rules. A short-video app must screen uploads. A dating app or a matrimonial website must catch fake profiles that ask people for money.
The work usually happens in three steps. First, software screens what is posted and flags what looks wrong. Users also report posts and accounts. Second, someone investigates each flag or report. This means collecting the evidence: the post, the account's history, other accounts linked to it and earlier reports. A trust and safety investigation tool can now do this collection with an AI agent, and draft a decision. Third, a trained human reviewer reads the draft and decides. This way of working is called human in the loop.
Some reports need a special check. A photo, a voice note or a video may have been made by AI. Deepfake detection checks whether the media was ever real. The same kind of AI model that creates fakes is used to catch them. This is described in AI in cybersecurity.
In India, the law sets some of this work. Under the IT Rules 2021, every social media platform must appoint a grievance officer. The grievance officer must acknowledge a user's complaint within 24 hours and resolve it within 15 days. Platforms with more than 50 lakh users in India must also publish a report every month on the complaints they received and the action they took. A user who disagrees with a platform's decision can appeal to a Grievance Appellate Committee set up by the government.
Trust and safety sits close to identity verification (IDV). Identity verification checks a person once, when the account is opened. Trust and safety watches what the person does after that.