Tech, ExplainedData platforms and AI
Product

Statistical software

Five decades of credit scores and drug trials.

BEFORE ITAFTER ITStatistical softwareSASCOVERSStatistical testsCredit scorecardsTrial tablesAudit trailClinical trialdesign software↗ CLINICAL TRIALSfixes theanalysis planData warehouse↗ DATA ENGINEERINGsupplies pastloan recordsAnalytics automationsendsprepared dataChecked tables for a trialreport, or a scorecard that abank's loan system uses toscore each applicant.BEFORE ITAFTER ITStatistical softwareSASCOVERSStatisticaltestsCreditscorecardsTrial tablesAudit trailClinical trialdesign software↗ CLINICAL TRIALSfixes theanalysis planData warehouse↗ DATA ENGINEERINGsupplies pastloan recordsAnalyticsautomationsendsprepared dataChecked tables fora trial report, ora scorecard that abank's loan systemuses to scoreeach applicant.BEFORE ITClinical trial design software↗ CLINICAL TRIALS↓ fixes the analysis planData warehouse↗ DATA ENGINEERING↓ supplies past loan recordsAnalytics automation↓ sends prepared dataStatistical softwareSASStatistical testsCredit scorecardsTrial tablesAudit trailAFTER IT↓Checked tables for a trial report, or ascorecard that a bank's loan system uses toscore each applicant.
Statistical software takes a trial's data, compares the average fall in blood pressure in two groups of patients, tests whether the gap could be chance and produces the tables for the regulator.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

Statistical software is software for analysing data with statistics: estimating, testing and building models, with every step recorded so that someone else can repeat it. Banks, insurers and drug companies have used one such platform, SAS, since the 1970s. Analysts usually write each analysis as a short program, in the SAS language, R or Python, rather than clicking through a spreadsheet.

Statistics answers questions that a chart cannot. Is this difference real, or could it have happened by chance? How much of the change in one number is explained by another? How likely is a borrower to stop paying?

Consider a drug company in Hyderabad that is testing a new tablet for high blood pressure. In its clinical trial, 600 patients at 30 hospitals are split at random into two groups. One group takes the new tablet. The other group takes the existing standard tablet. After 12 weeks, the trial's data arrives from the clinical trial data platform.

The company's statistical programmers write programs that produce every table in the trial report. One program works out the average fall in blood pressure in each group: 14 mmHg on the new tablet and 9 mmHg on the standard one. A test then asks how likely a gap of 5 mmHg would be if the two tablets actually worked equally well. This likelihood is called the p-value, and here it is below 0.001. So the gap is very unlikely to be due to chance. The test was chosen before the trial began, often with clinical trial design software. Choosing it in advance stops anyone from picking, afterwards, the test that gives the best-looking result.

Regulators check this work. In India the regulator is CDSCO, and in the US it is the FDA. Each program, its input data and its output must be kept, so that an inspector can run it again and get the same numbers. These rules on records are part of GxP compliance. Often a second programmer writes the same program independently, and the two outputs must match exactly.

Banks use the same software for underwriting, the decision whether to lend and at what price. Say a bank has the records of 2 lakh past personal loans, and knows which borrowers stopped paying. A model estimates how income, existing EMIs and repayment history change the chance that a borrower stops paying. The bank turns the model into a scorecard, which gives every new applicant a score, much like a credit score. The bank checks every year that the scorecard still predicts well.

How is this different from analytics automation? Analytics automation prepares and combines data without code. Statistical software is where the statistical model is built and tested. It is also different from a numerical computing environment such as MATLAB, which engineers use for matrices, signals and simulations rather than for trials and credit risk.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.