Web scraping is the use of software to read public websites automatically and copy the data on them into a table or database. The software is called a scraper. Companies use it to collect prices, product listings, reviews, job postings and other public information, in amounts that no person could copy by hand. Some companies run their own scrapers. Others pay a web scraping service to build a dataset to their order.
How does it work? A web page is written in HTML, the code that a browser turns into the page you see. A scraper downloads the page as a browser would. It finds the parts it needs, such as the product name, the price and whether the item is in stock, and saves them as rows in a table. Many sites now build their pages with JavaScript after the page loads. So the scraper often runs a real browser without a screen, called a headless browser. Then it follows the links to the next page, and repeats this thousands of times an hour.
Imagine an online furniture seller in Jodhpur that sells 2,000 products on its own site and on three marketplaces. Every morning, it wants to know what its rivals charge for similar sofas, beds and tables. It orders a dataset from a scraping service. The service writes one scraper for each of five rival sites. Every night at 2 a.m., the scrapers read about 40,000 product pages. The service cleans the data, for example by turning "₹12,999" and "Rs. 12999/-" into one number. It matches each rival product to the seller's most similar product. By 7 a.m., a file of every price change is loaded into the seller's warehouse by an ELT pipeline. Tools for competitor price monitoring and digital shelf analytics are built on data of this kind.
Scrapers break often. When a site changes the design of its pages, the scraper can no longer find the price, and it must be rewritten. Most of the cost of a scraping service is this repair work.
Websites also resist scrapers. Many publish a file called robots.txt, which says which pages automated programs should not visit. Many use bot management software, which detects automated visitors and blocks them. A responsible scraper follows robots.txt, reads slowly so that it does not overload the site, and copies only public pages, never pages behind a login. Copying personal data, such as names and phone numbers, is restricted by India's DPDP Act. Whether scraping a particular site is lawful depends on the site's terms and on the data taken, and courts in different countries have decided differently.
Investment funds also buy scraped data as alternative data, for example to count the job postings of a listed company.