Skip to main content
HomeThe WWS Daily

- News, tips, inspiration you can trust to thrive in today’s digital age.

Search form

Main menu

  • Home
  • News & Features
  • Business & Economy
  • Tech & Trends
  • Health & Style
  • Arts & Culture
  • Contact Us

How Web Scraping Can Support Macroeconomics and Be Used for Greater Good

Juras Juršėnas April 5, 2022

juras-jursenas.jpeg  COO at Oxylabs.io

  WWS contributor

hor-z.png

Web scraping lies within the domain of data, and data can be useful for more than just profit-driven purposes. It can also be used for macroeconomics and the greater good.

 

Scientists have carefully looked at the use of web scraping, and web scraping has had its praises sung for business applications. We play no small part in that ourselves. 

The world of business, however, isn’t just about turnover, profit, and the hustle. There’s some common good to be done as well. But hesitancy to take the plunge and deploy web scrapping for the common good could be driven by several reasons. 

Primarily, hesitancy to web scrapping is at least in part due to the “wild west” legal situation of the industry. Regulations and legislation have been a little sluggish, as our legal counsel would note. 

Slow adoption doesn’t mean it’s not going to happen, eventually. And it will happen because there’s plenty of utility for web scraping in the public sector. 

With some care, we’ll delve into a much-debated topic, both by professionals and laymen, on web scraping and how it can support macroeconomics and be used for the greater good.

 

Macroeconomics and Modeling

 

web-scraping-support-macroeconomics-experts.jpg

Web scraping refers to the various methods used to collect information from across the internet and extract data from a website. It builds the data scaffolding we never had before, being used to collect data about businesses, prices, and numerous other factors to track the slowly emerging changes.

Macroeconomics, on the other hand, is a branch of economics concerned with large-scale or general economic factors, such as interest rates and national productivity. It relies heavily on mathematical modelling to aggregate changes in the economy such as unemployment, inflation, growth rate, and gross domestic product (GDP).

Macroeconomics is often criticized for attempting to draw real-world conclusions from these models. In other words, attempting to fit conclusions from exceedingly simple systems to complex ones. Yet macroeconomics can be used for good using web scraping.

To a lot of people, macroeconomic approaches feel a little suspect. The discipline heavily relies on mathematical modeling, which may look like a fancy word for “oversimplification” to some. Basic supply and demand models are the most popular examples. 

Markets are significantly more complex and slower, the argument goes, than these macroeconomic models might imply. Supply doesn’t match demand instantaneously as there’s a certain delay and pass-through rate.

An esteemed example, however, has been the Consumer Price Index (CPI). The Bureau of Labor Statistics (BLS) started tracking CPI back in 1918 as a way to measure inflation. In a simplified sense, CPI tracks the changes of an allocated and economically weighted basket of goods across the country.

I should note that I’m not doing the complexity of CPI calculations justice. They include changing shopping and behavior patterns and numerous other mathematical instruments that inch the model closer to reality. 

Nevertheless, the Consumer Price Index is probably the most widely criticized metric. Some have criticized the choice of the basket of goods such as the removal of more volatile products from the CPI. Others criticize the weighting and substitution practices used in calculations since the 90s. Finally, some countries take completely different approaches to CPI.

In Alex M. Thomas’ Macroeconomics: An Introduction, he states that India uses several different metrics as signals for price levels. These indicators are the Wholesale Price Index (WPI) and CPI with various derivatives. India calculates CPI for industrial, rural, and agricultural laborers separately.

All of these are hedged against a base year. While they do have criteria to assess years, the usage of a base economic year is still somewhat susceptible to subjectivity. After all, at some point the base year has to be changed, which should move all indices accordingly.

In the end, every way to calculate CPI takes a slice of data out of the real world and attempts to make the conclusions as accurate as possible. Partly, modeling is a lack of the ability to acquire and analyze data.

 

Unfettered Access to Data

 

businessman-pointing-his-finger-growth-graph-data-access.jpg

Back in the day, causal determinists famously believed that if we knew everything there was to know, there would be no randomness. Coin flips could be predicted with perfect accuracy as long as the data was available.

Causal determinism has experienced a lot of changes since then. It has gone down a road a little more complicated than the initial stance. It, however, is partly reflective of modeling – if we knew everything there was to know, many, if not all, mathematical models would be obsolete.

CPI, I believe, collects data in such a manner, because getting information about all or a significant portion of goods across an entire country used to be nigh impossible. So, some deliberate and calculated liberties had to be taken.That may not be as relevant anymore. 

Today, web scraping can be used to acquire as much data as necessary. While physical stores might be out of the question, with the overall dominance of ecommerce, calculating prices online is no longer such an arduous task. Often, all it takes is just a scraper API that’s built for ecommerce.

 

Billion Prices and Other Projects

 

There have been successful attempts at using web scraping to predict inflation and CPI. The Billion Prices Project (BPP) did exactly that and led to the development of three other projects of which two provide accurate inflation tracking in countries where governments might not be as willing.

BPP is particularly interesting as it launched way back in the day – in 2007. Web scraping hadn’t then reached the heights of development it has now. Researchers today wouldn’t have to develop everything from scratch. Scraping solution providers could give them the opportunity to focus on macroeconomic research rather than overseeing development.

Unfortunately, whatever they did seems to be either lost to time or never fully disclosed. In one of the articles, they mention an online appendix for those more eager to learn about web scraping. The document only holds surface-level information about the process, however.

A short summary of the process might still be necessary. Cavallo and his team created automated programs that would run through major online stores and collect the prices of millions of products every day. Such data would be stored for comparison purposes.

Several years down the line, BPP had proven itself to be of immense practical use. Alberto Cavallo, the head of the project, has published numerous studies that show how web scraping can support macroeconomic research or even render some historical approaches obsolete.

His Using Online Prices for Measuring Real Consumption Across Countries is one such example. By scraping and collecting historical pricing data from 11 countries and a total of nearly 100 000 individual products, the authors were able to model consumption levels across countries. There was little-to-no estimation lag (i.e., the delay between data collection and publication), no need for CPI extrapolation, and the results were close to official reports.

The benefits don’t end there, however. As the authors themselves note, the data is more granular and delivers additional insights. For example, relative price levels across countries can be gleaned by using web scraped data – something that cannot be done with CPI.

The Billion Prices Project created several offshoots, some of which are still alive and well. Some of them are intended to solve specific problems in macroeconomic research. Others shed light on data that would otherwise be unavailable such as Inflacion Verdadera Argentina.

Due to some political complexities, Argentina’s government had started to publish inflation data that seemed suspect. Inflacion Verdadera Argentina was initiated to uncover the truth. Cavallo’s team discovered that the numbers were not just a little suspicious. The official reported inflation rate was three times lower than the one found by his team.

Finally, the last BPP offshoot, PriceStats, could solve the CPI delay problem. The US Bureau of Labor Statistics states that CPI is calculated based on data from about three years ago. Web scraping can remove any data lag and provide more accurate and timely data.

PriceStat’s data is unfortunately shared only on a request basis. While I haven’t acquired such data, they have a great example of aggregate inflation with the project’s and the official CPI data displayed. Both the problem and the solution are clearly displayed in their full glory.

Finally, web scraping seems like it could have a long-term career in macroeconomics. Cavallo recently published research stating that due to the sudden shift in shopping patterns caused by COVID-19, the usual CPI measures were inaccurate.

 

A Window to the Micro in Macroeconomics

 

There’s more to be done. Metrics, calculated through data collection methods, shouldn’t be taken as a way to “show the government.” Independent research can be used to enhance already existing methods and deepen our understanding of current macroeconomic phenomena.

Web scraping builds the data scaffolding we never had before. Monetary policy, a topic so complex and intricate that it by itself stands as an argument to split the Central Bank from any government entity, is turning the economic dials of an entire country (or even a group of countries). Small changes in things such as money supply or interest rates can have an enormous impact on economic health.

Unfortunately, we often don’t get to notice the direct impact of monetary policy. There are, usually, several types of lags outlined. Regardless of how many there truly are, the end-result is the same—the full impact of monetary policy isn’t immediately visible.

Additionally, there’s a gap in data. Models are used to predict outcomes. While they are mostly accurate, web scraping can do better. It can bridge the gap between monetary policy and its effects.

Web scraping can be used to collect data about businesses, prices, and numerous other factors in order to track the slowly-emerging changes. Such data would allow us to delve deeper into the minutia of monetary policy and to truly wrestle out the real-world effects of even the smallest changes.

In the end, there are two important and unique aspects of web scraping that remarkably enhance the possibilities of economic research. First, data is instant. Economics has been reliant on historical data that’s always a little late to the party, since it has to go through many parties and institutions before being made accessible.

Second, it’s more reliable. All data collected is “as is.” While data manipulation might be infrequent, there’s always the possibility of error, intentional or not. With web scraping, researchers can check and verify all information themselves independently.

 

Conclusion

 

Web scraping lies within the domain of data and data can be useful for more than just profit-driven purposes. Proper usage of these tools can aid in research and policy-making. Indeed, macroeconomics is just one discipline where automated data collection can be used for the greater good.


Juras Juršėnas is COO at Oxylabs.io, a global provider of premium proxies and data scraping solutions. With more than 16 years of experience in the field, he has established himself as an expert in the areas of IT and product management. Connect with Juras on LinkedIn.

 

SUBSCRIBE TO OUR NEWSLETTER  newsletter icon.png

Get our best content, news, tips, and inspiration in your inbox - free.

No spam. Just great stories. Promise!
 

 

Join Over 20,000 Subscribers!

Get our best content, tips, and inspiration free in your inbox. Subscribe ››

Connect with us:  twitter.gif linkedin-gray.jpg email.gif RSS feed

 

 

 

 

 

Most read this week


man-city-view-from-the-ship-achieve-financial-freedom-financial-bucketlist
Attain Financial Freedom: 4 Things to Do & Put in Your Bucket List
Molly Barnes

woman-smiling-entrepreneur-writer-things-successful-writers-do-differently
10 Things Successful Writers Do Differently
Alexis Davis

NEOM Launches Infrastructure Work for the World’s Leading Cognitive Cities with stc
NEOM Launches Infrastructure Work for the World’s Leading Cognitive Cities in An Agreement with stc
Alexis Davis

10 Common Business Blogging Pitfalls to Avoid
How to Dress for Success and Look Stylish as Men
George Mathews

 

Got a story or tip for us?

 

Tips_0_0_0.png

Here's how to submit it →

 

 

 

 

EXPLORE MORE ...

black-nav-bar1.png

News & Features  ›


81% of Brits Plan to Support Small Businesses this Christmas [Study]

81% of Brits Plan to Support Small Businesses this Christmas [Study]

Americans Need an Accounting of Trump Tax Cuts and Jobs Act of 2017

Americans Need an Accounting of Trump Tax Cuts and Jobs Act of 2017

Why More People Are Choosing Platter Food Delivery for Parties


The Digital Playground: Creating Safe and Engaging Online Spaces for Kids

Understanding Fathers’ Rights in the Child Custody Process

81% of Brits Plan to Support Small Businesses this Christmas [Study]

hor-line-blue

Tech & Trends  ›


SSD vs HDD: Which One Is Better?

Understanding VR and 360 Videos: How Your Business Can Benefit from Them

Should You Hire an IT Outsourcing Company for Your Business?

Should You Hire an IT Outsourcing Company for Your Business?


5 Web Accessibility Issues to Avoid

Ethics of Quality Assurance Tech Companies Need to Follow

Pros and Cons of Mobile Technologies in Healthcare
 

hor_line_yellow

Arts & Culture  ›


church-building-architecture-religious-sector-adapting-pandemic

How Religious Organizations Adapted to the Pandemic

calypso-goddess-nymph-of-the-island-of-ogygia

The Epic Greek Mythology of Calypso Goddess, Nymph of The Island of Ogygia

Why Downtown Los Angeles is The Place to Live

Why Downtown Los Angeles Is The Place to Live


10 Fun Hobbies & Activities for Couples to Enjoy Together

5 Ways to Make Writing a Lot More Fun

Could You Be Obsessed with Writing?

hor-line-brown

Business & Economy  ›


Coronavirus Unemployment Nearly 15%, Shy of Great Depression Record High

The Science Behind Why Some Business Photos Make Us Click (And Others Don't)

Woman Worried One Dollar Bill in Wallet Bankruptcy Signs Image for Bankruptcy Barometer - 5 Signs You Might Be Headed for Bankruptcy

Bankruptcy Barometer - 5 Signs You're Headed for Bankruptcy

How Life Insurance Can Secure-Proof Your Financial Future


How Salesforce Anywhere Can Transform Remote Work With Real-Time Collaboration

Maximizing Device Compatibility with Restreaming and Packaging: Benefits for OTT Operators

Smooth Operator: 5 Daily Habits that Dramatically Reduce Repair Frequency

hor-line-green

Health & Style  ›


male-medical-practitioner-holding-model-heart-health

Heart Health: Strategies for a Strong & Resilient Cardiovascular System

8 Ways to Calm Your Mind When Things Look Bad

8 Ways to Calm Your Mind When Things Look Bad

[node:title]

Packing for a Trip? 5 Ways to Keep Your Formals Wrinkle-Free

hori-3.jpg

7 Must-Haves for Hiking, Fishing, and Other Outdoor Activities

hori-3.jpg

The Different Types of Wine Explained in a Nutshell

hori-3.jpg

Stop the Clock or Let it Tick? The Pro-Aging vs. Anti-Aging Dilemma
 

Home | About Us | Contributors | Submissions | Advertise | Disclosure | Privacy Policy | Contact Us

Follow Us:

twitter_e.jpg linkedin-pg.jpg email-updates_icon.jpg

Committed to quality content and journalistic ethics.

RSS rss

Search WWS search-icon-trans_0_1.png

© 2025 The WWS Daily.