Challenges for the Modern Economist

• Big data (but what is it really?)
• Data availability and accessibility (or its disappearing act)
• The conundrum of confidentiality and privacy
• Replicability (because of all of the above)

*

• I don't run my own surveys
• I occasionally create official statistics (bias!)
• I did my Ph.D. in the 1990s

or rather…

• 8 GB? I can't run it on my laptop, my RA might
• 7.5TB? 2 years worth of stock trades
• 309.3 million people, measured once? 10 questions on US decennial census
• 150 million people, measured 80 times? LEHD US linked employer-employee data
• 16,721,787,543 tables? QWI (derived from LEHD)
• 100 countries, 5-15 times? *IPUMS
• 19.3 billion records reduced to 112 weekly data points? = 10% of tweets over 24 months, UM Economic Indicators from Social Media (now up to twice that)

• treating admin/organic as a noisy data source, different from surveys
• designing administrative data collection with statistics in mind
• handling large data flows in commonly accepted ways
• novel confidentiality issues <!– - reconceptualizing multiciplity of data sources –>

“Respondent load should always be considered when planning a statistical collection and there should be policies and practices in place to manage relationships with respondents. The aim should always be to keep reporting load to the minimum and to maintain the high quality of collections.”

Australian National Statistical Service

• ACS-based county-to-county flows (5-year horizon) [survey]
• Tax-filing-based county-to-county flows (Yearly horizon) [admin: IRS]
• Workplace to residence flows (Daily horizon?) [matched admin]
• Workplace based county-to-county (job) flows (Quarterly horizon published, others possible) [admin: states]

• Underlying population is the same
• Discrete data products
• Reconciling differences

What about using organic data?

A Metadata Question

Google Consumer Survey

• Question: On the Canadian Census 2016, did you consent to making your information available in 2108 (in 92 years)?
• Goal: 500 responses
• Fielded: 2016-05-16
• As of: 2016-05-17 11:18, 229 responses live

Live download from https://www.googleapis.com/consumersurveys/v2/surveys/qht3vffpx6peusl5jopzboksqm/results

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "required",
    "message": "Login Required",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Login Required"
 }
}

• immediate estimate
• timeliness
• “enhancement” possible
• Cost: $50…

Google

• Google Trends
• Google Correlate

Two challenges:

• computation and storage
• access

“● A billion hours ago, modern homo sapiens emerged. ● A billion minutes ago, Christianity began. ● A billion seconds ago, the IBM PC was released. ● A billion Google searches ago … was this morning.”

Hal Varian

D Antenucci et al (2014)

• compared to weekly frequency of the official (US) UI claims (high frequency and without sampling error)
• high-frequency social media index tracks events that affect the job market in real time
• index has greater signal to noise ratio than the official initial claims series

• cost of storing 60-70+ TB of data
• scraping
• maintaining software

See later

Based on quarterly wage reports from 50 states and DC:

• research file (LEHD Snapshot through 2012Q1) has information on
  • 1,579,393,000 jobs for
  • 262,106,000 people in
  • 21,793,000 firms
• statistics re-computed quarterly (full-information model)

Clearly explained in the article:

• 10 percent sample of all Tweets (first 28 months = 43.8TB)
• analyze k-grams (\( k\leq4 \)), aggregated first to days and then weeks
• narrow the feature space further by using domain knowledge Question:
• how can we second-guess?

• Arecibo Radiotelescope (since 1963)
• Large Hadron Collider (since 2008)

600 million collisions every second = 1 PB/s

1 in a million of interest

HARDWARE pre-selects (0.01%) - throws away 99.99% of data! Forever!

SOFTWARE selects (1%) using 15,000 processors

stop

Computing gap

• Census Bureau vs. XSEDE
• Statistics Canada vs. Compute Canada

Challenges in

• moving data to compute resource, or
• compute resource to data

Administrative data and organic data increase the challenge

• Google and Twitter provide
  • storage resources
  • implement data reduction
  • … without scientific consensus

How are administrative data collected and stored?

• often complex and … organic
• sometimes obfuscated
• very heterogeneous

How are organic data collected and stored?

• incidental
• selectively?

Many of the examples above are “siloed” because of computational constraints.

Administrative data is often “siloed” for a combination of (perceived and real) confidentiality constraints and legal barriers.

LEHD succeeded in brining together 51 (+) state administrations, sharing their data to a trusted party.

Success: QWI

Less so: Researcher access (14 out of 51)

Mostly in confidential silos

Broad consensus on legal framework

Still some issues

28 countries

• 
• very little common data
• no common access to administrative data

10 provinces + 3 territories + 1 federal government

• 14 silos
• 

• More detailed data implies more data on specific individuals and firms
• Are protection methods sufficient?
  • Statistics offices are concerned
  • Administrations are concerned
  • Private providers are taking action, because they are concerned.

• apply methods, produce more public-use statistics (move the data to the researcher)
• fail to apply methods, provide controlled direct access to data (move the researcher to the data; contracts/RDCs/etc.)

One view

• Privacy is not asking
• Confidentiality is not revealing

Statistics Canada

• asks the respondents about the data on the Census form (compulsory response!) = confidentiality
• makes the decision to use the respondents' tax data (also compulsory response!) = privacy (or not)

• older methods break down as published data become denser
• newer methods are still being developed
  • synthetic data
  • noise infusion
• more robust methods
  • differential privacy

• tradeoff utility - protection
• how much protection do data providers require (people, firms, etc.)
• how much utility do stakeholders request (people, firms, government, etc.)
• what technology is available to implement?

Abowd and Schmutte, Revisiting the Economics of Privacy: Population Statistics and Privacy as Public Goods

• Data are more complex
• Data are more difficult to share
• Data are more siloed

Or maybe: Pervasive data

• the era of public-use data is declining
  • because other data are more interesting, richer
• data sources multiply…
  • … but also hide in new silos

• Confidentiality by obscurity is dead
• the era of public-use data is declining
  • because confidentiality is too easy to compromise
• data sources multiply…
  • … making the risk of confidentiality breach more obvious
  • … enforcing the creation of silos

Challenge is to

• fuse the silos (f.i. provincial data available in national network)
• develop new methods that are robust to confidentiality protection methods

• new “big data” methods will become part of the canon of applied economics
• will include methods absorbed from computer science, biology, physics
• will need to iterate towards a consensus what data to keep (à la physics)

• all of the above constitute a challenge to the scientific integrity of our findings
• for old-style data, greater replicability will be required
• for new-style data, new methods to allow for replication need to be developed
  • building big data infrastructure may require building an access mechanism as well (synthetic data, validation server)
  • social science archives need to make a magnitude leap in archiving capability
• funding