Anda di halaman 1dari 138

THE PROMISE OF

EVIDENCE-BASED POLICYMAKING
Report of the Commission on Evidence-Based Policymaking
Te Promise of
Evidence-Based Policymaking
Report of the Commission on Evidence-Based Policymaking

September 2017
This page intentionally left blank
The Promise of Evidence-Based Policymaking i

COMMISSIONERS
KATHARINE G.
SHERRY GLIED
ABRAHAM
ROBERT M. GROVES
CHAIR ROBERT HAHN
HILARY HOYNES
JEFFREY B. LIEBMAN
BRUCE D. MEYER
RON HASKINS PAUL OHM
CO-CHAIR NANCY POTOK
KATHLEEN RICE MOSIER
Commission on Evidence-Based Policymaking ROBERT J. SHEA
4600 SILVER HILL ROAD LATANYA SWEENEY
WASHINGTON, DC 20233 KENNETH R. TROSKE
WASHINGTON, DC 20233 KIM R. WALLIN

September 7, 2017

Te President of the United States


Te Speaker of the House
Te President of the Senate

Dear Sirs:

On behalf of the Commission on Evidence-Based Policymaking, and pursuant to Public Law 114-
140, we are pleased to submit the following report detailing our fndings and recommendations. We
believe that adoption of the recommendations contained in our report will prove transformational,
improving both the privacy protections aforded to the American public and the availability of
rigorous evidence to inform policymaking.

Over the course of its work, the Commission heard from numerous expert witnesses and members
of the public, received extensive written public comment, and felded a survey of Federal agencies.
Tis fact-fnding phase was followed by several months of deliberations. Te recommendations
contained in the report have been agreed to unanimously by the 15 Members of the Commission.

Te Commission envisions a future in which rigorous evidence is created efciently, as a routine


part of government operations, and used to construct efective public policy. Advances in technology
and statistical methodology, coupled with a modern legal framework and a commitment to trans-
parency, make it possible to do this while simultaneously providing stronger protections for the
privacy and confdentiality of the people, businesses, and organizations from which the government
collects information.

We call on the Congress and the President to act to ensure that this vision becomes a reality.

Respectfully yours,

Katharine G. Abraham, Chair Ron Haskins, Co-Chair

Sherry Glied Robert M. Groves Robert Hahn Hilary Hoynes Jefrey B. Liebman

Bruce D. Meyer Paul Ohm Nancy Potok Kathleen Rice Mosier Robert J. Shea

Latanya Sweeney Kenneth R. Troske Kim R. Wallin


This page intentionally left blank
The Promise of Evidence-Based Policymaking iii

Contents
Executive Summary: The Promise of Evidence-Based Policymaking ..................................... 1
Commission on Evidence-Based Policymaking Recommendations ...................................... 5

Chapter 1. Introduction: Vision for Evidence-Based Policymaking ................................... 7


Defining Evidence-Based Policymaking .................................................................................... 8
Evolution of Evidence Building in the United States.............................................................12
Commissions Charge and Process............................................................................................. 16
Realizing the Promise of Evidence-Based Policymaking ....................................................17

Chapter 2. Secure, Restricted Access to Confidential Data ............................................. 23


Findings.............................................................................................................................................. 24
Recommendations ......................................................................................................................... 40

Chapter 3. Enhancing Privacy Protections for Federal Evidence Building.................. 47


Findings.............................................................................................................................................. 50
Recommendations ......................................................................................................................... 60

Chapter 4. Modernizing Americas Data Infrastructure for Accountability


and Privacy ...................................................................................................................................... 65
Findings ............................................................................................................................................. 68
Recommendations ......................................................................................................................... 80

Chapter 5. Strengthening the Evidence-Building Capacity within the


Federal Government .................................................................................................................... 87
Findings.............................................................................................................................................. 88
Recommendations .........................................................................................................................101

Chapter 6. Conclusion: Possibilities with More and Better Evidence .......................... 105

Acronyms .............................................................................................................................................. 107


Appendix A: Evidence-Based Policymaking Commission Act of 2016 ..............................109
Appendix B: Commissioner Biographies.......................................................................................115
Appendix C: Commissions Fact-Finding and Deliberative Processes ............................... 119
Appendix D: Examples of Data Productive for Evidence Building...................................... 125
This page intentionally left blank
The Promise of Evidence-Based Policymaking 1

Executive Summary:
The Promise of Evidence-Based Policymaking

T he American people want a government that


functions efciently and responsibly ad-
dresses the problems that face this country. Pol-
and statistical methods, combined with trans-
parency and a strong legal framework, create
the opportunity to use data for evidence build-
icymakers must have good information on which ing in ways that were not possible in the past.
to base their decisions about improving the viabil- Tis report describes the Commissions fndings
ity and efectiveness of government programs and and presents recommendations for fundamental
policies. Today, too little evidence is produced to improvements to the Federal governments evi-
meet this need. dence-building systems and capabilities. Specif-
Te Commission on Evidence-Based Policy- ically, the Commissions report includes recom-
making (the Commission) envisions a future in mendations on (1) how the Federal government
which rigorous evidence is created efciently, as a can provide the infrastructure for secure access to
routine part of government operations, and used data, (2) the mechanisms to improve privacy pro-
to construct efective public policy. Te Federal tections and transparency about the uses of data
government has already taken important steps for evidence building, and (3) the institutional
towards accomplishing this vision, but much capacity to support evidence building.
work remains. Te growing interest in producing
more and higher-quality evidence to support de- Recommendations for
cision-making led the Congress and the President
to enact the Evidence-Based Policymaking Com- Improving Secure, Private, and
mission Act of 2016, creating the Commission. Confdential Data Access
Te Commission was provided just over a year
to study and develop a strategy for strengthening Tere are many barriers to the efective use of gov-
governments evidence-building and policymak- ernment data to generate evidence. Better access
ing eforts. During the Commissions fact-fnding to these data holds the potential for substantial
phase, numerous experts, researchers, govern- gains for society. Te Commissions recommenda-
ment leaders, public and private organizations, tions recognize that the countrys laws and prac-
and members of the public ofered their perspec- tices are not currently optimized to support the
tives on the Commissions charge. use of data for evidence building, nor in a manner
Based on this collective input, the Commission that best protects privacy. To correct these prob-
determined that greater use of existing data is now lems, the Commission makes the following rec-
possible in conjunction with stronger privacy and ommendations:
legal protections, as well as increased transparen-
cy and accountability. Te Commission believes Establish a National Secure Data Service to
that improved access to data under more priva- facilitate access to data for evidence building
cy-protective conditions can lead to an increase while ensuring privacy and transparency in
in both the quantity and the quality of evidence to how those data are used. As a state-of-the-art
inform important program and policy decisions. resource for improving governments capacity
Traditionally, increasing access to confdential to use the data it already collects, the National
data presumed signifcantly increasing privacy Secure Data Service will be able to temporarily
risk. Te Commission rejects that idea. Te Com- link existing data and provide secure access to
mission believes there are steps that can be tak- those data for exclusively statistical purposes
en to improve data security and privacy protec- in connection with approved projects. Te Na-
tions beyond what exists today, while increasing tional Secure Data Service will do this without
the production of evidence. Modern technology creating a data clearinghouse or warehouse.
2 The Promise of Evidence-Based Policymaking

Require stringent privacy qualifcations for de-identifed confdential data intended for
acquiring and combining data for statistical public release to improve how data are protected
purposes at the National Secure Data Service and risk is managed.
to ensure that data continue to be efectively
protected while improving the governments Adopt modern privacy-enhancing technologies
ability to understand the impacts of programs for confdential data used for evidence build-
on a wider range of outcomes. At the same ing to ensure that governments capabilities to
time, consider additional statutory changes keep data secure and protect confdentiality
to enable ongoing statistical production that, are constantly improving.
under the same stringent privacy qualifca-
tions, may make use of combined data. Assign senior ofcials the responsibility for
stewarding data within government depart-
Review and, where needed, revise laws autho- ments. Agencies should improve leadership,
rizing Federal data collection and use to ensure coordination, and collaboration when imple-
that limited access to administrative and survey menting protections for the use of confdential
data is possible to return benefts to the public data.
through improved programs and policies, but
only under strict privacy controls. Codify policies for maintaining integrity and
objectivity in Federal statistics to promote con-
Ensure state-collected quarterly earnings data tinued public trust in the accuracy of informa-
are available for statistical purposes, including tion being used to guide government decision-
to support the many evidence-building activities making.
for which earnings are an important outcome.

Make additional state-collected data about Recommendations for


Federal programs available for evidence build- Implementing the National
ing. Where appropriate, states that administer
Secure Data Service
programs with substantial Federal investment
should in return provide the data necessary for Te Commissions recommendations for improved
evidence building. data access and strong privacy protections rely
heavily on the establishment of the National
Develop a uniform process for external research- Secure Data Service. Being able to combine data
ers to apply and qualify for secure access to con- within a secure environment will be an increas-
fdential government data for evidence-building ingly vital aspect of the evidence-building com-
purposes while protecting privacy by carefully munitys capacity to meet future demand from
restricting data access to qualifed and approved policymakers. Increased transparency will enable
researchers. the public to be informed about how data are being
used to improve their government, even as data
are being stringently protected. Te Commission
Recommendations for envisions that the National Secure Data Service
Modernizing Privacy Protections will operate an efective and efcient service that
can be held accountable by policymakers and the
for Evidence Building
American public. Te Commissions recommenda-
Enhancements to privacy, coupled with improved tions to implement the National Secure Data Ser-
methods for secure data access, will revolutionize vice include:
how government uses and protects the data it col-
lects. Among the Commissions recommendations Build on the infrastructure and expertise al-
to achieve this vision are: ready developed in government, including at
the U.S. Census Bureau, to ensure that data
Require comprehensive risk assessments on linkages and access to confdential data for
The Promise of Evidence-Based Policymaking 3

statistical purposes are conducted in the most Develop learning agendas in Federal depart-
secure manner possible. ments to support the generation and use of
evidence to address the range of policymakers
Require public input, guidance, and partici- questions.
pation in the policies and procedures for data
linkage activities through public and stake- Improve coordination of government-wide evi-
holder representation on the National Secure dence building by directing the Ofce of Man-
Data Services steering committee. agement and Budget to facilitate cross-govern-
ment coordination, and consider how a greater
Establish a new transparency and accountabil- commitment to foundational information pol-
ity portal for evidence-building activities to icy responsibilities can be achieved, including
ensure the public is notifed about how con- through any consolidation or reorganization at
fdential data are used for evidence building the Ofce of Management and Budget that may
and to document routine audits for compliance be necessary.
with rules governing privacy, confdentiality,
and data access. Align administrative processes with evidence-
building activities, including those relating to
Innovate continuously on privacy-protective the approval of information collections and the
data access approaches with sufcient admin- procurement of services for evidence building.
istrative fexibilities to ensure government can
adjust as technology advances. Ensure that sufcient resources to support evi-
dence-building activities are available, includ-
Increase eforts to make information available ing resources to support implementation of the
about the governments current data invento- Commissions recommendations.
ries and supply related data documentation to
help researchers inside and outside govern- Conclusion
ment know which data they need to evaluate
programs and policies. Generating and using evidence to inform govern-
ment policymaking and program administration is
not a partisan issue. Te strategy described in this
Recommendations for report ofers a non-partisan approach to improv-
Strengthening Federal ing how government staf, private researchers,
foundations, non-profts, the business communi-
Evidence-Building Capacity
ty, and the public interact to make sure govern-
More privacy protective approaches and improved ment delivers on its promises.
access to data alone will not improve the volume Te Commissions recommendations represent
and quality of evidence. Te evidence-building a comprehensive strategy for tackling the greatest
community also needs sufcient capacity, admin- problems facing evidence building todaydata
istrative fexibilities, and appropriate program de- access is limited, privacy-protecting practices are
sign to enable a strong and efective evidence-gen- inadequate, and the capacity to generate the evi-
eration system to operate. To strengthen the dence needed to support policy decisions is insuf-
evidence-building capacity within the Federal fcient. Te Congress, the President, and the Amer-
government, the Commission makes the follow- ican people are ill-served by this state of afairs.
ing recommendations: Government must do what it takes to increase
the quantity and quality of evidence building.
Identify or establish a Chief Evaluation Ofcer Te strategy outlined in the Commissions report
in each department to coordinate evaluation simultaneously improves privacy protections and
and policy research and to collaborate with makes better use of data the government already
other evidence-building functions within Fed- collects to support policymaking. Together with
eral departments. leadership from the President and the Congress
4 The Promise of Evidence-Based Policymaking

in calling for credible evidence to support policy how to improve processes for efciently providing
decisions throughout government, implementa- services, evidence should play an important role
tion of the Commissions recommendations is an in key decisions made by government ofcials.
important step in providing the country with an Te Commission proposes modernizing the coun-
efective government. trys evidence-building capacity to make sure our
Whether deciding on funding allocations, as- governments decision-making process is among
sessing proposed regulations, or understanding the best in the world, now and in the future.
The Promise of Evidence-Based Policymaking 5

Commission on Evidence-Based
Policymaking Recommendations

Note: Recommendations in this report are numbered REC. 2-6: Te Congress and the President should
sequentially to align with the discussion. For example, enact statutory or other changes to ensure that
2-1 refers to the frst recommendation in Chapter 2. state-collected administrative data on quarterly
earnings are available for solely statistical purpos-
es. Te data should be available through a single
Improving Secure, Private, and Federal source for solely statistical purposes.
Confdential Data Access
REC. 2-7: Te President should direct Federal de-
REC. 2-1: Te Congress and the President should partments that acquire state-collected adminis-
enact legislation establishing the National Secure trative data to make them available for statisti-
Data Service (NSDS) to facilitate data access for cal purposes. Where there is substantial Federal
evidence building while ensuring transparency investment in a program, Federal departments
and privacy. Te NSDS should model best prac- should, consistent with applicable law, direct
tices for secure record linkage and drive the im- states to provide the data necessary to support ev-
plementation of innovative privacy-enhancing idence building, such as complete administrative
technologies. data when samples are already provided.

REC. 2-2: Te NSDS should be a service, not a data REC. 2-8: Te Ofce of Management and Budget
clearinghouse or warehouse. Te NSDS should fa- should promulgate a single, streamlined process
cilitate temporary data linkages in support of dis- for researchers external to the government to ap-
tinct authorized projects. ply, become qualifed, and gain approval to access
government data that are not publicly available.
REC. 2-3: In establishing the NSDS, the Congress Approval would remain subject to any restrictions
and the President should amend the Privacy Act appropriate to the data in question.
and the Confdential Information Protection and
Statistical Efciency Act (CIPSEA) to require new
stringent privacy qualifcations as a precondition Modernizing Privacy Protections
for the NSDS to acquire and combine survey and for Evidence Building
administrative data for solely statistical purposes.
At the same time, the Congress should consider REC. 3-1: Te Congress and the President should
additional statutory changes to enable ongoing amend the Privacy Act and the Confdential In-
statistical production. formation Protection and Statistical Efciency
Act (CIPSEA) to require Federal departments to
REC. 2-4: Te Congress and the President should conduct a comprehensive risk assessment on
review and amend, as appropriate, statutes such de-identifed confdential data intended for public
as Title 13 of the U.S. Code to allow statistical release. De-identifed confdential data subject to
uses of survey and administrative data for evi- the Privacy Act and CIPSEA should only be made
dence building within the CIPSEA secure envi- available after a disclosure review board (1) ap-
ronment. proves the release and (2) publicly provides the
risk assessment and a description of steps taken
REC. 2-5: Te Congress and the President should to mitigate risk.
consider repealing current bans and limiting fu-
ture bans on the collection and use of data for ev- REC. 3-2: Te President should direct Federal de-
idence building. partments, in coordination with the National
6 The Promise of Evidence-Based Policymaking

Secure Data Service, to adopt state-of-the-art da- REC. 4-4: Te NSDS should have specifc adminis-
tabase, cryptography, privacy-preserving, and pri- trative and implementation fexibilities including
vacy-enhancing technologies for confdential data the ability to leverage public-private partnerships
used for evidence building. and to collect and retain user fees.

REC. 3-3: Te President should direct Federal de- REC. 4-5: Te Ofce of Management and Budget
partments to assign a senior ofcial the responsi- should increase eforts to make information avail-
bility for coordinating access to and stewardship able on existing Federal datasets including data
of the departments data resources for evidence inventories, metadata, and data documentation in
building in collaboration with senior department a searchable format.
information technology, privacy, and other lead-
ers. A Principal Statistical Agency head, or oth-
er appropriately qualifed senior ofcial, should Strengthening Federal
serve this function. Evidence-Building Capacity
REC. 3-4: Te Congress and the President should REC. 5-1: Te President should direct Federal
enact legislation to codify relevant portions of Of- departments to increase capacity for evidence
fce of Management and Budget Statistical Policy building through the identifcation or establish-
Directive #1 to protect public trust by ensuring ment of a Chief Evaluation Ofcer, in addition to
that data acquired under a pledge of confdential- needed authorities to build a high performing evi-
ity are kept confdential and used exclusively for dence-building workforce.
statistical purposes.
REC. 5-2: Te Congress and the President should
direct Federal departments to develop multi-year
Implementing the National learning agendas that support the generation and
Secure Data Service use of evidence.

REC. 4-1: Te National Secure Data Service (NSDS) REC. 5-3: Te Congress and the President should
should be established as a separate entity in the direct the Ofce of Management and Budget
Department of Commerce that builds upon and (OMB) to coordinate the Federal governments
enhances existing expertise and infrastructure in evidence-building activities across departments,
the Federal government, especially at the Census including through any reorganization or consoli-
Bureau, to ensure sufcient capacity in secure re- dation within OMB that may be necessary and by
cord linkage and data access for evidence building. bolstering the visibility and role of interagency
councils.
REC. 4-2: Te NSDS should establish a Steering
Committee that includes representatives of the REC. 5-4: Te Congress and the President should
public, Federal departments, state agencies, and align administrative processes to support evi-
academia. dence building, in particular by streamlining the
approval processes for new data collections and
REC. 4-3: To ensure exemplary transparency and using existing fexibilities in procurement policy.
accountability for the Federal governments use
of data for evidence building, the NSDS should REC. 5-5: Te Congress and the President should
maintain a searchable inventory of approved proj- ensure sufcient resources to support evi-
ects using confdential data and undergo regular dence-building activities about Federal govern-
auditing of compliance with rules governing pri- ment programs and policies.
vacy, confdentiality, and access.
The Promise of Evidence-Based Policymaking 7

1
Introduction:
Vision for Evidence-Based Policymaking

W ith the passage and signing of the Evi-


dence-Based Policymaking Commission
Act in the spring of 2016, elected leaders issued
in the country, it is notable that this legislation
was embraced by legislators on both sides of the
aisle and enacted without dissent. U.S. House of
a bipartisan call to improve the evidence avail- Representatives Speaker Paul Ryan, a co-spon-
able for making decisions about government sor of the Act, described the potential for evi-
programs and policies.1 (See the box Charge to dence-based policymaking as a sea change in
the Commission on Evidence-Based Policymak- how we solve problems. Likewise, co-sponsor
ing.) In an environment of growing partisanship Senator Patty Murray said: Whether you think
we need more government, or less government
1. Evidence-Based Policymaking Commission Act of 2016 (Public
you should agree that we should at least have
Law 114140, March 30, 2016). better government.

Charge to the Commission on Evidence-Based


Policymaking

In the Evidence-Based Policymaking Commis- and related statistical data series may be
sion Act of 2016 (see Appendix A), the Congress integrated and made available for evidence
and the President prescribed a number of duties building while protecting privacy and
to the Commission, including the following: confdentiality.

Study the data inventory, data Make recommendations on how best to


infrastructure, database security, and incorporate evidence building into program
statistical protocols related to Federal design.
policymaking. Make recommendations on
how data infrastructure, database security, Consider whether a clearinghouse
and statistical protocols should be modifed. for program and survey data should be
established and how to create such a
Determine the optimal arrangement for clearinghouse.
which administrative data, survey data,
8 The Promise of Evidence-Based Policymaking

Te American people want a government that ican publics ability to hold the government ac-
solves problems. Tis requires that decision mak- countable. Adhering to the highest possible stan-
ers have good information to guide their choic- dards with respect to privacy and accountability
es about how current programs and policies are is an important part of earning the publics trust.
working and how they can be improved. While the Te improvements to privacy and accountability
Federal government has already taken steps to- that the Commission envisions can occur simulta-
wards developing an evidence culture, much re- neously with providing policymakers the tools to
mains to be done. A particularly important barrier deliver more efective government services.
to governments further progress is lack of access Te Commission envisions a future in which
by researchers outside of government and by indi- rigorous evidence is created efciently, as a rou-
viduals within government to the data necessary tine part of government operations, and used to
for evidence building, even when those data have construct efective public policy. While this may
already been collected. sound like a daunting task, the Commissions vi-
While collecting taxes, determining eligibility sion for the future of evidence-based policymaking
for government benefts, engaging in econom- in the United States is well within reach (see the
ic development, and running programs, govern- box Examples of the Promise of Evidence-Based
ment necessarily collects a considerable amount Policymaking). Tis vision requires that new laws
of information. In 2017, the American public will and policies be put into place. When implemented,
spend nearly 12 billion hours responding to more the Commission is confdent that the approaches
than 100 billion individual requests for informa- proposed in this report will greatly improve both
tion from the Federal government.2 Even though the ability to produce evidence in support of bet-
the direct costs of collecting these data are funded ter policies and privacy protections for individuals
by taxpayers, these data are not generally available and organizations.
for producing evidence. Addressing barriers to the
use of already collected data is a path to unlocking Defning Evidence-Based
important insights for addressing societys great-
est challenges. Policymaking
As the use of existing government data to sup-
port policymaking grows, the American public Evidence can be defned broadly as information
will be concerned about exactly how those data that aids the generation of a conclusion. Trough-
are being used and whether the privacy and con- out this report, the Commission uses the term in a
fdentiality of individuals and organizations are more specifc waythis report uses the shorthand
being protected. Today, data are protected, in part, evidence to refer to information produced by
through pledges of confdentiality, privacy laws, statistical activities with a statistical purpose
and legal and policy limitations on how they are that is potentially useful when evaluating govern-
used, but the governments approach to data pro- ment programs and policies. Following U.S. Ofce
tection has not kept pace with important changes of Management and Budget (OMB) Statistical Pol-
in technology. icy Directive #1, which in turn follows the Con-
Capabilities now exist to improve privacy pro- fdential Information Protection and Statistical
tections while making better use of already collect- Efciency Act of 2002 (CIPSEA), we defne sta-
ed administrative data, including recent advanc- tistical activities as the collection, compilation,
es in statistical methodology, computer science, processing, analysis, or dissemination of data for
and computational capacity. Growing experience the purpose of describing or making estimates
with successful legal models for data stewardship concerning the whole, or relevant groups or com-
points in the same direction. Government also ponents within, the economy, society, or the nat-
can dramatically improve transparency about its ural environment, including the development of
collection and use of data, improving the Amer- methods or resources that support those activities,
such as measurement of methods, statistical clas-
sifcations, or sampling frames. A statistical pur-
2. Ofce of Information and Regulatory Afairs, Ofce of Man- pose is defned as the description, estimation, or
agement and Budget, Inventory of Currently Approved Infor-
mation Collections; https://www.reginfo.gov/public/do/PRAReport? analysis of the characteristics of groups, without
operation=11 (accessed August 10, 2017). identifying the individuals or organizations that
The Promise of Evidence-Based Policymaking 9

Examples of the Promise of Evidence-Based Policymaking


Trough transactions with the American pub- safety net.4 Tese studies were carried out us-
lic, governments collect a considerable amount ing a combination of survey and administra-
of data. Tese administrative data, collected tive data, including administrative data from
in the frst instance to serve routine program locally operated Homeless Management Infor-
operation purposes, also can be used to assess mation Systems.
how well programs are achieving their intend- As a direct result of this growing body of ev-
ed goals.1 Below are examples where admin- idence, in recent years, the U.S. Department of
istrative data were used to generate evidence Housing and Urban Development (HUD) has
that informed government policies. encouraged and incentivized communities to
Permanent Supportive Housing. Tere increase their supply of permanent supportive
is a growing body of research on the impact housing for chronically homeless individuals
of providing permanent supportive housing over the past several years.5 Notably, there has
to chronically homeless individuals.2 Tis re- been a 27 percent reduction in chronic home-
search demonstrates that an intervention lessness nationally between the years 2010
combining long-term housing assistance and 2016.6
with supportive services can help chronically Substance Abuse Education. Te Drug
homeless individuals maintain stable housing Abuse Resistance Education program (DARE),
and achieve other positive outcomes, such as created in 1983, originally aimed to prevent
improved health outcomes and reduced use drug use and gang membership for kindergarten
of crisis services, including costly emergen- through 12th grade students in Los Angeles. In
cy room visits or stays in a homeless shelter.3 partnership with local law enforcement ofcers,
Cost-efectiveness studies of the intervention DARE grew into a national program focused pri-
also suggest that ofering permanent support- marily on drug prevention that at its peak was
ive housing to chronically homeless individu- in over 75 percent of the schools in the United
als with the highest service needs can reduce States and in more than 50 countries.7
taxpayer costs for other components of the More than 30 rigorous evaluations conduct-
ed throughout the 1990s and 2000s suggested
that the original DARE program did not produce
continues

1. Te Evidence-Based Policymaking Commission Act of 2016


(Public Law 114140, March 30, 2016) defnes administrative
4. S. R. Poulin, M. Maguire, S. Metraux, and D. P. Culhane. Ser-
data as data (1) held by an agency or contractor or grantee of
vice Use and Costs for Persons Experiencing Chronic Home-
an agency (including a State or unit of local government); and
lessness in Philadelphia: A Population-Based Study, Psychiat-
(2) collected for other than statistical purposes.
ric Services 61, no. 11 (2010): 10931098; M.E. Larimer, D.K,
2. Te U.S. Department of Housing and Urban Development Malone, M.D. Garner and others, Health Care and Public Ser-
adopted the Federal defnition, which defnes a chronically vice Use and Costs Before and After Provision of Housing for
homeless person as either (1) an unaccompanied homeless Chronically Homeless Persons With Severe Alcohol Problems.
individual with a disabling condition who has been continu- Journal of the American Medical Association, 301, no. 13 (April
ously homeless for a year or more, or (2) an unaccompanied 1, 2009): 13491357.
individual with a disabling condition who has had at least
5. See HUDs Fiscal Year 2018 Congressional Justifcation for
four episodes of homelessness in the past three years. (See
the Homeless Assistance Grant Program for more information;
11 Code of Federal Regulations 91 and 578, 2015.)
https://portal.hud.gov/hudportal/documents/huddoc?id=22-
3. Dennis P. Culhane and Tomas Byrne, Ending Chronic HomelessAGrants.pdf (accessed August 10, 2017).
Homelessness: Cost-Efective Opportunities for Interagency Col-
6. HUD, Te 2016 Annual Homeless Assessment Report (AHAR) to
laboration. Federal Strategic Plan Supplemental Document
Congress; Part I: Point-in-Time Estimates of Homelessness (Wash-
No. 19 (Washington, D.C.: U.S. Interagency Council on Home-
ington, D.C. U.S. Government Printing Ofce, 2016).
lessness, 2010); https://www.usich.gov/resources/uploads/asset
_librar y/DennisCulhane_EndingChronicHomelessness.pdf 7. For information about DARE America, go to http://www.
(accessed August 10, 2017). dare.org/about-d-a-r-e (accessed August 10, 2017).
10 The Promise of Evidence-Based Policymaking

Examples of the Promise of Evidence-Based Policymaking


continued
substantial reductions in teenage substance participants can realize improved earnings
abuse over the long-term.8 One study carried and employment outcomes, though the ev-
out in a suburban setting even found that the idence is mixed on specifc strategies.14 Te
intervention could contribute to increases in U.S. Department of Labors Adult Program as-
drug use.9 In 2001, the Surgeon General sum- sists people who are economically disadvan-
marized the available research and designated taged facing barriers to employment. Te Dis-
DARE as an inefective primary prevention located Workers Program assists workers who
program but also stated its popularity persists have been laid of or who have been notifed
despite numerous well-designed evaluations that they will be terminated or laid of. Both
and meta-analyses that consistently show little programs provide a range of training and sup-
or no deterrent efects on substance abuse.10 portive services. In a study using administra-
Te DARE program partnered with Penn- tive data in a non-experimental program eval-
sylvania State University to adopt a new ele- uation, researchers found that participants in
mentary and middle school curriculum called the adult program experienced an increase in
keepin it REAL.11 Today, the DARE program quarterly earnings relative to a comparison
focuses on a broader vision of empowering group, while participants in the dislocated
students to respect others and choose to lead workers program actually saw reduced earn-
lives free from violence, substance abuse, and ings in several quarters.15 When the workforce
other dangerous behaviors.12 Preliminary ev- investment programs were reauthorized in
idence from the revised curriculum suggests 2014 through the Workforce Innovation and
more promising efectiveness at achieving the Opportunity Act (WIOA), the new law provid-
stated goals related to decision-making.13 ed states additional fexibility to shift fund-
Workforce Investment. A large portfolio ing between the adult and dislocated worker
of evidence about workforce investments and aspects of the program to better target local
job training programs suggests that program needs. WIOA included numerous other evi-
dence-informed strategies based on the exist-
8. Greg Berman and Aubrey Fox, Lessons from the Battle Over ing portfolio of evidence.
DARE (Center for Court Innovation and Bureau of Justice As- Implementation of the permanent support-
sistance of the U.S. Department of Justice, 2009); http://www. ive housing, DARE, and workforce investment
courtinnovation.org/sites/default/fles/DARE.pdf (accessed Au-
gust 10, 2017). programs each were infuenced by evidence
developed to inform the implementation of
9. Dennis P. Rosenbaum and Gordon S. Hanson, Assessing the
Efects of School-Based Drug Education: A Six Year Multi-Level Federal policies. With more evidence to inform
Analysis of Project DARE, Journal of Research in Crime and De- a range of policy interests and questions, poli-
linquency 35, no. 4 (1998): 381412. cymakers will have a stronger basis for making
10. U.S. Department of Health and Human Services, Youth Vi- decisions in the future.
olence: A Report of the Surgeon General (Washington, D.C: De-
partment of Health and Human Services, 2001): 110.
14. Ofce of Management and Budget (OMB), Using Admin-
11. DARE America, Keepin it Real Elementary School
istrative and Survey Data to Build Evidence, white paper for
Curriculum; https://www.dare.org/keepin-it-real-elementary-
the Commission on Evidence-Based Policymaking (Wash-
school-curriculum (accessed August 10, 2017).
ington, D.C.: OMB, Executive Ofce of the President, 2016);
12. DARE America, D.A.R.E.s keepin it REAL Elementary https://obamawhitehouse.archives.gov/sites/default/fles/omb/
and Middle School Curriculums Adhere to Lessons From mgmt-gpra/using_administrative_and_survey_data_to_build_
Prevention Research Principles; https://www.dare.org/d-a-r- evidence_0.pdf (accessed August 10, 2017).
e-s-keepin-it-real-elementary-and-middle-school-curriculums-
15. Caroline J. Heinrich, Peter R. Mueser, Kenneth R. Troske,
adhere-to-lessons-from-prevention-research-principles (accessed
Kyung-Seong Jeon, and Daver C. Kahvecioglu, Do Public
August 10, 2017).
Employment and Training Programs Work? IZA Journal of La-
13. Randy Borum and David Allan Verhaagen, Assessing and bor Economics 2, no. 6 (2013); https://izajole.springeropen.com/
Managing Violence Risk in Juvenile (New York: Guilford, 2006). articles/10.1186/2193-8997-2-6 (accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 11

comprise such groups; and includes the develop- purposes such as the use of individual-level in-
ment, implementation, or maintenance of meth- formation to determine beneft amounts, enforce
ods, technical or administrative procedures, or in- laws, or otherwise afect the rights or privileges
formation resources that support such purposes.3 of an individual. Te identifcation of a single in-
We return to these defnitions in Chapter 2 in the dividual means the information is not being used
discussion of CIPSEA. Te essence of a statistical for statistical activities and thus, would not be
activity with a statistical purpose is that the re- termed evidence in the Commissions defni-
sult summarizes information about a group rather tion. Troughout this report, the Commission has
than a single individual or organization. For exam- been mindful that, consistent with applicable law,
ple, a statistical activity could include analyzing eforts to make data available specifcally for sta-
a unit, such as a state or a grantee, in order to tistical purposes might also inadvertently put in-
generate average values for all of the individuals formation about individuals at increased risk for
included within that unit, such as residents, cli- use in other ways. Te Commissions proposals at-
ents, or frms. tempt to ensure strict structural and institutional
Data can be used for many purposes other than separation between statistical and non-statistical
evidence building. Tese include non-statistical uses of data. Te Commissions evidence-building
reforms are engineered to make data difcult to
repurpose for non-statistical uses.
3. Statistical Policy Directive No. 1, Federal Register 79 (Decem-
ber 2, 2014): 7160971616; https://www.gpo.gov/fdsys/pkg/ Te Commission defnes evidence-based poli-
FR-2014-12-02/pdf/FR-2014-12-02.pdf. cymaking as the application of evidence to inform

Questions to Answer with More and Better Evidence

Te Commissions fact-fnding process pro- Do Farm Service Agency programs ease


duced numerous examples of important ques- credit constraints for farmers?
tions that individuals who provided input to
the Commission reported cannot currently What impacts do Federal economic devel-
be adequately addressed because of difculty opment eforts, such as the Appalachian
accessing the right data: Regional Commissions grants, have on the
communities they are trying to help?
What efects does the Supplemental Nu-
trition Assistance Program have on health What enforcement approaches are most
outcomes? efective in improving clean air regulation
compliance?
Are the earnings of veterans improved by
training received while in the military? Each of these questions could be studied
using administrative data that the government
Which transition-to-adulthood experiences already collects. Too often, however, the capac-
make students with disabilities less likely ity and infrastructure to study pressing ques-
to rely on the Supplemental Security In- tions faced by decision-makers are lacking. Te
come program? Commissions vision for evidence-based poli-
cymaking would enable each of these import-
To what extent do eligible active-duty mil- ant policy questions to be addressed with ap-
itary households participate in antipoverty propriate information analyzed in a secure and
programs and how does this participation privacy-protected environment, and then used
afect their economic self-sufciency? to improve government policies and programs.
12 The Promise of Evidence-Based Policymaking

decisions in government. For evidence-based pol- Evolution of Evidence Building in


icymaking to occur, a supply of evidence must
frst exist. Tus, the Commission recognizes that
the United States
evidence-based policymaking requires the gen- Te nations founders recognized the importance
eration of evidence, which relies on access to of information for governance, requiring in the
data. As the evidence base becomes stronger, the U.S. Constitution a census of the population (see
American public should expect that policymakers Figure 1).5 James Madison argued that collecting
increasingly will incorporate new and better ev- more data about the populace could guide con-
idence into their decisions about the operation gressional decisions about government actions as
of government programs and funding for govern- the young country grew.6 Early censuses gathered
ment services. information about industry, agriculture, and the
Diferent types of evidence are relevant for population. Census questions changed over time,
policymaking and may involve a variety of meth- refecting important societal and governmental
ods.4 Descriptive statistics provide insights about information needs of the day.
trends and context. Performance metrics support By the mid-nineteenth century, the Congress
monitoring of policy outputs and efciency. Im- had established several permanent units to pro-
plementation and process studies can identify duce national statistics in specifc policy areas.
how well the application of programs and policies More statistical units were added in subsequent
aligns with their intended design and goals. Im- years and formed the basic information infra-
pact evaluations provide insights about wheth- structure of the nations frst two centuries. Over
er desired outcomes are achieved. Each of these the years, the statistics on population size, edu-
types of evidence and others are relevant for ev- cation, employment, gross domestic product, and
idence-based policymaking, and the appropriate others became a routine dashboard on what was
approach depends on the policymakers question happening in society.
(see the box Questions to Answer with More and Today, the evidence-building community oper-
Better Evidence). ates under a range of laws, regulations, and poli-
Tis report uses the term evidence-building cies that evolved over time, a state of afairs that
community, which is meant to describe the col- has contributed to a lack of coordination and col-
lective set of individuals located both inside and laboration across the community. Numerous com-
outside the Federal government who fulfll a set missions or committees were convened during
of roles key to generating evidence for use in pol- the 20th century to recommend improvements
icymaking. Te evidence-building community in- for the countrys evidence-building system (see
cludes individuals situated across government and online Appendix H). Tese included recommen-
in the business, non-proft, and academic sectors. dations for greater coordination of activities and
Te community includes individuals who perform for enhancements to the protection of privacy.7
statistical activities, such as collecting data to pro- Information policy setting and coordination
duce national indicators relevant to the country. across government began in earnest in 1939 when
Te community includes researchers who study President Franklin D. Roosevelt issued an Execu-
ways to improve governments programs and pol- tive Order directing the Bureau of the Budget to
icies and evaluators who assess whether those plan and promote the improvement, development
programs and policies are achieving their intend- and coordination of Federal and other statistical
ed goals. Te community also includes individuals
who support program administrators with analy-
sis to achieve targeted improvements to their pro-
grams and policies. 5. U.S. Constitution Article I, Section 2.
6. Annals of Congress, House of Representatives, 1st Congress,
2nd Session Remarks by James Madison on the Bill for the 1790
Census (1790): 1145.
4. OMB, Using Administrative and Survey Data to Build Evi- 7. Janet Norwood, Organizing to Count: Change in the Federal Statis-
dence, 2016. tical System (Washington, D.C.: Urban Institute Press, 1995).
The Promise of Evidence-Based Policymaking 13

services.8 Te Congress subsequently provided Figure 1. Key Milestones in the


additional authority to conduct central reviews U.S. Evidence-Building System
for new data collections in government with the
Federal Records Act of 1942.9
1787
U.S.Constitution Signed,
Calls for Decennial Census
1960s and 1970s: Expansion of Evidence
Building and Privacy Protections 1903
Commission Appointed by
As more data were collected and used by govern- Secretary of the U.S. Treasury to
ment for implementing programs and for statis- Investigate Accounting Methods
tical activities, the need for privacy protections
1942
became increasingly apparent. In 1973, the U.S. Federal Reports Act
Department of Health, Education, and Welfare
developed the Fair Information Practice Principles
Federal Records Act (FRA)
(FIPPs), recommending that the Congress enact 1950
them into what eventually became the Privacy Act Budget and Accounting
of 1974.10 Te principles include transparency, in- Procedures Act
dividual participation, purpose specifcation, data
minimization, use limitation, data quality and in-
tegrity, security, and accountability and auditing.
Dept. of Health, Education, and
Tese principles strive to balance the need for in- 1973 Welfare Develops Fair Information
formation with privacy protections for the beneft Practice Principles (FIPPs)
1974
of the American public.
Te Privacy Act also codifed some U.S. infor- Privacy Act
mation practices, establishing common require- 1977
ments related to collecting, maintaining, using, Privacy Protection Study
Commission
and disseminating government records about in-
dividuals. Te Privacy Act articulated basic trans- 1980
Paperwork Reduction Act (PRA)
parency requirements and limitations on how data
collected by the government may be disclosed. In
1977, the Privacy Protection Study Commission
created by the Privacy Act conveyed two central 1993
Government Performance and
tenets for evidence building. First, research and Results Act (GPRA)
statistical uses of data about individuals must ex- 1995
PRA Reauthorized
clude any result that would directly afect an in-
dividuals rights, privileges, or benefts. Second,
2000
government statistical and non-statistical uses of Information Quality Act
data should be separated by a bright line, a prin- Confdential Information
ciple referred to as functional separation.11 Tese 2002
Protection and Statistical
Eciency Act (CIPSEA)

8. Executive Order 824: Establishing the Divisions of the Exec- E-Government Act
utive Ofce of the President and Defning Teir Functions and
2010
Duties, 3 C.F.R. (September 8, 1939); https://www.archives.gov/ GPRA Modernization Act
federal-register/codifcation/executive-order/08248.html (accessed
August 10, 2017).
2016
9. Records Management by Federal Agencies, 44 USC 3101 et. seq. Commission on Evidence-Based
10. U.S. Department of Health, Education, and Welfare, Records, Policymaking Created
Computers, and the Rights of Citizens: Report of the Secretarys Adviso-
ry Committee on Automated Personal Data Systems (1973).
Legend Laws Commissions
11. Personal Privacy in an Information Society, Te Report of the Pri-
vacy Protection Study Commission, July 1977. *Timeline not to scale
14 The Promise of Evidence-Based Policymaking

two core concepts carry forward into the nations form the Federal Statistical System.15 Te PSAs
privacy protection and statistical laws, and remain beneft from explicit statutory authorities and
key to this Commissions work. an established culture that supports secure and
Amidst discussions about improving privacy, confdential data collection and stewardship. Ac-
approaches to generating and using evidence ex- cording to the American Statistical Association,
panded. Te U.S. Department of Defense intro- Federal statistical agencies will play a vital role in
duced cost-beneft analysis as part of budgeting evidence-based policymaking andthe historical
activities and the technique gained further trac- autonomy of these agencies [is needed] in order
tion in the regulatory infrastructure.12 Similarly, in to ensure the integrity of their work.16 Te Com-
the 1960s and 1970s, the Department of Health, mission agrees with this assessment. Coordina-
Education, and Welfare pioneered many of the ap- tion for the statistical system is supported by the
proaches for executing governments evaluation statutory Interagency Council on Statistical Policy
function in partnership with contractors, grant- and the Federal Committee on Statistical Method-
ees, and academics, to meet the need for better ology, and led by the Chief Statistician in OMB.17
information about the impacts of Great Society OMB also issues standards and directives to assure
programs. Many of these eforts created the basic the quality and integrity of activities conducted
infrastructure and resource mechanisms to sup- within the system.
port evaluation of education, human services, and With the Government Performance and Results
health programs. Act of 1993 (GPRA), the Congress also established
an expectation for continuous improvement in
1980s and 1990s: Improved Evidence government by creating government-wide re-
quirements for departments to set goals, measure
Coordination to Meet Demand
results, and report progress.18
As evidence-building activities became more
common across government, gaps in coordina- 2000s: Strengthened Protections for
tion mechanisms became more apparent. Te Pa-
Statistical Activities
perwork Reduction Act (PRA) of 1980 specifcally
recognized the role of the OMB in coordinating In recognition of the need to better protect data
government-wide information and statistical pol- collected by the Federal Statistical System, CIPSEA
icy, which the Congress reafrmed during a 1995 established common legal authorities for statisti-
reauthorization. Te PRA specifcally encourages cal activities. Consistent with the recommenda-
data sharing for statistical activities as govern- tions of the Presidents Reorganization Project for
ment policy, both to improve the efciency and the Federal Statistical System twenty years earlier,
the quality of statistical series and of evidence CIPSEA provides statutory protections that apply
more broadly.13 to all 13 PSAs and other designated units for data
Within the decentralized Federal evidence- collected under a pledge of confdentiality for sta-
building apparatus, 13 Principal Statistical Agen- tistical activities.19
cies (PSAs) in 11 Federal departments provide the
core infrastructure to support governments main 15. OMB, Statistical Programs of the United States Government
statistical activities and provide access to statis- (Washington, D.C.: 2017); https://obamawhitehouse.archives.gov/
tical datasets for research (see Figure 2).14 Tese sites/default/fles/omb/assets/information_and_regulatory_afairs/
statistical-programs-2017.pdf (accessed August 10, 2017).
13 agencies, along with nearly 100 smaller units
embedded within program agencies, collectively 16. Clyde Tucker, American Statistical Association, Commission
Public Hearing, Washington, D.C., October 21, 2016.
17. Te Chief Statistician role and the ICSP are established under
the Paperwork Reduction Act, 44 USC 3504(e) (1995).
18. P. G. Joyce, Te Obama Administration and PBB: Building on
the Legacy of Federal Performance-Informed Budgeting? Public
12. Edward P. Fuchs and James E. Anderson, Te Institutionaliza-
Administration Review, 71, no. 3 (2011): 356367.
tion of Cost-Beneft Analysis, Public Productivity Review 10, no. 4
(1987): 2533. 19. James T. Bonnen, Teodore G. Clemence, Ivan P. Fellegi, Tom-
as B. Jabine, Ronald E. Kutscher, Larry K. Roberson, Charles A.
13. Paperwork Reduction Act, 44 USC 3506 (1995).
Waite, Improving the Federal Statistical System: Report of the
14. Troughout this report, the term departments is generally Presidents Reorganization Project for the Federal Statistical Sys-
used to refer to the 24 Federal agencies subject to the Chief Finan- tem, American Statistician, 35, no. 4 (November 1981, reprint):
cial Ofcers Act of 1990 (Public Law 101576). 184196.
The Promise of Evidence-Based Policymaking 15

Te Privacy Act, CIPSEA, and the PRA ofer the Figure 2. Federal Principal
legal foundation for coordinating information pol- Statistical Agencies (PSAs)
icy and protecting data used in evidence building,
but much work remains to realize the promis- U.S. Department of Agriculture
es embodied in these laws. For example, Federal Econo ic Research Service
evaluation and policy research ofces are more
nascent in their development than the statistical National Agricultural Statistics Service
agencies. As such, none of them are recognized
under CIPSEA to employ the strong protections
U.S. Department of Commerce
that statistical agencies can provide for confden-
tial data. Bureau of Econo ic Analysis

Census Bureau
2010s: Increased Demand for Evidence
Reauthorizations of programs by the Congress U.S. Department of Education
during the 2010s provided numerous examples of
National Center for Education Statistics
the increased demand for evidence to inform pol-
icy. Te GPRA Modernization Act of 2010 reiterat-
ed the Congresss interest in better use of evidence U.S. Department of Energy
for continuous improvement in government.20
Energy Infor ation Ad inistration
Te Workforce Innovation and Opportunity Act in
2014 authorized numerous evaluation and mul-
tisite projects. Te Agricultural Act of 2014 called U.S. Department of Health and Human Services
for testing of innovative approaches to helping in-
National Center for Health Statistics
dividuals gain and retain employment that leads
to self-sufciency.21 Te Bipartisan Budget Act of
2015 reauthorized the Social Security Disability U.S. Department of Justice
Insurance programs demonstration authority to
Bureau of Justice Statistics
test policy improvements.22 Te Every Student
Succeeds Act in 2015 reauthorized education pro-
grams and expanded the use of evidence-based U.S. Department of Labor
grants.23 Numerous other laws and draft legisla-
Bureau of Labor Statistics
tion embody the increasing demand for evidence
such as by incorporating requirements about us-
ing evidence or supporting innovation in award- U.S. Department of Transportation
ing grants.
Bureau of Transportation Statistics
Within the Executive Branch, calls encouraging
more and better evidence also emerged. In 2013,
a memorandum to Federal departments specif- U.S. Department of the Treasury
cally encouraged using evidence that is available,
Internal Revenue Service
Statistics of Inco e Division
20. GPRA Modernization Act of 2010 (Public Law 111352, Jan-
uary 4, 2011); https://www.congress.gov/111/plaws/publ352/PLAW-
111publ352.pdf (accessed August 10, 2017). National Science Foundation
21. Agricultural Act of 2014 (Public Law 11379, February 7, 2014); National Center for Science and
https://www.congress.gov/113/plaws/publ79/PLAW-113publ79.pdf Engineering Statistics
(accessed August 10, 2017).
22. Bipartisan Budget Act of 2015 (Public Law 11474, Sec. 821,
November 2, 2015); https://www.congress.gov/114/plaws/publ74/ U.S. Social Security Administration
PLAW-114publ74.pdf (accessed August 10, 2017).
Oce of Research,
23. Every Student Succeeds Act (Public Law 11495, Decem-
Evaluation, and Statistics
ber 10, 2015); https://www.congress.gov/114/plaws/publ95/PLAW-
114publ95.pdf (accessed August 10, 2017).
16 The Promise of Evidence-Based Policymaking

producing new evidence to increase knowledge, search organizations hosting these centers. Te
and continually innovating to improve govern- centers provide researchers with secure access to
ment performance.24 And in 2017, the President confdential data, and the statistical agencies and
specifcally called for generating and using evi- general public beneft from research fndings that
dence to improve governments efectiveness.25 make important contributions to statistical prod-
ucts and general knowledge.
Partnerships for Policy Research and
Evaluation
Commissions Charge and
Te capacity within Federal departments to per- Process
form policy research and evaluation can be ar-
rayed along a continuum, ranging from a strong Continuing this history of evidence-building ac-
centralized approach to dispersed capacity across tions in the United States, the Commission on
bureaus and units. Discussing the existing track Evidence-Based Policymaking was established
record within the Federal government, the Amer- by the bipartisan Evidence-Based Policymaking
ican Evaluation Association observed that for the Commission Act of 2016 (Appendix A), jointly
most part, these evaluations have been sporadic, sponsored by Speaker of the House Paul Ryan
applied inconsistently, and supported inadequate- and Senator Patty Murray and signed by Presi-
ly. Training and capacity building for evaluation dent Barack Obama on March 30, 2016. Te Act
have been inconsistent across agencies and, in directed the Commission to consider how to
many cases, insufcient to achieve the needed strengthen governments evidence-building and
evaluation capacity and sustain it over time.26 Te policymaking eforts. Specifcally, the Act direct-
Commission agrees with this assessment. ed the Commission to study how the data that
Individuals outside of government, such as government already collects can be used to im-
non-governmental researchers, philanthropic prove government programs and policies. Te
organizations, universities, researchers, and oth- Commission was directed to submit to the Con-
er partners, play an essential role in supporting gress and the President a detailed statement of
and extending the evidence-building community its fndings and recommendations.
within government. Evidence produced by exter- Te Commission recognizes that data are an
nal actors can infuence policymakers, encourage important building block for the generation of
policy innovation, and drive the allocation of re- evidence and that many of the greatest gains for
sources across new and existing programs. More- evidence building in the near term can be accom-
over, external actors create a capacity that goes plished by addressing the challenge of secure, pri-
well beyond that of government staf. vate, and confdential data access. A theme that
Federal departments and universities have ini- runs throughout this report is that access to conf-
tiated collaborations to improve evidence-build- dential data for evidence-building purposes should
ing activities. For example, the Federal Statistical be increased, but only in the context of a modern
Research Data Centers operate today as partner- legal framework providing for strengthened priva-
ships between statistical agencies and the re- cy protections and increased transparency. Taking
up the privacy challenge, the report recommends
specifc steps to improve privacy and transparen-
24. OMB, Next Steps in the Evidence and Innovation Agenda cy with regard to evidence-building activities. Te
(memorandum M1317); https://obamawhitehouse.archives.gov/ report also recognizes the need to strengthen the
sites/default/f iles/omb/memoranda/2013/m-13-17.pdf (accessed
August 10, 2017).
governments institutional capacity to support the
evidence-building system.
25. Executive Ofce of the President, Building and Using Ev-
idence to Improve Government Efectiveness, in Analytical Per- Following the Commissions frst meeting in
spectives: Budget of the United States for Fiscal Year 2018 (March July 2016, Commissioners engaged in an eight-
2017); https://www.whitehouse.gov/sites/whitehouse.gov/fles/omb/ month-long fact-fnding process to learn about
budget/fy2018/ap_6_evidence.pdf (accessed August 10, 2017).
the current state of evidence production and use
26. American Evaluation Association (AEA), An Evaluation Road- in the Federal government, as well as the Federal
map for a More Efective Government (Washington, D.C.: AEA, re-
vised October 2013); http://www.eval.org/d/do/472 (accessed Au- governments policies and practices to protect data
gust 10, 2017). confdentiality (see Appendix C). Te Commission
The Promise of Evidence-Based Policymaking 17

solicited feedback from the public and experts by tions, the Commission developed a set of fve guid-
doing the following: ing principles that shaped its recommendations
(see the box Guiding Principles for Evidence-Based
Administering a survey to 209 ofces of the Policymaking). Te fact-fnding and deliberations
Federal government that the Commission using these principles culminated in the conclu-
identifed as likely to be generating or using sions presented in this report and a strategy for
evidence, in order to understand the current realizing the promise of evidence-based policy-
or potential capacity of Federal agencies to making. Each of the 15 appointed Commissioners,
engage in aspects of evidence-based policy- forming a bipartisan Commission with a range of
making (see online Appendix E). perspectives on the issues examined, approved this
fnal report and the recommendations herein.
Convening seven public meetings to hear
from 49 invited expert witnesses on a range of
issues (see online Appendix F). Realizing the Promise of
Evidence-Based Policymaking
Issuing a Request for Comments in the Federal
Register and accepted comments by email, Te Commission believes that enabling improved
which generated over 350 responses from the access to data under more modern privacy-pro-
public (see online Appendix G). tective conditions will lead to more and better
evidence. Improvement in access can occur safe-
Holding three open public hearings ly with improved privacy protections. Tis report
in Washington, D.C., Chicago, and San maps a path to realizing this vision.
Franciscoduring which 39 members of
the public presented information directly to Vision for Improving Secure Access to
Commissioners (see online Appendix G).
Confdential Data
Meeting with more than 40 organizations or During the Commissions fact-fnding phase, Com-
experts to solicit additional input. missioners heard about the many barriers to efec-
tive use of the data government already collects
Following the fact-fnding phase of the Commis- to generate evidence. Te Commission believes
sions work, Commissioners launched a fve-month that enabling better use of such data will ofer
deliberative process to consider all of the gathered substantial gains for society. Te Commission also
input and information. At the outset of its delibera- believes that by better coordinating the access to

Guiding Principles for Evidence-Based Policymaking

1. Privacy. Individual privacy and confden- ful channels for public input and comment
tiality must be respected in the generation and ensuring that evidence produced is
and use of data and evidence. made publicly available.
2. Rigor. Evidence should be developed using 4. Humility. Care should be taken not to
well-designed and well-implemented over-generalize from fndings that may be
methods tailored to the questions being specifc to a particular study or context.
asked. 5. Capacity. Te capacity to generate and
3. Transparency. Tose engaged in gener- use data and evidence should be integrated
ating and using data and evidence should within government institutions and ade-
operate transparently, providing meaning- quately funded and stafed.
18 The Promise of Evidence-Based Policymaking

administrative data, government can improve the art capacity for integrating existing data and pro-
security of these data while better protecting indi- viding secure data access for exclusively statistical
vidual privacy. purposes (REC. 2-1). Te NSDS will model best
Chapter 2 describes the current faws in the practices for secure record linkages, propel im-
countrys data infrastructure and capacity to use plementation of privacy-enhancing technologies,
data for evidence building. While the countrys and coordinate a new level of transparency about
data infrastructure and capacity have evolved sub- how government data are used for evidence build-
stantially over the past two and a half centuries, ing. Importantly, the NSDS will facilitate tempo-
the Federal government, as a whole, has not kept rary linkages only for authorized projects, and will
pace with emerging privacy-protecting technolo- not serve as a data warehouse (REC. 2-2).
gies and protocols to support secure access to data. As explained in Chapter 2, the countrys laws
While there are pockets of innovation, our govern- are not currently optimized either to support the
ment has not yet broadly adopted approaches for use of data for evidence building across programs
using data to generate insights that can enhance or to maximize privacy. Increasing connections
decision-making that are becoming much more between data used in evidence building across
prevalent in the private sector. Some other coun- topical areas will improve our ability to under-
tries also have more consistently implemented the stand the impacts of programs on a wider range
capabilities needed for developing and using high of outcomes. Some existing authorities enable
quality evidence to inform public policy. sharing of information across agencies within the
Federal government and allow access for non-Fed-
Data are the lifeblood of eral actors. Tese approaches could be expanded
to enhance the analytical capabilities to cut across
decision-making and the raw additional policy domains.
material for accountability. Inconsistent laws and interpretations of le-
Without high-quality data gal authorities for securely sharing and using
confdential data have inhibited the ability of
providing the right information
the evidence-building community to access the
on the right things at the right types of data. Te Commission proposes
righttime; designing, monitoring to amend the Privacy Act and CIPSEA to require
and evaluating efective policies stringent privacy qualifcations for acquiring and
combining data for statistical purposes, to en-
becomes almost impossible. sure that data continue to be efectively protect-
United Nations Secretary-Generals ed while improving the governments ability to
Independent Expert Advisory Group understand the impacts of programs on a wider
on a Data Revolution for Sustainable range of outcomes (REC. 2-3). In some cases, the
Development27 purposes for which administrative data may be
used are defned narrowly, preventing their use
As the amount of data available increases ex- for evidence building. Te Commission proposes
ponentially, improving governments capacity a review of such statutes to ensure that limita-
to securely analyze information that crosses the tions that preclude the use of administrative data
silos of government is an ongoing challenge. Te for evidence building are applied only when the
Commission proposes to answer this challenge by Congress and the President deem the limitations
building on and reorganizing existing resources still to be necessary (REC. 2-4). In some cases,
within government through the National Secure existing laws specifcally prohibit the collection
Data Service (NSDS), establishing a state-of-the- or analysis of information to support evidence
building. Again, the Commission calls for a re-
consideration of such bans and restraint in the
27. United Nations, Secretary Generals Independent Expert Ad-
visory Group on a Data Revolution for Sustainable Development. enactment of future bans (REC. 2-5).
A World Tat Counts: Mobilizing the Data Revolution for Sustainable Chapter 2 also highlights the value of data
Development (New York, NY: United Nations, November 2014); http:// collected by states and other jurisdictions about
www.undatarevolution.org/wp-content/uploads/2014/12/A-World-
Tat-Counts2.pdf (accessed August 10, 2017). Federal programs and policies, especially data
The Promise of Evidence-Based Policymaking 19

on income, wages, and earnings, while noting unevenly across government. Many are not dy-
that these data in most cases are not currently namic enough to meet the ever-changing risks
being used productively for evidence building.28 associated with the use of data. While the Federal
Multiple programs supported with Federal funds government is subject to many transparency re-
and administered by states and other jurisdic- quirements, government could be more open in
tions collect income, wage, or earnings data in explaining how individuals data are used for evi-
the course of program operations. Some of these dence building.
data are provided to the Federal government for Chapter 3 discusses privacy as a broad concept
specifc purposes, but numerous researchers and encompassing the full range of the Fair Informa-
evaluators reported to the Commission that ac- tion Practice Principles, which provide a useful
cessing the data for all states is nearly impossible. framework for the Commissions recommenda-
Te Commission believes that data about earnings tions. Confdentiality is a distinct but overlapping
are among the most important for enhancing the concept in that it refers specifcally to the protec-
generation of evidence about Federal government tion of information about individuals or other en-
programs. Te Commission strongly encourag- tities from unauthorized disclosure.29 Confden-
es the Congress and the President to make these tiality is a component of privacy, but protecting
data available for statistical purposes (REC 2-6). and respecting privacy entails substantially more
In addition, many Federal programs rely on states than just ensuring the confdentiality of informa-
and other jurisdictions to collect data, but those tion.For convenience, the Commission often uses
data are not currently available for evaluating the terms privacy or privacy protection in the
programs and policies. An expectation should be broader sense of the full range of Fair Informa-
established that these data will be available for ev- tionPractice Principles, including confdentiality.
idence building (REC. 2-7). Terefore, throughout this report, privacy and pri-
A related issue frequently reported to the vacy protection should be interpreted to include
Commission is the confusion caused by variation the confdentiality of individuals information,
across agencies in the approval processes through and, where appropriate, the protection of infor-
which external researchers establish eligibility to mation about businesses and other entities for
access confdential data held by Federal depart- which the term privacy in a narrower sense may
ments. Te Commission recognizes that while not normally apply.
embedding a certain amount of friction in these Chapter 3 describes how risks to privacy have
processes can enhance privacythat is, that ac- evolved and shows why they must be continu-
cess should not be too easybut needless varia- ally assessed as new information is made more
tion can create unintentional barriers that serve publicly available. Government has an obligation
no deliberate purpose. To better facilitate access to be open and honest with the American pub-
while protecting privacy, the Commission recom- lic. Te publics trust can be earned only through
mends the creation of a single process for exter- transparency about these risks. Te Commission
nal researchers to apply and qualify for access to recommends that, prior to any public release of
sensitive government data that are not otherwise de-identifed confdential data, Federal depart-
publicly available, subject to any restrictions ap- ments assess the risks that may be associated with
propriate to the data in question (REC. 2-8). the release and the steps taken to mitigate those
risks, then make these risk assessments publicly
Vision for Modernizing Privacy available (REC. 3-1). Tis recommendation is one
of several intended to improve how the Federal
Protections for Evidence Building
government manages privacy risks.
Many existing privacy protections in the Unit- As technologies evolve, the capabilities to keep
ed States are strong, but protections are applied data secure and protect confdentiality are con-
stantly improving. Te Commission calls on gov-
28. Troughout this report, the phrase other jurisdictions should
ernment to make better use of state-of-the-art
generally be interpreted to mean Tribes, territories, local govern-
ments (such as cities, counties, or districts), or other governing
entities. 29. Personal Privacy in an Information Society, 1977.
20 The Promise of Evidence-Based Policymaking

approaches that can be applied for enhancing se- held accountable for safely managing those risks.
curity and confdentiality for data used in evidence Te Commission recommends the creation of a
building (REC. 3-2). single new Federal transparency portal to ensure
With respect to all of these activities, the Com- that information about approved evidence-build-
mission recognizes a need for leadership with- ing activities that rely on confdential data is read-
in Federal departments. Te Commission rec- ily available. Ten, routine audits of government
ommends that a senior ofcial be designated to and external users of data should be carried out
ensure appropriate collaboration with regard to to ensure that, as promised, data are accessed in
decisions about data stewardship and the imple- the most privacy-protective fashion and for solely
mentation of privacy-protective measures within statistical purposes (REC. 4-3).
each department (REC. 3-3). Te Commission also Approaches to statistical analysis of govern-
recommends that existing policies for maintaining ment data that protect privacy will continue to
the integrity and objectivity of Federal statistics evolve as technologies advance. Government will
be codifed in law, maintaining the publics trust need to innovate continuously with respect to pri-
by ensuring that the confdential data used to pro- vacy-protective approaches. Te NSDS must play
duce these statistics continue to be protected and an important role in the process of continuous
used only for statistical purposes (REC. 3-4). improvement and be aforded the administrative
Tese enhancements to privacy, coupled with fexibilities to support innovation and to engage
the improved strategy for secure data access out- in partnerships that can bring needed expertise to
lined in Recommendation 2-1, will advance how bear (REC. 4-4).
government uses and protects the data it already Improving protections and capabilities for bet-
collects, allowing the data to be accessed securely. ter using data also requires new approaches to
understanding what data exist about Federal pro-
Implementing the Vision grams and policies. Te Commission recommends
the creation of a complete inventory of Federal
A core feature for implementing the Commissions data assets with technical information that will
vision is the creation of the NSDS as a new service help members of the evidence-building commu-
that builds on and enhances existing Federal gov- nity inside and outside government know which
ernment data infrastructure for statistical activi- data are available to evaluate programs and poli-
ties (REC 2-1). Te ability to temporarily combine cies (REC. 4-5).
identifable data within a secure environment, Even after the NSDS data capabilities have
then remove direct identifers for individual proj- been established, much work will remain to im-
ects is a vital element of the evidence-building plement other aspects of the Commissions vision.
communitys ability to meet future demand from More privacy protective approaches and improved
policymakers. For the NSDS to be a role model for access to data alone will not improve the volume
the use of data and held accountable by the Amer- and quality of evidence. Ultimately, the capacity
ican public, it should have fve core features. Tese of the Federal evidence-building community must
are described further in Chapter 4. be strengthened to increase its efcacy.
First, the NSDS should build on the infrastruc- Chapter 5 emphasizes the need for leadership,
ture and expertise already developed across gov- coordination, and resources to facilitate efcient
ernment, such as at the Census Bureau, to ensure evidence-building activities that support contin-
that statistical analyses of confdential data are uous learning in the Federal government. Depart-
conducted in the most secure manner possible ments need to coordinate their evidence-building
(REC. 4-1). As the data work of the NSDS pro- functions internally to ensure the activities can be
ceeds, the public must be aforded opportunities efectively and efciently implemented. Te Com-
to participate by providing input and guidance on mission recommends that Federal departments
the policies and procedures for conducting these identify or establish a Chief Evaluation Ofcer
eforts through representation on the NSDSs to encourage and coordinate policy research and
steering committee (REC. 4-2). evaluation (REC. 5-1). Programs can also be more
Te statistical uses of confdential data can efectively designed to enable evidence building.
pose risks to privacy, and government must be Te Commission recommends that departments
The Promise of Evidence-Based Policymaking 21

create learning agendas that outline priorities and be applied or created to tailor these processes to
support the development of all types of evidence better facilitate the need for increased evidence
to help address the range of policymakers ques- building (REC. 5-4). Finally, appropriate resourc-
tions (REC. 5-2). es must be available to support evidence build-
In addition to coordination within depart- ing, including resources to enable the full suite of
ments, better coordination is also needed across activities discussed throughout this report, from
government. OMB and Federal departments must data stewardship to risk assessments to the evalu-
be organized to implement this vision. Tis may ations of individual programs (REC. 5-5).
require consolidation or reorganization of certain In short, government must improve its exist-
activities at OMB where information policy and ing infrastructure and legal frameworks to enable
other evidence-building activities are currently more and better evidence building. Te strategy
spread across the organization (REC. 5-3). outlined in this report both improves privacy pro-
Te Commission also heard about administra- tections and makes better use of data the govern-
tive processes in government that inadvertently ment has already collected. Te remainder of the
inhibit evidence-building activities. Tese includ- report outlines the challenges to increasing the
ed processes for reviewing information collec- volume, quality, and utility of evidence generated
tions under the PRA and procurement policies. to support policymakers and makes specifc rec-
Te Commission recommends specifc fexibilities ommendations for improvements.
This page intentionally left blank
The Promise of Evidence-Based Policymaking 23

2
Secure, Restricted Access to
Confdential Data
Equip the evidence-building community with secure, restricted access to data to facilitate
the generation of high quality evidence about government programs and policies.

Recommendations
2-1: Te Congress and the President 2-5: Te Congress and the President
should enact legislation establishing the should consider repealing current bans
National Secure Data Service (NSDS) to facili- and limiting future bans on the collection and
tate data access for evidence building while use of data for evidence building.
ensuring transparency and privacy. Te NSDS 2-6: Te Congress and the President
should model best practices for secure record should enact statutory or other changes
linkage and drive the implementation of in- to ensure that state-collected administrative
novative privacy-enhancing technologies. data on quarterly earnings are available for
2-2: Te NSDS should be a service, not a solely statistical purposes. Te data should be
data clearinghouse or warehouse. Te available through a single Federal source for
NSDS should facilitate temporary data linkag- solely statistical purposes.
es in support of distinct authorized projects. 2-7: Te President should direct Federal
2-3: In establishing the NSDS, the Con- departments that acquire state-collect-
gress and the President should amend ed administrative data to make them avail-
the Privacy Act and the Confdential Informa- able for statistical purposes. Where there is
tion Protection and Statistical Efciency Act substantial Federal investment in a program,
(CIPSEA) to require new stringent privacy Federal departments should, consistent with
qualifcations as a precondition for the NSDS applicable law, direct states to provide the
to acquire and combine survey and adminis- data necessary to support evidence building,
trative data for solely statistical purposes. At such as complete administrative data when
the same time, the Congress should consider samples are already provided.
additional statutory changes to enable ongo- 2-8: Te Ofce of Management and
ing statistical production. Budget should promulgate a single,
2-4: Te Congress and the President streamlined process for researchers external
should review and amend, as appropri- to the government to apply, become qualifed,
ate, statutes such as Title 13 of the U.S. Code and gain approval to access government data
to allow statistical uses of survey and admin- that are not publicly available. Approval
istrative data for evidence building within the would remain subject to any restrictions ap-
CIPSEA secure environment. propriate to the data in question.
24 The Promise of Evidence-Based Policymaking

D ata are an essential ingredient for the


success of evidence-based policymaking,
providing the foundation for basic descriptive
that access to confdential data for evidence build-
ing should be enhanced, but in a fashion that also
improves privacy protections.
statistics that describe the status quo as well as
conclusions from the most sophisticated program Findings
evaluations. Te routine administrative opera-
tions associated with many Federal activities re- During the Commissions fact-fnding phase, nu-
sult in the collection of data from program partic- merous expert witnesses, commenters, and gov-
ipants. Tese administrative data, along with the ernment agencies identifed examples in which
multitude of surveys and other data collections, successful access to data generated information
have value for evidence building. important for improving programs and policies.
Data collection is resource intensive for the Tey often involved an individual or research
government and burdensome for the public. Tere team with sufcient persistence and creativity
is strong reason to carefully increase the use of ex- to successfully navigateor even pioneer the use
isting data to the extent possible before imposing ofavailable processes for accessing data. For ex-
new burdens on individuals, businesses, and other ample, Stanford University Professor of Econom-
organizations. Leveraging administrative data can ics Raj Chetty and his collaborators produced
improve statistical products, reduce the costs and path-breaking research on social mobility.1 Such
administrative burden associated with creating success stories, however, highlight common bar-
statistics and conducting research and evaluation, riers to accessing critical data that, in too many
and thereby greatly expand the evidence base for cases, have prevented researchers from produc-
setting Federal policy (see the box Reducing the ing analyses of important policy questions. Many
Respondent Burden of Income Questions on Sur- of these same challenges have been identifed by
veys). Te Commission believes the Federal gov- past commissions and panels tasked with explor-
ernment can more efciently and efectively use ing the Federal Statistical System or making rec-
the data it collects, while simultaneously enhanc- ommendations about policy in a particular area.2
ing privacy protection. Accumulated experience ofers valuable lessons
In a well-designed system, access to data and about how to establish processes that can make
the protection of privacy can work hand in hand. ongoing analyses of important programs and pol-
Te Commissions conception of secure data ac- icies a norm in government, rather than a rarity.
cess is prescribed narrowly to include access for To access confdential data for the development
exclusively statistical purposesthat is, the anal- of statistics, evaluation, and policy research, mem-
ysis of data to generate inferences about groups. A bers of the evidence-building community today,
well-designed system for secure data access is also
the means for achieving greater transparency with 1. Raj Chetty, Stanford University, Commission Meeting, Wash-
respect to how data are being used, an important ington, D.C., July 22, 2016.
element for increasing privacy and accountability. 2. National Academies of Sciences, Engineering, and Medicine, In-
Access to data held by the government should novations in Federal Statistics: Combining Data Sources While Protecting
occur only in service to the public interest. Deci- Privacy (Washington, D.C.: National Academies Press, 2017); Federal
Statistics: Report of the Presidents Commission, Volume III (Washing-
sions about allowing data access must be calibrat- ton, D.C.: U.S. Government Printing Ofce, 1971); https://catalog.
ed according to a projects potential public ben- hathitrust.org/Record/011325636 (accessed August 10, 2017); James
efts, the sensitivity of a particular dataset, and T. Bonnen, Teodore G. Clemence, Ivan P. Fellegi, Tomas B. Jabine,
Ronald E. Kutscher, Larry K. Roberson, and Charles A. Waite, Im-
any risk that allowing access could pose to conf-
proving the Federal Statistical System: Report of the Presidents
dentiality. Access can and should be restricted to Reorganization Project for the Federal Statistical System, American
eligible individuals who demonstrate an under- Statistician 35, no. 4 (November 1981, reprint): 184196; National
standing of their obligations for data stewardship. Commission on Hunger, Freedom From Hunger: An Achievable Goal for
the United States of America, Recommendations of the National Com-
Restricted access has been standard practice in mission on Hunger to Congress and the Secretary of the Depart-
many areas of government for decades. However, ment of Agriculture, 2015; https://digital.library.unt.edu/ark:/67531/
duplicative processes for accessing confdential metadc799756 (accessed August 10, 2017); Commission to Elimi-
nate Child Abuse and Neglect Fatalities, Within Our Reach: A National
data can distract from confdentiality and the pro- Strategy To Eliminate Child Abuse and Neglect Fatalities (Washington,
motion of transparency. Te Commission believes D.C.: Government Printing Ofce, 2016).
The Promise of Evidence-Based Policymaking 25

both inside and outside the Federal government, departments are also making such data available
must navigate a complex array of processes, pro- to external researchers. Still, signifcant barriers to
tocols, and approaches. Tey must negotiate legal data access remain. Starting a new statistical proj-
documents and bureaucratic processes that in- ect using administrative data is complicated and
crease in volume and complexity when using data time consuming. Even departments that regularly
from multiple policy domains, jurisdictions, or use administrative data for evidence building fnd
agencies. Often, such processes consider the value it onerous to navigate each originating agencies
proposition of data use only in the context of the existing processes for acquiring administrative
mission of the originating agency, irrespective of data for statistical use. Agencies just starting to
its broader value. For example, generally Title 26 explore the power of administrative data for im-
of the U.S. Code limits the use of tax data to those proving their programs and policies often waste
projects that would improve tax administration. months wading through various applicable laws
Te application of this narrow standard to research with little result.
on human services or transportation, for example, As the demand for statistics, evaluation, and
may undervalue the available public good. Tese policy research increase, so too must secure data
kinds of barriers limit the efective, efcient, and access with strong privacy protections. Te Com-
transparent use of existing data. mission identifed four overarching challenges to
Federal departments are making greater use of realizing the vision of secure access to confdential
their own administrative data for statistical activ- data for evidence building that appropriately cal-
ities today than in the past, and in some cases, ibrates public benefts, privacy, and transparency.

Reducing the Respondent Burden of


Income Questions on Surveys
Household survey data collection programs, cal agencies were able to rely more on the in-
including key U.S. Census Bureau programs, come data the government already maintains
are fnding it more difcult to obtain accurate to administer tax, income support, and social
income data from the survey population. Re- insurance programs. Recent work at the Cen-
spondents have become less willing to partic- sus Bureau has begun to explore the potential
ipate in surveys and are increasingly reluctant of administrative sources to replace questions
to respond to questions about income.1 When about selected income sources in household
they do answer questions about income, they surveys such as the American Community Sur-
are providing less accurate responses.2 Te vey and the Current Population Survey. While
burden on respondents could be reduced and this work is in the early stages and current data
the accuracy of the data improved if statisti- access is quite limited, the results are promis-
ing and suggest the administrative data can be
1. Christopher R. Bollinger, Barry T. Hirsch, Charles M.
a signifcant improvement.3
Hokayem, and James P. Ziliak, Trouble in the Tails? What
We Know about Earnings Nonresponse Tirty Years after Lil-
3. U.S. Census Bureau, American Community Survey Re-
lard, Smith, and Welch, paper (June 2017); http://www2.gsu.
search and Evaluation Report, Center for Administrative
edu/~ecobth/BHHZ_Trouble-in-the-Tails_6-8-2017.pdf (accessed
Records Research and Applications Memorandum Series
August 10, 2017).
1672015; Bruce D. Meyer, and Nikolas Mittag, Using Linked
2. National Research Council, Nonresponse in Social Science Survey and Administrative Data to Better Measure Income:
Surveys: A Research Agenda, edited by Roger Tourangeau and Implications for Poverty, Program Efectiveness, and Holes in
Tomas J. Plewes, Panel on a Research Agenda for the Future the Safety Net, National Bureau for Economic Research Work-
of Social Science Data Collection, Committee on National Sta- ing Paper 21676 (Washington, D.C.: 2015); C. Adam Bee and
tistics, Division of Behavioral and Social Sciences and Educa- Joshua Mitchell, Te Hidden Resources of Women Working
tion (Washington, D.C.: Te National Academies Press, 2013); Longer: Evidence from Linked Survey-Administrative Data,
Bruce D. Meyer, Wallace K.C. Mok, and James X. Sullivan, in Women Working Longer: Increased Employment at Older Ages,
Household Surveys in Crisis, Journal of Economic Perspectives edited by Claudia Goldin and Lawrence F. Katz (Chicago: Uni-
29, no. 4 (Fall 2015): 199226. versity of Chicago Press, 2016).
26 The Promise of Evidence-Based Policymaking

Challenge #1: Te current legal framework data readily among themselves. In some cases,
can stand in the way of data stewardship even when data sharing is allowable for agency
staf, impediments remain to providing external
and the sharing of data to support evidence
researchers with appropriate access to combined
building. data. In other countries with a more centralized
statistical system, data sharing is both less needed
Te current legal environment for data collection, and more straightforward.
protection, and sharing lacks consistency, lead- Both program and statistical agencies also must
ing to confusion and inefciency among depart- consider other applicable laws. Tese include the
ments, external researchers, and other members Paperwork Reduction Act (PRA), the Privacy Act,
of the evidence-building community. Laws gov- and the Confdential Information Protection and
erning the data lifecycle for any dataset include Statistical Efciency Act (CIPSEA). Generally the
the statute that authorized the collection of the PRA does not provide the authority to collect data,
data, statutes generally applicable to data col- but it does govern the process by which data col-
lection and management processes, and various lection occurs. Te PRA specifcally includes re-
legal provisions governing privacy and confden- quirements for transparency with regard to the
tiality protections. Te Commission recognizes information that government collects and why, by
that some variation in the laws that govern the extension implicating potential secondary uses of
protection of data is sensible given the contextu- collected data. As the name implies, the PRA also
al nature of privacy. It is unclear, however, that attempts to reduce the burden of data collection
all of the variation in the legal structure is inten- on the public by encouraging data sharing be-
tional; rather, some of the variation may be the tween agencies through the establishment of gov-
result of the complexity of government together ernment-wide statistical and information policies
with the various statutes having been developed and coordination procedures.
independently at particular times and with par- Similarly, the Privacy Act does not provide the
ticular goals in mind. Te Congress and the Pres- authority to collect data, but it does set require-
ident seemingly recognized as much when form- ments regarding how those data may be disclosed.
ing the Commission and charging it to review the Specifcally, the Privacy Act requires public and
net impact of accumulated laws and policies on individual notices about data held in government
evidence building.3 systems and limits disclosure of these data with-
Te authority to collect data typically is provid- out consent. Tis includes restricting the second-
ed in a program or statistical agencys authorizing ary uses of data without consent unless one of a
statute. In the case of program agencies, authoriz- limited number of exceptions applies. For exam-
ing statutes may stipulate the nature and opera- ple, the Congress exempted from individual con-
tions of the objectives and policies of the agency. sent requirements data provided by another agen-
Program statutes vary in their prescriptiveness cy to the U.S. Census Bureau for the purposes of
with regard to confdentiality and the use of data planning or carrying out a census or survey.4
for evidence building, with many statutes not ad- CIPSEA was enacted to address some gaps and
dressing the issue at all. variations in existing statistical laws. (See the
Te laws that authorize statistical agencies box, Te Confdential Information Protection
generally include restrictions on who can access and Statistical Efciency Act of 2002.) While the
data and for what purposes. While data collection authorizing statutes for some statistical agencies,
was a primary intent of the laws that created these such as the Census Bureau, provide collection
agencies, the existing structure has not produced authority together with strong confdentiality
the same level of coordination and integration protections, those for some other agencies do
that is typical in other countries because the Fed- not specifcally address privacy and confdenti-
eral Statistical System is decentralized. One result ality. Many statutes also limit data sharing, even
is the inability of U.S. statistical agencies to share among Principal Statistical Agencies (PSAs),
leading to inefciencies such as duplicative data

3. Evidence-Based Policymaking Commission Act of 2016 (Public


Law 114140, Section 4(a), March 30, 2016). 4. Privacy Act, 5 USC 552a(b)(4) (1974).
The Promise of Evidence-Based Policymaking 27

Figure 3. Functional Separation of Administrative and Statistical Data

Administrative Data Statistical Data


Information Gathered from One-Way Filter Data Collected, Compiled, and
Tax Filings and in Connection Provides Processed for the Purpose of
with Government Beneft Privacy and Describing, Estimating, or
Programs, Economic Data Format Analyzing Characteristics of
Development Programs, and Enhancements Groups, Without Identifying the
Other Federal Programs Individuals or Organizations
that Comprise Such Groups

Non-Statistical Purpose Statistical Purpose

The Confdential Information Protection and


Statistical Efciency Act of 2002
Te Confdential Information Protection and provide a prevailing standard for these con-
Statistical Efciency Act of 2002 (CIPSEA) is a cepts in law. Statistical purpose is defned un-
law with two components. Te frst provides a der CIPSEA as the description, estimation or
set of uniform confdentiality protections for analysis of the characteristics of groups, with-
data acquired under a pledge of confdentiality out identifying the individuals or organizations
and for exclusively statistical purposes. Te sec- that comprise such groups and includes the
ond provides a limited authority to share pro- development, implementation, or maintenance
tected data. Among the motivations for CIPSEA of methods, technical or administrative proce-
was a need to ensure a consistent statutory dures, or information resources that support
authority for agencies handling statistical data those purposes. Statistical activities are defned
to protect those data from non-statistical use.1 as the collection, compilation, processing, or
Te Federal government now has 15 years of analysis of data for the purposes of describing
demonstrated success with CIPSEA.2 or making estimates concerning the whole,
Te CIPSEA framework codifes the concept or relevant groups or components within the
of functional separation (see Figure 3). Specif- economy, society, or the natural environment
ically, CIPSEA defnes statistical activities, sta- and include the development of methods or
tistical purposes, non-statistical purposes, and resources that support those activities.
qualifed agents and agencies. Such defnitions CIPSEAs subtitle A fosters public trust by
are essential to the application of CIPSEA, and generally prohibiting disclosure of protected
information in identifable form, controlling
access to and use of protected information, and
1. U.S. House of Representatives, Report to Accompany H.R.
2458, House Report No. 107787 (2002); https://www.congress. ensuring that information is used exclusively
gov/congressional-report/107th-congress/house-report/787 for statistical purposes. While some statistical
(accessed August 10, 2017). statutes explicitly provide for data protection,
2. Brian Harris-Kojetin, CIPSEA at 15 Years: Benefts to others, such as the authorizing statute for the
Federal Statistics and Unmet Needs - Overview, (paper U.S. Bureau of Labor Statistics, do not. CIPSEA
presented at the 133rd Meeting of the Committee on National
Statistics, Washington, D.C., May 12, 2017); http://sites. Subtitle A provides a remedy for this.
nationalacademies.org/dbasse/cnstat/dbasse_178400 (accessed
August 10, 2017). continues
28 The Promise of Evidence-Based Policymaking

collections. CIPSEA was created as a partial an- for exclusively statistical purposes to a person or
swer. Te law provides common statutory protec- agency not entitled to receive it, including a fel-
tions for data acquired for exclusively statistical ony charge and imprisonment for up to 5 years
purposes under a pledge of confdentiality and and/or fnes up to $250,000. Te CIPSEA penalties
currently applies to all of the PSAs and to other are equal to, or exceed, other such provisions in
recognized statistical units. statutes, including provisions contained in Titles
CIPSEA also established common penalties for 13 and 26 of the U.S. Code.
any ofcers, employees, or agents of an agency For the Commissions purpose, the vital compo-
who, knowing that disclosure is prohibited under nents of the CIPSEA legal framework include the
CIPSEA, willfully disclose protected data collected ability to ofer strong confdentiality protections,

The Confdential Information Protection and


Statistical Efciency Act of 2002continued

Statistical Agencies or Units Recognized by OMB for CIPSEA Purposes 4

Bureau of Economic Analysis National Center for Science and Engineering


Statistics
Bureau of Justice Statistics
Ofce of Research, Evaluation, and Statistics,
Bureau of Labor Statistics
Social Security Administration
Bureau of Transportation Statistics
Statistics of Income Division, Internal
Census Bureau Revenue Service
Economic Research Service Center for Behavioral Health Statistics and
Energy Information Administration Quality, Substance Abuse and Mental Health
Services Administration*
National Agricultural Statistics Service
Microeconomic Surveys Statistical Unit,
National Center for Education Statistics Federal Reserve*
National Center for Health Statistics National Animal Health Monitoring System,
Animal and Plant Health Inspection Service*

* Recognized statistical units

CIPSEA grants the Director of the U.S. Of- CIPSEAs subtitle B provides for the limited
fce of Management and Budget (OMB) the sharing of business data among three desig-
authority to promulgate rules and provide im- nated statistical agencies: the U.S. Bureau of
plementing guidance, including the qualifca- Economic Analysis, the Bureau of Labor Statis-
tion and recognition of agencies or units that tics, and the U.S. Census Bureau. Te authori-
may exercise CIPSEA authority. In its CIPSEA ty is designed to reduce paperwork burden on
implementation guidance, OMB recognized the businesses, improve comparability of econom-
13 Principal Statistical Agencies (PSAs) as sta- ic statistics, and increase understanding of the
tistical agencies under CIPSEA, and also recog- economy.
nized three additional statistical units that had
demonstrated functional separation in accor-
dance with OMBs guidance.3

3. Implementation Guidance for Title V of the E-Govern-


ment Act, Confdential Information Protection and Statistical
Efciency Act of 2002 (CIPSEA), Federal Register 72, no. 115 4. List of Principal Statistical Agencies and Recognized Units;
(June 15, 2007): 33361; https://www.gpo.gov/fdsys/granule/FR- https://obamawhitehouse.archives.gov/omb/inforeg_statpolicy/
2007-06-15/E7-11542/content-detail.html (accessed August 10, bb-principal-statistical-agencies-recognized-units (accessed Au-
2017). gust 10, 2017).
The Promise of Evidence-Based Policymaking 29

ensure exclusively statistical uses, and within that words, even the parts of the Federal government
structure, enable limited and secure sharing of that today are leaders in producing evidence report
data. challenges in navigating the complex array of legal
CIPSEA permits the sharing of business data requirements to use data for evidence building.
exclusively for statistical purposes among three Perhaps more than any other single data need,
PSAs to create statistical efciency within the the Commission repeatedly heard calls for im-
context of CIPSEAs legal framework.5 Tis specifc proving access to income, wage, and earnings
provision enabled several new data sharing initia- data for evidence-building purposes. Efects on
tives that have improved efciency for all three income are central to the evaluation of numer-
agencies.6 ous Federal programs and activities, but existing
Except for that provided under CIPSEA, the au- laws and practices limit the ability of researchers
thority to share data for evidence building is rarely both internal and external to the government to
explicit. In cases where authorizing laws are am- access even the income data the Federal govern-
biguous, agency interpretations ultimately govern ment already collects (see the box Income Data:
access to and use of data. In some cases multiple Federal Tax and Unemployment Insurance Earn-
agencies interpret the same law diferently. Tis ings Data). Programs such as tax administration
can cause confusion and limit the efcient use of depend on compliance from the public, making
existing data for evidence building.7 Te complex it particularly important to ensure the privacy of
web of statutes, regulations, and implementing records generated in the course of administering
guidanceor absence thereofdrives risk aver- those programs. It is equally important, however,
sion in agencies, causes frustrations for the evi- to calibrate the need for privacy with the public
dence-building community, and limits the value good that research fndings based on such data
of data for statistical activities. In efect, the ex- can provide. Of the 22 ofces responding to the
isting legal environment limits the governments CEP Survey of Federal Ofces that it was hard to
ability to steward data responsibly as a valuable access Federal Tax Information for evidence build-
resource for the American people and for policy- ing, half cited statutes prohibiting data sharing as
makers. the reason for the difculty.
In the CEP Survey of Federal Ofces, 52 percent Another barrier identifed in testimony to the
of responding ofces identifed legal limitations as Commission is the explicit prohibition in some
a major or moderate barrier to using data for sta- laws on the collection or integration of data, which
tistics, evaluation, research, and policy analysis prevent the government from building evidence
purposes. While 10 out of 13 PSAs and 8 out of 10 to improve Federal programs and policies. For de-
evaluation ofces reported that legal limitations cades, diferent iterations of evidence-prohibiting
were a major or moderate barrier, less than half (41 provisions have been included by either the Con-
percent) of other types of ofces cited legal lim- gress in statutes or the Executive Branch in direc-
itations as a major or moderate barrier.8 In other tives and policies.Many of these prohibitions on
collecting or accessing data to develop evidence
5. Confdential Information Protection and Statistical Efciency exist in domains that have major implications for
Act of 2002 (CIPSEA)(Public Law 107347, Title V, 2002). policy areas of national importance and they of-
6. For a summary of several of the successful data sharing initia- ten involve programs that represent a substantial
tives that were enabled by CIPSEA Subtitle B, see U.S. Bureau of share of the Federal budget.
Labor Statistics (BLS), Confdentiality of Data Collected by BLS Trough the Commissions fact-fnding phase,
for Statistical Purposes, (May 16, 2016); https://www.bls.gov/bls/
cipsea-report.htm (accessed August 10, 2017). stakeholders identifed several examples of data
collection, integration, and analysis bans. Te
7. U.S. Ofce of Management and Budget (OMB), Barriers to
Using Administrative Data for Evidence Building, white paper Commission received substantial public com-
for the Commission on Evidence-Based Policymaking (Washing- ments regarding the student unit record ban, en-
ton, D.C.: OMB, Executive Ofce of the President, 2016); https:// acted in 2008.9 Te student unit record ban efec-
obamawhitehouse.archives.gov/sites/default/fles/omb/mgmt-gpra/
barriers_to_using_administrative_data_for_evidence_building.pdf (ac- tively limits some aspects of evidence production
cessed August 10, 2017). related to pre-kindergarten through 12th grade and
8. Based on 79 ofces that reported they collect or use data for sta-
tistics, evaluation, research, or policy analysis or spend a portion
of their budget for such purposes. 9. Database of Student Information Prohibited, 20 USC 1015c.
30 The Promise of Evidence-Based Policymaking

postsecondary education by prohibiting the Fed- ing the outcomes of government programs
eral governments development or maintenance and policies are collected at the state or lo-
of a new database with data on all students. Te
cal level and are inaccessible for evidence
Commission heard many substantive comments
about the student unit record ban, and received building.
more feedback on the issue than on any other sin-
gle topic within the Commissions scope. Nearly Many Federal programs are administered by states
two-thirds of the comments received in response and localities, most often in cooperation with
to the Commissions Request for Comments raised the Federal government. Examples of programs
concerns about student records, with the majority that operate under this model include Medicaid,
of those comments in opposition to overturning the Supplemental Nutrition Assistance Program
the student unit record ban or otherwise enabling (SNAP), Unemployment Insurance (UI) programs,
the Federal government to compile records about Housing Assistance programs, and the Temporary
individual students. Assistance for Needy Families (TANF) program.
Another ban brought to the attention of the Tese programs have a broad variety of funding
Commission is embedded in the Workforce Inno- and administrative structures that include the
vation and Opportunities Act (WIOA), a program Federal and state government as partners and of-
for workforce training and development.10 Te ban ten include cities, counties, and grantees as part-
is a prohibition on developing a national database ners as well.12 Te administrative data generated
of participants in WIOA training programs that by the operation of these programs are a valuable
includes personally identifable information.11 source of data for use in evidence building, but
With these bans, and a similar ban covering data there are numerous, well-documented barriers to
on Head Start participants, much of governments accessing and using these data.13
investment in human capital programsfrom early Barriers to accessing state-level data include:
childhood through adulthoodlargely is not sub- state laws or legal interpretations of Federal law
ject to the type of rigorous, national-scale evalua- that either expressly prohibit or tightly restrict
tion that leveraging administrative data could make data sharing for the purposes of evidence build-
possible. Other examples of bans or restrictions on ing; lack of capacity within states to transform
use of government data to conduct research popu- the administrative data to make them suitable
late the U.S. Code, although the Commission was for evidence building; and administrative and/
not approached in public comments or by expert or procedural variations for accessing data across
witnesses to address these other bans.
Prohibitions on data collection and use argu-
ably confict with the Commissions vision to im- 12. For example, the Federal government funds the full cost of
prove government based on credible evidence. SNAP benefts and reimburses the state for approximately 50
percent of its administrative costs. Medicaid is jointly funded by
Within the broad array of factors that infuence the Federal and state government, where the Federal government
policymaking, a ban may be an appropriate tool pays states for a specifed percentage of program expenditures
for setting priorities or achieving certain other calculated as the Federal Medical Assistance Percentage. TANF
operates as a block grant to states, and the program may be state
goals. In the context of evidence building, how-
or county administered, depending on the structure of the state.
ever, a ban on data collection or use cannot easily
13. OMB, Barriers to Using Administrative Data for Evi-
be reconciled with a goal of increased reliance on dence-Building; Ofce of Planning, Research and Evaluation
evidence to inform policymaking. (OPRE), Administration for Children and Families, U.S. Depart-
ment of Health and Human Services, Using Administrative Data
in Social Policy Research, OPRE Report 201662 (Washington,
Challenge #2: Many high-value administra- D.C.: OPRE, 2016); Northwestern University/University of Chi-
tive data associated with Federally funded cago Joint Center for Poverty Research, Administrative Data for
Policy-Relevant Research: Assessment of Current Utility and Rec-
programs that could be useful to measur- ommendations for Development, V. Joseph Hotz, Robert Goerge,
Julie Balzekas, and Francis Margolin, editors; Helen Lee, Anne
Warren, and Lakhpreet Gill, "Cheaper, Faster, Better: Are State
Administrative Data the Answer? Te Mother and Infant Home
10. Workforce Data Quality Campaign, submission to the Com-
Visiting Program Evaluation-Strong Start Second Annual Report,"
mission's Request for Comments.
OPRE Report 2015-09 (Washington, D.C.: OPRE, 2015); and Kel-
11. Prohibition on development of national database, 29 USC ly Maxwell, Issues in Accessing and Using Administrative Data,
3341(b). OPRE Report 201724 (Washington, D.C.: OPRE, 2017).
The Promise of Evidence-Based Policymaking 31

Figure 4. Examples of State-Collected Administrative


Data Productive for Evidence Building

Sharing with the


Federal Government: Adoption and Foster Care
Adoption and Foster Care Mandatory Analysis and Reporting System
Analysis and Reporting System Medicaid
Data Coverage:
Childrens Health Insurance National Directory of New Hires
Universe
Program
Medicaid
National Child Abuse and
Neglect Data System Supplemental Nutrition
Sharing with the
National Directory of Assistance Program
Sources of Federal Government:
New Hires Temporary Assistance for
Supplemental State-Collected Mandatory
Needy Families
Nutrition Assistance Administrative Data Coverage: Unemployment Insurance
Program Data Sample
Temporary Assistance Benefts
for Needy Families
Unemployment Insurance
Benefts Sharing with the Childrens Health Insurance Program
Unemployment Insurance Federal Government: National Child Abuse and Neglect
Quarterly Wages Voluntary Data System
Vital Statistics Cooperative Program Unemployment Insurance Quarterly
Data Coverage: Wages
Varied Vital Statistics Cooperative Program

For a list of additional datasets productive for evidence building, see Appendix D.

diferent state agencies or programs, leading to SNAP data submitted to the USDA may only be
protracted and often duplicative administrative used for administrative or enforcement purposes,
requirements.Tese barriers are compounded for not for evidence building.14
projects in which multiple administrative datasets Figure 4 describes the current landscape of
are to be combined, as each dataset has diferent data availability at the Federal level for a set of
access requirements. Similar barriers often exist high-priority state-collected administrative data-
to access data from local or other jurisdictions. sets. Te fgure illustrates where the data sources
Te Federal government invests signifcant diverge based on three important distinctions: (1)
funds to support the operation of many jointly ad- whether the provision of data from the state to
ministered Federal-state programs, providing a ba- the Federal government is mandatory or volun-
sis for some type of Federal reporting requirement. tary, (2) whether and to what extent the data be-
Currently, the volume and type of state-collected ing shared represent the universe or just a sample,
administrative data that are reported to Federal and (3) whether the data contain sufcient per-
program agencies vary considerably across pro- sonal identifers to enable integration with other
grams. Each program operates under its own set of data sources.
statutory and regulatory requirements for submit- Increasing access to administrative data doc-
ting program data to their Federal sponsor, rang- umenting individual earnings is of particular im-
ing from the Medicaid programfor which states portance, given the many Federal programs and
are required to submit data on all benefciaries to policies that seek to have an impact on earned
the Centers for Medicare and Medicaid Services income. Accessing quarterly earnings data di-
(CMS)to SNAP and TANFunder which states rectly from all 50 states, the District of Columbia,
are required to submit only a sample of data to a and the territories can be challenging and time
Federal agency. Additionally, some of the data sub-
mitted to the Federal government have statutory 14. See use restrictions in the Food, Conservation, and Energy Act
use restrictions; for example, the individual-level of 2008 (Public Law 110246).
32 The Promise of Evidence-Based Policymaking

Income Data: Federal Tax and Unemployment


Insurance Earnings Data
Wage and income data exist in at least two Researcher access points for FTI are the U.S.
forms, data from the Unemployment Insur- Treasury Department and the Census Bureau.
ance (UI) programs and Federal Tax Informa- Statutory restrictions on accessing data are the
tion (FTI). Tese data sources can be viewed as major barrier to the use of FTI. Te Federal tax
complementary, covering somewhat diferent code (Title 26 of the U.S. Code) designates FTI as
populations and diferent types of income with confdentialdisclosure to any party is prohib-
difering periodicity. Specifcally, the informa- ited, except under explicit statutory exceptions.
tion from the UI programs consists of individual FTI can be used by Treasury employees and a
wage records and is available quarterly, whereas limited set of approved researchers engaging in
FTI ofers a broader view of income earned from statistical projects in support of tax administra-
sources other than wages such as transfers, per- tion. Te Census Bureau and the U.S. Bureau of
mits the analysis of household as well as indi- Economic Analysis also are authorized limited
vidual income, and is available annually. access under Title 26 and associated regulations
Federal law requires states to collect quarterly for a narrow set of uses related to improving
wage data as part of their administration of state their statistical products. Te Title 26 limita-
UI programs operated as part of the Federal-state tions then transfer to any data products these
UI system.1 Te U.S. Department of Labor over- agencies produce with comingled FTI and access
sees and provides funds for the UI system, but to comingled microdata also requires Internal
each state administers its own UI program, which Revenue Service (IRS) approval.
includes the collection of data on individual em- Te Treasury Department is the primary ac-
ployees and their earnings in each quarter. cess point for FTI. Tere are several ways that
Researchers wishing to access UI data current- Treasury provides access to tax data for research.
ly have three primary access points: originating One way is through the IRS Statistics of Income
state systems, the National Directory of New Hires Divisions Joint Statistical Research Program. An
maintained by the U.S. Department of Health and important characteristic of the Statistics of In-
Human Services Administration for Children and come program is that it aims to limit, as much
Families, and the U.S. Census Bureaus Longitudi- as possible, non-IRS researchers direct access to
nal Employer-Household Dynamics Program. Yet, directly identifable taxpayer data by assigning
there are signifcant challenges to accessing and Statistics of Income employees as co-research-
using each of these sources of UI data. ers in all phases of the work, including assem-
Accessing quarterly earnings data direct- bling and cleaning the required data, performing
ly from multiple states can be challenging and analyses, and writing reports on the fndings.
time consuming, if not impossible. Both the Na- For this reason, the program has very limited
tional Directory of New Hires and the Longitu- capacity. For example, in 2014, only 12 projects
dinal Employer-Household Dynamics Program were selected from among 80 proposed. Numer-
have existing authority to allow researchers ac- ous researchers have gained access to tax data
cess to the wage record data for some statistical through alternative means, including through
activities, but the authority is narrow, making the Ofces of Tax Analysis and Tax Policy within
secure access to the data excessively restrictive.2 the Treasury Department, which also engage in
independent and occasionally joint research.
1. Te Federal Unemployment Tax Act (26 USC 3301 et seq.)
and titles III, IX, and XII of the Social Security Act (42 USC
enced by participants in Federally funded state employment and
chapter 7) form the framework of the system.
training programs. Either a state or the Employment and Train-
2. Te Wage Record Interchange System (WRIS), maintained ing Administration at DOL may propose research projects using
by the Department of Labor (DOL), facilitates the exchange of the WRIS, but each proposed project must demonstrate a direct
UI wage records between states for performance accountability beneft to a qualifying program or activity and approval must be
purposes, enabling improved reporting on the outcomes experi- obtained individually from each state whose data will be used.
The Promise of Evidence-Based Policymaking 33

Data About Births and Deaths in the Population


Data on vital events (births and deaths) are of In 2007, NCHS released a new policy on the
great value for statistics, evaluation, and pol- release of and access to vital statistics microda-
icy research. For studies of health, housing, ta to comply with state laws and policies.1 Te
and other policies, there is perhaps no more revised policy refects the dual goals to make
important outcome variable than death. Vital data available as widely as possible while being
records also are critical for public health pro- responsive to concerns about confdentiality.
grams and serve a variety of administrative Te current agreement with the states on
purposes. the re-release of restricted data containing geo-
Te U.S. National Center for Health Sta- graphic detail requires a review of all such data
tistics (NCHS) is the Federal agency mandat- requests by the National Association for Pub-
ed to produce national health statistics based lic Health Statistics and Information Systems
on a cooperative, decentralized system. Data (NAPHSIS), which represents state vital reg-
from more than six million vital-event records istrars.2 Te review by NAPHSIS is conducted
are collected each year by vital registrars in prior to the NCHS review, and applies to both
all states and U.S. territories and transmit- Federal and non-Federal requests for restrict-
ted to NCHS for processing and dissemina- ed data fles. Researchers in Federal agencies,
tion. NCHS has a statutory mandate (42 USC as well as their on-site or of-site contractors,
242k) to collect data annually from the re- also can submit project proposals that request
cords of births, deaths, marriages, and divorces exact dates of vital events. If needed, the fle
in registration areas. Currently the data col- with exact dates can include geographic detail.
lection is limited to data from birth and death Non-Federal researchers (including Federal
records (including fetal deaths), as NCHS dis- grantees) can gain controlled access to exact
continued the collection of individual-record dates of vital event fles only through the NCHS
marriage and divorce reports after 1995. Research Data Center, with approval.
Data are collected through the Vital Statis-
tics Cooperative Program. Tese data are pro-
vided through contracts between NCHS and
vital registration systems operated in the 57 1. NCHS defnes microdata as all raw data, including public
use and restricted data fles.
jurisdictions legally responsible for the regis-
tration of vital events, namely states. Te con- 2. NCHS, National Vital Statistics System, NCHS Data Release
and Access Policy for Microdata and Compressed Vital Statis-
tracts support the cost of training and techni- tics Files; https://www.cdc.gov/nchs/nvss/dvs_data_release.htm
cal assistance to help standardize data quality. (accessed August 10, 2017).

consuming, and in many cases, not possible. Many have enabled states to improve their capacity for
residents of one state work in a diferent state; evidence building. One example of this kind of in-
thus, evidence-building eforts frequently neces- vestment has been the grants administered by the
sitate accessing quarterly earnings data from mul- National Center for Education Statistics to support
tiple states. While there are currently two possible the development of Statewide Longitudinal Data
avenues by which researchers can access data on Systems to assess K12 education outcomes and
quarterly earnings from multiple states through investments, creating data systems at the state
a single access point (see the box Income Data: level similar to those prohibited by the student
Federal Tax and Unemployment Insurance Earn- unit record ban at the Federal level. Such invest-
ings Data), both avenues are relatively narrow ments also have been valuable in enabling states to
and include numerous restrictions. comply with reporting requirements. For example,
Federal resources and technical assistance to while states are required to submit extensive Med-
states for modernization of information systems icaid data to CMS, the Federal government spends
34 The Promise of Evidence-Based Policymaking

Longitudinal Employer-Household Dynamics Program


Te Longitudinal Employer-Household Dynam- state includes an option for the state to stream-
ics Program (LEHD) is an example of a part- line the review and approval process by allow-
nership that incorporates voluntarily provided ing the Census Bureau to make the determina-
state-held administrative data with survey and tion as to whether a proposed research project
other administrative data held by the U.S. Cen- is consistent with the uses authorized by the
sus Bureau to support statistical analysis and state, but currently only 12 states have selected
research. However, the current structure of the this option.1 All of the other states review and
program limits broader use of the underlying approve each project individually, meaning that
state data for evidence building beyond the nar- data access is cumbersome and time-consum-
row purposes of LEHD. ing, if it can be accomplished at all.
Under the Local Employment Dynamics Some of the individuals who commented to
Partnership, which is a voluntary Federal-state the Commission noted that UI wage records
partnership started in 1999, states agree to remain underutilized because of limitations to
share historical and ongoing administrative re- access. Commenters expressed support for in-
cords of Unemployment Insurance (UI) earn- creasing access to the LEHD data. Te data use
ings data and the Quarterly Census of Employ- agreements with each state typically need to be
ment and Wages data with the Census Bureau. renewed every fve years. While at present, 48
In return, the Census Bureau produces a lon- states, the District of Columbia, and two U.S.
gitudinal data infrastructure from which new territories have agreements in place with the
statistics about the dynamics of local employ- Census Bureau, their continued participation
ment and the locations of jobs and workers can is not guaranteed. Te UI program is governed
be produced. and partially funded by the Department of La-
Researchers seeking access to LEHD data, in- bor, but it does not have direct access to the
cluding the UI wage record microdata, can ap- microdata and is only able to access LEHD mi-
ply to access the data through a Federal Statisti- crodata for projects supporting the Census Bu-
cal Research Data Center. Te researcher must reau mission.
comply with the requirements for conducting
research at a research data center. Among other 1. If the research project will include the release of state or
things, research projects using the LEHD must substate-level specifc results (as opposed to results from a
advance the mission of the Census Bureau, a group of states), the state will have the opportunity to review
and approve the external research project. If the intent of the
requirement that precludes many types of valu- external researcher is to release pooled results only, states can
able research. It also must be consistent with choose to waive their opportunity to review each proposed
applicable state law. Te data use agreement project. If this option is not selected, even external projects
that will release pooled results only would need to be re-
that the Census Bureau enters into with each viewed and approved by the relevant states.

$23 billion per year to support more efcient state tive Program is based on the relationship between
information technology systems for tracking Med- the U.S. National Center for Health Statistics and
icaid eligibility, enrollment, and claims.15 the 57 jurisdictions that collect vital records cov-
Some Federal agencies also have developed ering births and deaths in the United States (see
productive partnerships with states and other the box Data about Births and Deaths in the Pop-
jurisdictions to establish mutually advantageous ulation). In exchange for collecting data in a uni-
systems for making data accessible for evidence form manner, states receive funding, training, and
building. For example, the Vital Statistics Coopera- technical assistance.
Other examples of Federal-state partnerships
are not necessarily fnancial in nature. For example,
15. CEP staf correspondence with Department of Health and
Human Services, Assistant Secretary for Planning and Evaluation the Census Bureau ofers data analysis services to
staf in March 2017. state programs willing to make their data accessible
The Promise of Evidence-Based Policymaking 35

for Federal evidence building. One witness during needed for comparability.17 Second, some states
a public Commission meetingErin Ulric from provide data on the full set of program partici-
Colorados Women, Infant, and Children (WIC) pants or benefciaries, while others provide only a
Programdescribed how Colorado benefts from representative sample. To be integrated with other
receiving statistics generated by the Census Bureau data and used for evidence building, administra-
to monitor program performance and identify tive datasets are more useful when they include
populations for targeted outreach. In this sense, all participants or benefciaries. Supplying the
secure access to the Colorado WIC data by eligible universe fle also can reduce the burden on states,
researchers enables the state to improve the oper- as implementing robust sampling procedures re-
ation of its program by receiving relevant aggre- quires technical expertise. Furthermore, when the
gate statistics that inform day-to-day operations. datasets include personal identifers so program
Te Longitudinal Employer-Household Dynamics participation can be statistically associated with
Program (LEHD) also operates as a non-fnancial outcomes of interest, such as college graduation
partnership (see the box Longitudinal Employ- or earnings, members of the evidence-building
er-Household Dynamics Program). community are better able to produce valid and
If the combination of laws that apply to Fed- reliable estimates about program and policy out-
eral programs and policies can create confusion, comes.
the same can be especially true for state-held data.
When individual jurisdictions interpret Federal Challenge #3: Cumbersome and idiosyn-
laws, inconsistencies can arise in whether they
cratic data access procedures create confu-
permit the use of data on the very same program.
Several Federal agencies have taken actions to ad- sion, impose unnecessary costs, and are a
dress inconsistent interpretations across states. barrier to evidence building, without always
For example, to encourage greater use of data from providing signifcant privacy benefts.
TANF and SNAP, the U.S. Department of Health
and Human Services Administration for Children Te Commission heard repeatedly about the dif-
and Families and the U.S. Department of Agricul- fculties that cumbersome and onerous proce-
tures Food and Nutrition Service issued memo- dures, often the result of idiosyncratic processes
randa to state program directors clarifying what that vary across government, cause for members
data uses are permissible under their respective of the evidence-building community seeking to
program laws.16 securely access confdential data. For researchers
Beyond accessibility, members of the evi- and evaluators external to government, no stan-
dence-building community noted two additional dard for applying for data access currently exists
challenges related to using state-collected ad- across government agencies, making it necessary
ministrative data for evidence-building activities. to navigate diferent and varied processes for each
First, Federal programs may not require states or agency.
local jurisdictions to collect the same kind of data Te CEP Survey of Federal Ofces found that
in the same way. As a result, while data within a all 13 PSAs, 5 out of 8 evaluation ofces, and 29
single state, program, or jurisdiction may be ana- percent of other ofces (14 out of 49) provide
lyzed with relative ease, when data from multiple external researchers access to the data they col-
sites or states are brought together for analysis, lect.18 Some agencies have formal programs for
substantial cleaning and consistency edits are external researcher access, with well-established
procedures for applying for access. Under these
programs, researchers apply for access to data
and the agency qualifes eligible researchers and
approves their projects based on agency-defned
16. Susan Golonka, Re: U.S. Census Bureau Request for TANF
Data, (memorandum, Washington, D.C.: U.S. Department of
Health and Human Services, Administration for Children and
17. Harry Hatry, Using Agency Records, in Handbook of Practical
Families, December 7, 2016); Tomas Louis and Richard Lucas,
Program Evaluation, edited by Kathryn E. Newcomer, Harry Hatry,
Memorandum of Record Regarding the Sharing of State SNAP
and Joe Wholey, 325343 (Hoboken, N.J.: Wiley, 2015).
and WIC Recipient Data With the U.S. Census Bureau, (memo-
randum, Washington, D.C.: U.S. Department of Agriculture and 18. Two of the 10 responding evaluation ofces and 18 of the 67
U.S. Department of Commerce, August 12, 2014). responding other ofces reported they do not collect data.
36 The Promise of Evidence-Based Policymaking

processes. Commenters said these processes can Act in 2017, which found that administrative
often be slow and confusing. Other agencies do costs faced by researchers may be reducing the re-
not have formal programs for external research- turn on investment of Federally funded research
er access, so the application process can be much and development and that it is a matter of crit-
more ad hoc. ical importance to United States competitiveness
that administrative costs of Federally funded re-
Key issues encountered include: search be streamlined so that a higher portion of
Federal funding is applied to direct research.19
inconsistent, but often lengthy Some of the idiosyncrasies of data access pro-
and cumbersome processes for cedures result from a lack of capacity within agen-
accessing data. Each agency cies to devote to this work. Te Commission found
that agencies vary signifcantly in their capacity to
negotiates its own data sharing
provide external researcher access to data. A lack
agreements, which can add up of sufcient capacity restrains agencies ability to
to signifcant time and cost for undertake key administrative functions necessary
national studies that require data for providing access to confdential data, including
the development and dissemination of metadata
from multiple agencies in many and even the governments ability to explain pro-
states, particularly when there cesses and procedures to eligible researchers. A
may be variations in exactly what lack of capacity can also result in delays through-
the process requires and how long out a research project, from approving a research-
ers access request to reviewing the researchers
it takes. output before it can be released.
Andrew Weigan, President and Chief Te Federal Statistical Research Data Centers
Executive Ofcer at Social Policy Research (FSRDCs) are an example of a successful partner-
Associates, Commission Public Hearing, San ship to expand the Federal governments capacity
Francisco, February 9, 2017 to facilitate external researcher access to data for
statistical purposes (see the box Federal Statisti-
Tese challenges are compounded when a re- cal Research Data Centers). Te Census Bureau
searcher seeks to access multiple agencies data. and other Federal agencies have collaborated with
In this situation, the researcher must apply to and host institutions to enable approved researchers
be approved for access by all of the agencies. Te to access confdential data from multiple agencies
result can create scenarios in which one agency through a network of secure data enclaves. While
provides an approval and another does not, neces- efective for facilitating data access, the FSRDCs
sitating ongoing negotiations between the agen- are still constrained by a lack of standard access
cies and the researcher. procedures across the agencies making their data
Not only does this result in an inefcient pro- available through an FSRDC. In practice, this
cess, these inefciencies might not enhance con- means that a researcher seeking data other than
fdentiality or privacy because the standards for or in addition to Census Bureau data must still
access and use are not based on a common un- apply to the originating agency for access, which
derstanding of risk for linked data. Nor do these can often result in delays and administrative hur-
access procedures result in standard information dles necessitating months or years of efort to gain
about use and practice to provide transparency access. Progress is being made on this front, with
and an ability to be audited. work on new data access agreements that will
Moreover, inefciencies in data access process- enable the FSRDC staf to approve projects on be-
es for evidence building create administrative ex- half of an originating agency. However, progress
penses and researcher burdens that can impede is only available to the extent applicable statutes
Federally funded research. Te costs associated provide agencies such discretion, as discussed ear-
with excessive administrative burdens were ac- lier in this chapter.
knowledged by the Congress and the President in
a somewhat diferent context with the enactment 19. American Innovation and Competitiveness Act (Public Law
of the American Innovation and Competitiveness 114329, Section 201, January 6, 2017).
The Promise of Evidence-Based Policymaking 37

Federal Statistical Research Data Centers


Te U.S. Census Bureau, on behalf of the Fed- to secure access to data, geographic proximi-
eral statistical agencies, operates a network of ty for researchers is key. Te FSRDC network
27 Federal Statistical Research Data Centers has grown organically with demand. Te frst
(FSRDCs) in 18 states and the District of Co- remote FSRDC opened in Boston in 1994. Te
lumbia to provide secure access to a range of most recent FSRDCs opened at the University
Federal restricted-use microdata for statistical of Kentucky and the University of Colorado
purposes. Boulder in August 2017. With these additions,
Te FSRDCs are partnerships between Fed- the FSRDC network will have doubled in size
eral statistical agencies and research institu- since 2013. As of 2016, the FSRDC network
tions across the country. Because the current supported about 900 researchers working on
structure requires physical presence in order about 250 diferent projects.

Federal Statistical Research Data Center (FSRDC) Locations

Northwest

New York -
Minnesota Cornell Boston
Wisconsin Michigan Yale
New York - Baruch
Central Plains Chicago
Penn State Philadelphia
California - Berkeley
Georgetown Maryland
California - Stanford Rocky Missouri Univ. of U.S. Census Bureau
Mountain Illinois
Kansas Kentucky
Urbana- Triangle - Duke
California - UCLA City
Champaign Triangle - RTI
California - USC
California - Irvine
Atlanta

UT - Austin Texas

Legend
Current FSRDC
Planned FSRDC

Once access has been approved, researchers en- fact-fnding phase, the Commission heard con-
counter other structural issues with FSRDCs such cerns that the FSRDC network, as it exists cur-
as the requirement to be physically present at an rently, would be overwhelmed by the additional
FSRDC site, the limited number of seats avail- demands of supporting an increase in data access
able at a given FSRDC, the limitations of current for a higher volume of evidence building.
computing resources available, and the wait time Members of the evidence-building community
to obtain disclosed results.20 Further, through its inside government face challenges similar to those
confronted by external researchers. Te non-stan-
20. According to the Census Bureau, the average wait time in 2017 dardized and lengthy processes by which access
is approximately three weeks for obtaining disclosed results. is negotiated and memorialized via an agreement
38 The Promise of Evidence-Based Policymaking

were among the most common data access barriers close to 30 percent (23 out of 79) of responding of-
cited by stakeholders. Formal data access agree- fces indicated that lack of staf, policies, and proce-
ments (e.g., Memoranda of Understanding or dures to establish data sharing agreements in their
MOUs) between two or more agencies can take own ofce made it hard for them to get access to
years to develop. Delays associated with negotiat- the specifc dataset they needed.24
ing MOUs are compounded by the challenges de-
scribed earlier in this chapter when legal authori- Challenge #4: Many Federal departments
ties are inconsistent. A lack of clear legal authority
do not sufciently or regularly assess their
can result in extended reviews and negotiations
by lawyers within multiple ofces or departments information resources to determine access
prior to granting access. An agencys ability to ef- controls and privacy protections appropri-
fciently execute MOUs can also be impacted by ate to the sensitivity of the data.
capacity constraints.
As described earlier in this chapter, a variety of laws
I think ideally you would have govern access to government data for evidence
building. Federal agencies follow these laws, asso-
a much more streamlined [data ciated regulations, and agency legal interpretations
access] procedure that was when making decisions about data security proto-
really oriented around statistical cols and access restrictions for the data they hold.
research. Te Commission fnds that many PSAs imple-
ment tiered access programs that set data access
Raj Chetty, Professor of Economics at
and security requirements based on an assessment
Stanford University, Commission Meeting,
of dataset sensitivity. Tiered access is an applica-
Washington, D.C., July 22, 2016
tion of data minimization, a key privacy safeguard
for evidence building as embodied in the Fair In-
In 2014, OMB developed an optional model formation Practice Principles (described in Chap-
agreement for agencies to consider, though uptake ter 3). Data minimization means giving access
has been limited to date.21 Some states also have to the least amount of data needed to complete
pursued similar agreements for use among internal an approved project. For example, an eligible re-
state agencies. Michael Basil, General Counsel in searchers project might earn approval for access
the Illinois Department of Innovation and Technol- to confdential information at a highly secure re-
ogy, described the states efort to develop an En- search data center that requires expert review of
terprise Memorandum of Understanding (eMOU) all output. Another researchers project may need
in an efort to improve the use of data at the state only access to a data query tool that runs an anal-
level.22 In 2016, an expert panel sponsored by Ac- ysis, checks for disclosure risk without ever show-
tionable Intelligence for Social Policy developed ing individual records, and provides group statis-
model templates for Memoranda of Understanding tics (see the box Data Query Tools). If researchers
to help states develop integrated data systems.23 All fnd that a data tool meets their needs, they may
of these eforts suggest opportunities for develop- never need to access the underlying confdential
ing a common MOU template for use in the Federal data. Similarly, researchers, journalists, and other
government. In the CEP Survey of Federal Ofces, members of the public may be able to answer their
questions by using data that have been masked,
21. Ofce of Management and Budget (OMB), Guidance for Pro- de-identifed, and approved for release as a pub-
viding and Using Administrative Data for Statistical Purposes lic use fle or by reference to published statistics.
(memorandum M1406, Washington, D.C.: OMB, Executive Of- A well-designed and properly implemented data
fce of the President, February 14, 2014).
minimization strategy like tiered access can reduce
22. Michael Basil, Illinois Department of Innovation and Technol- the risk of unauthorized use and unintended harm
ogy, Commission Meeting, Washington, D.C., September 9, 2016.
to individuals.
23. John Petrila, Barbara Choh, Wendell Pritchett, Paul Stiles,
Victoria Stodden, Jefrey Vagle, Mark Humowiecki, and Natassia
Rozario, "Legal Issues for IDS Use: Finding a Way Forward," (Phil-
adelphia: Actionable Intelligence for Social Policy, 2017); https:// 24. Based on 79 ofces that reported they collect or use data for
www.aisp.upenn.edu/wp-content/uploads/2016/07/Legal-Issues.pdf statistics, evaluation, research, or policy analysis or spend a por-
(accessed August 10, 2017). tion of their budget for such purposes.
The Promise of Evidence-Based Policymaking 39

Data Query Tools


Online query tools allow researchers to fnd conduct research on restricted access data
answers to their questions without ever ac- fles without needing access to the underlying
cessing individual records. Researchers submit data.Te simplest query tools typically include
requests for analysis through the tool. Te que- a table generator that allows the researcher to
ry tool conducts the analysis behind a secure select variables for rows and columns; the que-
server wall and returns only the result to the ry tools generate the results. More advanced
researcher. Tis approach can be applied to query tools allow researchers to submit code
support access to data in a way that limits dis- for statistical analyses on the restricted use
closure risks. data. Query tools using underlying confdential
Te National Center for Education Statis- data can automatically evaluate output for dis-
tics uses such technology in its DataLab ap- closure risk before returning the output to the
plication. Te application allows analysts to researcher.

Te National Center for Health Statistics uses a holders, including researchers and privacy advo-
variation of tiered access based on data sensitivity. cates, to establish access restrictions based on law,
Te agency produces a public use version of the context, and sensitivity. Such a review would en-
National Health and Nutrition Examination Survey sure that departments give appropriate access to
that has been masked to protect individual iden- Federal data needed for evidence building while
tities, but requires researchers needing access to protecting sensitive personal information.
human genome data to train and qualify as sworn Even with a tiered access approach, the Fed-
agents of the National Center for Health Statistics eral government currently lacks a consistent and
before accessing the data in a secure facility. Today, objective system for classifying statistical and ad-
Federal agencies can take advantage of new tech- ministrative data based on sensitivity. Te absence
nologies that allow versions of confdential data- of such a classifcation system limits governments
sets that are safe for wide public use. Some of those ability to describe access restrictions and data se-
technologies are described further in Chapter 3. curity and privacy protocols appropriate for each
To determine appropriate access, Federal de- level of sensitivity, in a way similar to the well-
partments should consider the sensitivity of the known classifcation and handling schedules for
data in context. A recent law review article on national security information.
sensitive information indicated that the conclu- Harvard University has developed a data tags
sion that a particular type of information should system to code the sensitivity of various datasets
be treated as sensitive gives rise to special rules of that illustrates the potential of such a system.
collection, use, and disclosure as a means to pre- Te assigned data tags govern the encryption
vent security or privacy harm. 25 Existing guide- protocols used for data storage and transmission
lines do not require data access decisions to be as well as the requirements for data access.26 Te
based on a formal analysis of the sensitivity of national archive of Denmark is considering using
the data. Te Commission did fnd, however, that a similar system to assess the sensitivity of its
most agencies conduct informal analyses of the datasets in preparation for the implementation in
sensitivity of data, and several agencies, including 2018 of the European Unions General Data Pro-
all PSAs, conduct formal analyses as part of their tection Regulation.27
standard disclosure review processes.
While staf in Federal departments include ex-
perts with the ability to assess the sensitivity of
26. Latanya Sweeney, Merc Crosas, Michael Bar-Sinai, Sharing
the data, they can beneft from input from stake- Sensitive Data with Confdence: Te Datatags System, Technology
Science (October 16, 2015); https://techscience.org/a/2015101601
(accessed August 10, 2017).
25. Paul Ohm, Sensitive Information, Southern California Law
Review 88 (2015): 7; https://papers.ssrn.com/sol3/papers.cfm?abstract- 27. EU General Data Protection Regulation (GDPR). EU GDPR
id=2501002 (accessed August 10, 2017). Portal; http://www.eugdpr.org/ (accessed August 10, 2017).
40 The Promise of Evidence-Based Policymaking

Figure 5. Model of Sensitivity when taken together with the privacy enhance-
Levels for Federal Data ments proposed in Chapter 3, the following rec-
ommendations will safely harness the potential
Sensi- of data to produce more and better evidence in
Level Description
tivity the future.
5 Crimson Maximally restricted.
Highly sensitive. Identifable REC. 2-1: Te Congress and the Presi-
records from data dent should enact legislation estab-
collected with a promise of
confdentiality.
lishing the National Secure Data Service
4 Red Restricted. Sensitive.
(NSDS) to facilitate data access for evi-
Identifable records from dence building while ensuring transparen-
data collected with a cy and privacy. Te NSDS should model
promise of confdentiality. best practices for secure record linkage
3 Yellow Restricted. Crimson or and drive the implementation of innova-
Red datasets modifed by
tive privacy-enhancing technologies.
technologies that mask
individual records (e.g. data
query tools, diferential Addressing the Federal governments capacity for
privacy). evidence building should begin by enabling con-
2 Green Minimally restricted. sistent, transparent, and accountable secure ac-
Not sensitive. Data fles cess to data. Recommendation 2-1 will establish
made available to the NSDS to help address the existing gaps in the ca-
public but subject to pacity to meet this need on a consistent, govern-
procedures designed to ment-wide basis. Because the evidence-building
raise accountability by users, community is highly decentralized, gaps exist in
such as registration before the consistent application of these functions and
accessing. a service with the primary mission to address
1 Blue Public data. Most safe. these gaps is needed (see Figure 6). Specifcally,
Open data. a service charged with facilitating access and en-
suring protection of data for evidence building is
a direct approach to ensure evidence is more rou-
Based on the Harvard model, Figure 5 provides tinely generated using privacy best practices.
an example of sensitivity levels that could be ap- Te NSDS should support secure access to con-
plied to Federal data. fdential datasets in a setting that meets strin-
gent protection standards. Te NSDS will apply
Recommendations strict data minimization techniques to ensure re-
searchers accessing combined data will use data-
Te Commission concludes that a priority for ev- sets with as much information removed as is pos-
idence building now and in the future is to equip sible while still meeting the research need. When
the evidence-building community with secure, re- two or more datasets will be combined, only a
stricted access to data to facilitate the generation of narrow group of qualifed and trained employees
high quality evidence about government programs will have access to direct identifers to conduct
and policies. Recognizing the benefts of maximiz- the linkage.
ing privacy protections and the utility of existing Te NSDS also should be responsible for devel-
data, the Commission identifed a series of rec- oping and implementing state-of-the-art meth-
ommendations to better enable secure, restricted ods to safely combine confdential administrative
access to increased amounts of the data already and survey data from two or more Federal depart-
collected by government. Many of these data are ments for evidence building. Tese combined data
highly relevant for informing and evaluating an will provide new opportunities for evidence build-
array of Federal programs and policies. Based ing. Te NSDSs existence will highlight how im-
on the evidence examined by the Commission, portant it is for the Federal government to use the
The Promise of Evidence-Based Policymaking 41

data it already collects with an unparalleled level Figure 6. Key Functions


of transparency about who is using which data for of the National Secure
what statistical purposes. Data Service (NSDS)
Te NSDS should help develop and apply cut-
ting-edge technologies to create highly priva-
cy-protective versions of combined datasets. A Privacy and
well-implemented tiered-access system will limit Transparency
the need for access to combined datasets contain-
ing identifable records.
Like the existing PSAs, the NSDS must have an
exclusively statistical mission. Distinct from PSAs, NSDS
which typically have a specifc topical focus, the National Secure Data
Data Service
NSDS will focus more functionally on data access Access
Secure and
and confdentiality for use in evidence building,
Linkage Analysis
especially across topical areas. Te NSDS should
be designated by the Congress and the President
as a PSA under CIPSEA to aford the entity protec-
tions already in law and embodied in other rec-
ommendations throughout this report. Te stat-
ute creating the NSDS should expressly prohibit
its use for non-statistical activities. REC. 2-2: Te NSDS should be a ser-
Te NSDS should be located in the U.S. Depart-
ment of Commerce, building upon the Depart- vice, not a data clearinghouse or
ments extensive portfolio of statistical and warehouse. Te NSDS should facilitate
data-related bureaus and expertise existing within temporary data linkages in support of dis-
the Census Bureau, the U.S. Bureau of Economic tinct authorized projects.
Analysis, the U.S. National Institute of Standards
and Technology, and the U.S. National Oceanic Te Evidence-Based Policymaking Commission Act
and Atmospheric Administration, among others. charged the Commission with considering whether
However, it should be situated in such a way as the country needed a data clearinghouse.28 Con-
to provide independence sufcient to set strate- sistent with this charge and after careful consider-
gic priorities distinct from any existing Commerce ation of the issue, the Commission has concluded
agency and to operate apart from policy and that a clearinghouse should not be created.
related ofces. Te NSDS also must be organized Specifcally, the Commission interprets the
in such a way as to prioritize support for evidence word clearinghouse as connoting a data storage
building across government, rather than support facility that permanently stores records from mul-
specifc to any one department, as is the case for tiple databases from multiple agencies and, there-
existing PSAs. fore, grows with each new data linkage. Previous
Te institutional placement and governance panels and commissions either have come close
of the NSDS should be considered in terms of any to recommending this type of clearinghouse or
implications for key objectives: (1) transparency did recommend one.29 Tese previous eforts pro-
and trust, (2) support for evidence building, (3) voked well-founded concerns about the potential
strategic coordination and cooperation, (4) conf- privacy harm such a clearinghouse could raise.
dentiality and security, (5) authority and fexibil- As further elaborated in Chapter 4, the Com-
ity, (6) scalable functionality, and (7) sustainabil- mission rejects the clearinghouse model in favor
ity. An institutional placement within an existing of the NSDS. Te NSDS should be designed to link
Federal department has a practical beneft, partic-
ularly the ability to leverage shared services for
28. Evidence-Based Policymaking Commission Act of 2016 (Public
administrative functions, as well as existing pro- Law 114140, Section 4(b), March 30, 2016).
fessional staf, established levels of public trust,
29. As described in Chapter 4, a committee in the 1960s proposed
and operational knowledge. creating a national data bank to address the decentralized nature
of the Federal governments data infrastructure.
42 The Promise of Evidence-Based Policymaking

data on an individual project basis only. In contrast Weighing its charge and the optimal arrangement
to a clearinghouse, it should not lead to the estab- for data collected by government, the Commission
lishment of a store of data that grows with every concluded that more of the data the government
research project conducted. Te data linked for a holds should be accessible for statistical purposes
project through the NSDS should be kept structur- to inform programs and policies. Such access op-
ally separate from other data linked through the timally will be grounded in clearly articulated au-
NSDS for other projects. By strictly enforcing this thority from the Congress for program agencies to
design, the NSDS will further the goal of increased make their data available for statistical purposes
access to and use of data for specifc research and and for agencies envisioned as recipients to use
evaluation eforts, without unduly increasing the the data for those purposes. Absent clear direction
potential for privacy harm. from the Congress, many Federal departments cur-
Te Commission envisions the NSDS working in rently may choose not to use existing authorities
concert with the existing network of PSAs to facili- or may interpret their statutes as lacking these au-
tate secure access to government data for evidence thorities. To provide clarity about permissible sta-
building conducted by those inside and outside of tistical uses, the Commission recommends build-
government. Te PSAs currently house and man- ing on the existing CIPSEA framework.
age much of the data critical to Federal evidence Te joint benefts to privacy and data access
building in order to fulfll their statutory missions. envisioned by the Commission can be realized si-
Tey collectively have appropriate infrastructure, multaneously within an appropriate enabling legal
technical expertise, and legal authority to protect framework. Te existing CIPSEA framework, in-
data. Terefore, the NSDS will both call upon PSAs cluding the provision that authorizes the sharing
to address requests by researchers and evaluators, of business data among three designated statistical
and assist PSAs in furthering their own research to agencies, permits efective statistical uses of data
improve the quality and efciency of existing sta- within a protected enclave. Te Commission rec-
tistical programs. Such a relationship will allow the ommends that the CIPSEA framework be extended
NSDS to facilitate access and create approved link- by providing NSDS with the powers and exemp-
ages, while also ensuring the underlying data re- tions necessary to use, for statistical purposes,
main dispersed in their originating agencies. data collected by other agencies. To ensure that the
Te NSDS may also support evidence building acquisition and use of data by the NSDS occurs in a
in states and localities, especially as part of cre- privacy-protecting manner that is at the frontier of
ating mutually benefcial partnerships around use what is feasible, the Commissions recommenda-
of Federal, state, and locally held data. Te NSDS tion also requires that NSDS adhere to specifc new
could also ofer analytical services to Federal, state, and stringent privacy qualifcations outlined else-
and local agencies that lack capacity to undertake where in the Commissions recommendations. Te
the work on their own, providing both needed ca- Commission does not intend for the new authori-
pacity and ensuring privacy best practices. ty given to the NSDS to curtail existing authorities
to acquire and link data currently being utilized
REC. 2-3: In establishing the NSDS, by the PSAs, but does see value in encouraging all
such activity to occur within the CIPSEA frame-
the Congress and the President
work. In fact, the Commission believes that some
should amend the Privacy Act and the PSAs will elect to use the NSDS to facilitate their
Confdential Information Protection and own research with potential to address the quality,
Statistical Efciency Act (CIPSEA) to re- burden, or cost-efciency of statistical production,
quire new stringent privacy qualifcations and that the NSDS will call upon the PSAs to pro-
as a precondition for the NSDS to acquire vide data to facilitate others research activities.
As a companion to the CIPSEA amendment, the
and combine survey and administrative Commission also recommends that the Congress
data for solely statistical purposes. At the and President amend the Privacy Act to extend to
same time, the Congress should consider NSDS the existing exception to consent require-
additional statutory changes to enable on- ments that applies when agencies disclose records
going statistical production. within a system of records to the Census Bureau.
CIPSEA authorization alone may not be sufcient to
The Promise of Evidence-Based Policymaking 43

enable needed access to data, as agencies may also programs.30 Similarly, under Title 26 of the U.S.
have to satisfy the Privacy Acts requirement to ob- Code, the predominant purpose of research con-
tain consent before disclosing information to third ducted using confdential Federal tax information
parties. Te recommendation acknowledges that must be tax administration.31 Likewise, the law
the evidence-building landscape has evolved sig- generally applicable to the Census Bureau, Title 13
nifcantly since the enactment of the Privacy Act. of the U.S. Code, permits access to data only for
With the Census Bureau exemption, the Congress the narrow purpose of improving Census Bureau
demonstrated its belief that the provision of ad- programs.
ministrative data under strong confdentiality pro- Where deemed appropriate, the Congress
tections for exclusively statistical purposes was safe should act to amend relevant statutes to autho-
enough to exempt it from the Privacy Acts consent rize access and use of data for statistical purposes.
requirements. In light of the stringent privacy and Te recommendation encourages the Congress to
transparency protections built into its procedures, review its prior determinations in the context of
the Commission believes there is justifcation to ex- the desire for increased use of evidence and the
tend the same narrow exemption to the NSDS. modernized legal and security environment con-
Te Commission does not envision that the templated in this report.
NSDS will be a locus of statistical production. Tat
role will remain, as today, with the statistical agen- REC. 2-5: Te Congress and the Presi-
cies in which the Congress has vested the responsi-
dent should consider repealing cur-
bility for such work. For the purpose of improving
the efciency and precision of key statistical indi- rent bans and limiting future bans on the
cators and datasets, these agencies should be able collection and use of data for evidence
to engage with the NSDS to carry out research on building.
and development of new statistical products using
data linked under the authorities newly granted Legislative bans on the collection and use of data
to the NSDS. But once research and development create barriers to the use of information for evi-
gives way to production, that activityalong with dence building. Te Commission recognizes that
the linking of data necessary to support itwill the creation of bans may be a mechanism for set-
need to revert to the relevant statistical agency or ting priorities for the collection and use of data in
agencies. Te Commission encourages the Congress a complex policy context. While it is clearly within
to consider conforming amendments, including to the purview and responsibility of the Congress to
CIPSEA and the Privacy Act, to allow these improve- establish such priorities, absent articulated crite-
ments in statistical production processes to occur. ria, bans on data collection and use create a seri-
ous impediment to evidence-based policymaking,
REC. 2-4: Te Congress and the Presi- and could make it difcult or impossible to hold
government activity accountable. Provided that
dent should review and amend, as ap-
the collection and use of data occur consistent
propriate, statutes such as Title 13 of the with the Commissions guiding principles, partic-
U.S. Code to allow statistical uses of survey ularly with regard to privacy protection, such col-
and administrative data for evidence build- lection and use generally should be allowed.
ing within the CIPSEA secure environment. Te Commission concluded that any use of data
may carry both value and the risk of harm. Tere
Te Commission identifed several datasets that are is also, however, a risk of harm associated with
highly valuable for evidence building (see Appendix the investment of billions of dollars in taxpay-
D). Several of the statutes that govern use of these er funds without adequate monitoring and study
datasets contain explicit prohibitions or strong of the efectiveness of the taxpayers investment.
restrictions on use for the purposes described in
this report. For example, the Higher Education
Opportunity Act limits the use of data collected to 30. Higher Education Opportunity Act (Public Law 110315, Au-
operate Federal student fnancial assistance gust 14, 2008); https://www.gpo.gov/fdsys/pkg/PLAW-110publ315/
pdf/PLAW-110publ315.pdf (accessed August 10, 2017).
programs to administration of fnancial aid
31. Internal Revenue Code, 26 USC 6103(h).
44 The Promise of Evidence-Based Policymaking

In considering the application of a ban on data REC. 2-6: Te Congress and the Presi-
collection and use, the Congress should create a
dent should enact statutory or other
means to weigh the various, and sometimes com-
peting, interests at stake. Te Commission empha- changes to ensure that state-collected ad-
sizes that encouraging the review of bans does not ministrative data on quarterly earnings
mean privacy protections should be undermined or are available solely for statistical purposes.
that data collection and use should be guaranteed. Te data should be available through a sin-
Rather, the Commission is encouraging the Con-
gle Federal source for solely statistical pur-
gress to develop reasonable criteria to help ensure
that legislators have the tools and opportunity to poses.
carefully weigh the implications of signifcant bans. Te Commission fnds that increasing access to
As noted previously, the Commission under- state-collected administrative data on quarterly
stands the complexity, motivations, and concerns earnings is of primary importance. Numerous Fed-
involved in weighing privacy against other consid- eral programs and policies seek to impact earned
erations. History is replete with examples of get- income. Data on quarterly earnings are already
ting this balance wrong. Yet, with modern priva- collected by states through their administration
cy-enhancing technologies, a well-developed legal of unemployment insurance programs, but these
framework, and a strong commitment to transpar- data are not easily accessible by researchers for use
ency, the Commission believes it is possible to use in evidence-building activities. For a more com-
data to evaluate the efectiveness of policies and plete picture of wage recipients, access to quarterly
ensure accountability to stakeholders, while also wage data from multiple states is critical.
protecting privacy. Te Commission also believes Te Commission identifed several strate-
it is incumbent on the Congress and the President gies for improving access to administrative data
to regularly and thoroughly consider the value of on quarterly earnings for statistical purposes,
policies that preclude evidence building against including expanding access to existing multi-state
the prospective public value of the evidence that data sources such as the National Directory of
could be produced. New Hires or LEHD or developing a new nation-
al UI wage record system (see the box Expanding

Expanding Access to Quarterly Earnings Data


Te Commission identifed several possible state wage record data for statistical activities.
ways to achieve the goal of improving access to Currently, they have uneven access to these
administrative data on quarterly earnings for data, despite the mandatory reporting require-
statistical purposes. One approach could be to ment for NDNH and the existence of voluntary
expand access for statistical purposes to wage reporting for the LEHD program.
records maintained by the National Directory Creating a single system to which states re-
of New Hires (NDNH) system or to wage re- port UI wage record data could reduce the bur-
cords maintained by the U.S. Census Bureau as den on states, who currently report the same
part of the Longitudinal Employer-Household data to multiple entities, and could reduce the
Dynamics Program (LEHD). burden on Federal agencies that spend a signif-
Alternatively, the Congress and the Presi- icant amount of resources negotiating multiple
dent could create a national Unemployment agreements and memoranda of understand-
Insurance (UI) quarterly earnings data system ing with the states. A single system also could
within the U.S. Department of Labor. A na- streamline access to this highly valuable data
tional UI quarterly earnings data system could source for statistical activities, while enhanc-
build on the same UI wage record data that cur- ing the security and privacy of the data through
rently support both the NDNH system and the development of standardized procedures for
LEHD program. In fact, states consistently have data submission and access.
expressed interest in being able to access inter-
The Promise of Evidence-Based Policymaking 45

Access to Quarterly Earnings Data). Each of these microdata from states on participants of all three
approaches has strengths and weaknesses. As programs, sample data have limited use in evidence
such, the Commission chose to focus its recom- building. Universe data, which includes all program
mendation on the desired outcomethe availabil- participants or benefciaries, are the data that have
ity of national administrative data on quarterly maximum value for evidence building. In addition,
earnings, accessible from a single Federal source, existing data are inaccessible for evidence building.
for use in evidence buildingrather than articulat- Recent eforts by the Census Bureau to enter into
ing a specifc approach. data sharing agreements with states to access these
data have been advancing, although slowly. As of
REC. 2-7: Te President should direct July 2017, there were 15 states sharing SNAP data
and 9 states sharing TANF data through this efort,
Federal departments that acquire
with varying years of data available, limiting the
state-collected administrative data to ability to easily conduct multi-state analysis.33
make them available for statistical purpos- In cases where barriers to access are signif-
es. Where there is substantial Federal in- cant and deemed to be misaligned with both the
vestment in a program, Federal depart- amount of Federal investment in a program and
ments should, consistent with applicable evidence-building needs, the Congress should con-
sider whether to enact statutory changes to require
law, direct states to provide the data nec- the submission by states of administrative data on
essary to support evidence building, such all program participants that could be combined
as complete administrative data when with other data sources for exclusively statistical
samples are already provided. activities. While signifcant, the challenges of ex-
panding the existing statutory authority to require
Te Commission strongly believes that increas- the submission of universe data in the case of SNAP,
ing access to state-collected administrative data UI benefts, and TANF, for example, could be less-
is vital to efciently increasing the volume and ened considerably with the provision of supports
quality of evidence that can be produced and to help states comply with a new requirement. Te
used for policymaking. Availability of state-col- Commission expects that doing so for these three
lected administrative data at the Federal level is programs could lead to especially valuable informa-
highly variable. In considering which administra- tion for policymakers and program managers.
tive data to prioritize for enhanced accessibility, Recognizing the potential value of state-held
policymakers might consider the level of Federal datasets related to Federal programs, the Com-
investment in the program, the value of the data mission recommends that Federal departments
for evidence-based policymaking, and the level conduct a thorough review of relevant statutes
of difculty that might be encountered in seek- and regulations governing the accessibility of
ing to increase access to a particular data source. state-collected data as well as any statutory use
For example, SNAP, UI, and TANF represent three restrictions related to evidence building to identi-
important programs in the current safety net in fy barriers that impede access to the data by qual-
the United States, and each involves substantial ifed researchers seeking to use the data for sta-
Federal and state expenditures. 32 tistical purposes. Te Congress and the President
Currently there are two distinct challenges in should require Federal agencies to submit a report
using state-collected administrative data for Fed- to the Congress summarizing the barriers they
eral evidence-building purposesthe data that are identify from a review of the relevant statutes and
available for evidence building are limited and what regulations governing these collections.
are available are not easily accessible. While the Across the existing jointly administered Fed-
Federal government already has the statutory au- eral-state programs in which the submission of
thority to collect a sample of linkable administrative universe administrative microdata is statutori-
ly required, the Federal government provides a

32. Trudy Renwick and Liana Fox, Te Supplemental Poverty Measure


2015 (Washington, D.C.: U.S. Census Bureau, September 2016) 33. U.S. Census Bureau, Administrative Data Inventory; https://
table 5b; https://www.census.gov/content/dam/Census/library/ www2.census.gov/about/linkage/data-fle-inventory.pdf (accessed
publications/2016/demo/p60-258.pdf (accessed August 10, 2017). August 10, 2017).
46 The Promise of Evidence-Based Policymaking

Figure 7. Research Process REC. 2-8: Te Ofce of Management


After Instituting NSDS and Budget should promulgate a single,
streamlined process for researchers external
Researcher Begins Project
Involving Two Agencies to the government to apply, become quali-
fed, and gain approval to access government
data that are not publicly available. Approval
Researcher Submits Application would remain subject to any restrictions
appropriate to the data in question.
NSDS When the regulatory framework permits data ac-
Qualifes Researcher cess for statistical purposes, researchers still need
Analyzes Legal Concerns clear and well-documented protocols and proce-
Analyzes Privacy Concerns
Analyzes Transparency Concerns dures for data access. A centralized and known set
of procedures for requesting access to data allows
greater focus on protecting privacy and avoids
inadvertently establishing onerous roadblocks to
Approved X Denied accessing data. A streamlined process will ensure
that data access requests are all adequately scruti-
nized. In fact, the approach for improving access
NSDS Solicits Data Researcher May would acknowledge diferences in the sensitivity
From Both Agencies Work to Address levels of data and set appropriate access require-
Data are Received,
Concerns and ments and data security and privacy protocols in
Privacy Protected, and Re-Apply
accordance with applicable laws or regulations.
Pre-Formatted as
Comparable Sets
Te Commission recognizes OMB as the pri-
mary actor for Recommendation 2-8 in order to
capitalize on OMBs existing legal authority and
Data Accessible experience in setting information and statistical
to Researcher policy that applies government wide. OMB should
Researcher Accesses include key stakeholders, both within and exter-
Data Within nal to the government, throughout the process of
Secure Environment establishing new procedures and standards relat-
ed to access and sensitivity.
Te NSDS will play a lead role in implementing
Researcher Provides Process Feedback these new standards (see Figure 7). NSDS will be
responsible for consulting and collaborating with
Federal departments to assign appropriate sensi-
signifcant amount of support to enable states to tivity levels to datasets they hold. Because sensitiv-
comply with reporting requirements. Te provi- ity may change over time, it would be appropriate
sion of funding for information technology infra- for sensitivity analyses to be reviewed whenever
structure and technical assistance can assist states important new information about potential risks
in building the necessary data collection systems becomes available. Sensitivity levels could be
and reporting structures to be able to comply with based on a standard classifcation system for data
a new requirement for increased data collection sensitivity, as discussed earlier in the chapter.
and transmission and to address issues of data in- Evaluators and researchers alike could similar-
tegrity. Federal departments should explore and ly be more readily informed about what data are
adopt incentives to support either compliance available at what levels of sensitivity with what
with a requirement for states to make data avail- access restrictions. As appropriate, the Federal
able to the Federal government or to incentivize government should provide detailed technical
the voluntary action of making administrative data documentation useful in planning projects
data available to the Federal government in cases and designing research studies, as discussed fur-
where a requirement is not established. ther in Chapter 4.
The Promise of Evidence-Based Policymaking 47

3
Enhancing Privacy Protections for Federal
Evidence Building

Improve data quality, integrity, and security by updating laws to require risk assessments
for data releases, embracing cutting-edge technologies, and prioritizing data
stewardship and the public trust.

Recommendations
3-1: Te Congress and the President 3-3: Te President should direct Federal
should amend the Privacy Act and the departments to assign a senior ofcial
Confdential Information Protection and Sta- the responsibility for coordinating access to
tistical Efciency Act (CIPSEA) to require Fed- and stewardship of the departments data re-
eral departments to conduct a comprehensive sources for evidence building in collaboration
risk assessment on de-identifed confdential with senior department information technolo-
data intended for public release. De-identifed gy, privacy, and other leaders. A Principal Sta-
confdential data subject to the Privacy Act and tistical Agency head, or other appropriately
CIPSEA should be made available only after a qualifed senior ofcial, should serve this func-
disclosure review board (1) approves the re- tion.
lease and (2) publicly provides the risk assess-
ment and a description of steps taken to miti- 3-4: Te Congress and the President
gate risk. should enact legislation to codify rele-
vant portions of Ofce of Management and
3-2: Te President should direct Federal Budget Statistical Policy Directive #1 to protect
departments, in coordination with the public trust by ensuring that data acquired un-
National Secure Data Service, to adopt state-of- der a pledge of confdentiality are kept conf-
the-art database, cryptography, privacy-preserv- dential and used exclusively for statistical pur-
ing, and privacy-enhancing technologies for poses.
confdential data used for evidence building.
48 The Promise of Evidence-Based Policymaking

T he Federal government has a legitimate need


and responsibility to use data for evidence
building. At the same time, the public has a legit-
Figure 8. The Privacy/Data Use
Frontier Curve
imate interest in knowing that their government More
is protecting their privacy while using their data.
With emerging technologies, the Commission be-
lieves that the country can have both (see Figure Future
8). Te knowledge and technology now exist to
achieve the Commissions vision for improving se-
cure access to confdential data while simultane- Data
ously enhancing privacy. Trough law and policy, it Use
is possible to ensure that Federal departments are
consistently adopting the most up-to-date meth- Current
ods to protect data used for evidence building.

Privacy, when properly Less


Privacy
More
implemented through good
Privacy-Enhancing Technologies
programs and early on, promotes Allow Greater Protection of Privacy
innovation in every case and While Increasing Access to Data
allows for and encourages the
wide scale adoption of new
provide secure access to confdential data. In addi-
technology. In fact, privacy, when tion, privacy considerations have underscored all
implemented correctly, fosters of the Commissions deliberations and priorities.
more information sharing, not Privacy, data security, confdentiality, and pub-
less. lic transparency were frequent themes in written
comments provided to the Commission and in
Marc Groman, then-Senior Advisor for
each of the three public hearings. Finally, Commis-
Privacy at OMB and Chair of the Federal sioners and staf sought out additional experts on
Privacy Council, Commission Meeting, privacy-protective technologies and approaches to
Washington, D.C., September 9, 2016 develop the recommendations in this chapter.
Just as the Fair Information Practice Principles
Troughout its deliberations, the Commission guide the Federal governments privacy policy, the
prioritized consideration of privacy on behalf of the Commission believes that these Principles should
American public. Te importance the Commission similarly guide the use of data for evidence-based
attached to privacy is captured in the frst of the policymaking in accordance with the Commis-
guiding principles for evidence-based policymak- sions guiding principle on privacy (see the box
ing stated in Chapter 1: Individual privacy and Fair Information Practice Principles). By con-
confdentiality must be respected in the generation sidering each of the Fair Information Practice
and use of data and evidence. Te Evidence-Based Principles in the context of evidence building,
Policymaking Commission Act called for the Presi- the Commission has taken an ethical approach to
dent and the Congress to appoint one-third of the data use that addresses both privacy and the need
Commissioners specifcally for their expertise in to generate accurate and reliable evidence. Many
protecting personally identifable information and of the Commissions recommendations in other
data minimization.1 Te Commission devoted two chapters address transparency, individual partic-
public meetings to testimony about privacy and two ipation, purpose specifcation, data minimization,
other meetings included substantial discussions of use limitation, and accountability and auditing.
how state, Federal, and international governments Te recommendations in this chapter specifcally
address the principle of data quality and integri-
1. Evidence-Based Policymaking Commission Act of 2016 (Public ty and the principle of security, and also relate to
Law 114140, Section 3(a), March 30, 2016). other principles, especially transparency.
The Promise of Evidence-Based Policymaking 49

Fair Information Practice Principles1


Transparency: notify individuals regard- Use Limitation: use PII solely for the pur-
ing collection, use, dissemination, and pose(s) specifed
maintenance of personally identifable
information (PII)
Data Quality and Integrity: ensure that PII
is accurate, relevant, timely, and complete
Individual Participation: involve the
Security: protect PII through appropriate
individual in the process of using PII and, security safeguards
to the extent practicable, seek individual
consent for the collection, use, dissemina- Accountability and Auditing: be account-
tion, and maintenance of PII able for complying with these principles
and audit the actual use of PII to demon-
Purpose Specifcation: articulate the strate compliance
authority that permits the collection of PII
and specifcally articulate the purpose(s)
for which the PII is intended to be used 1. U.S. Department of Health, Education, and Welfare, Records,
Computers, and the Rights of Citizens, Report of the Secretarys
Data Minimization: collect PII that is Advisory Committee on Automated Personal Data Systems
(Washington, D.C., July 1973); https://www.justice.gov/opcl/docs/
directly relevant and necessary to accom-
rec-com-rights.pdf (accessed August 10, 2017); Te White House,
plish the specifed purpose(s) and only re- Appendix A. Fair Information Practice Principles, National
tain PII for as long as is necessary to fulfll Strategy for Trusted Entities in Cyberspace: Enhancing Online Choice
the specifed purpose(s) Efciency, Security and Privacy (2011); https://obamawhitehouse.
archives.gov/sites/default/fles/rss_viewer/NSTICstrategy_041511.
pdf (accessed August 10, 2017).

While privacy harms can come from many evidence means minimizing the risk of such pri-
sources, the Commissions recommendations fo- vacy harms.
cus on establishing laws, policies, and procedures Risk is a measure of the extent to which an
to minimize the risk that confdential data used entity is threatened by a potential circumstance
for evidence building may be misused to cause or event, and typically is a function of: (i) the ad-
privacy harm. Privacy scholar Ryan Calo notes verse impact, or magnitude of harm, that would
that privacy harms can be both subjective and ob- arise if the circumstance or event occurs; and
jective: Te subjective category of privacy harm is (ii) the likelihood of occurrence. 3 Risk can come
the perception of unwanted observation. Tis cat- from bad actors who actively attempt to break
egory describes unwelcome mental statesanxi- into confdential data that they know they are
ety, embarrassment, fearthat stem from the be- not authorized to use and from well-intentioned
lief that one is being watched or monitoredTe people who inadvertently violate the procedures
objective category of privacy harm is the unantic- established to protect privacy. Risk also can come
ipated or coerced use of information concerning from outdated data encryption protocols and un-
a person against that person. Tese are negative, even capacity to apply up-to-date data protection
external actions justifed by reference to personal techniques. An increasingly important source of
information.2 Privacy harm occurs when an indi- risk is the growing number of publicly available
vidual is adversely afected because of misuse of data sources containing personal information that
data; aspects of this harm can also afect business- can be combined with publicly available datasets
es or other entities if, for example, their confden- designed to be de-identifed, thus permitting the
tial data are inappropriately accessed and used.
Respecting privacy in the generation and use of
3. Ofce of Management and Budget, Managing Information as
a Strategic Resource, Circular A130 (updated July 27, 2016);
2. Ryan Calo, Te Boundaries of Privacy Harm, Indiana Law Jour- https://obamawhitehouse.archives.gov/sites/default/f iles/omb/
nal 86, no. 3 (2011); https://ssrn.com/abstract=1641487 (accessed assets/OMB/circulars/a130/a130revised.pdf (accessed August 10,
August 10, 2017). 2017).
50 The Promise of Evidence-Based Policymaking

connection of identities with previously private, are respectful of privacy.5 Not surprisingly, public
sensitive information. Tis risk must be managed. trust in the accuracy and validity of statistical data
This chapter describes a need to better protect refect the publics trust in the statistical agencies
individuals from privacy harms caused by unau- that produced them.6
thorized use or disclosure of Federal confdential Te laws governing PSAs give them the author-
data. The Commission developed specifc recom- ity to protect confdential data by allowing use of
mendations about the quality, integrity, and se- the data for exclusively statistical purposes. Sta-
curity of confdential data to prevent individuals tistical purposes include analyzing individual re-
and organizations from being identifed without cords along with many other individual records to
authorization. Private sector research has contrib- produce descriptions of groups of people or cre-
uted to rapid advances in computer science that ating averages about groups within society, the
make data safer and easier to access. Adoption of economy, or the environment. PSAs cannot and
these emerging technologies can accelerate se- do not use data records about individual people,
cure access to the governments confdential data businesses, or other entities for non-statistical
resources for evidence building. Federal depart- purposes, such as administering a program, deter-
ments need leaders with the skills and vision to mining benefts, or enforcing laws.
fully harness the promise of privacy-protective PSAs pioneered and continue to advance sta-
data stewardship. tistical disclosure limitation techniques, dis-
cussed below, with the purpose to ensure that the
What people care most about is risk of disclosing confdential information about
not simply restricting the fow of identifable persons, businesses or other units will
be very small.7 PSAs have a long history of pro-
information but ensuring that it viding secure access to the least amount of data
fows appropriately. 4 needed for approved statistical purposes and of
Helen Nissenbaum, Professor of ensuring that researchers abide by strict rules to
Information Science at Cornell Tech protect confdentiality.
Reducing barriers to accessing and using ad-
ministrative data for evidence building requires
Findings the same kind of attention to confdentiality that
is embedded in the culture and history of the PSAs.
Protecting privacy has been a priority for Federal Trough the establishment of disclosure review
evidence building for many decades. Many Feder- boards, PSAs apply best practice statistical disclo-
al departments use modern methods to keep data sure limitation techniques and assess the risk of
secure and confdential, but government practices public data releases. Disclosure review boards as-
must adapt to new threats and take advantage of sess proposed public data releases to ensure that
technologies that better protect data. enough identifying information has been removed
For decades, Principal Statistical Agencies or masked to protect confdentiality. All Federal
(PSAs) such as the U.S. Census Bureau have
demonstrated responsible stewardship of data
collected through censuses and surveys. Tese 5. Jennifer Hunter Childs, Understanding Trust in Official Sta-
tistics in the U.S. and Implications For Administrative Record
data are used to create important national statis- Use (prepared for the Commission, Washington, D.C., December
tical indicators, such as the unemployment rate, 2016).
gross domestic product, health outcome statis- 6. Melissa Mitchell, Jennifer Hunter Childs, and Morgan Earp,
tics, and crime statistics. Public trust in the Fed- Monitoring and Detecting Shocks That Influence Public Trust To-
eral Statistical System historically has been high, wards the Federal Statistical System (paper presented at the 68th
Annual Conference of the American Association of Public Opinion
with a majority of Americans concluding that gov- Research, Boston, MA, May 2013); http://www.aapor.org/
ernment can be trusted to produce statistics that AAPOR_Main/media/AnnualMeetingProceedings/2013/
Session_H-5-2- Mitchell.pdf (accessed August 10, 2017).
7. Federal Committee on Statistical Methodology (FCSM), Re-
port on Statistical Disclosure Limitation Methodology, Statisti-
4. Helen Nissenbaum, Privacy in Context: Technology, Policy, and cal Policy Working Paper 22 (Washington, D.C.: FCSM, December
the Integrity of Social Life (Stanford, CA: Stanford University Press, 2005); https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/
2010). 242/2014/04/spwp22.pdf (accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 51

departments should do the same. When govern- and Confdential Information Protection and Sta-
ment pledges to keep data confdential, the data tistical Efciency Act of 2002 (CIPSEA).10 Tese
should have strong protections, and data use should laws seek to protect confdentiality by requiring
generally be made known to the American public. that released records be in a form that is not indi-
Te Federal governments Open Data initiative vidually identifable11 (Privacy Act) or prevents
has made more government data publicly avail- the identity of the respondent[from being]
able than ever before. Even so, agencies must reasonably inferred by either direct or indirect
review the information collected or created for means12 (CIPSEA). Te Privacy Act and CIPSEA,
valid restrictions to release to determine whether as well as other Federal laws governing confden-
it can be made publicly available, consistent with tial data, share a basic goal: to ensure that infor-
the Open Government Directives presumption in mation given to the Federal government under a
favor of openness, and to the extent permitted by promise of confdentiality is not released to any-
law and subject to privacy, confdentiality pledge, one in identifable form, except as allowed by law.
security, trade secret, contractual, or other valid As described in Chapter 2, CIPSEA authorizes ac-
restrictions to release.8 cess to identifable confdential data for exclusive-
Troughout its fact-fnding phase, the Commis- ly statistical purposes by (1) trained staf in PSAs
sion learned about potential threats to the conf- and recognized statistical units and (2) by external
dentiality of Federal data used for evidence building researchers who are trained and qualifed by PSAs.
and heard from experts about new and emerging Even then, direct identifers are removed from
methods to address such threats. Te fndings pre- analysis fles and their access protected and limit-
sented in this chapter highlight three challenges ed to a few technical staf. Several Federal agencies
that keep the Federal governments confdentiality that collect confdential administrative data under
protections from being as strong as possible. First, the Privacy Act also provide secure access to identi-
laws governing Federal data sometimes do not re- fable information for research purposes.
quire a risk assessment or disclosure review prior PSAs release de-identifed confdential data
to the public release of de-identifed confdential collected under CIPSEA as part of their mission and
data. Second, Federal departments must adapt to legal authority to disseminate data in their topical
new threats to information security and privacy and domains. Many program agencies that collect con-
take advantage of emerging technologies that better fdential information under the Privacy Act also re-
protect data. Tird, many Federal departments lack lease de-identifed data under their own statutory
senior leadership focused on or experienced in data authorities. Tese public-use fles are easily acces-
stewardship for evidence building, and the PSAs, sible and widely available; many public-use fles
while often best positioned to provide such leader- can be downloaded from department websites.
ship, have uneven independence and authority. In the Federal government today, however, there
is great variation in the meaning of de-identifed
Challenge #1: Key laws governing Federal data. Following CIPSEA implementation guidance
data do not require the application of best from OMB, statistical agencies routinely go beyond
practice statistical disclosure limitation removing direct identifers to apply appropriate
techniques or a risk assessment prior to statistical disclosure limitation techniques before
the public release of de-identifed conf- seeking disclosure review board approval to re-
dential data. lease de-identifed datasets.13 In contrast, program

As discussed in Chapters 1 and 2, two laws that


govern much of the confdential Federal data used 10. Confdential Information Protection and Statistical Efciency
Act of 2002, (CIPSEA), (Public Law 107347, Title V, 2002).
for evidence building are the Privacy Act of 19749
11. Privacy Act, 5 USC 552a (b)(5) (1974).
12. Confdential Information Protection and Statistical Efciency
8. Ofce of Management and Budget (OMB), Open Data Policy: Act of 2002, (CIPSEA), (Public Law 107347, Section 502, 2002).
Managing Information as an Asset (memorandum M1313,
13. Implementation Guidance for Title V of the E-Government
Washington, D.C.: OMB, Executive Ofce of the President, Novem-
Act, Confdential Information Protection and Statistical Efcien-
ber 8, 2016); https://project-open-data.cio.gov/policy-memo (accessed
cy Act of 2002 (CIPSEA), Federal Register 72, no. 115 (June 15,
August 10, 2017).
2007): 33361; https://www.gpo.gov/fdsys/granule/FR-2007-06-15/
9. Privacy Act, 5 USC 552a et seq.(1974). E7-11542/content-detail.html (accessed August 10, 2017).
52 The Promise of Evidence-Based Policymaking

agencies routinely remove direct identifers follow- Federal departments that collect and store conf-
ing legal guidance from general counsels about im- dential data are increasingly concerned about the
plementing the Privacy Act, but only some apply risk of privacy harm that comes from data security
rigorous statistical disclosure limitation techniques breaches and unauthorized re-identifcation of in-
before releasing de-identifed data. According to dividual identities in confdential data. Te Federal
the Federal Committee on Statistical Methodolo- government has an obligation to develop strategies
gys Working Paper 22, agencies that need to pro- to assess and minimize the impact of these risks
tect data should move as far as possible toward the when it uses government data for evidence build-
use of a small number of standardized disclosure ing. Te Commissions witnesses indicated that
limitation methods whose efectiveness has been new technologies can be part of the solution.
demonstrated.14
Federal departments run the continuum of Preventing Unauthorized
capacity and expertise to apply state-of-the-art
Re-Identifcation
methods for creating de-identifed data that min-
imizes the risk of re-identifcationthat is, the Early laws on privacy in the United States gener-
chance that any individual in a publicly released ally described a relatively short and fnite list of
dataset could be uniquely identifed. Minimizing direct identifers that could easily be removed to
the risk of re-identifcation is important because create de-identifed data for broad distribution.
of the potential that a bad actor could inappropri- Accepted practice for many years meant simply re-
ately fgure out which record belongs to an indi- moving direct identifers such as names, address-
vidual and infict harm as a result. es, and dates of birth. By the late 1970s, however,
Statisticians, computer scientists, and the the Federal Committee on Statistical Methodology
recognized that de-identifcation of data required
far more than simply removing elements from a
14. FCSM, Report on Statistical Disclosure Limitation Methodology: 6. dataset and advocated the adoption of more formal

Statistical Disclosure Limitation Techniques from


Working Paper 22

To reduce the potential for disclosure 3. Adding or multiplying by random num-


public use data fles: bers (noise),
1. Include data from only a sample of the 4. Swapping or rank swapping (also called
population, switching),
2. Do not include obvious identifers, 5. Selecting records at random, blanking
out selected variables and imputing for
3. Limit geographic detail, and
them (also called blank and impute),
4. Limit the number and detailed break- and
down of categories within variables on
6. Aggregating across small groups of
the fle.
respondents and replacing one individ-
Additional methods typically used to uals reported value with the average
disguise high risk variables include: (also called blurring).1

1. Truncation of extreme codes for certain


variables (top or bottom-coding),
1. FCSM, Report on Statistical Disclosure Limitation Methodology:
2. Recoding into intervals or rounding, 2425.
The Promise of Evidence-Based Policymaking 53

statistical disclosure limitation methods to protect efort In general, de-identifcation


individuals privacy in public data releases.15 approaches are designed to push
Te potential danger of re-identifcation from data to the left [towards not being
indirect identifers and contextual information related to individuals] while retain-
in datasets stripped of direct identifers became ing some desired utility, lowering
more widely recognized in the 1990s. As a result, the risk of distributing de-identifed
PSAs increasingly began to rely on statistical dis- data to a broader population or the
closure limitation techniques to mask the data. general public.18
Te Federal Committee on Statistical Methodolo-
gys Working Paper 22, issued in 2005, stated: Ev- Current best practice statistical disclosure lim-
ery agency or unit within an agency that releases itation methods may become less efective in the
statistical data should be capable of selecting and future because of the growing availability of pub-
applying suitable disclosure limitation procedures lic and private information about individuals and
to all the data it releases.16 By the early 2000s, the computing techniques to exploit them. Adding to
prevailing way to implement strong de-identifca- currently available information, Federal depart-
tion protocols was to use statistical disclosure lim- ments are now directed to release as much data
itation techniques on a release-by-release basis to as possible to the public under the Open Data ini-
reduce the risk of re-identifcation that could po- tiative. However, advances in computer science
tentially cause privacy harm. and technology have produced promising new
Today, a variety of statistical disclosure lim- enhanced statistical disclosure limitation tech-
itation techniques, methods, and technologies niques that have the potential to provide stronger
are used to reduce the identifability of a dataset. confdentiality protections by minimizing the risk
One relatively new technique, k-anonymity, pro- of re-identifcation.
tects confdentiality by adjusting the data to make
sure that all combinations of potentially identi- Assessing the Risk of Re-Identifcation
fable information refect a minimum number of
individuals (k).17 No existing statistical disclosure Te Privacy Act and CIPSEA do not currently re-
limitation method, however, is able to complete- quire agencies to assess the risk of unauthorized
ly eliminate the risk of re-identifcation. As Sim- re-identifcation for de-identifed confdential
son Garfnkel, formerly of the National Institute data; however, the CIPSEA implementation guid-
of Standards and Technology, the Federal gov- ance includes the requirement that for CIPSEA
ernments standard-setting body for information protected information, the agency as well as any
technology, explains: agent accessing the information shall ensure that
any dissemination of information based on con-
all data exist on an identifability fdential information is done in a manner that
spectrum. At one endare data that preserves the confdentiality of the informa-
are not related to individuals (for tion.19 Today, each agency that releases de-iden-
example, historical weather records) tifed confdential data has to decide for itself how
and therefore pose no privacy risk. it will assess the risk of re-identifcation and the
At the other endare data that are resulting potential for harm to individuals from
linked directly to specifc individu- misuse of individual information.
als. Between these two endpoints One demonstrated approach for carrying out the
are data that can be linked with responsibility to assess risk is to establish a disclosure
review board. Most of the 13 PSAs and a few other
Federal agencies have established formal disclosure
15. FCSM, Report on Statistical Disclosure and Disclosure Avoid-
ance Techniques, Statistical Policy Working Paper 2 (Washington,
D.C.: FCSM, 1978); https://s3.amazonaws.com/sitesusa/wp-content/
18. Simson L. Garfnkel, De-Identifcation of Personal Informa-
uploads/sites/242/2014/04/spwp2.pdf (accessed August 10, 2017).
tion, National Institute of Standards and Technology (NIST) In-
16. FCSM, Report on Statistical Disclosure Limitation Methodology: 2. ternal Report 8053 (Gaithersburg, MD: NIST, October 2015): 5;
https://dx.doi.org/10.6028/NIST.IR.8053 (accessed August 10, 2017).
17. Latanya Sweeney, K-Anonymity: A Model for Protecting Pri-
vacy, International Journal on Uncertainty, Fuzziness and Knowl- 19. Implementation Guidance for Title V of the E-Government
edge-Based Systems 10, no. 5 (2002): 557570. Act, CIPSEA.
54 The Promise of Evidence-Based Policymaking

review boards that assess the risk of re-identifca- Data collected and released by state and local
tion in the agencies public data releases.20 Having governments, private companies, non-proft or-
a disclosure review board allows a department to ganizations, and researchers contribute to the cu-
assess the underlying risk of re-identifcation prior mulative amount of information available about
to public data release and assess whether the sta- individuals and businesses that could be used for
tistical disclosure limitation methods applied to the re-identifcation. Researchers recently demon-
data have efectively mitigated the risk. All PSAs strated how data on air and dust samples from 50
conduct disclosure review and typically document homes in two communities in California could be
the results in safe to release memos or disclosure combined with data released under the Safe Har-
review reports for public data releases. bor provisions of the Health Insurance Portabil-
Te establishment of a centralized disclosure ity and Accountability Act (HIPAA) to uniquely
review board within a department has added ben- and correctly identify [in one community] 8 of 32
efts, in that each data release can be considered (25 percent) by name and 9 of 32 (28 percent) by
within the context of all other planned data re- address.21
leases across the department (see the box Estab- Many programs have released de-identifed
lishing a Centralized Disclosure Review Board"). public-use data fles for decades without formally
In addition, a centralized disclosure review board assessing risk. However, the broader context with-
can leverage expertise from across the depart- in which these releases are occurring has changed
ment, bringing together staf that have technical as the amount of information about individuals
skills, such as mastery of statistical disclosure lim-
itation methods, and specialized program knowl-
edge about the contents of a particular dataset. 21. Latanya Sweeney, Ji Su Yon, Laura Perovich, Katherine E Bor-
onow, Phil Brown, and Julia Green Brody, Re-identifcation Risks
in HIPAA Safe Harbor Data: A Study of Data From One Environ-
20. FCSM, Report on Statistical Disclosure Limitation Methodology. mental Health Study, Technology Science (August 28, 2017).

Establishing a Centralized Disclosure Review Board


In 2013, the U.S. Department of Education es- essary to ensure that the department releases
tablished a centralized disclosure review board as much usable data as possible while protect-
(the ED-DRB) to review and approve privacy ing privacy. Te ED-DRB consists of staf with
protections for public releases of the depart- expertise in privacy law, policy priorities, dis-
ments administrative data. Prior to the ED- closure risk assessment, and the application
DRBs creation, each of the departments prin- of statistical disclosure limitation techniques,
cipal ofces was responsible for selecting and as well as subject matter experts from data-re-
applying privacy-protection methods for their leasing ofces in the department. Education
own data releases. Tis decentralized approach Department staf participate in the ED-DRB as
was problematic, as data releases by one ofce one part of their regular duties. Since its cre-
could afect the re-identifcation risk of related ation, the ED-DRB has been a model for other
data released by other ofces. Senior depart- agencies seeking to improve the consistency
ment leaders recognized that decentralization and efcacy of the disclosure review process for
of risk assessments resulted in substantial vari- administrative data releases.1
ation across ofces about the acceptable level
of re-identifcation risk in public data releases.
1. Federal CIO Council Innovation Committee, Te Data
Te Education Department chartered the Disclosure Decision: Use Case Study (Washington, D.C.: Gov-
ED-DRB to be the central point for review and ernment Printing Ofce, 2015); https://s3.amazonaws.com/
approval of privacy protections for the majority sitesusa/wp-content/uploads/sites/1151/downloads/2015/03/
The-Data-Disclosure-Decision-Department-of-Education-
of the departments administrative data releas- Case-Study_Mar-2015. pdf (accessed August 10, 2017).
es. Te board has the technical expertise nec-
The Promise of Evidence-Based Policymaking 55

that is publicly available has grown and the tech- record. In the absence of strict protocols for using
nology that can permit unauthorized re-identif- confdential data, an inexperienced researcher
cation has improved. Within the Federal govern- may attempt to take a screen shot of fndings that
ment alone, the Open Data initiative made over have not been through the required disclosure re-
150,000 datasets accessible through a single web- view or use other means to remove information
site, including many administrative datasets nev- not approved for release. Federal departments are
er before released to the public.22 While releasing required to have internal controls to prevent and
these data can generate tremendous value, en- address poor data handling practices and to im-
abling entrepreneurs to produce better products pose penalties for any procedural violations. Fed-
and departments to understand their work better, eral guidance, however, does not prescribe how
it is important that the risks that may accompany management designs, implements, and operates
data releases also be assessed.23 Te Commission an internal control system.25
learned that a robust approach to risk assessment, To keep data safe from misuse, the Federal gov-
one that considers the availability of external data ernment must keep data secure and ensure appro-
in deciding what is safe to release, is paramount to priate training, controls, and accountability. Data
the success of all future evidence building. security is the development and implementation
Agencies that link electronic records for inter- of management, operational, and technical as-
nal government administrative purposes related pects of protecting the confdentiality, integrity,
to fnancial benefts are required today to assess and availability of federal information and infor-
the risk of the linkage and develop procedures for mation systems.26 Many Federal agencies collect
safeguarding the data.24 Te system used for the identifable information in accordance with their
assessment is cumbersome and could be simpli- statutory missions and store those data on govern-
fed, and statistical activities are exempt. However, ment servers. A breach of a data system, or a mis-
the transparent accountability portion of the pro- use of data, resulting in the release of identifable
cess is instructive for evidence building. Agencies information could lead to an increased risk of indi-
must issue a Federal Register notice and post the viduals being re-identifed in a dataset released to
written agreement between the relevant agencies the public under CIPSEA or the Privacy Act.
on each agencys website. For example, the U.S. Data security in Federal departments today
Department of Education and the U.S. Department is generally the responsibility of Chief Infor-
of Veterans Afairs have a publicly available agree- mation Ofcers who implement standards pro-
ment to link administrative data to determine eli- mulgated by OMB in consultation with the Na-
gibility for veterans educational benefts. tional Institute of Standards and Technology. A
recent report from the Commission on Enhanc-
Challenge #2: Federal departments must ing National Cybersecurity sought to develop
adapt to new threats to information se- actionable recommendations for securing and
curity and privacy and take advantage of growing the digital economy by strengthening
emerging technologies that better protect cybersecurity in the public and private sectors.27
data. Cyberattacks on confdential data increase the
risk of privacy harm and may erode public trust.
Te risk of re-identifcation is not the only source
of potential privacy harm from personal informa- 25. Government Accountability Ofce (GAO), Standards for Inter-
tion collected by the Federal government. Staf nal Control in the Federal Government (Washington, D.C.: GAO, Sep-
with otherwise legitimate reasons to access per- tember 2014): 8; http://www.gao.gov/assets/670/665712.pdf.
sonal information may misuse such access, for ex- 26. Marianne Swanson, Joan Hash, and Pauline Bowen, Guide for
ample, by browsing a celebritys or acquaintances Developing Security Plans for Federal Information Systems, National
Institute of Standards and Technology (NIST) Special Publication
80018 (Gaithersburg, MD: NIST, February 2006): 13; http://
nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-18r1.
22. DATA.gov, Federal Agency Participation; https://www.data.
pdf (accessed August 10, 2017).
gov/metrics (accessed August 10, 2017).
27. Commission on Enhancing National Cybersecurity, Report on
23. DATA.gov, Federal Agency Participation.
Securing and Growing the Digital Economy (December 1, 2016):
24. Computer Matching and Privacy Protection Act of 1988 (Pub- i; https://www.nist.gov/sites/default/fles/documents/2016/12/02/
lic Law 100503); https://www.gpo.gov/fdsys/pkg/STATUTE-102/ cybersecurity-commission-report-fnal-post.pdf (accessed August 10,
pdf/STATUTE-102-Pg2507.pdf (accessed August 10, 2017). 2017).
56 The Promise of Evidence-Based Policymaking

Well-publicized cybersecurity incidents in the of re-identifcation, guard against other forms of


past have afected Federal systems and resulted privacy harm, and allow for increased use of data
in the unintentional release of millions of conf- to support evidence-based policymaking. In par-
dential records, raising concerns about vulner- ticular, privacy-protective technologies and proto-
abilities in the level of data security of Federal cols can expand access to data while potentially
systems. Nearly all Federal agencies must assess improving, rather than adversely afecting, pri-
and secure information systems under the Federal vacy. Because technology is always evolving, the
Information Security Management Act (FISMA).28 Commission does not endorse the adoption of any
PSAs and the three other statistical units autho- one specifc approach. Rather, the Commission
rized to protect data under CIPSEA have an explicit acknowledges the critical need for leadership and
requirement to ensure the physical security and partnerships that will advance and accelerate the
information systems security where data protect- adoption of technologies that improve authorized
ed under CIPSEA are accessed and stored.29 Te access and protect privacy.
National Institute of Standards and Technology is As the Commission learned, new approaches
currently collaborating with Federal departments hold the potential for enhancing privacy protec-
and the private sector to establish new engineer- tion. Examples include diferential privacy, Secure
ing standards and privacy risk assessments intend- Multiparty Computation, and synthetic data (see
ed to minimize security risk in Federal information the box Emerging Approaches Tat Enhance Pri-
technology systems.30 vacy Protections).
PSAs respond to data security mandates under
CIPSEA and individual authorizing statutes by re- Diferential privacy is a
quiring computer systems that store confdential mathematical guarantee that
data be kept completely separate from other sys-
tems. In practice, this means that PSAs store conf-
an individual data contributor
dential data used for evidence building on servers will not be afected, adversely or
that are not connected to other data systems with- otherwise, by allowing her data to
in their departments. Data can be extracted from be used in any study or analysis,
these servers only by authorized individuals for ex-
clusively statistical purposes such as to create the no matter what other studies,
PSAs own production statistics or to combine data datasets, or information sources,
across authorized agencies for approved purposes. areor will becomeavailable.
Te Federal evidence-building community Cynthia Dwork, Microsoft Research (former),
must keep pace with evolving technology to main- Commission Meeting, Washington, D.C.,
tain the security of data and the trust of the Ameri-
September 9, 2016
can public. While there are multiple approaches to
protecting data against misuse, the Commission
believes that technology ofers part of the solution Te Census Bureau has led eforts among
for enhancing privacy while allowing more use of PSAs to use enhanced statistical disclosure lim-
data to satisfy the growing demand for evidence itation tools now made possible by cutting-edge
to support policymaking. technologies. For example, the Census Bureau
New and evolving technologies have the po- is beginning to use synthetic data to create data
tential to improve data security, minimize risk tools that answer interesting policy questions
while using diferential privacy to measure the
risk of re-identifcation and keep it below a de-
28. E-Government Act of 2002 (Public Law 107347, Title III). fned threshold (see the box OnTeMap: Difer-
29. Implementation Guidance for Title V of the E-Government ential Privacy in Practice at the Census Bureau).
Act, CIPSEA. Adopting new privacy-protective technologies
30. Sean Brooks, Michael Garcia, Naomi Lefovitz, Suzanne Light- will pose implementation challenges even once
man, and Ellen Nadeau, An Introduction to Privacy Engineering and the methods have been sufciently demonstrat-
Risk Management in Federal Systems, NIST Internal Report 8062
(Gaithersburg, MD: NIST, January 2017); http://nvlpubs.nist.gov/
ed in practice. One substantial challenge, for ex-
nistpubs/ir/2017/NIST.IR.8062.pdf (accessed August 10, 2017). ample, is that many departments currently use
The Promise of Evidence-Based Policymaking 57

Emerging Approaches That Enhance Privacy Protections


Differential privacy is a formal privacy con- SMC in the real world.3 Te technique allows
cept that applies to analyses performed over one party to jointly compute a function with
collections of sensitive personal information.1 another party, while never revealing one an-
It provides a quantifable measure of the ex- others private data. SMC and its variations rely
cess privacy risk any individual may incur due on mathematical principles based on cryptog-
to their data being included in an analysis, as raphy to compute answers over databases from
compared with their data not being included. diferent locations and controlled by diferent
Because the application of diferential pri- organizations without either side needing to
vacy includes the mathematical computa- see individual records of the other. In short,
tion of the risk to privacy loss in any given SMC enables distributive computing that al-
analysis, parameters are established that set lows data owners to maintain their data in
bounds on what amount of privacy loss is ac- their own silos and yet compute results on the
ceptable. Tese parameters are then applied combined data.
to controlthe tradeof between protection of Synthetic Data: Verification Model: A
individual privacy on the one hand and accu- synthetic dataset mimics an original dataset,
racy of the performed analyses on the other. by replacing actual values from the original
Research over the last decade has provid- data with altered values that still retain many
ed solid theoretical grounds for diferential (but not necessarily all) important statistical
privacy. Algorithms realizing optimal priva- properties of the real dataset.4 Synthetic data-
cy-accuracy tradeofs are emerging for a vari- sets include more detailed data with lower
ety of statistical and machine learning tasks. disclosure risk because the original data are
In particular, these algorithms can enable pri- changed. But for the same reason, analysts may
vacy-preserving computations using data that be uncertain about the precision of the results.
more traditional techniques would need to re- Using a verifcation server is one way to ad-
dact. Notably, diferential privacy is currently dress this uncertainty. Te verifcation server
the only framework with mathematically prov- runs the analysis on the original data and then
en bounds on the accumulated privacy risk re- tells the analyst how diferent the results are
sulting from the composition of several analy- between the two datasets. If the analyst feels
ses. Te frst real-world uses now exist (by the that the result using synthetic data is good
U.S. Census Bureau and Internet companies enough, then they may choose to work solely
such as Apple, Google, and Uber). with the synthetic data; but if the analyst feels
Secure Multiparty Computation, or SMC, that the result using synthetic data is not good
is a method for conducting privacy-preserv- enough, they may choose to pursue access to
ing data analysis.2 Te concept of SMC has the original dataset.
been in existence for over 40 years. However,
recent advances in algorithms and computa-
tional power have made it feasible to achieve
3. Dan Bogdanov, Liina Kamm, Baldur Kubo, Reimo Rebane,
Ville Sokk, and Riivo Talviste, Students and Taxes: a Priva-
cy-Preserving Study Using Secure Computation, Proceedings
1. National Academies of Sciences, Engineering, and Medi- on Privacy Enhancing Technologies no. 3 (July 2016); https://
cine. Innovations in Federal Statistics: Combining Data Sourc- doi:10.1515/popets-2016-0019 (accessed August 10, 2017).
es While Protecting Privacy (Washington, D.C.: National Acad-
4. National Institutes of Health Ofce of Behavioral and Social
emies Press, 2017).
Science Research, Synthetic Data: Protecting Data Privacy in
2. Yehuda Lindell and Benny Pinkas, Secure Multiparty Com- an Era of Big Data (October 15, 2015); https://obssr.od.nih.gov/
putation for Privacy-Preserving Data Mining, in Encyclopedia synthetic-data-protecting-data-privacy-in-an-era-of-big-data
of Data Warehousing and Mining. (accessed August 10, 2017).
58 The Promise of Evidence-Based Policymaking

techniques that assess and seek to minimize the technologies and techniques to provide secure ac-
risk of re-identifcation of a particular data release cess to data, the government must consider the
as a stand-alone matter. In contrast, full imple- need for reproducibility.
mentation of diferential privacy requires deter-
mining the cumulative re-identifcation risk of Challenge #3: Many Federal departments
data releases over the entire life-span of the data.
lack senior leadership focused on data
Te widespread adoption of techniques such as
diferential privacy would require departments to stewardship, and PSAs have uneven inde-
consider and manage the myriad planned and un- pendence and authority to protect conf-
planned future uses of data. dential data used for evidence building.
Even with these limitations, the promise of
provable privacy methods such as diferential Te Commission found that evidence-based pol-
privacy is that the level of risk of re-identifcation icymaking requires leaders who recognize the
can be quantifed, thus allowing any dataset to be value of generating more and better evidence as
used for evidence building while minimizing pri- a mechanism for improving government services
vacy risk. and holding government accountable. Feder-
Another implementation challenge is ensuring al leadership that acknowledges and values the
that enhanced statistical disclosure methods do importance of evidence building must support
not change the data in ways that increase the dif- the use of Federal data assets for statistical pur-
fculty of reproducing research results. Te Com- poses. Without sustained leadership and support
missions guiding principle on humility points within Federal departments for data stewardship,
to the need for multiple researchers to replicate opportunities to improve and appropriately lever-
results to verify the credibility of evidence. Fur- age data resources for evidence building will be
ther, professional journals sometimes require missed.
researchers to demonstrate that analyses can be Meaningful collaboration within departments
reproduced prior to publication.31 In adopting new is essential if administrative data are to be har-
nessed as an integral part of evidence building.
31. For example, the American Journal of Political Sciences For example, the Commission to Eliminate Child
Replication and Verif ication Policy; https://ajps.org/ajps- Abuse and Neglect Fatalities concluded in 2016
replication-policy (accessed August 10, 2017). Additionally, the that the inability to coordinate data collection and
American Economic Review only publishes papers if the data used in
the analysis are documented and available for replication; https://
analysis activities across child protection programs
www.aeaweb.org/journals/policies/data-availability-policy (accessed posed a real risk to childrens lives. One of the
August 10, 2017).

OnTheMap: Diferential Privacy in Practice at the


Census Bureau
OnTheMap is an online mapping and report- which is becoming popular in the statistical
ing application that shows areas where peo- disclosure limitation community.1 Without
ple work and where workers live. OnTeMap the use of privacy-preserving technologies,
was developed through a partnership between OnTeMap would not be able to provide the
the Census Bureau and its Local Employment detail it does.
Dynamics partner states. OnTeMap is an ex-
ample of diferential privacy in practice. In
Privacy: Teory meets Practice on the Map, 1. Ashwin Machanavajjhala, Daniel Kifer, John Abowd,
researchers describe how technology was used Johannes Gehrke, and Lars Vilhuber, Privacy: Teory Meets
to create OnTeMap. Te algorithm used to Practice on the Map (paper presented at the IEEE 24th
International Conference on Data Engineering, Cancun,
anonymize the data for the mapping applica- Mexico, April 2008); http://www.cse.psu.edu/~duk17/papers/
tion is known as the synthetic data generation, PrivacyOnTeMap.pdf (accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 59

Commissions recommendations was to improve evidence about programs and policies.


and support data collection about child abuse and Te Commission found that PSAs are well posi-
neglect fatalities ofchildren, and integrate the tioned to become department leaders to facilitate
data into national databases for analysis, research, the transformation and analysis of administrative
and the development of efective prevention strat- data for evidence building. Currently, however,
egies. 32 Coordinating evidence-building activities PSAs vary in readiness to take on greater respon-
and supporting use of administrative data across sibility for supporting the larger evidence-build-
departments leads to useable inventories, data re- ing community. Some PSAs have strong statutes
sources, and analytic capabilities for responding that ensure their independence, place them high
to specifc policy and research questions. in department hierarchies, and authorize them
All Federal departments have a Chief Informa- to protect their confdential data. Other PSAs as-
tion Ofcer (CIO) and a Senior Agency Ofcial for sert their need for independence and authority
Privacy (sometimes called a Chief Privacy Of- through policy and practice but may lack strong
cer). According to OMB Circular A-130, CIOs play legal protections.
an important role in the management of Federal Lack of independence can be a challenge for
information resources. In addition to advising some PSAs. Te Federal Information Technology
department heads on the design, development, Acquisition Act (FITARA) of 2014 led department
and implementation of information resources, information technology leadership to consolidate
CIOs help prioritize department expenditures on Federal data centers and coordinate information
information resources.33 Te Senior Agency Of- technology acquisitions to promote consisten-
cial for Privacy has responsibility for implemen- cy and efciency.35 Tese well-intentioned eforts
tation of privacy protections; compliance with could hamper the ability of the evidence-building
Federal laws, regulations, and policies relating to community to limit access to confdential data. As
privacy; [and] management of privacy risks at the Katherine (Kitty) Smith of the Council of Profes-
agency.34 Other senior leaders with infuence on sional Associations on Federal Statistics (COPAFS)
department information resources include Chief noted in comments to OMB about the roles and
Financial Ofcers and Chief Performance Ofcers. responsibilities of Federal statistical agencies, in-
Te Commission also recommends that depart- formation technology systems that are out of the
ments identify a senior ofcial to serve as Chief direct control of the statistical agency can: result in
Evaluation Ofcer (Recommendation 5-1). delays in the retrieval of and dissemination of sta-
While each of these ofcials has a role to play tistical data; impose restrictions on the accompani-
in data stewardship, each also has an expansive ment of transparent explanations of methodology
portfolio of responsibilities that prevents their with the data; violate the integrity of statistical in-
prioritizing data management and secure access formation; and, very importantly, endanger the sta-
to data resources for evidence-building activities. tistical agencies ability to follow through on their
Te Commission learned that none of these senior pledges of confdentiality and non-disclosure.36
leaders is explicitly charged with managing the In public comment, witnesses expressed con-
portfolio of department data resources in support cern about independence for PSAs and recommend-
of Federal evidence building. Because senior lead- ed that PSAs should have sufcient legal authority
ers with a focus on data stewardship for evidence implemented or restored. Margaret Levenstein of
building are not at the table when priorities are ICPSR, a consortium of more than 750 academic
discussed and resources allocated, departments institutions and research organizations, said that
cannot take full advantage of Federal data to build undermining [the publics trust in Federal statisti-
cal agencies] undermines statistical measurement
32. Commission to Eliminate Child Abuse and Neglect Fatalities,
Within Our Reach: A National Strategy To Eliminate Child Abuse and
35. Federal Information Technology Acquisition Reform Act, 40
Neglect Fatalities (Washington, D.C.: Government Printing Ofce,
USC 11319(b)(1)(A).
2016): 14.
36. Katherine R. Smith (COPAFS) to Katherine K. Wallman,
33. U.S. Ofce of Management and Budget (OMB), Managing In-
Comment on OMB Statistical Policy Directive, June 18, 2014;
formation as a Strategic Resource: 14.
http://www.copafs.org/UserFiles/fle/handouts/CommentonOMBSta-
34. OMB, Managing Information as a Strategic Resource: 36. tisticalDirective%20.pdf (accessed August 10, 2017).
60 The Promise of Evidence-Based Policymaking

as well as the efectiveness of the programs upon elements of Statistical Policy Directive #1 would
which statistics are based.37 Clyde Tucker of the help PSAs further protect confdential data used
American Statistical Association cited a 2015 letter in evidence building.
from 20 former PSA heads that said the indepen-
dence of a federal statistical agency is a critical ele- Recommendations
ment in an agency producing objective and credible
statistical data.such autonomy should include Achieving the Commissions vision for enhancing
control over an agencys planning, budget, press re- privacy while also making better use of data for
leases, and information technology.38 evidence building will require some signifcant
Several commenters asked the Commission to changes. Departments will need to conduct risk
reinforce the importance of PSAs and the laws and assessments for public releases of de-identifed
policies that undergird them. Te publics willing- confdential data and to adopt cutting-edge tech-
ness to trust PSAs to protect confdential informa- nology for data security, integrity, and confden-
tion must be maintained and strengthened. tiality. Tis vision also calls for data stewardship
OMB Statistical Policy Directive #1 encourag- coordinated among senior leaders within Federal
es Federal departments to recognize the need for departments and the independence of PSAs to
integrity and objectivity in Federal statistics (see protect confdential data. Te Commission con-
box, Four Responsibilities of Statistical Agencies cluded that the Federal government must improve
from OMB Statistical Policy Directive #1). Poli- data quality, integrity, and security by updating laws
cy Directives do not have the same force as stat- to require risk assessments for data releases, embrac-
utes, however, so legislation to codify certain key ing cutting-edge technologies, and prioritizing data
stewardship and the public trust.
37. Margaret Levenstein, ICPSR, submission to the Commissions
Te recommendations below support the
Request for Comments. Commissions vision by allowing secure access to
38. Clyde Tucker, American Statistical Association, Commission confdential data in a privacy-protective manner,
Public Hearing, Washington, D.C., October 21, 2016. enabling the accountable production of evidence.

Four Responsibilities of Statistical Agencies from OMB


Statistical Policy Directive #1 1

1. Relevance requires communication tical units must function in an environment


across and within departments with plan- that is clearly separate and autonomous
ning information collection and dissemina- from the other administrative, regulatory,
tion activities and that statistical agen- law enforcement, or policymaking activities
cies collect program and policy-relevant within their respective Departments.
information from administrative records 4. Te trust of information providers requires
collected and maintained by the agency, or agencies to ensure confdentiality and
other government agencies. exclusive statistical use of their data.
2. Credibility requires that agencies apply
sound statistical methods to ensure statis-
tical products are accurate.
3. Objectivity requires that agencies pro- 1. OMB, Statistical Policy Directive No. 1: Fundamental
duce data that are impartial, clear, and com- Responsibilities of Federal Statistical Agencies and Recog-
nized Statistical Units, Federal Register 79 (December 2,
plete and are readily perceived as such by 2014): 71610; https://www.gpo.gov/fdsys/pkg/FR-2014-12-02/
the public and further requires that statis- pdf/2014-28326.pdf (accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 61

When taken together with the privacy-protective Figure 9. Improving Confidentiality


data access and use recommendations proposed Protections During Public Release of
in Chapters 2 and 4, these recommendations seek De-Identified Data
to safely harness the potential of data already col-
lected by government to produce more and better
evidence in the future. Data Acquired Under
Pledge of Confidentiality
REC. 3-1: The Congress and the Presi-
dent should amend the Privacy Act
and the Confidential Information Protec- Statistical Disclosure Limitation
Techniques Applied
tion and Statistical Efficiency Act (CIPSEA)
to require Federal departments to conduct
a comprehensive risk assessment on Formal Comprehensive
de-identified confidential data intended Risk Assessment Conducted
for public release. De-identified confiden-
tial data subject to the Privacy Act and CIP-
Data Release Approved
SEA should be made available only after a
disclosure review board (1) approves the
release and (2) publicly provides the risk Risk Assessment Posted to
assessment and a description of steps tak- Transparency Portal
en to mitigate risk.
Public Release Cleared
The Commission concludes that existing laws
such as the Privacy Act and CIPSEA should require
a comprehensive risk assessment and disclo- De-Identified Confidential
sure review board approval before public release Data Released to Public
of de-identified confidential data. With a small
change in how government approaches the man-
agement of confidential data, public data releases
can be made much more secure. Many PSAs have of all known sources of risk (see Figure 9). Depart-
the expertise to do this now and are a valuable ments should consider the risk of unauthorized
resource for program agencies that want to make re-identification in the context of other publicly
their data releases more secure. available data. As a starting point, departments
Establishing a requirement in law for risk as- should prioritize risk assessments for de-identified
sessment provides a basis for the entire Execu- confidential datasets planned for public release
tive Branch to more seriously and deliberately and expand risk assessment requirements over
approach risk management for publicly releasing time to include all public data releases.
data. OMB should collaborate with Federal depart- Departments may choose to go through exist-
ments and other stakeholders to establish imple- ing disclosure review boards, such as those in the
mentation guidance that requires a risk assess- PSAs, for approval to release de-identified con-
ment and approval by a disclosure review board fidential data under the Privacy Act or CIPSEA.
of any de-identified data released under the Pri- Other departments may choose to establish a sep-
vacy Act or CIPSEA. The Congress should consider arate disclosure review board for this purpose or
whether other statutes governing privacy for Fed- establish an entirely new disclosure review board
eral confidential data should be subject to a simi- if no such centralized function currently exists. In
lar requirement. addition, the National Secure Data Service (NSDS)
Under the approach envisioned by the Commis- will establish a disclosure review board to provide
sion, Federal departments will conduct compre- review services, extending the capacity of Federal
hensive risk assessments that include an analysis departments.
62 The Promise of Evidence-Based Policymaking

Departments should be required to public- Federal governments lead on the implementa-


ly post risk assessments as well as the general tion and application of cutting-edge data manage-
steps taken to mitigate risk from public releases ment technologies. OMB should review publicly
of de-identifed confdential data covered by the available risk assessments to determine if Federal
Privacy Act and CIPSEA. Risk assessments should departments are adopting the most advanced dis-
be posted to the NSDS transparency portal and in- closure limitation methods, techniques, and tech-
clude a mechanism for the public to report infor- nologies. OMB should set standards that require all
mation about additional sources of risk (see Rec- agencies to adopt new methods as they are shown
ommendation 4-3). Either OMB or a department to be feasible in the Federal context. Te National
could request the initiation of a new risk assess- Institute of Standards and Technology should con-
ment when new information comes to light. tinue its eforts to ensure security protection for
Transparency about risk assessments means data exchange, transmission, and storage.
that the public will know what the government Te evidence-building community should be
considered in its assessment of risk when it as- at the forefront of applying cutting-edge tech-
sesses risk and why the government now believes nologies to improve data security and privacy.
the data can be released as de-identifed. Public A National Academies of Sciences, Engineering,
risk assessments also make it easy for agencies to and Medicine report recently recommended that
learn from one another about the implementation Federal statistical agencies should adopt modern
of enhanced statistical disclosure limitation tech- database, cryptography, privacy-preserving, and
niques. privacy-enhancing technologies.39 Te Commis-
sions Recommendation 3-2 echoes and extends
REC. 3-2: Te President should direct the reports recommendation. Standards govern-
Federal departments, in coordination ing data security for all Federal agencies should
with the National Secure Data Service, to be adapted and strengthened as new technologies
emerge that provide greater privacy protections.
adopt state-of-the-art database, cryptog-
Leaders within Federal departments should ac-
raphy, privacy-preserving, and privacy-en- knowledge and adhere to special data stewardship
hancing technologies for confdential data requirements for confdential data used for evi-
used for evidence building. dence building.

To implement this recommendation, the Com- REC. 3-3: Te President should direct
mission proposes that the NSDS should have the Federal departments to assign a
initial and generative role in this enhancement of senior ofcial the responsibility for coor-
privacy protections. Te NSDS should lead the im-
dinating access to and stewardship of the
plementation and application of new privacy-re-
lated technologies in the Federal evidence-build- departments data resources for evidence
ing community. OMB should consult with the building in collaboration with senior
National Institute of Standards and Technology department information technology, pri-
on the most appropriate privacy-protective data vacy, and other leaders. A Principal Statis-
exchange standard for use across Federal depart- tical Agency head, or other appropriately
ments and Federally funded projects in support of qualifed senior ofcial, should serve this
evidence-building, consistent with or useful for function.
operational or administrative data exchange stan-
dards. Finally, this recommendation requires the Each department should assign to a senior of-
active participation of all Federal departments, as cial the duties of Senior Agency Ofcial for Data
they will need to periodically review their compli- Policy and the responsibility for coordinating the
ance with the National Institute of Standards and
Technology standards for the security of confden-
tial data in transmission and storage. 39. National Academies of Sciences, Engineering, and Medicine.
Innovations in Federal Statistics: Combining Data Sources While
Te proposal to establish the NSDS in statute Protecting Privacy (Washington, D.C.: National Academies Press,
should include authority for the NSDS to be the 2017): 93.
The Promise of Evidence-Based Policymaking 63

departments statistical needs and information serve as the departments liaison to the NSDS and
policy in support of evidence building. advise the NSDS, other Federal departments, and
Te person assigned this responsibility would OMB about the best way to use existing depart-
need to have the technical and substantive exper- ment data benefcially for exclusively statistical
tise to be efective in promoting secure evidence purposes.
building within the department. Because of the In many cases, departments would not have
imperative to protect confdential data, the desig- to create a new position or hire a new person to
nated senior leader would need extensive knowl- fll the role. Because of the expertise required, in
edge of data confdentiality laws, policies, and most cases the head of a PSA should fll the po-
practices including experience with applying sta- sition. In other cases, or in departments that do
tistical disclosure limitation techniques in public not have a PSA, a senior ofcial with expertise
releases of data. Other necessary technical skills in evaluation or policy-relevant research may be
would include knowledge of strategies to collect qualifed to provide such leadership. Regardless,
and manage data used for evidence building, data collaborative leadership around the coordination
analysis methods appropriate to data used for ev- and management of data resources within a de-
idence building, and emerging strategies to pro- partment is important for the efcient production
vide secure access to confdential data in a manner of evidence.
useful for analysis. In addition, this senior ofcial
would need a connection to and understanding REC. 3-4: Te Congress and the Presi-
of the broader evidence-building community, in- dent should enact legislation to codi-
cluding a deep appreciation for the importance fy relevant portions of Ofce of Manage-
of keeping Federal data authorized for evidence
ment and Budget Statistical Policy Directive
building on the exclusively statistical purposes
side of the functional wall of separation described #1 to protect public trust by ensuring that
in Chapter 1. data acquired under a pledge of confdenti-
Te Commission believes that this function is ality are kept confdential and used exclu-
essential as the Federal government increasingly sively for statistical purposes.
uses its existing data resources to generate valid
evidence. Today, leadership for evidence building Te Congress and the President should enact
is fragmented in or absent from many depart- legislation to codify those elements of Statisti-
ments. Te senior data policy ofcial would be cal Policy Directive #1 related to supporting the
charged to anticipate that all of the departments quality and objectivity of Federal statistical in-
data may be assets for evidence building. Tey also formation.40 OMB issued the directive in 2014
would be charged to ensure that legal, policy, and to afrm the long-acknowledged, fundamental
technical data stewardship requirements are met responsibilities of Federal statistical agencies and
when using data for evidence building. recognized statistical units in the design, collec-
Senior Agency Ofcials for Data Policy would tion, processing, editing, compilation, storage,
have both inward- and outward-facing roles. analysis, release, and dissemination of statistical
Within their own departments, the senior data data and information. Te Commission recom-
policy ofcials would: (1) collaborate with oth- mends that the requirements embodied in the
er senior department leaders in information directive be embedded in law. Te Congress and
technology, fnance, privacy, and evaluation to the President should provide each PSA with the
prioritize the use of data for evidence building; authority and institutional support necessary for
(2) ensure that program and statistical agencies ensuring confdentiality and maintaining the in-
determine the sensitivity of data resources and tegrity and objectivity of Federal statistics.
establish appropriate access controls (see Rec- Recommendation 3-4 further strengthens
ommendation 2-8); and (3) promote the use of statistical institutions by placing in law the ex-
enhanced statistical disclosure limitations tech- pectations of independence outlined in OMBs
niques to protect confdentiality and expand ac-
cess for evidence building. In their external co-
ordination role, senior data policy ofcials would 40. OMB, Statistical Policy Directive No. 1.
64 The Promise of Evidence-Based Policymaking

Statistical Policy Directive #1. Te placement employees established in Title 5 of the U.S. Code.41
of these responsibilities explicitly in statute Placing the directive in law also provides PSAs a
strengthens the governments commitment to stronger basis from which to defend their need for
the policies, makes the principles more difcult independent information technology resources
to change, and increases the likelihood that vi- and tailored procedures to secure the confdenti-
olations of the responsibilities would be report- ality of Federal data used for evidence building.
ed under whistleblower protections for Federal
41. Prohibited Personnel Practices, 5 USC 2302.
The Promise of Evidence-Based Policymaking 65

4
Modernizing Americas Data Infrastructure
for Accountability and Privacy

Empower government to develop state-of-the-art capacity to securely combine


existing data and provide secure data access for exclusively statistical purposes
in a privacy-protective and transparent way.

Recommendations
4-1: Te National Secure Data Service the NSDS should maintain a searchable in-
(NSDS) should be established as a sepa- ventory of approved projects using confden-
rate entity in the Department of Commerce tial data and undergo regular auditing of
that builds upon and enhances existing ex- compliance with rules governing privacy,
pertise and infrastructure in the Federal gov- confdentiality, and access.
ernment, especially at the Census Bureau, to
ensure sufcient capacity in secure record 4-4: Te NSDS should have specifc ad-
linkage and data access for evidence build- ministrative and implementation fexi-
ing. bilities including the ability to leverage pub-
lic-private partnerships and to collect and
4-2: Te NSDS should establish a Steer- retain user fees.
ing Committee that includes represen-
tatives of the public, Federal departments, 4-5: Te Ofce of Management and
state agencies, and academia. Budget should increase eforts to make
information available on existing Federal
4-3: To ensure exemplary transparency datasets including data inventories, metada-
and accountability for the Federal gov- ta, and data documentation in a searchable
ernments use of data for evidence building, format.
66 The Promise of Evidence-Based Policymaking

T he Federal evidence-building community


needs a modern institutional infrastructure
that facilitates evidence building and incorporates
A developer and implementer of new and
secure approaches to data access and analysis.
Te NSDS will be designated as a Principal Sta-
strong privacy protections. Today, some elements tistical Agency (PSA) with an exclusively sta-
of that infrastructure exist within individual si- tistical mission, with the authority to protect
los, focused on particular agency topics or tasks. data under the Confdential Information Pro-
Achieving the Commissions vision of routine, se- tection and Statistical Efciency Act (CIPSEA)
cure evidence production for the public good re- and the resources and capacity to implement
quires a new infrastructure aiding Federal depart- emerging privacy-protective technologies.
ments and outside researchers by improving the
amount and quality of evidence while enhancing Te Congress and the President specifcally
privacy. charged the Commission to consider whether the
While the risk assessment described in Chapter country needed a data clearinghouse.1 While the
3 (Recommendation 3-1) was designed for pub- term was not defned in the Commissions statute,
lic releases of de-identifed confdential data, the clearinghouse evokes images of past proposals
recommendations in this chapter seek to promote to consolidate large amounts of confdential data
evidence generation possible only by securely har- into a central location. For example, in the 1960s,
nessing the value of detailed data that cannot be the Kaysen Committee proposed creating a na-
made publicly available, and only for exclusively tional Data Bank to address the decentralized na-
statistical purposes. ture of the Federal Statistical System and the Fed-
Tis chapter expands on Recommendation eral governments data infrastructure. Te Kaysen
2-1 to establish the National Secure Data Service Committee determined that decentralization
(NSDS) by describing several of the implementa- prohibited efective use of socioeconomic data by
tion steps necessary to make the envisioned ser- researchers both inside and outside government,
vice possible. Te NSDS described in Chapter 2 is and recommended the creation of a large ware-
intended to be: house of data about the American public.2 While
the Commission on Evidence-Based Policymaking
A service for qualifed researchers seeking came to a similar conclusion about the value of
approval to securely analyze confdential data, it concluded that setting up a data warehouse
government data for exclusively statistical of the sort envisioned by the Kaysen Committee
purposes. Te NSDS will apply data minimi- would create an attractive target for misuse of
zation techniques to protect combined data, private data. Te Commission on Evidence-Based
require researchers to complete training, and Policymaking opposes the creation of any such
subject researchers to stringent monitoring for clearinghouse and proposes an entirely difer-
adherence to privacy-protective protocols. ent solution for addressing the challenges of data
access. (See the box How is the National Secure
A center for unparalleled transparency about Data Service Diferent from the Data Bank?.)
government uses of confdential data for Technology and privacy-protective approaches
exclusively statistical purposes. Te NSDS will have advanced greatly over the past 50 years since
make information about data sensitivity, risk the Data Bank proposal was rejected, and they
assessments for public release of de-iden- will continue to evolve. New approaches to secure
tifed confdential data, and individual evi- access to data for evidence building feature stron-
dence-building projects available in ways that ger privacy protections than at any point in histo-
exceed previous transparency and accountabil- ry. Te Commissions approach to implementing
ity for evidence building in the United States.

A developer and implementer of state-of-the- 1. Evidence-Based Policymaking Commission Act of 2016 (Public
art methods to safely combine confdential Law 114140, Section 4(b), March 30, 2016).
data. Te NSDS will temporarily and securely 2. Rebecca S. Kraus, Statistical Dj Vu: Te National Data Cen-
combine data and create analysis fles with the ter Proposal of 1965 and Its Descendants (paper presented at
the Joint Statistical Meetings, Miami Beach, FL, August 1, 2011);
minimum amount of data needed for approved https://www.census.gov/history/pdf/kraus-natdatacenter.pdf (ac-
projects with exclusively statistical purposes. cessed August 10, 2017).
The Promise of Evidence-Based Policymaking 67

How is the National Secure Data Service Diferent from the


Data Bank?
While both the 1960s Data Bank and the Na- remove personal identifers before the data can
tional Secure Data Service (NSDS) proposals be analyzed for exclusively statistical purposes.
seek better use of government data for statis- Te NSDS takes privacy to a new level by apply-
tical activities, that is where their similarity ing state-of-the-art protections and ensuring
ends. Te Data Bank was designed as a massive that, at every step, the American public is noti-
warehouse of constantly cumulating govern- fed about the uses of data to hold government
ment-held data about the American public that accountable on a project-by-project basis. Te
could have been used for diverse purposes. Te NSDS proposal requires a privacy audit for pub-
NSDS instead is a service that brings together lic transparency and accountability.
as little data as possible for as little time as pos- In short, the Commission does not recom-
sible for exclusively statistical purposes. Te mend a single clearinghouse that collects
NSDS is not intended to store data and users of massive stores of data to be warehoused in-
the service would not be permitted to conduct defnitely, as was envisioned in the 1960s Data
analyses for non-statistical purposes under the Bank proposal. In fact, the Commission con-
Commissions proposed legal and operational cluded that a single national clearinghouse
framework. with all types of data is simply not necessary,
While the Data Bank proposal would have nor does it represent best practices in privacy
stored individual-level data, the proposed protection.
NSDS is designed to temporarily link data and

Key Similarities and Diferences Between the Data Bank and the NSDS

1960s
2017
Proposal Feature Data
NSDS
Bank

Limits data uses to exclusively statistical purposes

Allows public participation in setting policies and priorities

Intended to improve researcher access to confdential data


Restricts access to data with personally identifable
information (PII) and keeps access temporary
Approves projects individually

Includes an appeals process for project approval

Includes a publicly available inventory of approved projects

Implements state-of-the-art privacy protection techniques

Makes transparent all data access for statistical purposes

Includes an auditing function


68 The Promise of Evidence-Based Policymaking

Recommendation 2-1, elaborated upon in this cies abide by legal requirements for transparency
chapter, aims to take full advantage of the oppor- about data systems, but have made only limited
tunity created by these new approaches. attempts at public transparency about individu-
However, the Federal government currently al projects using confdential data for exclusive-
lacks the complete infrastructure needed to rou- ly statistical purposes. Finally, the lack of a com-
tinely, efciently, and securely generate evidence prehensive inventory of data useful for evidence
about government policies and programs using building, lack of complete and accurate technical
confdential data. Te infrastructure that does documentation for many government datasets,
exist is not designed to meet government-wide and inconsistent defnitions of key concepts
needs; does not have the capacity to meet expand- across datasets inhibits routine evidence building.
ed demand from inside and outside government;
is uneven in its application of best practice tech- Challenge #1: Existing Federal government
nologies and techniques for secure data access,
privacy-protective data linkage, access,
linkage, and analysis; and does not always priori-
tize public transparency and accountability. and analysis capabilities do not meet gov-
Tis chapter includes recommendations that ernment-wide needs for evidence.
elaborate on aspects of NSDSs implementation
and build on a vision of the NSDS as a source of Te Commission received testimony from Federal
important shared services to improve govern- agencies using state-of-the-art methods to pro-
ment-wide data infrastructure for evidence build- vide privacy-protective access to securely linked
ing. Te Commission envisions that NSDS will confdential data for exclusively statistical pur-
help government increase the appropriate use poses. Within the Federal government, pockets of
of data for exclusively statistical purposes, while excellence can be found for most of the statistical
keeping confdential data safe and secure. NSDS activities needed to routinely generate evidence.
will have the capacity and capability to support Some Federal departments are using state-of-
secure analysis of single or combined confden- the-art methods for secure data linkage, access,
tial, restricted-use datasets while minimizing the and analysis in response to their specifc statuto-
risks of privacy harm from evidence-building ac- ry missions. Tese departments have expert staf
tivities. Given the breadth of the existing gaps in with experience implementing privacy-protective
infrastructure, the Commission recommends the techniques in a Federal context. Tis expertise
establishment of NSDS to address multiple capac- constitutes a valuable resource for the Federal
ity needs simultaneously to usher in a new era of evidence-building community, but it is currently
evidence building in the United States. concentrated in diferent agency silos and focused
on specifc agency purposes. Tere are pockets of
Findings excellence; there is no center of excellence capa-
ble of serving government-wide needs for priva-
Te Commission learned from witnesses and sub- cy-protective data linkage, access, and analysis for
mitted statements about gaps and unevenness in evidence building.
infrastructure supporting evidence building. Te A great deal of expertise relevant to Federal
fndings presented in this chapter highlight four evidence building resides in the U.S. Department
challenges to routine and secure evidence gen- of Commerce. Te U.S. Census Bureau has built
eration to improve government policies and pro- the capacity to securely link demographic data
grams. Tese fndings relate specifcally to the use for exclusively statistical purposes in support of
of confdential government data for exclusively Census Bureau operations and demographic and
statistical purposes. First, the Federal government socioeconomic research (see the box Center for
currently has pockets of expertise in privacy-pro- Administrative Records Research and Applica-
tective data linkage, access, and analysis, but this tions). Te Census Bureaus Center for Econom-
capacity does not meet government-wide needs ic Studies (CES) links and archives business and
for evidence. Second, the identifcation and adop- economic data used for research and analysis. Te
tion of new technologies that can increase data Census Bureau also coordinates a network of Fed-
access while enhancing privacy is slow and un- eral Statistical Research Data Centers (FSRDCs) to
even across Federal agencies. Tird, Federal agen- provide secure access to confdential data (see the
The Promise of Evidence-Based Policymaking 69

Center for Administrative Records Research and Applications


Te Center for Administrative Records Research organizations; and directs Census to seek out
and Applications (CARRA) at the U.S. Census this information instead of conducting direct
Bureau strategically uses administrative data inquiries. Further, the Census Bureau has a
from Federal, state, and third party providers specifc exemption under the Privacy Act that
for exclusively statistical activities in support of allows Federal agencies to disclose records to
the Census Bureaus operations and demograph- the Census Bureau without prior written con-
ic and socioeconomic research. Linked data are sent of the individual if it is for the purposes of
also used by researchers at the Census Bureaus planning or carrying out a census or survey or
Center for Economic Studies and through the related activity.
Federal Statistical Research Data Centers. Census is authorized to access Federal Tax
As of 2017, CARRAs portfolio includes 12 Information for individuals and businesses
pilot research and evaluation projects span- under Title 26 (IRS statute). CIPSEA authoriz-
ning housing, health, welfare, education, and es the Census Bureau, the U.S. Bureau of La-
labor. Te initiative also includes six pilot proj- bor Statistics, and the U.S. Bureau of Economic
ects in partnership with Chapin Hall at the Analysis to share business data for exclusively
University of Chicago that combine state and statistical purposes. While these authorities
local administrative data with information ac- permit the Census Bureau to access many dif-
cessible to the Census Bureau. Existing CARRA ferent datasets, in practice data sharing agree-
partnerships with experts in academia support ments are negotiated separately and governed
CARRAs ongoing eforts to improve capabili- by applicable laws and policies.
ties for securely combining confdential data. Under Title 13, CARRA can establish agree-
CARRA benefts from a number of legal ments to access data held by Federal agencies,
authorities applicable to the Census Bureau. states, and some other jurisdictions for exclu-
Foremost, the Census Bureau has broad au- sively statistical purposes. Te list below high-
thority under Title 13 (Census statute) to ac- lights a portion of the data CARRA currently ac-
quire administrative records. Title 13 authoriz- quires to support the Census Bureaus mission.1
es the Census Bureau to use records previously State administrative records not only vary on
collected by other Federal agencies and state, the number of states participating, but also in
tribal, or local governments, as well as private the years of data available.

Federal administrative data State administrative data


Medicaid Statistical Information System Child Care Development Fund (1 state)
Ofce of Personnel Management Low Income Home Energy Assistance
Personnel System Program (1 state)
Public and Indian Housing Information Center Special Supplemental Nutrition Program
Selective Service System for Women, Infants, and Children (6 states)

Small Business Administration Loan Supplemental Nutrition Assistance


Guarantee Program Program (15 states)

Social Security Old Age, Survivors, and Temporary Assistance for Needy Families
Disability Insurance (9 states)

Tax Information Unemployment Insurance quarterly wages


(48 states)
United States Postal Service Change of
Address

1. U.S. Census Bureau, Administrative Data Inventory as of


July 18, 2017; https://www2.census.gov/about/linkage/data-fle-
inventory.pdf (accessed August 10, 2017).
70 The Promise of Evidence-Based Policymaking

box Federal Examples of Secure Access to Con- 49) of other responding ofces have provided oth-
fdential Data). In addition, the U.S. Bureau of er Federal agencies access to data they collect.3 All
Economic Analysis, a PSA that produces a broad PSAs have external researchers who access data,
range of economic data;the U.S. National Oceanic while the same is true of 75 percent of respond-
and Atmospheric Administration, an agency with ing evaluation ofces and 31 percent of other re-
many scientifc and technical data resources; and sponding ofces.
the National Institute of Standards and Technolo- Te Commission considered whether this
gy are all part of Commerce. existing infrastructure could, with little new
PSAs, and some program agencies, provide se- additional authority or resources, meet govern-
cure access to their confdential data for statistical ment-wide needs for evidence building. Much of
purposes authorized by law (see examples in the the governments staf expertise on securely link-
box Federal Examples of Secure Access to Conf- ing confdential data resides at the Census Bureau,
dential Data). Te CEP Survey of Federal Ofces
found that 10 out of 13 PSAs, 5 out of 8 respond-
3. Two of the 10 responding evaluation ofces and 18 of the 67
ing evaluation ofces, and 39 percent (19 out of responding other ofces reported that they do not collect data.

Federal Examples of Secure Access to Confdential Data

Administrative Data Research Facility: Te data related to education in the United States
Administrative Data Research Facility is a pilot and other nations. NCES makes detailed data
project that enables secure access to analytical available to external researchers through the
tools, data storage and discovery services, and IES restricted-use data licensing system. NCES
general computing resources for users, includ- loans researchers restricted-use data through a
ing Federal, state, and local government ana- license (contract) among IES, the user, and the
lysts and academic researchers. Te Census users organization (e.g., university, research
Bureau and academic partners developed the institution, or company). Licensed users agree
project as part of the collaborative Training to the terms of the IES confdentiality law; IES
Program in Applied Data Analytics sponsored performs regular audits to confrm compliance
by the University of Chicago, New York Univer- with required procedures. Because of this audit
sity, and the University of Maryland.1 It is cur- requirement, IES will only loan restricted-use
rently operating as a pilot with users accessing data to qualifed organizations in the 50 States
the Facility as part of the training program. Te and Washington, D.C. Researchers seeking a
Facility operates as a cloud-based computing data license must complete an online formal
environment, with Federal security approvals, request and research proposal.2
which currently hosts selected confdential Federal Statistical Research Data Cen-
data from the U.S. Department of Housing and ters: Te Federal Statistical Research Data
Urban Development and the Census Bureau, as Centers (FSRDCs) are partnerships between
well as state, city, and county agencies, and an Federal statistical agencies and leading re-
array of public use data. search institutions. Tey are secure facilities
Data Licensing System: Te National Cen- housed in partner institutions and managed
ter for Education Statistics (NCES), one of the by the U.S. Census Bureau to provide secure
Principal Statistical Agencies (PSAs), is part of access to a range of Federal restricted-use mi-
the Institute of Education Sciences (IES) in the crodata for statistical purposes only. All FSRDC
U.S. Department of Education. It is the prima- continues
ry Federal entity for collecting and analyzing
2. U.S. Department of Education, National Center for Educa-
tion Statistics, Learning Procedures: Frequently Asked Ques-
1. Training Program in Applied Data Analytics; http://www. tions; http://nces.ed.gov/statprog/instruct_licensing_faq.asp (ac-
applieddataanalytics.org (accessed August 10, 2017). cessed August 10, 2017).
The Promise of Evidence-Based Policymaking 71

and particularly in the Center for Administrative at the Census Bureau in Chapter 3), but no one
Records Research and Applications (CARRA). As agency today serves as a model of best-practice
currently constituted, however, CARRA cannot application of cutting edge privacy-protecting
meet all of the project linkage needs of other Fed- technologies.
eral departments and qualifed external research- Te Commissions public meetings, hearings,
ers. It has neither the authority in law nor the Request for Comments, and Survey of Federal Of-
capacity to do so. Te FSRDCs and PSAs provide fces yielded numerous examples of unintentional
secure access to confdential data; none of them barriers to evidence building erected due to unclear
has the authority or capacity to provide expand- legal frameworks as described in Chapter 2. Other
ed access for increased evidence generation gov- barriers arise because of uneven capacity across
ernment wide. Some individual Federal projects Federal departments. Some Federal departments
at the Census Bureau and elsewhere are piloting lack PSAs; some evaluation and policy research
new technologies for secure data analysis (see the ofces lack technical capacity; and some program
box OnTeMap: Diferential Privacy in Practice agencies fail to prioritize evidence building.

Federal Examples of Secure Access to Confdential Data


continued
researchers must obtain Census Bureau Special years, resources permitting.3
Sworn Status. FSRDC data enclaves meet the Virtual Research Data Center: Te Cen-
physical and environmental protections re- ters for Medicare and Medicaid Services (CMS)
quired by Federal statistical agencies for use of is responsible for administering the Medicare,
restricted-access microdata, including Federal Medicaid, and Childrens Health Insurance
Tax Information. FSRDC researchers can col- programs. To support these functions, CMS
laborate with other FSRDC researchers across collects, generates, and stores fnancial, health
the United States through the secure comput- care, and other sensitive information. With ap-
ing environment. propriate safeguards, CMS provides access to
Joint Statistical Research Program: Te microdata for statistical and research uses to
Statistics of Income Division at the Internal Federal agencies, Federal grantees, and other
Revenue Service collects and processes tax data approved researchers on encrypted data fles
and provides access to tax microdata to those provided on external media or through a Vir-
users authorized under statuteincluding tual Research Data Center. Trough the virtual
certain government agenciesfor tax policy data center, approved researchers can access
analysis, research, and statistical purposes. and perform their own analysis and manip-
In addition, the Statistics of Income Division ulation of CMS data virtually from their own
makes microdata available to researchers workstations. Te virtual data center helps
through its Joint Statistical Research Program. CMS meet additional demand from data users
Te goals of the program are to (1) provide new while also ensuring data privacy and security.
insights and advance the understanding of the It also provides researchers with more timely
ways existing tax policies afect individuals, access to data in a more cost-efective manner.4
businesses, and the economy, (2) suggest tax
policy solutions to advance the common good,
and (3) provide new understanding of taxpayer 3. Internal Revenue Service, Statistics of Income, Joint Statis-
behavior that could impact the administration tical Research Program Call for Proposals; https://www.irs.gov/
pub/irs-soi/16jsrpapplication.pdf (accessed August 10, 2017).
of the U.S. tax system. Applications for the
program are accepted approximately every two 4. Centers for Medicare and Medicaid Services (CMS),
Virtual Research Data Center (VRDC); http://www.
resdac.org/cms-data/request/cms-virtual-research-data-center
(accessed August 10, 2017).
72 The Promise of Evidence-Based Policymaking

Some states and local jurisdictions told the and burden for local entities due to the expertise
Commission that basic technical services could required and may [disincentivize] research part-
substantially improve their capacity to support ev- nerships. A central service could lend consistent
idence-based policymaking about their own pro- technical expertise to the task of data matching.5
grams and policies as well as those implemented Such technical services could be useful to Federal
in partnership with the Federal government.4 At a departments facing capacity challenges as well.
Commission public hearing in February of 2017, CARRAs approach to using statistical and ad-
Joy Bonaguro, Chief Data Ofcer for the City and ministrative data for statistical activities over the
County of San Francisco, testifed that matching past several years demonstrates the possibility and
and linking data across systems can be a challenge potential of making better use of these data for

4. Communication with National Association of State Workforce


Agencies Labor Market Information Committee, Chicago, IL, April 5. Joy Bonaguro, City and County of San Francisco, Commission
20, 2017. Public Hearing, Washington, D.C., February 9, 2017.

Examples of Secure Access to Confdential Data in


Other Countries

Administrative Data Research Network Research Data and Service Centre


(United Kingdom): Te Administrative Data (Germany): Te German Bundesbank compiles
Research Network is a national partnership data for monetary, fnancial, and external sta-
in the United Kingdom that provides trained tistics and produces a comprehensive set of
social and economic researchers access to indicators under its legal mandate to develop
de-identifed, linked administrative data in a a broad spectrum of user-oriented economic
secure environment. Te Network was founded data. Te Bundesbank established the Research
in 2013 by the Economic and Social Research Data and Service Centre to facilitate data link-
Councilan independent, publicly funded en- age and researcher access. Te Bundesbanks
tityto allow more use of government data program provides visiting researchers with
in social and economic research, with safety on-site access to microdata on banks, securi-
measures in place to protect sensitive data. A ties, investment funds, enterprises, and house-
goal of the Networks work is to develop high holds. Researchers seeking to access Bundes-
standards for sharing, linking, and matching bank data must submit an application that
records securely and consistently. Te Network includes a description of the project, its hy-
continues to research data linkage methodol- potheses and methods, and a justifcation for
ogies and evaluate new technologies. It also the datasets required. Staf at the Bundesbanks
recognizes that public trust through public program work with researchers in advance of
engagement is the most important element their application to help them refne their re-
for success. Te Networks processes were de- search questions and identify the data sources
signed with a great deal of public consultation. they will need.2
Te Networks public engagement strategy is Research Services Division (Denmark):
designed to promote sustainability through Statistics Denmark, under the Ministry for Eco-
transparency and public understanding.1 nomic and Interior Afairs, is responsible for
creating ofcial statistics on Danish society. Te
continues
1. United Kingdom, Administrative Data Research Network,
Communication and Public Engagement; https://adrn.ac.uk/ 2. Germany, Bundesbank, Research Data and Service
about/network/ads/communication-and-public-engagement (ac- Centre; http://www.bundesbank.de/Navigation/EN/Bundesbank/
cessed August 10, 2017). Research/RDSC/rdsc.html (accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 73

government-wide evidence building. Tat CARRA sure limitation techniques to protect confdential
safely and confdentially links data for exclusively data and provide highly restricted access to con-
statistical purposes under the law without priva- fdential data for exclusively statistical purposes
cy breaches demonstrates this can be done well (see Chapter 3).
when appropriate authorities and privacy protec- Some of todays most innovative and informa-
tions are in place. tive evidence-building projects rely on highly re-
stricted access for researchers to confdential data.
Challenge #2: Identifcation and adoption Examples include (1) research on intergeneration-
al mobility by race and ethnicity using tax and
of emerging technologies and techniques
survey data,6 (2) assessments of the long-term ef-
for privacy-protective data access, link- fects of antipoverty policies that connect welfare
age, and analysis are too slow and uneven data with earnings records,7 and (3) development
in government to efectively support evi-
dence building. 6. Raj Chetty, Nathaniel Hendren, Patrick Kline, and Emmanuel
Saez, Where is the Land of Opportunity? Te Geography of In-
As described in Challenge #1, few Federal agencies tergenerational Mobility in the United States, National Bureau of
are providing privacy-protective access to conf- Economic Research (NBER) Working Paper 19843 (January 2014);
http://www.nber.org/papers/w19843 (accessed August 10, 2017).
dential data for exclusively statistical purposes,
and even fewer are securely linking confdential 7. For example, Experimental Estimates of the Long-Run Impacts
of Welfare Reform on Participants and Teir Children and on-
data for evidence building. All PSAs and a few oth- going work by Hilary Hoynes, Jordan Matsudaira, Pauline Leung,
er Federal agencies use rigorous statistical disclo- and Zhuan Pei.

Examples of Secure Access to Confdential Data in


Other Countriescontinued
Research Services Division of Statistics Den- dAccs Scuris aux Donnes, or CASD) was
mark provides researcher access to linked data founded in 2009. One distinct feature of this
with identifers removed for statistical purposes system is a secure device, the SD-Box, that al-
only. Since 1968, every person in Denmark has lows researchers to remotely access and analyze
received a Personal Identifcation Number. Each confdential data from CASDs servers. Te SD-
person is also associated with a dwelling identi- Box is installed at approved researchers own in-
fcation and possibly an enterprise or business stitutions. With their access card and fngerprint
identifcation. Tese three keys can be used to confrmation, researchers access everything
link multiple sources of administrative and sur- they need at their local workstationincluding
vey data for individuals and family units over data, statistical software, storage, and comput-
time. Researchers afliated with an approved ing powerwhen, in fact, all of these are on
Danish research environment may submit an the secure servers inside CASD. CASD remote-
application to the Research Services Division ly authenticates, monitors, and confgures the
to access Statistics Denmark data. Data may be SD-Box and all communications between it and
accessed only through authorized research en- CASD servers take place through encrypted tun-
vironments, which include universities, sector nels. Because there is centralized security man-
research institutes, ministries, and non-proft agement, CASD can push upgrades and patches
foundations.3 to each device. SD-Boxes can also be remotely
Secure Data Access Centre (France): disabled. All SD-Boxes are identical and easily
Frances Secure Data Access Centre (the Centre replaced in the event of an equipment failure.4

3. Denmark, Statistics Denmark Research Services


Division, Data for Research; http://www.dst.dk/en/TilSalg/ 4. France, Secure Data Access Centre; https://casd.eu/en
Forskningsservice (accessed August 10, 2017). (accessed August 10, 2017).
74 The Promise of Evidence-Based Policymaking

of improved measures of poverty and the income However, the absence of past breaches does not
distribution that connect survey, tax, and pro- mean future breaches are impossible, particularly
gram data.8 Such projects can only be conducted if as technologies evolve or bad actors focus on
linkages among datasets can be made in an envi- diferent types of data.
ronment in which privacy and confdentiality are As described in Chapters 2 and 3, a legal in-
protected. frastructure must exist to enable secure data ac-
cess for evidence building and to take advantage
The collection of accurate of new privacy-protective technologies. Among
and complete information those agencies providing secure access to conf-
dential data, some have the expertise and resourc-
is predicated on trustWithout es to identify and implement feasible new tech-
that trust, data collection will nologies that hold the promise of allowing more
be undermined before it even evidence generation in more secure ways (see the
begins 9 box Emerging Approaches Tat Enhance Privacy
Protections in Chapter 3). While these priva-
Thomas L. Mesenbourg, Former U.S. Census
cy-protective approaches are in use today, they
Bureau Deputy Director
are not available at the scale necessary to enhance
Today, PSAs and a few other Federal agencies evidence production across government. Te Cen-
that provide secure access to confdential data for sus Bureaus capacity, for example, is limited to a
statistical purposes have a number of protections handful of projects, with new cutting-edge tech-
in place as required by statute and policy. nologies limited to even fewer applications.
Federal staf and outside researchers accessing Te Commission heard from several other coun-
confdential data must complete annual training tries about models for using new technologies for
in information technology security and data secure data access (see the box Examples of Secure
stewardship. Agencies restrict access to person Access to Confdential Data in Other Countries).
identifers such as name and date of birth to a Today, the Federal evidence-building commu-
very few technical staf under tightly controlled nity lacks a coordinated focus on harnessing new
conditions. Other staf and outside researchers technologies to provide more secure data access for
almost never receive approval to access personal exclusively statistical purposes. Te governments
identifers. Any statistical research output that capacity to take advantage of emerging privacy-pro-
uses confdential data undergoes statistical tective technologies is spread across a number of
disclosure review before public release to ensure diferent agencies. Te focus of individual agencies
confdentiality is not compromised. During the is rightfully on their own missions and not orient-
Commissions December meeting, government ed towards government-wide evidence building.
ofcials, including from the Census Bureau No one agency has sufcient resources and staf
and the Statistics of Income Division in the expertise to develop and implement state-of-the-
Internal Revenue Service, indicated that to their art technologies or to push the evidence-building
knowledge no breach of their confdential data community to continually innovate to address po-
used for evidence building had ever occurred.10 tential threats and risks to the American publics
confdential data. Achieving the Commissions vi-
sion means addressing these challenges.
8. Bruce D. Meyer and Nikolas Mittag, Using Linked Survey and
Administrative Data to Better Measure Income: Implications for
Poverty, Program Efectiveness, and Holes in the Safety Net,
Challenge #3: Te existing infrastructure
NBER Working Paper 21676 (October 2015); http://www.nber.org/ for accessing, linking, and analyzing conf-
papers/w21676 (accessed August 10, 2017).
dential data for evidence building does not
9. Tomas L. Mesenbourg, How Statutory and Regulatory Chang-
es May Create New and Unintended Challenges for Statistical always prioritize state-of-the-art transpar-
Agencies (paper presented at the International Society of Scien- ency and oversight.
tometrics and Informetrics Conference, Durban, South Africa, July
2011).
From collecting data to generating evidence about
10. Barry Johnson, Internal Revenue Service, and Ron Jarmin,
Census Bureau, Commission Meeting, Washington, D.C., Decem- government programs and policies, members of
ber 12, 2016. the Federal evidence-building community should
The Promise of Evidence-Based Policymaking 75

tell the American people how their information is fdential information from the American people.
being used and for what beneft to society. Trans- To maintain the publics trust, the evidence-build-
parency means giving the public information ing community must take reasonable steps to en-
about how the government is using their data to sure the public is aware of how data are collected,
improve its efectiveness and efciency. Te Fed- safeguarded, shared, and used, and what benefts
eral evidence-building community also should es- those uses may provide for society. Tese are long
tablish oversight and accountability mechanisms recognized aspects of transparency for evidence
to verify that it is using best practice procedures building. However, the Commission fnds that the
to protect personal information. Federal evidence-building community today does
For data to be useful for evidence building, they not consistently nor efectively use best practice
must be complete and accurate. Te cooperation transparency processes for public accountability.
of the American people is central to the quality During the Commissions September 2016
and accuracy of survey and administrative data meeting, then-Senior Advisor for Privacy at the
used for evidence building. Telling the American U.S. Ofce of Management and Budget (OMB) and
people about how the government is using their Chair of the Federal Privacy Council, Marc Gro-
data to improve accountability and serve the pub- man, emphasized the importance of transparency:
lic is ethical and conforms to the Fair Information
Practice Principles (FIPPs) on transparency and [As] the United States Government
accountability and auditing described in Chapter we have an obligation to betrans-
3. Only with such transparency and accountabil- parent to our citizens about what
ity can the public trust needed to collect and use we collect and how we use it.We
personal data for evidence building be preserved. make representations about use and
Further, for evidence to make a diference in we have to honor thoserepresenta-
government efectiveness and efciency, policy- tions...[by explaining]how the data
makers, program administrators, and the public collect[ed] will be used. And that is
must trust the results. Without trust, the results acore principle of the Privacy Act or
will lack credibility. Without credibility, the results other laws and of privacy generally.11
are more likely to be ignored, wasting the time and
money used to generate them. Even worse, lack of Te Privacy Act requires a System of Records
credibility could jeopardize the goal of generating Notice to promote public transparency. Govern-
more and better evidence to improve government ment agencies issue public notices when a sys-
and monitor the state of the nation. Trust is criti- tem of records containing personally identifable
cal to evidence-based policymaking. information is being created or expanded. Tese
Te tradition and culture of data stewardship public notices disclose what information is col-
in some parts of the Federal evidence-building lected and for what purpose. However, the notices
community today should inform the eforts of all are made available in a way that can be difcult for
Federal departments that provide secure access to the public to access; members of the public may
confdential data. Te PSAs responsible for much not know where to fnd them or understand the
of the current data infrastructure operate within technical information they contain. While Gro-
a framework of laws, regulations, policies, and man discounted the publics interest in the notic-
practices that emphasize the importance of re- es given this challenge, he added: I can tell you
specting individuals privacy and protecting the that advocacy groups do read it; [inspectors gen-
confdentiality of their information. Tese histor- eral] do readit. And it provides actually a beneft,
ic protections must be augmented with additional because its about transparency.
measures to increase transparency and account- An important limitation is that these notices
ability about how data are accessed and used for are for systems of records, not for individual proj-
exclusively statistical purposes. ects. Today, no notice is required for individual sta-
tistical or evidence-building projects. Some other
Transparency countries provide public notice about individual

Te Congress and the Executive Branch require 11. Marc Groman, U.S. Ofce of Management and Budget, Com-
public notice when the government collects con- mission Meeting, Washington, D.C., September 9, 2016.
76 The Promise of Evidence-Based Policymaking

projects that link confdential data (see the box and improve the quality of the data collection.
Case Study: Transparency at Statistics Canada). In sum, members of the public currently have
Federal departments also are required to gain multiple opportunities to learn about and provide
approval for Information Collection Requests un- feedback on governments information collection
der the Paperwork Reduction Act for new data activities before they are initiated. Such transpar-
collections from 10 or more individuals. Tese re- ency is valuable but applies to systems of records,
quests must go through two public comment peri- rather than to individual linkage and analysis
ods before fnal approval. Some statistical agencies projects, and public notices are sometimes dif-
go even further when conducting new primary cult to fnd, access, and understand.
data collections and engage stakeholder groups When collecting administrative information
prior to data collection to incorporate feedback from program participants or benefciaries,

Case Study: Transparency at Statistics Canada


Statistics Canadas policies on using adminis- administrative data are combined with other
trative data address how that agency protects data. It is transparent in its use of administra-
privacy and provides transparency on the use tive data, and communicates benefts of using
and linkage of its administrative data about Ca- existing data as well as measures taken to pro-
nadians. Te policys objective is to maximize tect confdentiality.
the efciency and efectiveness of the use of Statistics Canada employs individuals with
administrative data in Statistics Canadas sta- dedicated roles to promote transparency. Te
tistical programs. Statistics Canada achieves Director General of the Communication Divi-
this objective through a corporate strategy in- sion is responsible for developing and imple-
tended to infuence, access, use, and manage menting a plan that shows Statistics Canadas
the administrative data supplied to Statistics commitment to the transparent use of admin-
Canada, and to maintain public trust while do- istrative data. Statistical and legal analysts in
ing so. 1 Statistics Canada intends the policy the Information Management Division are
to communicate its statistical use of adminis- responsible for maintaining and managing an
trative data to stakeholders and the public in inventory of all approved data linkages. Dedi-
a proactive, coherent, and transparent manner. cated websites provide another layer of trans-
Under Statistics Canadas Principles for the parency.2
Statistical Uses of Administrative Data, statis- As part of its governance over microdata
tical uses are kept functionally separate from linkages, Statistics Canada has pre-approved
administrative uses through strong legal, poli- specifc types of linkages. Te linkages involved
cy and organizational safeguards. In particular, are those in which privacy risks and situations
the confdentiality of all identifable adminis- of potential confict of interest are low and
trative data obtained by Statistics Canada for where procedures to mitigate risk to confden-
statistical purposes is protected by law, and the tiality and privacy are in place. All other mi-
data are only accessed within Statistics Canada crodata linkages must undergo a prescribed re-
based on demonstrated needs. Statistics Can- view and approval process, which involves the
ada carefully considers the impact on privacy submission of documented proposals to senior
in using identifable administrative data for management. When such linkages include per-
a purpose that was not envisaged at the time sonal information, a summary of the linkage is
of original data collection, particularly when posted on Statistics Canadas website.

2. Statistics Canada, Linkage depository; http://www.statcan.


1. Statistics Canada, Policy on the Use of Administrative Data gc.ca/eng/sdle/status. See also Studies, research papers and
Obtained under the Statistics Act; http://www.statcan.gc.ca/ technical papers; http://www5.statcan.gc.ca/researchers-
eng/about/policy/admin_data (accessed August 10, 2017). chercheurs/ (both accessed August 10, 2017).
The Promise of Evidence-Based Policymaking 77

government agencies sometimes indicate that data when an independent group helped to ensure
data will only be used for specifc purposes, such the presence of privacy protections.13
as only for processing a beneft application. Such CARRA and CES operate with only the pub-
promises generally mean the Federal government lic transparency and oversight that are currently
cannot use those data for any other purpose, in- standard practices for Federal agencies, such as
cluding statistical analysis for evidence building, through the annual appropriations process when
without re-contacting the individual. However, the Congress reviews programs and activities. Te
when an agency uses its own administrative data public does not have meaningful opportunities to
for exclusively statistical purposes, it generally provide guidance or feedback about the individual
does not need to re-contact individuals. An ex- projects that are underway. Some states have be-
ception under the Privacy Act allows agencies, in gun to develop procedures for public transparency
some cases, to engage in an intra-agency disclo- about evidence building that could be a model (see
sure of data for statistical purposes.12 box Case Study: Mississippis LifeTracks System).
Federal agencies today operate with limited
transparency about individual projects seeking to Privacy Auditing
link and analyze confdential data for exclusive-
ly statistical purposes. Even agencies at the fore- Transparency in Federal evidence building requires
front of Federal evidence building, like CARRA verifying that the Federal government implements
and CES, do not currently have easily accessible clear, strict rules about data access and follows
information about approved projects, qualifed re- through on its promises to the American people to
searchers, and datasets linked for evidence build- keep data for evidence building safe and secure.
ing. Further, because CARRA and CES are located One way to promote public trust through ac-
within the Census Bureau, they do not have their countability is to require privacy audits, which
own independent advisory boards. A recent Gal- assess available protections within an organiza-
lup survey reported that nearly two-thirds of re- tion against legal requirements and privacy best
spondents were more likely to support combining practices. Privacy audits document the status of
an organizations risk associated with information
12. U.S. Ofce of Management and Budget (OMB), Guidance for misuse and make recommendations to limit this
Providing and Using Administrative Data for Statistical Purposes risk. A privacy audit also reviews an organizations
(memorandum M1406, Washington, D.C.: OMB, Executive Of-
fce of the President, February 14, 2014); https://obamawhitehouse.
archives.gov/sites/default/fles/omb/memoranda/2014/m-14-06.pdf 13. CEP staf correspondence with Census Bureau staf in April
(accessed August 10, 2017). 2017.

Case Study: Mississippis LifeTracks System


Mississippi LifeTracks is a state longitudinal devoted a portion of their website to pub-
data system that, in addition to programmatic lic accountability. Te website lists approved
uses, allows for the analysis of administrative projects and completed projects and cites
data from multiple state agencies to assess ed- state-level statistics based on their results. Tis
ucation and workforce outcomes in the state.1 demonstration of the value of the program and
LifeTracks is funded through a combination of the useful information it provides has been
U.S. National Center for Education Statistics suggested as the key to the systems sustain-
grants and annual state appropriations. One ability. LifeTracks aims to hold state govern-
challenge Mississippi faced in creating the ment accountable for implementing efective
LifeTracks system was conveying the value of programs and policies. At the same time, the
the system. To accomplish this, Mississippi public can hold government accountable for
the use of data in the LifeTracks System be-
1. Domenico Parisi, Mississippi State University, Commission cause of the programs transparency eforts.
Meeting, Washington, D.C., January 13, 2017.
78 The Promise of Evidence-Based Policymaking

compliance with its own privacy-related polices.14 building. Simply providing a way for researchers
In the private sector, privacy audits provide a to see what data are potentially available for evi-
means of benchmarking corporate privacy practic- dence building, along with appropriate technical
es against what the law requires and what indus- documentation on each dataset to assess relevance
try best practices demand.15 Conducting a privacy and quality, can allow researchers to identify how
audit and correcting any identifed weaknesses existing data can be used to answer research and
demonstrates to the public that an organization is evaluation questions about government programs
serious about both transparency and privacy. and policies.
Today, there is no coordinated auditing func- Te Commissions study of the existing inven-
tion for Federal evidence building. Tis lack of ac- tory of government data relied on recent work by
countability means that there is no consistency in OMB. In 2016, OMB summarized the results of an
whether and how Federal departments review and attempt to document existing datasets across gov-
audit access to and use of confdential data, mon- ernment through a white paper for the Commis-
itor agency compliance with transparency and sion.16 Federal departments already are required
privacy rules and procedures, monitor research- to create data inventories to the extent practica-
er compliance with the terms of their data access ble, with the ultimate goal of including all agency
agreements, recommend appropriate penalties for datasets.17 Today, however, departments inven-
violations of policies and procedures, or report to tories are of uneven quality and completeness,
the Congress and the American people. especially for datasets that agencies determined
cannot be publicly released. In addition, the exist-
Challenge #4: Datasets that could be used ing inventory does not include datasets that may
be relevant for generating evidence about Federal
for evidence building do not all have ade-
programs and policies but are collected by state
quate technical documentation. and local governments or other jurisdictions. A
more comprehensive inventory of data assets that
Te Commission routinely heard about challeng- are available for evidence buildingor not avail-
es associated with researchers accessing data due able but should bewould be extremely valuable
to a lack of information about what datasets are for Federal staf and external researchers. Com-
available, what data elements are included in a mission witnesses suggested that any catalog of
dataset, the quality of those data, and how the data useful for evidence building should indicate
dataset can be accessed. Notwithstanding recent whether the data are restricted so researchers can
attempts to develop a data inventory and calls for determine if they are requesting data that may re-
improved technical information about datasets, quire additional hurdles to access.
government lags behind the private sector in its
standards for managing and documenting data Metadata
that could be used for evidence building. Techni-
cal documentation standards are a necessary pre- Once an inventory is created, potential data users
cursor to routine secure evidence generation. still need to understand the contents, character-
istics, and quality of individual datasets. One par-
Data Inventory ticular kind of technical documentation consists
of data about the data, often called metadata.
During the Commissions fact-fnding phase, com- Especially when administrative data are initially
menters stated that a data inventory with basic
technical information is fundamental to evidence
16. U.S. Ofce of Management and Budget, Comprehensive Data
Inventory, (white paper for the Commission on Evidence-Based
14. Muzamil Riffat, Privacy AuditMethodology and Related Policymaking, Washington, D.C., 2016); https://obamawhitehouse.
Concerns, ISACA Journal 1 (2014); https://www.isaca.org/Journal/ archives.gov/sites/default/fles/omb/mgmt-gpra/comprehensive_data_
archives/2014/Volume-1/Pages/default.aspx (accessed August 10, inventory.pdf (accessed August 10, 2017).
2017).
17. U.S. Ofce of Management and Budget, Open Data Policy
15. Michael L. Whitener, Conducting a Privacy Audit, The Managing Information as an Asset (memorandum M1313,
Cor-porate Counselor 27, no. 3 (July 2012); https://iapp.org/media/ Washington, D.C.: OMB, Executive Ofce of the President, 2013);
pdf/knowledge_center/Conducting_a_Privacy_Audit_- https://obamawhitehouse.archives.gov/sites/default/f iles/omb/
_The_Corporate_Counselor_-_July_2012.pdf (accessed August 10, memoranda/2013/m-13-13.pdf (accessed August 10, 2017).
2017).
The Promise of Evidence-Based Policymaking 79

provided and used for statistical activities, infor- cludes variable defnitions, units of measurement,
mation about their meaning and intent can be and response ranges. Also, responses to the CEP
lost. Making appropriate use of administrative survey suggested that overall, ofces that used a
data for evidence building requires the develop- higher percentage of their budget for statistical,
ment of data documentation, which can be time evaluation, research, and analysis purposes more
intensive and require a deep knowledge of the frequently created robust metadata. But among
administrative data collected.18 OMBs report on a Federal departments administering programs and
comprehensive government inventory described collecting data that can be useful for evidence
the minimal amount of metadata required under building, the completeness and accuracy of meta-
the Open Data initiative; however, not all agen- data vary widely.
cies have met this minimum requirement.19 Com-
mission witnesses requested much more robust Harmonization of Defnitions
metadata for evidence building.
Complete and accurate metadata can help re- Diferent Federal agencies have diferent defni-
searchers determine whether access to certain tions of common terms and key concepts in their
data even makes sense for their specifc project data systems. Sometimes, the same agency has
and allow them to assess data quality prior to multiple diferent defnitions of the same con-
seeking access to microdata. In fact, incomplete cept. Tese diferences evolved because of how
or inaccurate metadata may result in researchers the terms are defned in law or because of specifc
seeking access to confdential data that will not program needs.
answer their research question. Having complete Tis variation in defnitions for common
and accurate metadata is vital to improving se- terms and key concepts in government presents
cure data linkage and access. In fact, standardized challenges in analyzing data across agencies and
metadata is an essential element of employing sometimes across jurisdictions. At times, these
privacy-preserving technologies such as Secure variations are acknowledged and planned for or
Multiparty Computation (see Chapter 3). addressed by programs. In 2008, the Special Sup-
In addition to emphasizing the importance of plemental Nutrition Program for Women, Infants,
the Federal government maintaining a list of gov- and Children at the U.S. Department of Agricul-
ernment-held datasets and providing basic infor- ture updated guidance to states about what data
mation about mechanisms for accessing the data- elements should be included in data systems, and
sets, many stakeholders called for a list of variables clarifed defnitions needed for managing case-
in the datasets as the most basic form of metadata. loads.21 For some concepts with government-wide
Stakeholders described metadata as a critical com- applicability, OMB issues standards to create com-
ponent of assessing data quality and ftness for use; mon defnitions, such as the Standard Occupa-
the Commission heard numerous recommenda- tional Classifcation or a recently revisited stan-
tions to create robust administrative dataset docu- dard on race and ethnicity.22
mentation by standardizing metadata.20 Multiple witnesses in the Commissions public
Te CEP Survey of Federal Ofces found that hearings and other commenters highlighted the
all 12 PSAs that answered the survey item on need to harmonize defnitions and the meaning
documentation already create metadata that in-

21. U.S. Department of Agriculture (USDA), Food and Nutrition


Service, Functional Requirements Document for a Model WIC Infor-
18. Aileen Rothbard, Quality Issues in the Use of Administrative
mation System (Washington, D.C.: USDA, 2008); https://www.fns.
Data Records, in Actionable Intelligence: Using Integrated Data Sys-
usda.gov/sites/default/f iles/apd/FReD%20v2.0%20Final.pdf (ac-
tems to Achieve a More Efective, Efcient, and Ethical Government,
cessed August 10, 2017).
edited by John Fantuzzo and Dennis P. Culhane (New York, NY:
Palgrave MacMillan, 2016): 72103. 22. U.S. Ofce of Management and Budget, Standards for Main-
taining, Collecting, and Presenting Federal Data on Race and
19. U.S. Ofce of Management and Budget, Comprehensive Data
Ethnicity, Federal Register 81 (September 30, 2016); https://
Inventory.
www.federalregister.gov/documents/2016/09/30/2016-23672/
20. Data quality pertains to data completeness, data accuracy, and standards-for-maintaining-collecting-and-presenting-federal-data-
consistent defnitions of data elements. Te ftness for use is de- on-race-and-ethnicity; OMB, 2010 Standard Occupational Classif-
termined based on the intended use, and may be limited by data cation (SOC)OMBs Final Decisions, Federal Register 74 (January
quality or other challenges associated with overly aggregated data 21, 2009); https://www.bls.gov/soc/soc2010fnal.pdf (both accessed
forms, use restrictions, or ethical considerations. August 10, 2017).
80 The Promise of Evidence-Based Policymaking

Interagency Employer Data Matching Workgroup

An interagency workgroup developed strate- ent-child relationships within frms, and the
gies for matching and reusing data on U.S. em- dynamic nature of such relationships. Te
ployers across Federal datasets for statistical workgroup found two primary challenges in
purposes. In 2016, the group published a white matching employer data collected by the Fed-
paper that summarized key issues for match- eral government: (1) the lack of a common
ing employer data and identifed best practices universal identifer and (2) poor quality of the
that could be implemented across agencies.1 underlying identifying data.
Matching employer data can be complex Te interagency group identifed the most
because of the diferent levels of data (estab- important data elements for matching em-
lishment versus frm or enterprise), the par- ployer data across Federal agencies. Te most
common elements that are most essential
to matching include Employer Identifcation
1. Employer Data Matching Workgroup. Employer Data Number, Legal Entity Identifer, legal business
Matching Workgroup White Paper. (white paper for the Com-
mission on Evidence-Based Policymaking, Washington, D.C. name, trade name, physical and mailing ad-
2017). dress, and country code.

of outcome metrics across Federal laws. Several imizing the possible risk of harm to individuals or
witnesses expressed a desire for the Federal gov- organizations. Te Commission concluded that
ernment to establish common defnitions for per- the Congress and the President must empower
formance metrics across the Higher Education government to develop state-of-the-art capacity to se-
Opportunity Act, the Workforce Innovation and curely combine existing data and provide secure data
Opportunity Act, and the Perkins Career and Tech- access for exclusively statistical purposes in a priva-
nical Education Act to reduce burden and improve cy-protective and transparent way.
opportunities for integrating data across programs In the Commissions vision, linkages of conf-
with similar goals and target populations. Another dential data and access to those data for solely sta-
witness identifed inconsistencies in the applica- tistical purposes will occur in a highly protective
tion of geographic defnitions across the Federal privacy, legal, and technological environment, de-
government and suggested they be further har- scribed in Chapters 2 and 3. Project-by-project re-
monized.23 As described in the box Interagency views will help the evidence-building community
Employer Data Matching Workgroup, some Fed- work toward the ideal of permitting access to data
eral agency personnel recognize the importance of for projects with great promise for improving the
consistent defnitions for key matching variables public good, while limiting the risk of re-identif-
to combine economic data on frms and business- cation and other privacy harms. At the same time,
es for statistical analysis. a more transparent process for statistical analysis
of confdential government data by researchers
Recommendations and government agencies will reduce unintended
inefciencies that can inadvertently hamper the
Te Commissions strategy seeks to extract the generation of evidence.
maximum value of data for public good while min- Te Commission ofers fve recommendations
regarding the implementation of the NSDS as a
service to support privacy-protective, transparent,
23. Timothy Slaper, Indiana Business Research Center, Indiana
University, Commission Public Hearing, Chicago, January 5, 2017. and accountable evidence generation.
The Promise of Evidence-Based Policymaking 81

REC. 4-1: Te National Secure Data mission also recommends that NSDS should have
(1) an institutional placement that gives it neces-
Service (NSDS) should be established
sary independence, (2) designation as a PSA with
as a separate entity in the Department of authority to protect data under CIPSEA, (3) sep-
Commerce that builds upon and enhances arate funding, and (4) an improved governance
existing expertise and infrastructure in the structure ofering enhanced transparency.
Federal government, especially at the Cen- Using the expertise that already exists, NSDS
will be well placed to quickly begin implement-
sus Bureau, to ensure sufcient capacity in
ing activities, such as developing agreements with
secure record linkage and data access for data providers, developing model approaches for
evidence building. securely sharing data across agencies, and facilitat-
ing secure access to data for researchers external to
Te Census Bureau already has established, in pi- government with approved projects that serve the
lot form, some of the capacities that are central public good (see Figure 10). Te Commission envi-
to the Commissions vision for the NSDS. CARRA sions that when the relevant parts of CARRA and
and the CES at the Census Bureau each work to CES become part of the NSDS they would continue
combine data that can be made available for re- to manage the FSRDC network and that research-
searchers, primarily through the FSRDCs. Te ers external to government will continue to access
Commission concluded that the Census Bureaus confdential data through the FSRDC network.
current operations, and other capacity through- Te Commissions vision requires that the
out the Federal government, form a starting point Federal government stay on the cutting edge of
for implementing Recommendations 2-1 and 2-2. technological approaches to protecting data pri-
Building on the Census Bureaus history and cul- vacy and security. Te technical approaches for
ture of data stewardship and public trust, NSDS combining data require a high degree of expertise
can be a cornerstone of future Federal evidence and continuous training to keep abreast of rapidly
building with a focus on secure access to Federal evolving methods.
data, public transparency, and a consistent set of Concentrating this expertise in a Federal center
legal and technological privacy protections. for excellence means that the government can de-
Te existing Census Bureau infrastructure in velop best practices for data protection and secure
CARRA and CES supports, on a small scale, inter- access. Substantially restricting the individuals
nal research and operational uses of administra- who have access to identifable records on indi-
tive data and, in partnership with the FSRDC net- viduals or businesses will provide a higher degree
work, some externally motivated research projects of safety and security for confdential data.
using administrative data. Te current infrastruc- Bringing expertise together within govern-
ture should be expanded to support increased ment will help identify and implement feasible
evidence building. Rather than build an entirely new techniques and technologies. Specifcally, the
new and potentially duplicative set of functions, NSDS should be a center for excellence in develop-
a cost-efective approach to implementing the ing, using, and sharing best practices for transpar-
NSDS is to adapt and scale the existing elements ently and securely using confdential data for evi-
of CARRA and CES and build upon additional ex- dence building. New privacy-protective techniques
pertise throughout the Federal government and such as Secure Multiparty Computation (described
its partners in the evidence-building community. in Chapter 3) may allow individuals in the Feder-
To maximize efciency, the President and the al evidence-building community to combine data
Congress should reallocate designated staf and and conduct analyses without directly accessing or
resources from the Census Bureau to form the core storing information. Te NSDS should be a lead-
infrastructure for NSDS, supplementing those re- er within government for advancing that goal.
sources as needed with expertise from other large Te NSDS could also implement provable privacy
data collecting and disseminating components of methods such as diferential privacy (described in
the government. To achieve an optimal arrange- Chapter 3) to create datasets with minimal and
ment for expanding evidence building, the Com- quantifable risk of re-identifcation. Furthermore,
82 The Promise of Evidence-Based Policymaking

Figure 10. The NSDS in Action

Agency A Agency B

Direct Identifiers Direct Identifiers

Admin. Admin.
Data Data

National Secure Data Service (NSDS)

Combines Data and Formats into Linkable Datasets

Direct Identifiers Direct Identifiers


Removes and Statistical Statistical
Deletes Direct Data Data 15
Identifiers Assigns Expiration
Date to Data
Housed in NSDS
Final Checks
Data Minimization
Quality Control
Documentation
Transparency Portal

Data Cleared for Statistical Use 30


Assigns Expiration
Project Dataset Date to All
Project Data

Secure Use of Datasets

Disclosure Review and


Clearance of Results

Public Release of
Statistical Results
The Promise of Evidence-Based Policymaking 83

best practices should be shared throughout the develops and expands at NSDS.
government and lead to more secure data access Building on the existing capacity in the Federal
and better privacy protections across the Federal government, and particularly at the Census Bu-
evidence-building community. reau, and supplementing it with resources drawn
Because of the existing pilot activities at Cen- from across government will ensure that the ex-
sus, the implementation of NSDS can begin im- pertise for evidence building can be available not
mediately once the Congress and the President just for projects to advance the Census Bureaus
authorize its establishment. Work may then con- mission, but government wide.
tinue in phases, with the core activities that sup-
port NSDS fully devolved from the Census Bureau REC. 4-2: Te NSDS should establish a
within two to three years.
Steering Committee that includes
A phased approach to establishing and im-
plementing NSDS will allow it to conduct its own representatives of the public, Federal de-
pilot projects, develop data documentation and partments, state agencies, and academia.
integration procedures, and develop and enhance
relationships with data providers that allow op- Te public must have a voice in how the Federal
portunities for public input (see the box Potential government uses data for evidence building. Pub-
Phases for NSDS Implementation). lic participation ensures the NSDSs policies and
Te NSDS should also have the capacity to pro- practices remain focused on exclusively statisti-
vide technical services such as secure data link- cal purposes for the public good. Te Commission
age and analytical services on a fee-for-service concluded that a service like NSDS requires input
basis for states, local governments, and other ju- about how data are used from representatives of
risdictions. While technical and analytic services the public, privacy and technology experts, Fed-
should not be an initial priority for the NSDS, a eral departments, state agencies, research institu-
basic level of such services may be both necessary tions, and other stakeholders.
and desirable in the long term to help improve the A representative steering committee for NSDS
capacity for evidence building throughout the na- will allow individuals to advise the service on
tion and to encourage collaboration across levels strategic policies and provides one avenue for
of government. Researchers may also wish to avail organizational accountability. Te Commission
themselves of technical and analytic capacity as it specifcally recommends that such an advisory

Potential Phases for NSDS Implementation


Phase I Legal Authority. New legal authority Phase III Pilot Projects and Data Doc-
is enacted to establish the National Secure Data umentation. NSDS works with researchers
Service (NSDS) in its new location, with a stated and other stakeholders to build out new pri-
transition period of 2-3 years. vacy-protecting capacities and conduct a series
of pilot projects for scaling implementation.
Phase II Relocation and Staffing. Te Sec- Te NSDS launches a transparency portal.
retary of Commerce hires an administrator.
NSDS appoints the steering committee. Te rel- Phase IV New Development and Imple-
evant assets, personnel, and resources from the mentation. NSDS develops new relationships
Census Bureau are transferred to NSDS and the with data providers to meet user needs and
administrator brings in staf from other gov- continues to scale activities as necessary based
ernment agencies and outside government as on demand. NSDS expands eforts to imple-
needed. ment state-of-the-art technologies and tech-
niques for secure data linkage and access and
assesses the quality of the approaches.
84 The Promise of Evidence-Based Policymaking

committee be charged with the following respon- propriate use or failure to follow procedures. Fur-
sibilities: ther, the administrator must ensure compliance
with applicable Federal laws and policies, espe-
Consult with the Secretary of Commerce about
cially regarding data confdentiality and security.
budget and resource allocation decisions relat-
ed to NSDS operations and privacy protections. Te administrator should provide leadership
across the Federal evidence-building community
Conduct strategic planning for NSDS. to use state-of-the-art data protection and access
methods and technologies. However, the adminis-
Develop and ensure appeals processes are trator should not be expected to also fll the role of
implemented for applications that are denied, privacy ofcer for the NSDS. A separate, designat-
for privacy concerns raised after a project is ed NSDS ofcial should be appointed to enforce
approved, and for administrative sanctions for the full breadth of privacy-protective approaches
non-compliance with applicable policies and necessary and expected for the level of data use
procedures. in the NSDS anticipated by the Commission, the
Congress, and the President. Te designated pri-
Oversee and approve an annual report to the vacy ofcial should be given a role in scrutinizing
Congress and the President providing appropri- all data linkage activities, data access activities,
ate performance indicators, evaluating progress and other data uses for evidence building.
toward fulflling NSDSs purpose, and describ- A designated privacy ofcial at the NSDS could:
ing implementation challenges including those (1) support OMB in privacy standard setting ac-
that may require new statutory authority. tivities; (2) establish and promulgate NSDS priva-
cy policies, including for researcher qualifcation,
Receive and review audit reports on priva- project approval, and data stewardship training;
cy-protective protocols and practices; identify (3) ensure NSDS compliance with Federal privacy
any ameliorating steps to be taken to reduce laws, regulations, and policies; (4) develop man-
privacy risk within the mission of NSDS. datory privacy and data stewardship training for
NSDS staf and researchers; (5) review risk assess-
While a representative steering committee ments when confdential data are combined across
can provide high-level direction and support for departments; and (6) communicate and coordi-
NSDS, it would not be feasible or desirable for nate with privacy ofcers in statistical agencies
such a committee to manage the day-to-day op- and departments.
erations and provide leadership for the NSDS. Te
Commission believes there should be a senior ad- REC. 4-3: To ensure exemplary trans-
ministrator empowered to administer, oversee, parency and accountability for the
and coordinate the activities of the NSDS, such as Federal governments use of data for evi-
implementing regulations and developing proce-
dence building, the NSDS should maintain
dures to acquire access to data from Federal de-
partments and state agencies. a searchable inventory of approved projects
As part of the public participation process, the using confdential data and undergo regular
administrator should ensure that individual proj- auditing of compliance with rules govern-
ects are approved in consultation with data provid- ing privacy, confdentiality, and access.
ers. Information about those projects will then be
made available as described in Recommendation Te Commission believes that advancing beyond
4-3, and will include who the eligible researcher the status quo and achieving unparalleled trans-
is and what data they are using for what purpose. parency means (1) telling the public about how
Because the American public expects the Fed- government data are used for evidence building
eral government to manage the risks of using data and (2) regularly auditing adherence to priva-
for evidence building, the administrator must also cy-protecting policies to assure the public that the
prescribe data access control methods consistent Federal evidence-building community only uses
with OMB policies and standards, including estab- data for approved exclusively statistical purposes.
lishing data use agreements, training for staf and Te President, through collaboration between
researchers, and determining penalties for inap- the NSDS and OMB, in its government-wide
The Promise of Evidence-Based Policymaking 85

standard setting role, should use existing statutory mitigate those cause(s), and a process for revising
authority to establish a clear example of transpar- procedures to reduce the risk of similar future in-
ency in Federal evidence building. Te President cidents.
should authorize a single internet-based portal Finally, NSDS should report annually to the
that gives details about each project using integrat- Congress, the President, and the public about
ed confdential data for evidence building or each uses of confdential data for evidence building
external researcher accessing confdential data for and identify and discuss any problems that arise.
evidence building. Te NSDS is the logical place to Other accountability mechanisms are established
develop, host, and maintain the transparency por- through other Commission recommendations; for
tal because of its role in combining data across the example, Recommendation 2-8 calls for a single,
topical domains of Federal departments. streamlined process for external researcher access
Approved projects facilitated by NSDS will be an that is transparent about data sensitivity and sets
important input to the portal. In addition, other data access restrictions appropriately, and Rec-
PSAs should provide NSDS details on projects they ommendation 3-1 provides for the public posting
undertake with other agencies to link confden- and use of risk assessments for public release of
tial data as well as approved projects for external de-identifed confdential data.
researcher access to confdential data. Te infor-
mation in the portal should minimally include for REC. 4-4: Te NSDS should have spe-
each approved project: the project name (which
cifc administrative and implementa-
some Federal agencies already publish), a project
abstract, the datasets involved, the products of the tion fexibilities including the ability to
analyses, and the name(s) and contact information leverage public-private partnerships and to
of the approved researcher(s). Te transparency collect and retain user fees.
portal should also feature the transparency strate-
gies discussed as part of Recommendation 3-1 and In order to implement the NSDS in a manner that
could include those discussed under Recommen- allows it to serve the government as a whole and
dation 2-8. Tese strategies include a public inven- continually adapt to emerging needs and chang-
tory of data available for evidence, including an ing technology, some administrative fexibility
analysis of the sensitivity of the data and publicly beyond the usual government practices will be
available risk assessments for the public release of required. Specifcally, success for the NSDS will
de-identifed confdential data under the Privacy require legal authority for grant-making, cooper-
Act and CIPSEA. Te portal also should include a ative agreements, workforce development, and
mechanism for public feedback. other activities that facilitate engagement with
Once established, the NSDS transparency por- partner organizations to accelerate the develop-
tal will provide a valuable resource for the public ment of new methods and technologies. Such
to understand how data are being used and doc- authority could include the ability to sponsor
umentation to establish an audit trail. Te Con- a Federally Funded Research and Development
gress and the President should choose to assign Center to support research and development re-
responsibility for auditing to the Government lated to innovative privacy-protective approaches.
Accountability Ofce or to a designated inspector By encouraging research and development in pri-
general. Auditors should ensure compliance with vacy-enhancing and secure access technologies,
privacy protocols and ensure that enforcement the NSDS can both capitalize on and encourage
mechanisms are appropriately applied. Further, innovation in the private sector with joint ben-
auditors should be responsible for reviewing the efts to government. Development of new tech-
data use and handling procedures and processes nologies related to providing data access, such as
in place for confdential data. Te NSDS should es- virtual data access technologies, will help address
tablish and auditors should monitor a near miss increased demand.
reporting system for procedural or process viola- Te NSDS should have the authority to collect
tions to create a feedback and learning cycle about and spend user fees, with sufcient fexibility to
risks. Such a system could include descriptions of adjust rates based on changes in demand or other
any violations and re-identifcations that may oc- factors. Explicit authority should be provided in
cur, determinations of cause(s), actions taken to appropriations bills for other Federal departments
86 The Promise of Evidence-Based Policymaking

to transfer funds to the NSDS to support its cen- secure environment. Te technical documentation
tralized activities, as noted in Chapter 5. requirements of new technologies for securely ac-
Te NSDS can provide a clear public beneft cessing data such as Secure Multiparty Computa-
by facilitating generation of evidence about gov- tion further guide the Commissions conclusion
ernment programs and policies. It will likely not that improvements to the governments data in-
be feasible, however, for the NSDS to cover its ventory, metadata, and defnitions are necessary
expenses entirely with user fees. Terefore, the precursors to increased evidence generation.
Commission believes that while user fees for OMBs ongoing efort beginning in 2013 to de-
some costs associated with the NSDS can support velop a data inventory is a productive start, but
self-funding, other funding mechanisms will be much more needs to be done to make this infor-
necessary to achieve the Commissions vision. mation complete and useful for evidence building.
Such mechanisms could include a direct appropri- Te establishment of a comprehensive searchable
ation or reimbursable funding from other depart- inventory, through which the public can learn
ments. Te NSDS will need some level of fexible about the data that government collects, would
resources to ensure it can implement cutting-edge improve transparency. With robust metadata, re-
technologies that enhance privacy. searchers inside and outside government will be
better able to identify which data are needed and
REC. 4-5: Te Ofce of Management useful for answering policy questions, conduct-
ing program evaluations, and reducing inefcient
and Budget should increase eforts to
and unnecessary data requests. Tese metadata
make information available on existing Fed- should be sufcient to allow users to assess quali-
eral datasets including data inventories, ty and implement privacy-enhancing data match-
metadata, and data documentation in a ing technologies, such as Secure Multiparty Com-
searchable format. putation. Te metadata must also allow the NSDS
and OMB to assess the sensitivity of data in spe-
Administrative datasets often contain extensive cifc contexts and to determine access restrictions
records about individuals and businesses receiv- and privacy protection protocols as described in
ing government benefts and services, sometimes Recommendation 2-8.
including multiple entries on transactions. Given Te Commission recognizes that, despite their
their size, frequent updates and revisions, and potential for adding value, developing a data in-
data structures, use of administrative data for ev- ventory, technical documentation, and consis-
idence building requires knowing which data are tent defnitions for key concepts across datasets
available and what they mean. is a burden and challenge. In the short term, the
Te Commission intends that while data will be Commission recommends prioritizing metadata
accessible through the NSDS, most of those data and other technical documentation related to the
will remain in their original locations in the fu- datasets researchers consider important for evi-
ture. Te NSDS will only maintain a minimal core dence building, including those described in Ap-
set of data for linking confdential data within its pendix D.
The Promise of Evidence-Based Policymaking 87

5
Strengthening the Evidence-Building
Capacity within the Federal Government

Build and maintain a strong Federal infrastructure for the sustained production and
use of evidence.

Recommendations
5-1: Te President should direct Federal ments, including through any reorganization or
departments to increase capacity for evi- consolidation within OMB that may be neces-
dence building through the identifcation or es- sary and by bolstering the visibility and role of
tablishment of a Chief Evaluation Ofcer, in ad- interagency councils.
dition to needed authorities to build a high
performing evidence-building workforce. 5-4: Te Congress and the President
should align administrative processes to
5-2: Te Congress and the President support evidence building, in particular by
should direct Federal departments to de- streamlining the approval processes for new
velop multi-year learning agendas that support data collections and using existing fexibilities
the generation and use of evidence. in procurement policy.

5-3: Te Congress and the President should 5-5: Te Congress and the President
direct the Ofce of Management and Bud- should ensure sufcient resources to sup-
get (OMB) to coordinate the Federal govern- port evidence-building activities about Federal
ments evidence-building activities across depart- government programs and policies.
88 The Promise of Evidence-Based Policymaking

I mplementing the features of the Commissions


vision to improve the use of data for evidence
building requires more than just improving access
maximize evidence building, Federal departments
must have the capacity to support the full range
of analytic approaches required for evidence
to data and privacy protections. Government must building, including the development of statistics,
have the capacity to analyze data, and then apply evaluation, and policy research. Tese functions
insights to inform policymaking. Tis chapter ad- must be operational, appropriately resourced, and
dresses the alignment and empowerment of the well-coordinated both within and across depart-
Federal evidence-building communityincluding ments. Strong leadership within government that
those who generate, manage, and analyze data, prioritizes evidence building and creates the de-
those who transform information into evidence, mand for evidence is vital for institutionalizing
and those who support those functions through these functions. Without a strong institutional
the routine processes of government. foundation, other recommendations related to
improving data access, establishing the National
Data are necessary, but not Secure Data Service (NSDS), and implementing
sufcient, to create evidence. enhanced privacy protections will not have the
comprehensive impact that is needed.
Naomi Goldstein, Deputy Assistant
Secretary for Planning, Research, and
Evaluation in the Administration for Children Findings
and Families, Commission Meeting, Te Commission received input from numerous
November 4, 2016 stakeholders, both internal and external to the
Federal government, identifying challenges to in-
Today, evidence building takes place unevenly creasing the capacity of the Federal evidence-build-
across the government, both within the internal ing community and numerous ways in which the
Federal evidence-building community and across practices of the Federal government could be
partnerships with external members of the evi- streamlined to increase the volume, quality, and
dence building community. Within departments timeliness of evidence production. Te collective
inside the Federal government, key actors include input stressed the importance of addressing these
Principal Statistical Agencies (PSAs) and other sta- challenges directly. Based on this feedback, the
tistical programs, evaluation and policy research Commission identifed fve primary challenges
ofces, program administrators, performance that currently restrict the capacity of the Federal
management ofces, policy analysis ofces, and governments evidence-building community.
privacy ofces. Often the work of these ofces
occurs in silos, leading to duplication of efort or Challenge #1: Te capacity of Federal de-
missed opportunities for collaboration, and eforts
partments to support the full range of ev-
to coordinate evidence building across the Federal
government have been challenging. In addition, idence-building functions is uneven, and
administrative functions that support Federal where capacity for evidence building does
government operations across departments are exist, it is often poorly coordinated within
poorly aligned to support and prioritize evidence departments.
building. Tus, major gaps remain in the Federal
evidence-building communitys capacity, which Currently, some departments operate all, and
hamper the ability to build on the communitys other departments operate only some, of the ev-
many strengths to increase evidence production. idence-building functions necessary to support
Federal departments must enhance their ca- evidence-based policymaking. Where these ac-
pacity for evidence building to support the growth tivities are taking place in diferent units across a
of evidence-based policymaking. Tose generat- department, these eforts must also be coordinat-
ing, supporting, and using evidence within and ed to maximize the departments capacity to ful-
across departments throughout the Federal gov- ly address a specifc research or policy question.
ernment must be empowered and organized to In pursuit of more and better evidence, depart-
work together and accomplish shared goals. To ments may choose to pursue diferent avenues for
The Promise of Evidence-Based Policymaking 89

increasing their capacity for evidence building. In Organizational culture is an


some cases, departments will opt to strengthen
their human capital by hiring staf or establish- ongoing dialogue between leaders
ing units with particular expertise. In other cases, and others in their organization
departments may decide that leveraging exper- about how we do things around
tise outside of government is a more efcient ap-
proach to expanding capacity. here. Data and evidence are
Regardless of how a department may pur- necessary but not sufcient.
sue expanded capacity, the coordination of ev- Continuing dialogue based on the
idence-building functions is vital for achieving
maximum efciency. Currently, some departments data and evidence is critical.
prioritize coordination across units while other de- Seth Harris, Former U.S. Department of
partments lack a coordinating infrastructure. Te Labor Acting Secretary, Commission Meeting,
following section (1) describes each of the evi- March 13, 2017
dence-building functions needed within Federal
departments, (2) ofers existing approaches for ex- Te evaluation and policy research functions
panding the evidence-building capacity of depart- are newer additions to the evidence-building
ments through human capital strategies and exter- community in many departments. Te American
nal partnership arrangements, and (3) addresses Evaluation Association defnes evaluation as as-
the importance of coordinating evidence-building sessing the strengths and weaknesses of individ-
eforts within a Federal department. ual programs, policies, and organizations to im-
prove their efectiveness, efciency, and worth
Evidence-Building Functions within It uses systematic data collection and analysis to
address questions about how well government
Federal Departments
programs and policies are working, whether they
Key actors within the Federal evidence-building are achieving their objectives, and, no less impor-
community include: the PSAs and other statistical tantly, why they are or are not efective. 1 Policy
programs, evaluation and policy research ofces, research is related to evaluation and often pre-
program administrators, performance manage- cedes formal evaluation, but plays a fundamen-
ment ofces, and policy analysis ofces. Tese are tally diferent role in evidence building.2 Because
central entities that support an organizational cul- these functions are less institutionalized in many
ture that enables evidence building. Federal departments, they require diferent solu-
Tirteen PSAs form the centerpiece of the larg- tions to bolster their development compared with
er Federal Statistical System, which also includes longer standing functions such as the develop-
dozens of statistical programs spread across Fed- ment of statistics. While some excellent examples
eral departments, embedded within programmat- of evaluation and policy research ofces exist in
ic agencies. For many decades, the PSAs have col- the Federal government today, many departments
lected most of their data by conducting surveys of will likely need additional capacity to meet the
individuals, businesses, and other organizations. Commissions vision of cultivating these two es-
PSAs now increasingly supplement these surveys pecially important types of evidence building.
with administrative data, creating new opportu-
nities, and also new relationships with program
agencies. Because of their existing legal conf- 1. American Evaluation Association, An Evaluation Roadmap for
dentiality protections, culture of data protection, a More Efective Government (Washington, D.C.: AEA, October
2013); http://www.eval.org/d/do/472 (accessed August 10, 2017).
and substantive and technical expertise, PSAs can
serve as building blocks for the modern system of 2. Research includes basic and applied activities conducted in the
pursuit of new knowledge, the former directed at general appli-
evidence building in the United States. Te grow- cations or theory and the latter focused on practical objectives.
ing capabilities of many PSAs to transform and an- Policy research is used synonymously with policy-relevant re-
alyze administrative data should be expanded to search both of which are considered gaining knowledge about
programs and policies. See OMB, Section 84: Character Classifca-
meet the needs for government evidence building tion (Schedule C) in Preparation, Submission, and Execution of
and the public good. the Budget (Circular A11, Washington, D.C.: OMB, July 2016).
90 The Promise of Evidence-Based Policymaking

Program administrators play an increasingly tions, geographies, or contexts.5


important role in evidence-building activities, as To the extent that capacity is uneven in gov-
more administrative data collected for program ernment, important evidence-building activities
administration purposes are being transformed may be crowded out by other priorities within
for use in evidence building. In addition, program departments. Departments face numerous pres-
administrators are one audience for the evidence sures to respond to immediate and important
being generated, as they have the capacity to ad- requests that limit the ability to deploy staf or
ministratively implement program and policy ad- funding for evidence building. For example, pri-
justments based on the results of research and oritizing funding for new programs and initia-
evaluation. tives may mean longstanding programs receive
Performance management ofces are responsi- less attention for measuring or assessing program
ble for advising and assisting departmental lead- outcomes. Program administrators whose perfor-
ership to ensure that the mission and goals are mance is assessed based on their meeting annual
achieved cost efectively through strategic and targets may be reluctant to devote program re-
performance planning, measurement, analysis, sources to longer-term evidence-building activ-
regular assessment of progress, and the use of ities. In most departments, the development of
high-quality performance information and other annual performance measures aimed at provid-
evidence to improve results.3 ing timely information for decisions in the bud-
Centralized policy analysis ofces support get process itself consumes an array of resources.
many departments today and typically perform While performance measurement (or monitoring)
an important translation function. Policy analy- is an important component of the broader feld of
sis translates knowledge gained from evaluation evaluation,6 a focus on short-term activities may
and policy research both inside and outside gov- jeopardize the ability to prioritize long-term proj-
ernment, making the information accessible to ects that do not align with the time horizons of
program administrators and operators, policy- policymakers and program administrators. Te
makers, including the Congress, and the public. capacity must exist for both short- and long-term
Te dissemination of information helps inform activities.
and spur action. Tese knowledge brokers serve
as intermediaries between evidence users and Mechanisms for Expanding Capacity
producers, and play an important role within
for Evidence Building in Federal
the evidence-building community.4 Increasingly,
as a greater volume of evidence is produced in Departments
the future, knowledge brokering eforts also will Strengthening the program evaluation function
need to expand. Policy analysis ofces sometimes within the Federal government is an important
sponsor or rely on formal dissemination tools for frst step in expanding evidence building. Sever-
evidence, such as the What Works Clearinghouse al strong evaluation units currently exist across
operated by the U.S. Department of Education, government and work with contractors, other
and the Clearinghouse for Labor Evaluation and government staf, and academic institutions to
Research, operated by the U.S. Department of implement governments evaluation function.
Labor, which ofer compilations of research and Such ofces include the Ofce of Planning, Re-
evaluation to provide grantees with a valuable search, and Evaluation (OPRE) within the Admin-
starting point for determining which interven- istration for Children and Families (ACF) of the
tions may be most appropriate to their popula- U.S. Department of Health and Human Services
(HHS) and the Chief Evaluation Ofce within the

3. OMB, Overview of Federal Evidence-Building Eforts (white


paper for the Commission on Evidence-Based Policymaking, 5. What Works Clearinghouse; https://ies.ed.gov/ncee/wwc; Clear-
Washington, D.C.: OMB, 2016); https://obamawhitehouse.archives. inghouse for Labor Evaluation and Research; https://clear.dol.gov
gov/sites/default/fles/omb/mgmt-gpra/overview_of_federal_evidence_ (both accessed August 10, 2017).
building_eforts.pdf (accessed August 10, 2017).
6. Kathryn Newcomer and Clinton T. Brass, Forging a Strategic
4. Karol Olejniczak, Estelle Raimondo, and Tomasz Kupiec, Eval- and Comprehensive Approach to Evaluation Within Public and
uation Units as Knowledge Brokers: Testing and Calibrating an In- Nonproft Organizations, American Journal of Evaluation 37, no. 1
novative Framework, Evaluation 22, no. 2 (April 2016): 168189. (March 2015): 8099.
The Promise of Evidence-Based Policymaking 91

Department of Labor (DOL). (See the box U.S. To both support and beneft from routine ev-
Department of LaborChief Evaluation Ofcer). idence building, Federal departments require a
OMB supports evaluation activities across govern- range of expertise including in-house expertise
ment and encouraged the administrative creation to design and conduct evaluations or surveys, ad-
of a voluntary coordinating unit called the Inter- vanced technical knowledge required for combin-
agency Council on Evaluation Policy. ing or analyzing data, and expertise in writing and
As noted in the Analytical Perspectives sec- managing highly technical evidence-building con-
tion of the Presidents Fiscal Year 2018 Budget, tracts. In seeking to increase capacity for evidence
Centralized or chief evaluation ofces play an building, some Federal departments may choose
important role in an evidence infrastructure that to establish new units or expand the functions of
can develop and sustain agency capacity to build an existing unit. Te CEP Survey of Federal Ofces
and use evidence. 7 In this vein, the Commission showed that over half (54 percent) of responding
received testimony suggesting that a Chief Eval- ofces reported difculty hiring staf with the ap-
uation Ofcer could assume responsibility for (1) propriate skill set to match the work requirements
establishing department-wide evaluation and re- for evidence-building activities.8 Functions that
search policies that encourage rigor, credibility, are newer in government, such as program eval-
independence, and transparency; (2) coordinat- uation, have particular challenges in attracting
ing and supporting technical expertise for evalua- and retaining qualifed staf and, therefore, may
tion and research within the department; and (3) require some adjustments in existing Federal hu-
identifying and setting priorities for departmental man resource policies. Today, while the U.S. Ofce
program evaluation and policy research, with ap- of Personnel Management (OPM) recognizes stat-
propriate attention to the mission and context of isticians, economists, information specialists, and
each department. Particularly in very large or de- policy analysts as ofcial and distinct occupations
centralized organizations, departments may have within government, no occupational series exists
multiple centers for evaluation, which could be specifcally for the feld of program evaluation.
coordinated by a Chief Evaluation Ofcer. External partnerships also can be an efective
way to expand government capacity for production
7. OMB, Building and Using Evidence To Improve Government
Efectiveness, in Analytical Perspectives, Budget of the United States
Fiscal Year 2018 (Washington, D.C.: OMB, May 2017); https://www. 8. Based on 79 ofces that reported that they collect or use data
whitehouse.gov/sites/whitehouse.gov/fles/omb/budget/fy2018/ap_6_ for statistics, evaluation, research, or policy analysis or spend a
evidence.pdf (accessed August 10, 2017). portion of their budget for such purposes.

U.S. Department of LaborChief Evaluation Ofcer


Te U.S. Department of Labor (DOL) is cur- ects designed to build evidence about DOL
rently the only Cabinet-level department to programs and policies.
have a Chief Evaluation Ofcer. Established in
2. Complements, not duplicates, agency eval-
2010, the Chief Evaluation Ofce, led by the
uation functions.
Chief Evaluation Ofcer, coordinates, manag-
es, and implements DOLs evaluation program. 3. Raises the quality of evaluations and aware-
Te Chief Evaluation Ofce is an independent ness and knowledge of evaluation method-
evaluation ofce, located organizationally in ology and standards.
the Ofce of the Assistant Secretary for Policy. 4. Improves use and dissemination of evalua-
At the department level, the Chief Evaluation tion results.
Ofcer:
5. Improves access to, quality of, and use of
1. Designs, funds, and oversees the imple- data.
mentation of a portfolio of evaluation proj-
92 The Promise of Evidence-Based Policymaking

of evidence by leveraging the expertise of the expand staf capacity and build staf expertise.
non-Federal evidence-building community. Te Federal departments have mechanisms for hir-
Robert Wood Johnson Foundation encouraged ing staf from colleges and universities and oth-
the Commission to consider how partnerships er institutions on a temporary basis, such as the
between researchers, service providers, and large Intergovernment Personnel Act (IPA) Mobility
institutionsboth governmental and corporate Program. Assignments under the IPA program are
can improve the capacity to develop evidence. 9 intended to facilitate cooperation between the
Engaging successfully with external partners re- Federal government and the non-Federal entity
lies on having staf and structures in place to es- and to serve a sound public purpose. 10
tablish and support collaboration with academics,
non-profts, foundations, and businesses. While Coordination of Evidence-Building
tools like grant-making or cooperative agreements
Functions within Federal Departments
have been used to good efect at some agencies to
build external partnerships, not all agencies have Currently, the coordination of evidence-building
the necessary authority to use these tools for part- activities within departments varies widely. In
nerships. some departments, very little coordination exists,
Partnerships can be especially useful for small- whereas other departments place a high priority
er agencies with limited capacity. Agencies must on coordination. For example, HHS supports an
continually develop their workforce and increas- entire unit, the Ofce of the Assistant Secretary
ingly need staf with specialized expertise in pri- for Planning and Evaluation (ASPE), which is re-
vacy protections, data security, computer science, sponsible for major activities in policy coordina-
and statistical methods. Partnering with non- tion, legislation development, strategic planning,
governmental entities can be an efcient way to
10. Ofce of Personnel Management, Hiring Information: In-
tergovernment Personnel Act; https://www.opm.gov/policy-data-
9. Robert Wood Johnson Foundation, submission to the Commis- oversight/hiring-information/intergovernment-personnel-act (ac-
sions Request for Comments. cessed August 10, 2017).

Case Study: HHS Data Council


Te Data Council within the U.S. Department of data standards and statistical policies, and pri-
Health and Human Services (HHS) is an exam- vacy protection policies. An important activity
ple of successful coordination of data resources of the Data Council includes leveraging HHS
within a Federal department.1 Te HHS Data statistical and administrative data collections
Council is the principal internal advisory body to support evidence building. Te HHS Data
to the Secretary for health and human services Council develops recommendations regarding
data and statistical policy, and facilitates collab- the collection, analysis, and dissemination of
oration and coordination among various ofces data to guide future decisions and enhance the
involved in evidence building, meeting month- health and well-being of Americans.
ly to discuss department-wide issues related to Te Council is co-chaired by the Assistant
data and statistical policy. Te HHS Data Coun- Secretary for Planning and Evaluation (ASPE)
cil plays an essential role leading the develop- or their designee, the head of an operating divi-
ment and implementation of an integrated data sion or their designee (currently the Director of
collection strategy, coordination of analysis ac- the Agency for Healthcare Research and Quali-
tivities, development and implementation of ty), and the Director of the National Center for
Health Statistics, who also serves as the senior
advisor to the Secretary on health statistics.
1. U.S. Department of Health and Human Services, HHS
Data Council: An Introduction; https://aspe.hhs.gov/hhs-data- Te ofce of the ASPE serves as Executive Di-
council-introduction (accessed August 10, 2017). rector for the Council.
The Promise of Evidence-Based Policymaking 93

policy research, evaluation, and economic anal- cision-making. 11 Such learning agendas typically
ysis across the entire department. (See the box identify the most pressing research and policy
Case Study: HHS Data Council). questions facing a department at a particular time,
Harnessing administrative data as a more inte- and provide departmental leaders with a mecha-
gral part of evidence building heightens the need nism to prioritize research questions within bud-
for meaningful collaboration within departments. get and policy timeframes.
Coordination within a department is vital for the Te Ofce of Policy Development and Research
efcient production of evidence. within the U.S. Department of Housing and Ur-
ban Development (HUD) developed a learning
agenda titled the Research Roadmap. 12 Te Re-
Challenge #2: Federal departments fre-
search Roadmap outlines the key research ques-
quently do not have an integrated approach tions relevant to HUDs mission, and enables HUD
or a long-range plan for evidence building. leadership to be responsive to OMBs direction to
strengthen the use of evidence and technology to
To make evidence-based policymaking routine in improve government performance. HUD refreshes
the Federal government, a broad base of evidence the Research Roadmap every fve years, engaging a
and information must be available in order to in- broad stakeholder group both inside and outside of
form decision-making. Departments can beneft the department, to ensure that the priorities cap-
from an organized approach by which they regular- tured refect the needs of the department and the
ly identify short- and long-term priority research broader feld.Learning agendas also can be used to
and policy questions relevant to the departments communicate research priorities to external part-
mission and legal responsibilities. Te result of ners to help catalyze targeted evidence-building ac-
this exercise is a learning agenda, which can be tivities outside the Federal government.
used by leadership to prioritize the set of research
and policy questions to be pursued by the depart- A Portfolio of Evidence
ment over a given period. Te evidence-building
community within a department can in turn use Evidence building and use must become a regu-
the learning agenda as a coordination tool. lar feature of program oversight and operations to
Te efective implementation of learning agen- promote continual learning, program refnement,
das requires sustained leadership and support. and accountability. Today, nearly every Feder-
Te various units within the evidence-building al government program collects information to
community in a department must work togeth- support program operations and monitor perfor-
er to determine the best approach to answering mance. Te data gathered through these activi-
a priority research or policy question, and to al- ties provide valuable basic information about the
locate the work appropriately across the diferent scope and reach of programs, such as identifying
evidence-building functions. A well-coordinated the number of participants in a program. Depart-
evidence-building community within a Federal ments generally need more in-depth information
department will be able to leverage the various to understand how a program is implemented,
strengths of each function, resulting in a more ro- the costs of administering the program, and the
bust project. extent to which policies yield anticipated results.
Policy decisions should be reviewed using a broad
Learning Agendas set of methodologies, including descriptive statis-
tics, process studies, implementation evaluations,
Te development of a learning agenda is a con-
crete step that some Federal departments have 11. U.S. Agency for International Development, Implementing
taken towards prioritizing evidence building. a Learning Agenda Approach; https://usaidlearninglab.org/sites/
Learning agendas are defned as a set of broad default/fles/resource/fles/defning_a_learning_agenda.pdf (accessed
August 10, 2017).
questions directly related to the work an agency
conducts that, when answered, enable the agency 12. U.S. Department of Housing and Urban Development (HUD),
HUD Research Roadmap: 2017 Update (Washington, D.C., January
to work more efectively and efciently, particu- 2017); https://www.huduser.gov/portal/pdf/ResearchRoadmap-
larly pertaining to evaluation, evidence, and de- 2017Update.pdf (accessed August 10, 2017).
94 The Promise of Evidence-Based Policymaking

impact evaluations, using randomized control tri- Te Commission identifed numerous exam-
als where appropriate, and meta-analysis. Te re- ples of Federal programs developing increasingly
sults of all relevant evidence-building activities is rigorous portfolios of evidence, but these pro-
referred to as a portfolio of evidence. grams are still the exception, rather than the rule.
Te American Evaluation Association suggests One example is the Teen Pregnancy Prevention
that in selecting a methodological approach, the Program administered by HHS, which was de-
frst step is to identify the important evaluation signed to address high teen pregnancy rates in
questions [needing to be] answered to efective- the United States by replicating evidence-based
ly direct the future of Federal programs allowing models and testing innovative strategies. Evi-
evaluators to then identify which scientifc meth- dence building was woven into the program from
ods are best suited to answer those questions. 13 the start, including a full range of studies from
Te Commission fnds that the need for evidence implementation assessments to impact evalua-
about programs is context specifc and should be tions, using random assignment when appropri-
tailored to produce the most relevant, valid, and ate. Te program is administered by the Ofce of
reliable information possible for evaluating indi- Adolescent Health, which used a multipronged
vidual programs and policies. approach to evaluation, including the establish-
Two of the Commissions guiding principles for ment of uniform performance measures, a series
evidence-based policymaking outlined in Chapter 1 of grantee-led evaluations, and multiple cross-site
are particularly relevant when considering both the evaluations. Te purpose of this framework was
generation of evidence and the application of evi- to address the question of whether the replicated
dence by policymakers. Te principle of rigor states evidence-based teen pregnancy prevention pro-
evidence should be developed using well-designed grams and the new, innovative strategies for pre-
and well-implemented methods tailored to the venting teen pregnancy were efective.
question being asked. For example, when seeking Evelyn Kappeler, former Director of the Ofce
to determine program impacts, random assignment of Adolescent Health at HHS, described the pro-
generally is preferable for developing causal state- gram as follows:
ments. Te Institute of Education Sciences and the
Directorate for Education and Human Resources Our experience with the frst cohort
of the National Science Foundation established of [Teen Pregnancy Prevention grant-
guidelines for evaluation that conclude Efcacy, ees] brings to light several important
efectiveness, and scale-up research should use issues regarding the capacity of the
study designs that will yield impact estimates with organizations to conduct rigorous
strong causal validity . . . generally and when feasi- evaluations. Many of these lessons
ble, they should use designs in which the treatment have applications for government
and comparison groups are randomly assigned. 14 entities, which are conducting evalua-
In another principle adopted by the Commis- tions in community settings. Te chal-
sion, policymakers are urged to approach evidence lenges and lessons learned include the
with humility. Te Commissions humility princi- importance of providing for a plan-
ple says Care should be taken not to over-gener- ning and piloting period early in the
alize from fndings that may be specifc to a partic- grant cycle, measuring and monitor-
ular study or context. Even when evaluated using ing fdelity and adaptations, ensuring
methods appropriate to stated questions, contexts high quality program implementation,
may difer, circumstances can change, and fdelity incorporating evaluation efectively
to program design may diminish over time while once program implementation has
conducting an evaluation. already begun, ensuring strong con-
trast between treatment and control,
providing intensive evaluation tech-
13. George Grob, American Evaluation Association, Commission nical assistance, and disseminating all
Public Hearing, Washington, D.C., October 21, 2016.
evaluation results transparently.15
14. Institute of Education Sciences, U.S. Department of Education,
and the National Science Foundation, Common Guidelines for Edu-
cation Research and Development (August 2013); https://ies.ed.gov/ 15. Evelyn Kappeler, Ofce of Adolescent Health, Commission
pdf/CommonGuidelines.pdf (accessed August 10, 2017). Meeting, Washington D.C., November 4, 2016.
The Promise of Evidence-Based Policymaking 95

Te challenges cited with regard to the Teen evaluation and policy research within program
Pregnancy Prevention program demonstrate the design. One challenge is that programs, policies,
complexity of program evaluation. Tese challenges and regulations are generally designed without in-
can make it difcult to keep the attention of leader- volving the evidence-building community. Engag-
ship on evidence building over time. Even with an ing the evidence-building community as partners
appropriate organizational structure and sufcient during the program design process will help en-
staf capacity, Federal departments still may not sure that the program can be rigorously evaluated
prioritize evidence building among the competing and that research questions will meet the needs
demands on their resources without the right set of of program managers and policymakers. Barriers
incentives and expectations. As the quote from Ms. to early collaboration between evaluators and pro-
Kappeler suggests, developing and sustaining Fed- gram designers include limited evaluation capac-
eral leadership in support of evidence building is a ity and expertise, program ofces concerns about
formidable undertaking. Senior department leaders how evaluation results will be used, and lack of
play an important role in making the generation and access to quality data. Several public comments
use of evidence routine in Federal policymaking. submitted to the Commission highlighted core el-
Today, Federal departments lack individual ements that must be incorporated into program
incentives for senior leaders to produce and use design, including the following:17
evaluation and policy-relevant research. While
the Commission does not ofer recommendations Clearly stated goals, objectives, logic, outputs,
to address individual motivation, it learned that and desired outcomes.
departmental approaches could include incorpo-
rating measures into the performance reviews of Recognition of roles for diferent types of
career senior leaders that refect their efective- analyses based on program, policy, or regula-
ness in producing and using evidence. Similarly, tion stage and development (e.g., prospective,
the Senate could use the confrmation process to pilots, implementation, retrospective, system-
seek afrmative responses regarding a political atic reviews/meta-analysis).
appointees support for producing and using evi-
dence to inform decision-making. Incentives for evidence production.
Te political process can itself introduce uncer-
tainties about the value of evidence and its use in Sufcient legislative authority or fexibility to
policymaking. A report from the Bipartisan Policy enable evaluation, including mechanisms to
Center observes that the fragmented nature of encourage innovation and to conduct impact
the legislative process, the clash of partisan inter- evaluations, using random assignment when
ests, and the labyrinth of legislative procedures appropriate.
and practices make it more difcult to establish a
coordinated structure for using research and eval- Incorporation of data collection needs and
uations in a more systematic fashion in the leg- requirements; a focus on data quality to
islative process.16 Establishing clear expectations ensure meaningful and useful information is
through reauthorizations of programs, develop- gathered in order to assess program costs and
ment of regulations, the budgeting process, and outcomes.
other routine interactions between the President,
the Congress, and Federal departments will rein- Nearly all programs collect some form of ad-
force the demand for more and better evidence. ministrative data about program benefciaries
or the services ofered. How programs use that
Program Design information, however, varies greatly. As the ev-
idence-building community increasingly relies
Te statute that established the Commissions on those data, programs and evidence building
charge includes multiple references to embedding
17. Based on comments submitted to the Commission by J-PAL
16. Bipartisan Policy Center, Congress and Evidence-Based Policy- North America, Laura and John Arnold Foundation, and American
making: Creating a 21st Century Legislature (Washington, D.C.: Bi- Evaluation Association; https://www.cep.gov/library/testimony.html
partisan Policy Center, 2017): 13. (accessed August 10, 2017).
96 The Promise of Evidence-Based Policymaking

become increasingly interdependent. Te infor- policy through required transactional reviews of


mation collected by programs must be analyzed or agency information collection requests, systems
made available to those who can interpret the data of record notices, and other related reporting
in a manner that does not compete with, but rath- and by chairing interagency councils, such as the
er enhances, programs ability to implement day- Privacy Council and the Interagency Council on
to-day operations and achieve their intended goals Statistical Policy. Te law also requires OIRA to en-
in the most cost efective manner. Leadership that sure the sufciency of budget proposals for statis-
recognizes the value of evaluation and other evi- tical activities. Tese responsibilities coexist with
dence building, and the role of administrative data OIRAs regulatory review and coordination func-
in those activities, can ensure that this approach tion, which accounts for the majority of OIRAs
becomes the norm. Te Commission fnds that ex- resources. In addition to OIRA, OMB also houses
pectations for evaluation established by the Con- several other statutorily created ofces that play a
gress and the President can promote continuous role in evidence building. Tese other ofces in-
production of valid and reliable evidence about clude the CIO, who oversees information technol-
programs and policies and their costs. ogy policies that have direct bearing on informa-
tion privacy, confdentiality, metadata and Open
Challenge #3: Te current organizational Data; the Chief Performance Ofcer, charged in
law with overseeing performance management
structure of OMB does not optimize the
activities within the Federal government; and
agencys ability to coordinate evidence the Ofce of Federal Procurement Policy, which
building across the Federal government. helps set Federal contracting policies. OMB also
houses a small team focused primarily on Federal
Federal departments often struggle to coordinate program evaluation. Tese ofces work together
evidence-building activities within their own de- with OMBs Resource Management Ofces to pri-
partments, and coordinating across departments oritize policy and resource allocations across the
can be even more challenging. Building evidence Executive Branch. Each ofce reports to a diferent
within topical silos like labor, transportation, and political appointee, and each can issue guidance
education alone misses opportunities to address or directives to agencies that enable or discourage
cross-cutting research and policy questions. For activities related to evidence building.
example, questions about the impact of housing Te capacity of OMB to efectively coordinate
assistance on health outcomes or the impact of the Federal evidence-building community has
food security on child development would nec- been complicated by the ways in which the roles
essarily rely on the integration of data resources of these ofces have evolved over time, resulting
managed by two separate Federal departments. A in confusion or inconsistent guidance for agen-
lack of coordination across government for evi- cies. For example, the PRA expressly directs that
dence building leads to unnecessary burden and OIRA facilitate government-wide information
cost from duplicative data collection, missed op- management policies.18 Te E-government Act
portunities for programmatic collaboration, and of 2002 fractured implementation responsibili-
a less robust response to a crosscutting policy or ties within OMB by establishing a separate ofce
programmatic question. charged with responsibility for all information
technology policy, overlapping with the infor-
The Critical Role of OMB mation policy function in OIRA. Perhaps more
problematic, many agency organizational struc-
OMB serves as the central coordinator for many tures developed to mirror OMBs organizational
government-wide processes, but currently, re- arrangement. As the demand for evidence to sup-
sponsibilities related to evidence building are dis- port the policymaking process continues to grow,
persed across the organization without consistent, the operational silos within OMB will likely only
sustained internal coordination. By law, the Ofce become more constraining for the timely produc-
of Information and Regulatory Afairs (OIRA) is tion of evidence across government.
the unit within OMB responsible for developing
government-wide information, privacy, confden-
tiality, and statistical policy. It implements such 18. Paperwork Reduction Act, 44 USC 3503 (1995).
The Promise of Evidence-Based Policymaking 97

Due to OMBs current organizational structure, Coordinating Evidence Building Across


departments typically need to coordinate with Federal Departments
multiple ofces within OMB before launching
new evidence-building activities. Tis structure In addition to improving the organizational struc-
places the burden of coordination on individual ture of OMB, the Commission fnds that interagen-
departments, rather than OMB, and further chal- cy coordination is vital for the efcient operation
lenges the ability to coordinate throughout the of the Federal evidence-building community (see
evidence-building community. Because OMB is the box Why Is Coordinating Evidence Building
the hub of evidence and information policy in the across Government Important?). OMB already
Federal government, the Commission fnds that convenes and participates in several interagency
fragmentation of its evidence-building functions coordination groups related to evidence building.
hampers its ability to sufciently prioritize and One key group, the Interagency Council on Statis-
coordinate evidence building. Te Commission tical Policy,19 has a statutory role in advising OMB
fnds further that ensuring OMB is structured to on the needs and policies of the statistical system.
efectively coordinate the evidence-building com-
munity is crucial for realizing the full benefts of
the Commissions recommendations. 19. Paperwork Reduction Act, 44 USC 3504(e)(8) (1995).

Why is Coordinating Evidence Building Across


Government Important?

Coordination of evidence-building activities dents attending them.VA tracks colleges using


across government is particularly important VA Facility Codes, which have no relation to
when a research or policy question spans mul- either of the two diferent methods of track-
tiple topical domains. For example, to under- ing colleges used by ED. ED ofcials have spent
stand the impact of federal student aid and years trying to build a crosswalk between the
the GI Bill on veteran educational outcomes, two separate systems within the department
one approach would be to combine data from for tracking colleges, and currently, both ap-
both the U.S. Department of Education (ED) pear in EDs Integrated Postsecondary Educa-
and the U.S. Department of Veterans Afairs tion Data System database. Crosswalking the
(VA). However, both agencies track veterans data collected by ED and VA is even more com-
who are students diferently. Specifcally, ED plicated, and is especially difcult in the case
tracks student outcomes, but cannot identify of for-proft education companies with numer-
which students in its database are veterans, in ous campus and online locations.Harmonizing
part because of a skip pattern introduced on the VA and ED data collections to permit the
EDs Free Application for Federal Student Aid tracking of colleges and the students who at-
around the same time that the Post 9/11 GI Bill tend them is important to understanding the
was enacted, causing most students not to be success of hundreds of billions of dollars of
asked if they are veterans. Te exact opposite taxpayer investment in Federal student aid and
is true at the VA. TeVA tracks which veterans the GI Bill.1
are using the GI Bill, but does not track student
outcomes, at least not robustly.
Te problem becomes more prominent
1. Carrie Woford, Veterans Education Success; Mark Schnei-
when one considers the difering methods by der, American Institutes for Research, Commission Public
which ED and VA track colleges and the stu- Hearing, Washington D.C., October 21, 2016.
98 The Promise of Evidence-Based Policymaking

Additional coordinating committees, including the eral government awarded just under a half trillion
Privacy Council, Performance Improvement Coun- dollars in contracts to entities to carry out work
cil, and Interagency Council on Evaluation Policy, on behalf of the Federal government.23 Contracts
emerged more recently.20 Tese groups provide are an important mechanism used by Federal de-
valuable communities of practice within the Fed- partments to carry out research and evaluation
eral government to support knowledge generation, activities. Across government, numerous ofces
problem solving, best practice dissemination, and often identify contract procurement as a barrier
shared services. Te Commission encourages the to the efcient production of evidence. In the CEP
continuation and prioritization of the leadership Survey of Federal Ofces, 40 percent of responses
of these Councils in order to bring maximum value identifed lack of ability to execute and manage
to the Federal evidence-building community. contracts as a barrier to their ability to use data
for evidence building, including 17 percent who
identifed this as a moderate or major barrier.24
Challenge #4: Administrative processes One challenge with contracts relates to how
are not tailored or aligned to support evi- to best structure a procurement at the outset of
dence-building activities. a project to support evidence building. Tis chal-
lenge can be addressed through better applica-
During the Commissions fact-fnding phase, both tion of existing fexible guidance related to pro-
governmental and non-governmental witnesses curement applied to research and development
identifed numerous administrative barriers within contracts.25 Te Commission fnds that existing
the Federal government that hamper the efcient regulations acknowledge that certain types of
production of evidence. In a follow-up to the Com- services, specifcally those identifed as research
mission public meeting held in November 2016, and development, may be more appropriately
the Interagency Council on Evaluation Policy sub- procured with a more fexible, yet riskier, con-
mitted a list of the top barriers to increasing Feder- tract. Te Federal Acquisition Regulation states
al evidence-building capacity, highlighting bureau- because of the importance of technical consider-
cratic barriers that discourage evaluation, create ations in research and development, the choice of
inefciencies and impose additional costs when contract type should be made after obtaining the
conducting a Federal evaluation, particularly issues recommendations of technical personnel.26 Ap-
related to PRA, interagency agreements, and pro- plying this standard consistently throughout the
curement. 21 Evidence building requires support evidence-building community holds the potential
from administrative functions, but these functions to improve the governments ability to issue con-
are not always aligned or tailored to fulfll the par- tracts that are properly structured to support evi-
ticular needs of the evidence-building community. dence-building activities.
A second challenge relates to staf familiarity
Procurement with and ability to manage contracts that support
Procurement is the process by which government evidence-building activities. Acquisition staf who
acquires supplies and services using funding ap- may be unfamiliar with contracts that are not
propriated by the Congress, including through structured as the more typical fxed-price variety
grants and contracts.22 In fscal year 2016, the Fed- and with the fexibilities available under research
and development contracts may seek to limit an
agencys preference to use a riskier contract type in
20. Te Performance Improvement Council was established support of evidence-building activities. Similarly,
by Executive Order 13450 in 2007 and was later codifed in the
GPRA Modernization Act of 2010. Te Interagency Council on
Evaluation Policy was established administratively in 2015. Te
23. For the total value of awarded contracts (as well as grants,
Privacy Council was established by executive order in 2016;
loans, and other assistance) made by the Federal government by
https://obamawhitehouse.archives.gov/the-press-ofce/2016/02/09/
fscal year, go to www.usaspending.gov.
executive-order-establishment-federal-privacy-council (accessed Au-
gust 10, 2017). 24. Includes ofces that reported that they collect or use data for
statistics, evaluation, research, or policy analysis or spend a por-
21. Interagency Council on Evaluation Policy, Comments to the
tion of their budget for such purposes.
Commission, Top-Five List of Issues and Solutions Related to
Federal Evaluation Activity, November 22, 2016. 25. Federal Acquisition Regulation, 48 C.F.R. 35.000 (2005).
22. Federal Acquisition Regulation, 48 C.F.R. 2.101 (2005). 26. Federal Acquisition Regulation, 48 C.F.R. 35.006 (2005).
The Promise of Evidence-Based Policymaking 99

technical staf responsible for overseeing the day- be administered under the collection, also must
to-day management of the contract may be unfa- be made available for two public comment periods
miliar with proper techniques for managing the of 60 and 30 days during this sequential process.
performance of a riskier contract type. Difering Completing these steps requires four months at a
goals between acquisition professionals and tech- minimum, but six to nine months is considered a
nical staf may lead to difculties in identifying more realistic time frame for approval, including
the right contract type and raise concerns about incorporation of public comments as well as nec-
the capacity of the government to properly man- essary departmental and OMB clearances.29
age the contract. During the Commissions fact-fnding phase,
members of the evidence-building community
Information Collection Reviews from both inside and outside the Federal gov-
ernment mentioned that PRA requirements are
Te PRA directs OMB to coordinate requests for burdensome and time-consuming. Te Laura
information collections initiated by Federal de- and John Arnold Foundation noted that the PRA
partments.27 Te law requires that any collection processes cause long delays and bottlenecks for
for ten or more individuals from a survey, ques- agencies seeking approval of evaluations that have
tionnaire, or form undergo a review to ensure stymied eforts by agencies to increase the num-
collected information achieves public beneft and ber of high-quality studies of important research
maximum utility. Te vast majority of data collec- questions. 30 Six of the 10 evaluation ofces that
tions undertaken by the evidence-building com- responded to the CEP Survey of Federal Ofces
munity fall within the scope of the PRA, as do noted that information collection requirements
administrative data collections undertaken, spon- acted as a moderate or major barrier to their evi-
sored, or required by Federal departments. Te dence-building activities.
purposes embodied by the PRA are important for Tere are two primary areas of concern for
ensuring that Federal information collections are those who identify the Information Collection Re-
necessary, useful, and of high quality. However, quest review and approval process as a barrier to
the Commission fnds that the current structure engaging in evidence building with maximum ef-
of the information collection review and approv- ciency. Te frst issue relates to the broad net cast
al process can be inefcient. Tese inefciencies by the PRA regarding the size of information col-
lead to problematic delays in data collection and lections requiring OMB review and approval. Te
challenges to OMBs ability to maximize the coor- low threshold number of respondents that trig-
dination and transparency of Federal information gers the Information Collection Request clearance
collections. process (10 or more) means most collections the
By law, there are three primary actors involved Federal government wishes to undertake require
in processing an information collection for ap- OMB review and approval; few collections for ev-
proval. Te agency wishing to sponsor the collec- idence building can be implemented with fewer
tion prepares an Information Collection Request, than 10 respondents. Te broad defnition of what
which must meet the standards outlined in the constitutes an information collection means
PRA, including elements such as the need, the that almost every efort to collect standardized in-
plan, and the public burden estimated for the col- formation from the public, including information
lection. Te departmental CIO is responsible for from grantees spending Federal funds, triggers
certifying that the collection meets these stan- the Information Collection Request review and
dards prior to submitting the package to OMB. OMB approval process.
is responsible for review and approval of the pack- Te second issue is the length of time required
age.28 Te Information Collection Request, along to gain approval for an information collection.
with the proposed data collection instruments to Delays prior to starting data collection may be

27. Paperwork Reduction Act, 44 USC 35013520 (1995). 29. Stuart Shapiro, Te Paperwork Reduction Act: Benefts, Costs,
and Directions for Reform, Government Information Quarterly 30
28. Te 1995 amendments to the PRA used the phrase senior of-
(2013): 204210.
fcial, which was later changed to Chief Information Ofcer in
the Clinger-Cohen Act (Public Law 104106, February 10, 1996; 30. Letter submitted to the Commission by the Laura and John
Public Law 104208, September 30, 1996). Arnold Foundation.
100 The Promise of Evidence-Based Policymaking

compounded by Institutional Review Board re- produced. Te Commission believes that a respon-
views also required for certain types of research. sible investment of resources in more and better
Tese delays often mean that the window of op- evidence holds the potential to yield substantial
portunity passes for collecting critical baseline savings in the longer term as programs that are
data from participants in a new program and also improved become more cost-efective, and as pro-
can inhibit the ability of agencies to gather re- grams that are not efective are discontinued.
al-time information about program operations.
As Katherine ORegan, former Assistant Secretary Using evidence to improve
at HUD, noted in her testimony to the Commis- government is what taxpayers
sion, if the collection of such information cannot
take place in a timely manner, agencies are left to expectsmart and careful use of
make policy decisions and program changes with limited resources to best address
very little information; often reacting to anecdote
national priorities.
rather than a more complete picture.31
Within OMB, OIRA is tasked with the review Analytical Perspectives, Budget of the
and approval of new information collections. Be- United States Government, Fiscal Year 2018
cause of the high priority of the regulatory func-
tion within OIRA, the swift review of new collec- Te Commission recognizes that resource pri-
tions often is not a priority. Te statutory language oritization is essential to ensuring the goals of
of the PRA intentionally casts a broad net to en- the Commission are achieved, but it is not rec-
sure that the majority of information collections ommending an infusion of large sums of funding
undertaken by the Federal government are includ- to create new agencies or to launch massive new
ed in the required review and approval process. evidence-building endeavors. Instead, throughout
However, fexibilities currently exist within the this report, the recommendations of the Commis-
PRA statute that could be appropriately applied sion balance the need to prioritize evidence build-
for evidence building activities. For example, the ing while recognizing fscal constraints. In some
PRA provides OMB with the ability to delegate departments, sufcient resources already exist to
some authority to approve proposed collections of enable evidence building, though such resources
information.32 Exploring opportunities to delegate may have use restrictions that inhibit the most
authorities could create an opportunity to focus cost-efective approach for evidence building. In
additional attention within OMB on other purpos- several instances shared with the Commission,
es and benefts of the PRA, such as coordinating restrictive funding procedures appear to have
information collections across government and hampered or terminated otherwise positive evi-
improving quality. dence-building practices. Specifcally, with regard
to the procurement challenges discussed previ-
Challenge #5: Te Federal evidence-build- ously, the lack of funds that are available across
multiple years is one considerable barrier to carry-
ing community has insufcient resources
ing out long-term projects. Evidence-building ac-
and limited fexibilities that restrict the tivities are generally today considered non-sev-
ability to expand evidence-building activ- erable services under Federal appropriations law,
ities. which means that they must be fully funded up-
front or structured as a multi-year contract. De-
Trough the course of the Commissions research partments seldom have sufcient funding to fully
and deliberations, the topic of resources repeat- fund the entire cost of large or complex studies
edly emerged as a major perceived need for the up front; rather, departments must break those
evidence-building community and a challenge contracts into phases. Doing so, however, requires
for improving the volume and quality of evidence considerable extra staf time and additional costs
to the government and its contractors. Forcing de-
partments to issue multiple contracts to conduct
31. Katherine ORegan, HUD, Commission Meeting, Washington, a single evaluation or research project distracts
D.C., November 4, 2016. from efciently implementing projects aimed at
32. Paperwork Reduction Act, 44 USC 3507(i)(1) (1995). generating more and better evidence.
The Promise of Evidence-Based Policymaking 101

Some Federal departments struggle to identi- than hinder, evidence-building and sufcient re-
fy resources that can be made available for evi- sources and fexibilities must be provided to sup-
dence-building purposes. Flexibilities in funding port the entire enterprise. To build capacity across
have allowed other departments to pursue evi- the Federal evidence-building community, the
dence-building activities within existing resourc- Commission identifed fve recommendations:
es. Several departments have the legal authority to
set-aside, or allocate, a set amount of funding for REC. 5-1: Te President should direct
their evidence-building budgets that can be trans-
Federal departments to increase ca-
ferred across budget accounts. Te explicit trans-
fer authority in appropriations is one approach to pacity for evidence building through the
reducing the need for burdensome eforts to justi- identifcation or establishment of a Chief
fy budget transfer under general authorities, such Evaluation Ofcer, in addition to needed
as the Economy Act. DOL, for example, receives authorities to build a high performing evi-
funding available for multiple years for evalua- dence-building workforce.
tion through the set-aside and transfer authority
in its appropriation, with a requirement to noti-
fy appropriations committees about evaluations Ensuring the routine and continuous production of
to be conducted.33 Te Department of Justice and evidence to support policy decisions and program
HHS also have programs that include small fund- administration will require the capacity to leverage
ing set-asides for evidence building. Agencies each evidence-building functionstatistics, evalu-
like the Social Security Administration create a ation, and policy researchwithin Federal depart-
reliable funding stream for information technol- ments. Today, some combination of these functions
ogy improvements through set-asides. Te same exists in all major departments, including through
strategy could be used to support the full suite of collaborations with non-governmental partners,
evidence-building activities, including data collec- but these functions are not necessarily operating
tion and curation, policy-relevant research, and at the level needed. Tis recommendation directs
evaluation. Federal departments to establish the capacity to
undertake the full range of evidence-building ac-
Recommendations tivities through internal human resource strate-
gies and by leveraging partnerships with external
Te Commission identifed a series of opportu- partners. Federal departments should also con-
nities to build and maintain a strong Federal in- duct a regular inventory of units that perform evi-
frastructure for the sustained production and use of dence-building functions across the department as
evidence. To maximize the generation of evidence, part of their strategic planning process.
the Federal evidence-building community must To advance the goal of strengthening the gov-
support a full range of analytic approaches, in- ernments more nascent program evaluation func-
cluding statistics, evaluation, and policy research. tion, the Commission recommends that Federal
Such expertise may be established within Federal departments identify or establish a Chief Evalu-
departments, and may be strengthened through ation Ofcer. A Chief Evaluation Ofcer can help
partnerships with the non-Federal evidence-build- lead eforts to coordinate the departments evi-
ing community. To achieve the greatest gains, ev- dence-building activities. Further, OPM should
idence building must be well-coordinated both take immediate steps to support the growth and
within and across departments. Strong leadership development of the program evaluation feld, in-
that prioritizes evidence building and creates the cluding the establishment of occupational struc-
demand for evidence is vital for institutionaliz- tures to address critical needs regarding technical
ing these functions within departments and en- expertise and contract management. One approach
suring coordination across the evidence-building to address these stafng challenges would be for
community. In addition, administrative functions OPM to collaborate with agencies that already
must be aligned and tailored to support, rather have high-functioning program evaluation ofc-
es to identify human capital strategies to support
33. Division H, Section 107, Consolidated Appropriations Act, the development of the evaluation feld. For newer
2017 (Public Law 11531). evaluation ofces that may be struggling to grow
102 The Promise of Evidence-Based Policymaking

their capacity, OPM could identify hiring strategies learning agendas through routine strategic plan-
focused on recruiting and retaining professionals ning reduces the burden for identifying knowledge
with the specialized skills needed to directly exe- gaps and helps align evidence building with depart-
cute evaluations on behalf of the Federal govern- mental strategic priorities. Te resulting document
ment or manage evaluation contracts. Such strate- also helps to clearly communicate the high-priority
gies might include the establishment of a targeted research and policy questions of a department both
occupational series that would allow departments to the public and to the many diferent actors with-
fexibility to hire based on specifc criteria most in the evidence-building community.
relevant for the position and the function. Te Congress and the President should pro-
Tools and mechanisms to leverage external vide sufcient and appropriate authority for de-
partnerships and build agency capacity should also partments to design programs and policies that
be made available to Federal agencies in the evi- enable a portfolio of evidence to support contin-
dence-building community, including grant-mak- uous learning and information needed to ensure
ing authority and cooperative agreement authority, accountability. Te Commission acknowledges
to encourage the use of partnerships with founda- the value of research and evaluation that takes
tions, universities, and others. Te President should place at various points throughout the lifecycle of
also encourage the increased use of programs, such a program, as well as the broad variety of research
as the IPA program, that allow agencies to bring on methods that may be employed to address difer-
temporary personnel from outside the Federal gov- ent types of research questions. When appropriate
ernment to help expand capacity and advance the and feasible, randomized controlled trials should
departments learning agenda. be conducted to understand program impacts.
Federal departments should be encouraged to
REC. 5-2: Te Congress and the Presi- routinely evaluate programs and policies, including
their cost efectiveness. Continuous improvement
dent should direct Federal depart-
can be facilitated by supporting a cycle of frst pilot
ments to develop multi-year learning agen- testing a new program, policy, or regulation, then
das that support the generation and use of conducting research to learn from the pilot test, and
evidence. fnally, adapting the program, policy, or regulation
based on what was learned through the research. In
Te supply of evidence to support policymaking is establishing new programs in law or reauthorizing
more likely to increase when there are consistent existing programs, the Congress and the President
signals from policymakers that the production of should strongly encourage that pilot and demon-
evidence is a priority. Te Congress and the Pres- stration projects are evaluated, and provide fexi-
ident can take steps to encourage departments bility to design policies that allow experimentation,
to develop more and better evidence. As a start- including phased implementation options.
ing point, the Congress and the President should
encourage Federal departments to inventory the REC. 5-3: Te Congress and the Presi-
units responsible for various evidence-building
dent should direct the Ofce of Man-
activities to ensure that each unit is recognized
and operating in concert with other units engag- agement and Budget (OMB) to coordinate
ing in evidence-building activities. Such an inven- the Federal governments evidence-build-
tory could be developed through the regular qua- ing activities across departments, includ-
drennial strategic planning process. ing through any reorganization or consoli-
Te President should encourage Federal depart- dation within OMB that may be necessary
ments to develop multi-year learning agendas in
consultation with program and evidence-building and by bolstering the visibility and role of
units. Te learning agenda should be updated on interagency councils.
a regular basis and contain both short- and long-
term evidence-building priorities for each depart- Te Commission recognizes the Presidents and
ment. Trough the development of learning agen- OMBs prerogative in organizing and optimizing
das, departments should proactively identify where OMBs structure and resources. Implementing a
the need for more evidence is greatest. Developing government-wide vision for evidence building
The Promise of Evidence-Based Policymaking 103

consistent with the Commissions vision, though, To generate a greater volume of evidence in a
may require substantial changes in how govern- more efcient manner, foundational adminis-
ment operates. Te Commission believes that trative processes must be aligned and tailored
having this efort championed at OMB is critical. to better support evidence building. Te Com-
Many of the Commissions recommendations rely mission identifed a specifc set of actions relat-
implicitly on the integration of evidence building ed to procurement and streamlining the review
functions within departments. Diferent compo- and approval processes for new data collections
nents of the evidence-building community, in- that would require little cost, but ofer substan-
cluding statistical agencies, evaluation ofces, pri- tial benefts and savings while making it easier to
vacy ofces, and performance management units produce evidence. Specifcally, OMB should clarify
must increasingly work together to realize the the applicability of the research and development
Commissions vision. Because many department procurement policies to contracts that support ev-
structures for implementing evidence-building ac- idence-building activities, and should support the
tivities mirror the structures at OMB, careful con- establishment of training for technical staf who
sideration should be given to whether a consolida- are responsible for managing research and devel-
tion of activities necessary for evidence building opment contracts that support evidence building.
should occur in the near term at OMB. A focus on In partnership with this approach, training should
evidence building can get crowded out by other be developed for technical staf responsible for
priorities. Te President and OMB should careful- managing evidence-building contracts, including
ly consider how a greater commitment to founda- efective management techniques for oversight
tional, critical information policy setting and co- and accountability of riskier service contracts. An
ordination responsibilities and resources can be additional strategy would be to establish a Gov-
achieved, in light of the signifcant, high stakes ernment-wide Acquisition Contract (GWAC) for
and fast-moving regulatory responsibilities with- use in issuing contracts for evidence-building ac-
in OIRA, responsibilities that currently comprise tivities and services.
the overwhelming majority of OIRAs work. Tis Te Congress and the President also should
includes any reorganizations or required statutory take steps to streamline the PRAs review and
changes that could improve the impact of OMBs approval process for Information Collection Re-
investment in evidence-based policymaking. quests, while ensuring that information coordi-
Efciently implementing evidence-building nation, transparency, and data quality remain
activities across government requires a strong co- central features of PRA. OMB should pilot ap-
ordination function to address cross-cutting re- proaches for better meeting the needs of the ev-
search and policy questions, minimize duplicative idence-building community in conducting ICR
eforts, and reduce the burden on the public. For reviews, including expanding the use of the del-
the Federal government to maximize resources egated authority already authorized under the
available for evidence building across government, PRA. Te process of review and approval internal
OMB must provide leadership that spans the Ex- to Federal departments might also be strength-
ecutive Branch. In addition, OMBs role in leading ened by removing the departmental CIO from the
many interagency councils must be strengthened ICR review and approval process and instead as-
to ensure these eforts are appropriately visible signing this responsibility to the senior data poli-
and infuential in improving how government cy ofcial proposed in Recommendation 3-3. At a
agencies collaborate. minimum, OMB should issue guidance to Feder-
al agencies describing the fexibilities under the
REC. 5-4: Te Congress and the Pres- PRA, including the process for acquiring delegat-
ident should align administrative ed approval authority, and strengthening depart-
mental capabilities to design rigorous studies to
processes to support evidence building, in
advance evidence building.
particular by streamlining the approval Te Congress should consider additional statu-
processes for new data collections and us- tory changes to the PRA to streamline the review
ing existing fexibilities in procurement process. Tese include assessing the rationale for
policy. the current standard under the PRA that requires
104 The Promise of Evidence-Based Policymaking

the review and approval of all new data collections Appropriators can further support evidence build-
of 10 or more respondents, and considering short- ing that often spans the domain of multiple poli-
ening the initial public comment period for new cies and jurisdictional silos by enabling the transfer
collections from 60 days to 30 days. of funding across the evidence-building communi-
ty. Te Congress and the President should enable
REC. 5-5: Te Congress and the Presi- and encourage transfers across budget accounts
that support multi-departmental evidence-build-
dent should ensure sufcient resourc-
ing needs. Explicit transfer authority in appropri-
es to support evidence-building activities ation bills will enable departments to spend less
about Federal government programs and time justifying transfers under general authorities
policies. (e.g., the Economy Act), and enable them to focus
on the production of evidence.
Insufcient resources for evidence building can be In establishing new programs in law or reau-
perceived as an insurmountable limitation to the thorizing existing programs, the Congress and the
pursuit of more and better evidence about gov- President should enable the use of new set-aside
ernment programs and policies. Tere are several authorities of up to 1 percent of program admin-
actions that should be taken by the Congress and istration resources to support the full suite of evi-
the President to signal the importance of evidence dence-building activities, including data collection
building within Federal departments, and set the and curation, policy-relevant research, and evalu-
expectation for the production and application of ation. In addition, the Congress and the President
evidence through the provision of fexibilities or a should establish and grant Federal departments
commitment of resources. access to Evidence Incentive Funds to supplement
Te Congress, through the appropriations pro- the production of future research, evaluation, and
cess, and the President should provide depart- related activities identifed in departmental learn-
ments in active pursuit of a learning agenda access ing agendas. Evidence Incentive Funds in each de-
to multi-year funding to pursue articulated evi- partment are conceptualized by the Commission
dence-building priorities. Te availability of multi- to operate similarly to Working Capital Funds or
year funds for departments that establish learning Salary and Expense accounts. Te funds could be
agendas both creates an incentive for the genera- created by taking up to 10 percent of unobligated
tion of learning agendas and enables departments balances at the end of a fscal year to be allocated
to more efciently contract for multiyear studies.34 for future evidence-generating activities.

34. Interagency Council on Evaluation Policy, Comments to the


Commission, Top-Five List of Issues and Solutions Related to
Federal Evaluation Activity, November 22, 2016.
The Promise of Evidence-Based Policymaking 105

6
Conclusion:
Possibilities with More and Better Evidence

T he Commissions recommendations pres-


ent a comprehensive strategy for addressing
the greatest problems facing evidence building
in all evidence-building activities. Te risks of
re-identifying individuals in data used for evi-
dence building will be continuously reviewed and
today: unintentional limits on data access, inad- mitigated. Te evidence-building community will
equate privacy practices, and insufcient capacity have avenues for secure access to the information
to generate the amount of quality evidence need- they need to answer questions from policymak-
ed to support policy decisions. Te Congress, the ers, program administrators, and the public. Te
President, and the American people are ill-served institutions of government will further enhance
by this state of afairs. Te Commission believes their systems for stewarding data and enabling ev-
that fully implementing the Commissions recom- idence building. Principal Statistical Agencies will
mendations will lead to substantial progress in ad- take on increased leadership within the Federal
dressing these challenges, enabling more and bet- government in facilitating the secure statistical
ter evidence for our society, generated in a more use of administrative data. Program ofces with-
secure fashion. in the Federal government, state ofcials, and the
Over the past century, numerous commissions public will demand and use statistical analysis to
and panels ofered strategies to improve the U.S. support routine decision-making. Perhaps most
evidence-building system. Te changes made in importantly, there will be greater accountability
response to their recommendations have not kept and transparency about evidence-building ap-
pace with the need. Twenty years ago, many of the proaches and uses of data for the Congress, the
recommendations included in this Commissions President, and the American public.
strategy could not have been feasibly implement- In developing recommendations, the Commis-
ed. In this report, the Commission proposes a sion relied on fve guiding principles related to the
modernization of the countrys evidence-produc- concepts of privacy, rigor, transparency, humility,
ing capacity that uses available approaches and and capacity (described in Chapter 1). Each of
that incorporates new technologies and methods these principles is refected in the recommenda-
as they come on line. Tis strategy will enrich the tions developed by the Commission for building
capabilities for producing evidence in the U.S. and and using evidence to improve our society. Te
ensure a formidable ability to use evidence to im- principle for respecting individual privacy and
prove government policies. confdentiality is captured in virtually every rec-
Te Commission is confdent that, with full ommendation in this report, particularly those in
and ongoing implementation of the Commis- Chapters 2, 3, and 4 that seek to improve priva-
sions recommendations, privacy-protective ap- cy protections. Te principle of rigor encourages
proaches will continuously improve to ensure well-designed and well-implemented methods;
the confdentiality of individuals information improvedaccess to administrative data to answer
106 The Promise of Evidence-Based Policymaking

important research questions, as recommended in ment spending be treated diferently? Taxpayers


Chapters 2 and 3, advances this principle. Te rec- and policymakers should receive credible infor-
ommendations in Chapter 5 concerning learning mation to know and understand how well the
agendas and portfolios of evidence as tools for ad- programs and policies they fund achieve their in-
dressing policymakers questions are grounded in tended goals.
the principle of humility in the development and Generating and using evidence to inform pol-
use of evidence. In all of the recommendations, icymaking and program administration is not a
the Commission encourages increased transpar- partisan issue. Te strategy described in this re-
ency about evidence-building eforts, particularly port ofers a responsible approach to improving
with the new transparency portal recommended how government ofcials, private researchers,
in Chapter 4. In addition to the creation of the foundations, non-profts, the business communi-
National Secure Data Service which will enable ty, and the public interact to make sure govern-
greater capacity for the entire evidence-building ment delivers on its promises.
community, Chapter 5 includes numerous recom- All of these recommendations will depend on
mendations that seek to improve governments the leadership of the President and the Congress
capacity for evidence building while encouraging in calling for credible evidence to support policy
the use of information. decisions throughout government. Whether mak-
Te Commissions recommendations recognize ing decisions on funding allocations, assessing
the complexity of improving evidence about gov- new regulations, or understanding how to im-
ernment programs and policies. Te Commissions prove processes for efciently providing services,
recommendations seek to create an environment evidence is needed in every decision made by gov-
that enables routine production of evidence to ernment ofcialscareer civil servant, political
meet the countrys informational needs. And appointee, or elected ofcial. Without the use of
the Commissions recommendations embody the evidence in our democracy, we are only guessing
spirit of bipartisan cooperation. at whether government programs and policies are
People need credible information to inform achieving their intended goals.
their actions. We review ratings on websites be- Tis is a milestone moment. Te Congress and
fore making online purchases. We inspect homes the President should seize it by working togeth-
before moving in. We seek second opinions before er to enact the laws and develop the regulations
major surgeries. Businesses rely on data and anal- necessary to implement the Commissions recom-
ysis to make their decisions. Why should govern- mendations.
The Promise of Evidence-Based Policymaking 107

Acronyms
ACF: Administration for Children and Families (U.S. HUD: U.S. Department of Housing and Urban
Department of Health and Human Services) Development

ASPE: Office of the Assistant Secretary for Planning ICR: Information Collection Request
and Evaluation (U.S. Department of Health and
Human Services) IES: Institute of Education Sciences (U.S.
Department of Education)
CARRA: Center for Administrative Records Research
and Applications (Census Bureau, U.S. Department IPA: Intergovernment Personnel Act
of Commerce)
LEHD: Longitudinal Employer-Household Dynamics
CASD: Centre dAccs Scuris aux Donnes Program
(Secure Access Data Center France)
NCES: National Center for Education Statistics (U.S.
CEP: U.S. Commission on Evidence-Based Department of Education)
Policymaking
NDNH: National Directory of New Hires
CES: Center for Economic Studies (Census Bureau,
NSDS: National Secure Data Service (proposed by
U.S. Department of Commerce)
recommendations)
CIO: Chief Information Officer
NSF: National Science Foundation
CIPSEA: Confidential Information Protection and
OIRA: Office of Information and Regulatory Affairs
Statistical Efficiency Act of 2002
(Office of Management and Budget, Executive
CMS: Centers for Medicare and Medicaid Services Office of the President)
(U.S. Department of Health and Human Services)
OMB: U.S. Office of Management and Budget
DARE: Drug Abuse Resistance Education (Executive Office of the President)

DOL: U.S. Department of Labor OPM: U.S. Office of Personnel Management

ED: U.S. Department of Education OPRE: Office of Planning, Research, and Evaluation
(Administration for Children and Families, U.S.
ED-DRB: disclosure review board at the U.S. Department of Health and Human Services)
Department of Education
PII: Personally identifiable information
eMOU: Enterprise Memorandum of Understanding
PRA: Paperwork Reduction Act of 1995
FIPPs: Fair Information Practice Principles
PSA: Principal statistical agency
FITARA: Federal Information Technology Acquisition
Reform Act SMC: Secure Multiparty Computation

FNS: Food and Nutrition Service (U.S. Department of SNAP: Supplemental Nutrition Assistance Program
Agriculture)
TANF: Temporary Assistance for Needy Families
FSRDC: Federal Statistical Research Data Center
UI: Unemployment Insurance
FTI: Federal tax information
USDA: U.S. Department of Agriculture
GPRA: Government Performance and Results Act of
VA: U.S. Department of Veterans Affairs
1993
WIC: Special Supplemental Nutrition Program for
GWAC: Government-wide Acquisition Contract
Women, Infants, and Children
HHS: U.S. Department of Health and Human
WIOA: Workforce Innovation and Opportunity Act
Services
of 2014
This page intentionally left blank
The Promise of Evidence-Based Policymaking 109

Appendix A: Evidence-Based Policymaking


Commission Act of 2016

PUBLIC LAW 114140MAR. 30, 2016 130 STAT. 317

Public Law 114140


114th Congress
An Act
To establish the Commission on Evidence-Based Policymaking, and for other pur- Mar. 30, 2016
poses. [H.R. 1831]
Be it enacted by the Senate and House of Representatives of
the United States of America in Congress assembled, Evidence-Based
Policymaking
SECTION 1. SHORT TITLE. Commission Act
of 2016.
This Act may be cited as the Evidence-Based Policymaking
Commission Act of 2016.
SEC. 2. ESTABLISHMENT.
There is established in the executive branch a commission
to be known as the Commission on Evidence-Based Policymaking
(in this Act referred to as the Commission).
SEC. 3. MEMBERS OF THE COMMISSION.
(a) NUMBER AND APPOINTMENT.The Commission shall be com-
prised of 15 members as follows:
(1) Three shall be appointed by the President, of whom President.
(A) one shall be an academic researcher, data expert,
or have experience in administering programs;
(B) one shall be an expert in protecting personally-
identifiable information and data minimization; and
(C) one shall be the Director of the Office of Manage-
ment and Budget (or the Directors designee).
(2) Three shall be appointed by the Speaker of the House
of Representatives, of whom
(A) two shall be academic researchers, data experts,
or have experience in administering programs; and
(B) one shall be an expert in protecting personally-
identifiable information and data minimization.
(3) Three shall be appointed by the Minority Leader of
the House of Representatives, of whom
(A) two shall be academic researchers, data experts,
or have experience in administering programs; and
(B) one shall be an expert in protecting personally-
identifiable information and data minimization.
(4) Three shall be appointed by the Majority Leader of
the Senate, of whom
(A) two shall be academic researchers, data experts,
or have experience in administering programs; and
(B) one shall be an expert in protecting personally-
identifiable information and data minimization.
(5) Three shall be appointed by the Minority Leader of
the Senate, of whom
33CYQ1 with PUBLAW
110 The Promise of Evidence-Based Policymaking

130 STAT. 318 PUBLIC LAW 114140MAR. 30, 2016

(A) two shall be academic researchers, data experts,


or have experience in administering programs; and
(B) one shall be an expert in protecting personally-
identifiable information and data minimization.
(b) EXPERTISE.In making appointments under this section,
consideration should be given to individuals with expertise in
economics, statistics, program evaluation, data security, confiden-
tiality, or database management.
President. (c) CHAIRPERSON AND CO-CHAIRPERSON.The President shall
select the chairperson of the Commission and the Speaker of the
House of Representatives shall select the co-chairperson.
Deadline. (d) TIMING OF APPOINTMENTS.Appointments to the Commis-
sion shall be made not later than 45 days after the date of enact-
ment of this Act.
(e) TERMS; VACANCIES.Each member shall be appointed for
the duration of the Commission. Any vacancy in the Commission
shall not affect its powers, and shall be filled in the manner in
which the original appointment was made.
(f) COMPENSATION.Members of the Commission shall serve
without pay.
(g) TRAVEL EXPENSES.Each member of the Commission shall
be allowed travel expenses, including per diem in lieu of subsistence,
at rates authorized for employees of agencies under subchapter
I of chapter 57 of title 5, United States Code, while away from
their homes or regular places of business in the performance of
services for the Commission.
SEC. 4. DUTIES OF THE COMMISSION.
Recommenda- (a) STUDY OF DATA.The Commission shall conduct a com-
tions. prehensive study of the data inventory, data infrastructure, data-
base security, and statistical protocols related to Federal policy-
making and the agencies responsible for maintaining that data
to
(1) determine the optimal arrangement for which adminis-
trative data on Federal programs and tax expenditures, survey
data, and related statistical data series may be integrated
and made available to facilitate program evaluation, continuous
improvement, policy-relevant research, and cost-benefit anal-
yses by qualified researchers and institutions while weighing
how integration might lead to the intentional or unintentional
access, breach, or release of personally-identifiable information
or records;
(2) make recommendations on how data infrastructure,
database security, and statistical protocols should be modified
to best fulfill the objectives identified in paragraph (1); and
(3) make recommendations on how best to incorporate out-
comes measurement, institutionalize randomized controlled
trials, and rigorous impact analysis into program design.
(b) CLEARINGHOUSE.In undertaking the study required by
subsection (a), the Commission shall
(1) consider whether a clearinghouse for program and
survey data should be established and how to create such
a clearinghouse; and
Evaluation. (2) evaluate
(A) what administrative data and survey data are rel-
evant for program evaluation and Federal policy-making
and should be included in a potential clearinghouse;
kgrant on DSKB33CYQ1 with PUBLAW

VerDate Sep 11 2014 11:52 May 06, 2016 Jkt 059139 PO 00140 Frm 00002 Fmt 6580 Sfmt 6581 E:\PUBLAW\PUBL140.114 PUBL140
The Promise of Evidence-Based Policymaking 111

PUBLIC LAW 114140MAR. 30, 2016 130 STAT. 319

(B) which survey data the administrative data identi-


fied in subparagraph (A) may be linked to, in addition
to linkages across administrative data series, including
the effect such linkages may have on the security of those
data;
(C) what are the legal and administrative barriers
to including or linking these data series;
(D) what data-sharing infrastructure should be used
to facilitate data merging and access for research purposes;
(E) how a clearinghouse could be self-funded;
(F) which types of researchers, officials, and institu-
tions should have access to data and what the qualifications
of the researchers, officials, and institutions should be;
(G) what limitations should be placed on the use of
data provided;
(H) how to protect information and ensure individual
privacy and confidentiality;
(I) how data and results of research can be used to
inform program administrators and policymakers to
improve program design;
(J) what incentives may facilitate interagency sharing
of information to improve programmatic effectiveness and
enhance data accuracy and comprehensiveness; and
(K) how individuals whose data are used should be
notified of its usages.
(c) REPORT.Upon the affirmative vote of at least three-quar- Recommenda-
ters of the members of the Commission, the Commission shall tions.
submit to the President and Congress a detailed statement of its
findings and conclusions as a result of the activities required by
subsections (a) and (b), together with its recommendations for such
legislation or administrative actions as the Commission considers
appropriate in light of the results of the study.
(d) DEADLINE.The report under subsection (c) shall be sub-
mitted not later than the date that is 15 months after the date
a majority of the members of the Commission are appointed pursu-
ant to section 3.
(e) DEFINITION.In this section, the term administrative data
means data
(1) held by an agency or a contractor or grantee of an
agency (including a State or unit of local government); and
(2) collected for other than statistical purposes.
SEC. 5. OPERATION AND POWERS OF THE COMMISSION.
(a) EXECUTIVE BRANCH ASSISTANCE.The heads of the following Consultation.
agencies shall advise and consult with the Commission on matters
within their respective areas of responsibility:
(1) The Bureau of the Census.
(2) The Internal Revenue Service.
(3) The Department of Health and Human Services.
(4) The Department of Agriculture.
(5) The Department of Housing and Urban Development.
(6) The Social Security Administration.
(7) The Department of Education.
(8) The Department of Justice.
(9) The Office of Management and Budget.
(10) The Bureau of Economic Analysis.
(11) The Bureau of Labor Statistics.
kgrant on DSKB33CYQ1 with PUBLAW

VerDate Mar 15 2010 01:04 Apr 26, 2016 Jkt 059139 PO 00140 Frm 00003 Fmt 6580 Sfmt 6581 E:\PUBLAW\PUBL140.114 PUBL140
112 The Promise of Evidence-Based Policymaking

130 STAT. 320 PUBLIC LAW 114140MAR. 30, 2016

(12) Any other agency, as determined by the Commission.


Deadline. (b) MEETINGS.The Commission shall meet not later than 30
days after the date upon which a majority of its members have
been appointed and at such times thereafter as the chairperson
or co-chairperson shall determine.
(c) RULES OF PROCEDURE.The chairperson and co-chairperson
shall, with the approval of a majority of the members of the
Commission, establish written rules of procedure for the Commis-
sion, which shall include a quorum requirement to conduct the
business of the Commission.
(d) HEARINGS.The Commission may, for the purpose of car-
rying out this Act, hold hearings, sit and act at times and places,
take testimony, and receive evidence as the Commission considers
appropriate.
(e) CONTRACTS.The Commission may contract with and com-
pensate government and private agencies or persons for any purpose
necessary to enable it to carry out this Act.
(f) MAILS.The Commission may use the United States mails
in the same manner and under the same conditions as other agen-
cies of the Federal Government.
(g) GIFTS.The Commission may accept, use, and dispose of
gifts or donations of services or property.
SEC. 6. FUNDING.
(a) IN GENERAL.Subject to subsection (b) and the availability
of appropriations
(1) at the request of the Director of the Census, the agencies
identified as Principal Statistical Agencies in the report, pub-
lished by the Office of Management and Budget, entitled
Statistical Programs of the United States Government, Fiscal
Year 2015 shall transfer funds, as specified in advance in
appropriations Acts and in a total amount not to exceed
$3,000,000, to the Bureau of the Census for purposes of carrying
out the activities of the Commission as provided in this Act;
and
(2) the Bureau of the Census shall provide administrative
support to the Commission, which may include providing phys-
ical space at, and access to, the headquarters of the Bureau
of the Census, located in Suitland, Maryland.
(b) PROHIBITION ON NEW FUNDING.No additional funds are
authorized to be appropriated to carry out this Act. This Act shall
be carried out using amounts otherwise available for the Bureau
of the Census or the agencies described in subsection (a)(1).
SEC. 7. PERSONNEL.
Appointment. (a) DIRECTOR.The Commission shall have a Director who
shall be appointed by the chairperson with the concurrence of
the co-chairperson. The Director shall be paid at a rate of pay
established by the chairperson and co-chairperson, not to exceed
the annual rate of basic pay payable for level V of the Executive
Schedule (section 5316 of title 5, United States Code).
(b) STAFF.The Director may appoint and fix the pay of addi-
tional staff as the Director considers appropriate.
(c) EXPERTS AND CONSULTANTS.The Commission may procure
temporary and intermittent services under section 3109(b) of title
5, United States Code, at rates for individuals which do not exceed
the daily equivalent of the annual rate of basic pay for a comparable
position paid under the General Schedule.
kgrant on DSKB33CYQ1 with PUBLAW

VerDate Sep 11 2014 11:52 May 06, 2016 Jkt 059139 PO 00140 Frm 00004 Fmt 6580 Sfmt 6581 E:\PUBLAW\PUBL140.114 PUBL140
The Promise of Evidence-Based Policymaking 113

PUBLIC LAW 114140MAR. 30, 2016 130 STAT. 321


SEC. 8. TERMINATION.
The Commission shall terminate not later than 18 months
after the date of enactment of this Act.

Approved March 30, 2016.

LEGISLATIVE HISTORYH.R. 1831 (S. 991):


HOUSE REPORTS: No. 114211 (Comm. on Oversight and Government Reform).
SENATE REPORTS: No. 114151 (Comm. on Homeland Security and Governmental
Affairs) accompanying S. 991.
CONGRESSIONAL RECORD:
Vol. 161 (2015): July 27, considered and passed House.
Vol. 162 (2016): Mar. 16, considered and passed Senate, amended.
Mar. 17, House concurred in Senate amendment.


kgrant on DSKB33CYQ1 with PUBLAW

VerDate Mar 15 2010 01:04 Apr 26, 2016 Jkt 059139 PO 00140 Frm 00005 Fmt 6580 Sfmt 6580 E:\PUBLAW\PUBL140.114 PUBL140
This page intentionally left blank
The Promise of Evidence-Based Policymaking 115

Appendix B: Commissioner Biographies


Katharine G. Abraham Sherry Glied
University of Maryland New York University
Commissioner and Chair Commissioner
Katharine G. Abraham is Professor of Econom- In August 2013, Sherry Glied became Dean of New
ics and Survey Methodology at the University of York Universitys Robert F. Wagner Graduate School
Maryland. She has written extensively on the ef- of Public Service. From 1989-2013, she was Profes-
fects of labor market policies and institutions on sor of Health Policy and Management at Columbia
frm and worker behavior, labor market adjust- Universitys Mailman School of Public Health. She
ment over the business cycle, and the measure- was Chair of the department from 1998-2009. On
ment of economic activity. Abraham frst joined June 22, 2010, Glied was confrmed by the U.S.
the University of Maryland faculty in 1987, after Senate as Assistant Secretary for Planning and
holding prior positions at the Brookings Institu- Evaluation at the U.S. Department of Health and
tion from 1985 to 1987 and at the Massachusetts Human Services, and served in that capacity from
Institute of Technologys Sloan School of Man- July 2010 through August 2012. She has previous-
agement from 1980 to 1985. From 2011 to 2013, ly served as Senior Economist for health care and
Abraham served as a Member of the Council of labor market policy on the Council of Economic
Economic Advisers and from 1993 to 2001, she Advisers in 1992-1993, under Presidents Bush and
served as Commissioner of the Bureau of Labor Clinton. Glieds principal areas of research are in
Statistics. Abraham is a Fellow of the Society of health policy reform and mental health care policy.
Labor Economists and of the American Statistical
Association and has served as a Vice President of
the American Economic Association. Abraham re- Robert M. Groves
ceived a Bachelors degree from Iowa State Uni- Georgetown University
versity and a Ph.D. from Harvard University. Commissioner
Robert M. Groves is the Provost of Georgetown
University and the Gerard J. Campbell, S.J. Profes-
Ron Haskins sor in the Math and Statistics Department as well
Brookings Institution as the Sociology Department. He served as director
Commissioner and Co-Chair of the U.S. Census Bureau during the conduct of the
Ron Haskins is a Senior Fellow and holds the 2010 Census of Population. Groves studies how to
Cabot Family Chair in Economic Studies at the improve large-scale sample surveys and censuses.
Brookings Institution, where he co-directs the His research has focused on the impact of mode of
Center on Children and Families. Haskins is also data collection on responses in sample surveys, the
a senior consultant at the Annie E. Casey Founda- infuences on survey participation, the use of adap-
tion and is Past-President of the Association for tive research designs to improve the cost and error
Public Policy Analysis and Management. Prior to properties of statistics, and public concerns about
joining Brookings and Casey, he spent 14 years on privacy afecting attitudes toward statistical agen-
the staf of the House Ways and Means Human cies. Groves is an elected member of the National
Resources Subcommittee, serving as the subcom- Academy of Sciences and the National Academy of
mittees Staf Director after Republicans became Medicine. He is also the chair of the Committee
the majority party in the House after the 1994 on National Statistics at the National Academies
elections. He holds a Bachelors degree in Histo- of Sciences, Engineering, and Medicine. He is an
ry, a Master of Arts in Teaching, and a Ph.D. in appointed member of the National Science Board,
Developmental Psychology from the University of overseeing the National Science Foundation, as
North Carolina at Chapel Hill. In his Washington well as the Board of the Pew Research Center.
career, he has focused on evidence-based policy,
early childhood education, marriage and family
formation, poverty, equal opportunity, abused and
neglected children, and budget issues.
116 The Promise of Evidence-Based Policymaking

Robert Hahn Poverty by Half in 10 Years. Previously, she was


University of Oxford a member of the Advisory Committee for Social,
Commissioner Behavioral and Economic Sciences for the Nation-
Robert Hahn is Professor and Director of Economics al Science Foundation and the National Advisory
at the Smith School of Enterprise and the Environ- Committee of the Robert Wood Johnson Founda-
ment at the University of Oxford; a senior fellow at tion Scholars in Health Policy Research Program.
the Institute for New Economic Tinking at Oxford;
a senior fellow at the Georgetown University Cen-
ter for Business and Public Policy, and a non-resi- Jeffrey Liebman
dent senior fellow at the Brookings Institution. Harvard University
Hahn worked at American Enterprise Institute for Commissioner
two decades, where he co-founded and directed Jefrey Liebman is the Malcolm Wiener Professor
the AEI-Brookings Joint Center. Previously, Hahn of Public Policy at the Harvard Kennedy School
worked for the Council of Economic Advisers and where he directs the Taubman Center for State
was the chief economist on the White House draft- and Local Government and the Rappaport Insti-
ing team for the 1990 Clean Air Act Amendments. tute for Greater Boston. Liebman teaches courses
He also has served on the faculties of Harvard Uni- in social policy, public sector economics, Ameri-
versity and Carnegie Mellon University. Hahn is can economic policy, and public sector manage-
currently conducting several behavioral economics ment. In his research, he studies tax and budget
experiments aimed at encouraging the conservation policy, social insurance, and public sector inno-
of energy and water resources, improving labor pro- vation. During the frst two years of the Obama
ductivity, improving health outcomes, and under- Administration, Liebman served at the U.S. Ofce
standing the welfare benefts of new technologies. of Management and Budget, frst as Executive As-
In addition, he co-founded the Community Prepa- sociate Director and Chief Economist and then as
ratory School, an inner-city middle school in Provi- Acting Deputy Director. From 1998 to 1999, Lieb-
dence, Rhode Island, that provides opportunities for man served as Special Assistant to the President
disadvantaged youth to achieve their full potential. for economic policy and coordinated the Clinton
Administrations Social Security reform technical
working group. For the past fve years, his Harvard
Hilary Hoynes Kennedy School Government Performance Lab
University of California, Berkeley has been providing pro bono technical assistance
Commissioner to state and local governments interested in im-
Hilary Hoynes is a Professor of Economics and proving the results they achieve for their citizens.
Public Policy and holds the Haas Distinguished
Chair in Economic Disparities at the University of
California, Berkeley. From 2011 to 2016, she was a Bruce D. Meyer
co-editor of the leading journal in economics, the University of Chicago
American Economic Review. Hoynes specializes Commissioner
in the study of poverty, inequality, food and nutri- Bruce D. Meyer is the McCormick Foundation Pro-
tion programs, and the impacts of government tax fessor at the Harris School of Public Policy at the
and transfer programs on low-income families. University of Chicago where he has been since 2004.
Current projects include evaluating the efects of From 1987 to 2004, Meyer was a professor in the
access to the social safety net in early life on later Economics Department at Northwestern University.
life health and human capital outcomes, examin- He is also a Research Associate of the National Bu-
ing the efects of the Great Recession on poverty, reau of Economic Research and a Visiting Scholar at
and the role of the safety net in mitigating in- the American Enterprise Institute. He studies pover-
come losses. Professor Hoynes is a member of the ty and inequality, tax policy, government safety net
American Economic Associations Executive Com- programs such as unemployment insurance, work-
mittee and a panel member of the Committee on ers compensation, food stamps, and Medicaid, and
National Statistics at the National Academies of the accuracy of household surveys. His most recent
Sciences, Engineering, and Medicine on Building work includes research on trends in poverty and in-
an Agenda to Reduce the Number of Children in equality, the consequences of disability, the efects
The Promise of Evidence-Based Policymaking 117

of Medicaid, and errors in household surveys. He uty Director and Chief Operating Officer of the
currently serves on the Technical Advisory Commit- U.S. Census Bureau. She has also served as Deputy
tee to the Bureau of Labor Statistics, the American Under Secretary for Economic Affairs at the U.S.
Economic Association Committee on Government Department of Commerce; Principal Associate Di-
Relations, and as an ofcer of the Business and Eco- rector and CFO at the U.S. Census Bureau; Senior
nomic Statistics Section of the American Statistical Vice President for Economic, Labor, and Population
Association. Meyer received his Bachelors degree Studies at NORC at the University of Chicago; and
and Masters degree in economics from Northwest- Chief Operating Officer at McManis & Monsalve
ern University and his Ph.D. in economics from the Associates, a business analytics consulting firm.
Massachusetts Institute of Technology. She is an adjunct professor at the Trachtenberg
School of Public Policy and Public Administration
at The George Washington University. Potok is an
Paul Ohm elected Fellow of the National Academy of Public
Georgetown University Administration. She currently serves on the Board
Commissioner of Directors for the Institute of Pure and Applied
Paul Ohm is a Professor of Law at the Georgetown Mathematics at the University of California Los
University Law Center. He specializes in infor- Angeles; the Board of Trustees for the Arthur S.
mation privacy, computer crime law, intellectual Flemming Award, which recognizes outstanding
property, and criminal procedure. achievement by federal employees with fewer than
15 years of service; and The George Washington
University Trachtenberg School Advisory Board.
Allison B. Orris Potok received her Ph.D. in public policy and public
U.S. Office of Management and Budget administration at The George Washington Univer-
Commissioner sity, with an emphasis on program evaluation.
(Departed the Commission on January 20, 2017)
Allison B. Orris served as the Associate Adminis-
trator of the Ofce of Information and Regulato- Kathleen Rice Mosier
ry Afairs at the U.S. Ofce of Management and Faegre Baker Daniels, LLP
Budget until January 2017. Orris served in several Commissioner
roles at the U.S. Department of Health and Hu- Kathleen Rice Mosier is a Counsel at Faegre Baker
man Services Centers for Medicare & Medicaid Daniels, LLP, where she provides advice and assis-
Services from 2009 to 2014, including Senior Poli- tance to organizations on issues relating to privacy,
cy Advisor, Acting Director of the Division of State data security, risk management, and compliance
Demonstrations and Waivers, and Director of the with applicable laws and regulations. Before join-
Low Income Programs Analysis Group in the Of- ing FaegreBD, she spent nearly 20 years in law
fce of Legislation. Before joining CMS, Orris was enforcement and national security. She served as
a Senior Legislative Associate at the Center on a Counsel on the Senate Select Committee on In-
Budget and Policy Priorities from 2006 to 2009. telligence, as an Assistant General Counsel for the
From 2002 to 2006, she practiced law at Powell Federal Bureau of Investigation, and as an Assistant
Goldstein LLP in Washington, D.C. Orris received U.S. Attorney for the Southern District of Florida.
a Bachelors degree in History from Columbia She is a frequent author and speaker on privacy
University and a J.D. from Yale University. and data security matters. She is also a Certifed
Information Privacy Professional, International As-
sociation of Privacy Professionals, and an Adjunct
Nancy Potok Professor of Law at Notre Dame Law School.
U.S. Office of Management and Budget
Commissioner
(Appointed to the Commission on March 10, 2017) Robert Shea
Nancy Potok is Chief Statistician of the United Grant Thornton, LLP
States and Chief of the Statistical and Science Pol- Commissioner
icy Branch in the U.S. Office of Management and Robert Shea is a principal and a member of the
Budget. Prior to January 2017, she served as Dep- Public Sector practice at Grant Tornton, LLP. He
118 The Promise of Evidence-Based Policymaking

leads Strategy and Communications for Grant his Ph.D. in economics in 1992 from the Univer-
Tornton Public Sector and provides performance sity of Chicago.
improvement services to international, federal,
and state and local government agencies. Before
joining Grant Tornton, Shea served as Associ- Kim R. Wallin
ate Director for Management at the U.S. Ofce of D.K. Wallin, Ltd.
Management and Budget (OMB). Before joining Commissioner
OMB, Shea served as counsel to the Senate Com- Kim R. Wallin is currently in private practice at her
mittee on Governmental Afairs, legislative direc- CPA firm, D.K. Wallin Ltd., which she founded in
tor for Congressman Pete Sessions (TX), and pro- 1984. She took eight years off to be the Controller
fessional staf member for the House Committee for the State of Nevada. Wallin was the first CPA to
on Government Reform and Oversight. be elected to the office of Nevada State Controller
in 50 years and the first Certified Management
Accountant to ever hold this office. As Control-
Latanya Sweeney ler, she received the Association of Government
Harvard University Accountants Presidents Award and Excellence in
Commissioner Government Award and the National Association
Latanya Sweeney is Professor of Government and of Comptrollers Presidents Award for her work to
Technology in Residence at Harvard University. improve transparency in government with the use
Her mission is to create and use technology to as- of technology. She also received the National As-
sess and solve societal, political and governance sociation of State Auditors, Comptrollers and Trea-
problems, and to teach others how to do the same. surers Presidents Award for her work on various
Her primary area of focus is the scientific study of U.S. Presidential work groups and for her continued
technologys impact on humankind. She serves as efforts to improve efficiency and accountability in
the Editor-in-Chief of the Journal of Technology Government. Wallin is a graduate of the University
Science, Director of the Data Privacy Lab at Har- of Nevada, Las Vegas with a degree in Business
vard, and Faculty Dean at Currier House. Sweeney Administration with a major in accounting.
formerly served as the Chief Technology Officer,
also known as the Chief Technologist, at the U.S.
Federal Trade Commission. Commission Staf
Rochelle (Shelly) Wilkie Martinez,
Kenneth R. Troske Executive Director
University of Kentucky H. Lucas Hitt, Deputy Executive Director
Commissioner Sharon A. Boivin, Senior Policy and
Kenneth R. Troske is the Associate Dean for Grad- Research Analyst
uate Programs and Outreach and Richard W. and
Anne Fletcher, Senior Policy and
Janis H. Furst Endowed Chair in Economics at
Research Analyst
the Gatton College of Business and Economics at
the University of Kentucky, as well as a Research Nicholas Hart, Policy and Research Director
Fellow with the Institute for the Study of Labor Michael Hawes, Privacy Consultant
(IZA) in Bonn, Germany. Troske served as a mem- Kristy L. Howell, Senior Policy and
ber of the Congressional Oversight Panel whose Research Analyst
task was to assess the existing condition of Amer-
icas financial markets and the regulatory system Mary D. McKoy, Chief, External Afairs
as well as to monitor the actions of the Treasury Kathryn McNamara, Librarian and
Department and financial institutions to deter- Records Coordinator
mine if their actions are in the best interest of Sara Stefanik, Policy and Research Analyst
the American economy. His primary research ar-
Robin L. Wyvill, Meetings and Events
eas are labor and human resource economics. He
Coordinator
received his undergraduate degree in economics
from the University of Washington in 1984 and
The Promise of Evidence-Based Policymaking 119

Appendix C: Commissions Fact-Finding and


Deliberative Processes
Fact-Finding Process o Justin ErlichCalifornia Attorney Generals
Ofce
Following the initial appointments of commis-
sioners, the Commission convened its frst public o Katherine WallmanOfce of Management
meeting in July 2016. Over the course of the follow- and Budget, Executive Ofce of the
ing eight months, the Commission studied issues President (former)
about the current state of evidence production and o Marc GromanOfce of Management and
use in the Federal government as well as the Fed- Budget, Executive Ofce of the President
eral governments policies and practices to protect (former)
data confdentiality. Te Commissions fact-fnd-
o Marc RotenbergElectronic Privacy
ing process included public meetings, public hear-
Information Center
ings, and meetings with organizations to which
Commissioners or staf were invited or that staf o Michael BasilIllinois Department of
initiated, a survey of Federal ofces, a Request for Innovation & Technology
Comments in the Federal Register, and other public
input received through email correspondence.
November 4, 2016: Considerations for the
Commission Related to Evaluation
Public Meetings o Adam GamoranWilliam T. Grant
Te Commissions seven public meetings included Foundation
a total of 49 invited witnesses. o Demetra NightingaleU.S. Department of
July 22, 2016: Introductory Meeting Labor (former)
o Devin OConnorOfce of Management o Evelyn KappelerOfce of Adolescent
and Budget, Executive Ofce of the Presi- Health, U.S. Department of Health and
dent (former) Human Services
o Jeri MulrowBureau of Justice Statistics, o Jim SullivanUniversity of Notre Dame
U.S. Department of Justice o Katherine OReganU.S. Department of
o John RighterCommittee on Health, Edu- Housing and Urban Development (former)
cation, Labor & Pensions, U.S. Senate o Kelly FitzsimmonsTe Edna McConnell
o Mary BohmanEconomic Research Ser- Clark Foundation
vice, U.S. Department of Agriculture o Matthew KleinNew York City Center for
o Nancy PotokCensus Bureau, U.S. Depart- Economic Opportunity
ment of Commerce (former) o Naomi GoldsteinAdministration for
o Raj ChettyStanford University Children and Families, U.S. Department of
Health and Human Services
o Ted McCannOfce of the Speaker, U.S.
House of Representatives o Tanya BeerCenter for Evaluation
Innovation
September 9, 2016: Key Considerations in
Privacy Relevant to the Commissions Charge December 12, 2016: Considerations for the
Commission Related to Federal Models
o Aimee GuideraData Quality Campaign
(former) o Barry JohnsonStatistics of Income
Division, Internal Revenue Service, U.S.
o Cynthia DworkMicrosoft Research
Department of Treasury
(former)
120 The Promise of Evidence-Based Policymaking

o David GruskyStanford University o Andrew ReamerTe George Washington


o Erica GroshenBureau of Labor Statistics, University
U.S. Department of Labor (former) o Brian MoyerBureau of Economic Analysis,
o Marilyn SeastromNational Center for U.S. Department of Commerce
Education Statistics, U.S. Department of o Erin UlrichColorado Department of Public
Education Health and Environment
o Niall BrennanCenters for Medicare and o Maria CancianUniversity of Wisconsin
Medicaid Services, U.S. Department of o Seth HarrisU.S. Department of Labor
Health and Human Services (former) (former)
o Ron JarminCensus Bureau, U.S.
Department of Commerce Public Hearings
January 13, 2017: International and State Te Commission convened three open public
hearingsin Washington, D.C., Chicago, and San
Models for Managing Data
Franciscoduring which any member of the public
o Charles RothwellNational Center for
who requested to testify before the Commission
Health Statistics, U.S. Department of
was allowed to present. A total of 39 members of
Health and Human Services
the public presented information during the three
o David MancusoWashington State hearings.
Department of Social and Health Services
o Domenico ParisiMississippi State October 21, 2016: Washington, D.C.
University o Amanda Janice RobersonInstitute for
o Ivan TaulowStatistics Denmark Higher Education Policy
o Kenneth DodgeDuke University o Carrie WofordVeterans Education
Success
o Robert GoergeUniversity of Chicago
o Christine KellerAssociation of Public &
o Roxane SilbermanSecure Data Access
Land-grant Universities
Centre, France
o Clyde TuckerAmerican Statistical
o Shawna WebsterNational Association for
Association
Public Health Statistics and Information
Systems o Daniel CrowleyNational Prevention
Science Coalition to Improve Lives
o Stefan BenderDeutsche Bundesbank
o David MedinaResults for America
o Tanvi DesaiAdministrative Data Research
Network, United Kingdom o Emmett McGroartyAmerican Principles
Project
February 24, 2017: Role of Legal Standards
o Erin KnowlesUnited States Parents
and Technology in Maximizing Data Security
Involved in Education
and Privacy
o George GrobAmerican Evaluation
o Alexandra WoodHarvard University
Association
o Bradley MalinVanderbilt University
o Kelleen KayeTe National Campaign to
o Daniel GorofAlfred P. Sloan Foundation Prevent Teen and Unplanned Pregnancy
o Jerome ReiterDuke University o Mark SchneiderAmerican Institutes for
o Lars VilhuberCornell University Research
o Quentin WilsonPublic Performance
March 13, 2017: Federal Statistical and Part-
Improvement Researcher
nership Infrastructure for Evidence Building
Opportunities and Limitations o Rachel FishmanNew America
The Promise of Evidence-Based Policymaking 121

o Rachel ZinnWorkforce Data Quality Additional Meetings


Campaign
In addition, Commissioners and staf met with
o RK PaleruBooz Allen Hamilton experts, and participated in conference panel pre-
o Sara DubePew-MacArthur Results First sentations to solicit additional input from more
Initiative than 40 organizations. Tese meetings supple-
mented information gathered during the Com-
o Tifany JonesTe Education Trust missions fact-fnding phase.
o Tom AllisonYoung Invincibles o Actionable Intelligence for Social Policy
January 5, 2017: Chicago, IL PublicAcademic Research Colloquium

o Cassie CreswellRaise Your Hand Action/ o American Evaluation Association


Parent Coalition for Student Privacy o American Public Human Services
o Donna K. GintherUniversity of Kansas Association National Summit

o Lint BarrageRhode Island Innovative o Amy OHara, Census Bureau , U.S.


Policy Lab Department of Commerce (former)

o Margaret LevensteinICPSR o Anna McDowell, Statistics New Zealand

o Matthew StagnerMathematica Policy o Army Analytics Group


Research o Association for Public Policy Analysis &
o Quentin PalfreyJ-PAL North America Management

o Timothy SlaperIndiana Business Research o Association of Public Data Users Webinar


Center o Bipartisan Policy Center
o Tom Schenk, Jr.City of Chicago o Bureau of Economic Analysis Advisory
o V. Joseph HotzDuke University Committee

o Virginia KnoxMDRC o Census Scientifc Advisory Committee


o Center for Regional Economic
February 9, 2017: San Francisco, CA Competitiveness
o Andrew WiegandSocial Policy Research o Center for the Study of Social Policy
Associates
o Committee on Population Statistics
o Cindy GuyAnnie E. Casey Foundation
o Council of Professional Associations on
o David JohnsonUniversity of Michigan Federal Statistics
o Jim HillProofpoint Systems, Inc. o Cross Agency Learning Community on
o Joy BonaguroCity and County of San Research and Evaluation
Francisco o Defense Manpower Data Center, U.S.
o Karen LevesqueRTI International Department of Defense
o Karen R. EfremEducation Liberty Watch o Defense Personnel and Security Research
o Mary Ann BatesJ-PAL North America Center, U.S. Department of Defense

o Maryann FeldmanUniversity of North o Federal Committee on Statistical


CarolinaChapel Hill Methodology Policy Conference

o Sandra TorosianUnited Parents Involved o Federal Economic Statistics Advisory


in Education Committee

o Susan DreyfusAlliance for Strong Families o Federal Evaluators


and Communities o Federation of Associations in Behavioral &
Brain Sciences
122 The Promise of Evidence-Based Policymaking

o Frauke Kreuter, University of Maryland o Transportation Research Board Special


o Health and Human Services Administrative Task Force on Data for Decisions and
Data Technical Expert Panel Performance Measures

o Interagency Council on Evaluation Policy o U.S. Census Bureau and Economic Research
Service Administrative Data Conference
o Jenny Hunter Childs, Census Bureau, U.S.
Department of Commerce o U.S. Department of Health and Human
Services Data Council
o John Abowd, Census Bureau, U.S.
Department of Commerce o Workforce Data Quality Campaign: FlyIn
Conference
o John Mitchell, Stanford University
o Julia Lane, New York University Other Stakeholder Input
o Kathy Stack, Laura and John Arnold Te Commission issued a Request for Comments
Foundation in the Federal Register and accepted comments
o Laura and John Arnold Foundation and Bill by email, which generated over 350 responses
and Melinda Gates Foundation Workshop from the public (see online Appendix G). Finally,
on the use of data to understand the current or potential capacity
of Federal agencies to engage in aspects of evi-
o Liz McPherson, Statistics New Zealand
dence-based policymaking, the Commission ad-
o Mark Mazur, U.S. Department of the ministered a survey to 209 ofces of the Federal
Treasury (former) government that the Commission identifed as
o MDRC likely to be generating or using evidence (see on-
line Appendix E).
o Michael Duf, Stanford University
o Michigan Retirement Research Center,
Financial Research Symposium
Deliberative Process
Following the formal stakeholder input processes,
o Minister Hiro Matsumoto, Embassy of
the Commission compiled and reviewed the body
Japan
of information collected. A series of 12 memoranda
o National Association of State Workforce that compiled existing literature, stakeholder in-
Agencies Labor Market Information put, and options for recommendations were devel-
Committee oped. Each memorandum was initially discussed
o National Conference of State Legislatures in small workgroups then formally discussed by
Commissioners during closed session meetings as
o National Governors Association
the Commission deliberated. Multiple memoranda
o National Institute of Standards and involved discussion during more than one meeting;
Technology, Privacy Risk Assessment the list below refects the initial discussion points:
Meeting
o Naval Postgraduate School April 3, 2017
o Vision Statement and Evidence Principles
o Open Data Center Korea
o National Secure Data Service Implementation
o Pew Charitable Trusts
o Data Access and UseLegal and Statutory
o Pew Results First & State Data Project Team
Issues
o Results for America
o Data Access and UseFederal Policy and
o Ross Young, United Kingdom Statistics Standards
Authority
o Increasing Access to StateCollected
o Society for Beneft Cost Analysis Annual Administrative Data
Conference o Tiered Access with a National Secure Data
Service
The Promise of Evidence-Based Policymaking 123

May 15, 2017 June 27, 2017


o Enhancing Collaboration in the Federal o Collaborating for Evidence Building
Evidence Ecosystem o Incorporating Evaluation and Policy
o Te Role of the Federal Statistical System Research in Program Design
within the Evidence Ecosystem o Administration
o Protecting Privacy and Increasing
Confdentiality Once the Commission developed tentative recom-
mendations, the fnal two closed session meetings
o Protecting Privacy by Increasing of the Commission were dedicated to discussing
Transparency About Evidence Generation report text.
and Benefts
o Establishing a National Secure Data Service
July 11, 2017
July 20, 2017
June 7, 2017
Commissioners collaborated on draft report text
o Role of the National Secure Data Service during this time. Commissioners cast digital votes
and the PSAs in the Federal Evidence of approval from July 28 to August 2. All 15 Com-
Ecosystem missioners cast votes in the afrmative approving
o Institutionalizing and Incentivizing the the fndings and recommendations in the report.
Evaluation and Policy Research Functions
Across Government
This page intentionally left blank
The Promise of Evidence-Based Policymaking 125

Appendix D: Examples of Data Productive for


Evidence Building

During the course of the Commissions fact-fnd- and individual information. Statutory Citation:
ing phase, numerous experts and members of 26 USC 6103.
the public suggested specifc sources of data that
could be useful for evidence-building activities. Te National Directory of New Hires*, main-
Statistical uses of administrative data collected tained by the Administration for Children
through program operationssuch as enrollment and Families (ACF) Ofce of Child Support
and utilization informationas well as statistical Enforcement in the Department of Health and
data collected directly through topical surveys in- Human Services (HHS), includes quarterly
tended to support evidence building ofer oppor- wage information, Unemployment Insurance
tunities to enhance evidence-based policymaking beneft information, and information on new-
in the United States. Secure access to data, includ- ly hired employees. Statutory Citation: 42 USC
ing data in the domains listed below, is addressed 653(j)(5).
throughout the Commissions report.
Te data described below are not intended Unemployment Insurance (UI)* quarterly
to refect a comprehensive inventory of govern- wage records are collected by states as part of
ment data, but rather refect suggestions provid- their administration of state UI programs oper-
ed to the Commission about some data that have ated through the Federal-state UI partnership.
the potential to be useful for evidence building.
Numerous other administrative and survey data Te Wage Record Interchange System
sources are likely to be no less valuable for evi- (WRIS)*, maintained by the Department of
dence building in their respective policy domains. Labor, facilitates the exchange of wage data
In some cases, the purposes for which data may between states for performance accountability
be used are defned narrowly, limiting their use purposes, enabling improved reporting on the
for evidence building. Te Commission includes outcomes experienced by participants in Fed-
recommendations to reconsider these limitations. erally funded state employment and training
Recommendation 2-4 proposes a review of such programs. It allows researchers access to wage
statutes to ensure that limitations that preclude data on a state-by-state approval basis for proj-
the use of administrative data for evidence build- ects with a direct beneft to these programs.
ing are applied only when the Congress and the Te WRIS is being revised to refect changes
President deem the limitations still to be nec- introduced by the Workforce Innovation and
essary. In some cases, existing laws specifcally Opportunity Act, passed in 2014, and will be
prohibit the collection or analysis of information called the State Wage Interchange System.
to support evidence building. Recommendation Statutory Citation: 42 USC 1111.
2-5 calls for a reconsideration of such bans and
restraint in the enactment of future bans. Data Health
relevant for implementing Recommendations 2-4
and 2-5 include those indicated with an asterisk. Health Services Research Program data are
collected by the Substance Abuse and Mental
Health Services Administration (SAMHSA)
Administrative Data in HHS. Data include research related to
substance abuse treatment, access to mental
Income, Wages, and Earnings health care, healthcare costs, and patient
outcomes. Statutory Citation: 42 USC 290aa.
Federal Tax Information* is collected by the
Internal Revenue Service (IRS) in the Depart- Medicaid, Medicare, and Childrens Health
ment of the Treasury and includes business Insurance Program (CHIP) data are collected
126 The Promise of Evidence-Based Policymaking

by the Centers for Medicare and Medicaid Ser- tures Family and Nutrition Services. SNAP data
vices (CMS) in HHS. Data include information include information about program eligibility,
on Medicare claims, benefciaries and provid- length of services, and amounts of benefts.
ers, Medicaid eligibility, and claims. Statutory Statutory Citation: 7 USC 2011 et seq.
Citations: 42 USC 1396 (Medicaid and CHIP), 42
USC 1395 (Medicare). Supplemental Security Income (SSI) Program
data collected by states and SSA includes
Vital Records administrative data are collected income assistance information for those partic-
by the National Center for Health Statistics ipants who are needy, aged, blind, or disabled.
(NCHS) in HHS from vital registrars in all states Statutory Citation: 42 USC 1383.
and U.S. territories. Vital Records fles include
information about births and deaths. Statutory Temporary Assistance for Needy Families
Citation: 42 USC 242k(h). (TANF) is administered by ACFs Ofce of
Family Assistance in HHS. TANF program data
Human Services includes caseloads, fnancial information,
workforce participation rates and other pro-
Child Welfare Program data include statistics gram participant characteristics. Statutory Cita-
collected through the Adoption and Foster tion: 42 USC 611-611a.
Care Analysis and Reporting System (AFCARS)
and through the National Child Abuse and Ne- Unemployment Insurance (UI) payments
glect Data System (NCANDS). AFCARS collects (benefts) provide temporary fnancial assis-
case-level information on all foster children and tance to unemployed workers who meet the
those who have been adopted in the United requirements of State law. Information about
States. NCANDS includes data on child abuse payments includes the benefts paid and dura-
and neglect cases in the United States. Both data tion. Statutory Citation: 29 USC 49b(b).
collection systems are administered by the Chil-
drens Bureau of ACF. Statutory Citations: 42 USC Other
679 (AFCARS) and 42 USC 5119 (NCANDS).
Business Program Participation data, such as
Head Start Program* data are collected by local the small business loan program at the Small
jurisdictions through a program funded by the Business Administration includes information
ACF in HHS about program participants. Statu- on lenders, borrowers, and loan amounts.
tory Citation: 42 USC 9836a and 9844. Data collected by the Economic Development
Administration refects information about
Social Security Disability Insurance (SSDI) grantees use of funds to support workforce
data are collected about program length of ser- development and other regional economic
vice, benefts, eligibility, appeals, and denials opportunities. Statutory Citations: 15 USC 638
through the Social Security Administration (Small Business Administration) and 42 USC
(SSA) and Disability Determination Services in 3192 (Economic Development Administration).
each state. Statutory Citation: 42 USC 401.
Criminal justice data include crime incident
Social Security Old Age and Survivors Insur- reports and criminal history record informa-
ance (OASI) Program data includes informa- tion, or rap sheets. Crime incident data are
tion on retired and survivor benefciaries and collected through the Federal Bureau of Inves-
beneft amounts received. Statutory Citation: 42 tigation (FBI) National Incident-Based Report-
USC 401. ing System (NIBRS). Data include the nature
and types of specifc ofenses in the incident,
Supplemental Nutrition Assistance Program characteristics of the victim(s) and ofender(s),
(SNAP) data are collected by all states through a types and value of property stolen and recov-
program funded by the Department of Agricul- ered, and characteristics of persons arrested in
The Promise of Evidence-Based Policymaking 127

connection with a crime incident. Rap sheets to serve a statistical purpose. In some cases, the sta-
are created and maintained by law enforce- tistical purposes for which a statistical agency may
ment agencies and accessible via the FBI Inter- grant access to confdential data under applicable
state Identifcation Index for law enforcement law are defned very narrowly; the Census Bureaus
and other purposes. Tey contain information law, for example, permits the use of sworn agents
about an individuals arrests and subsequent such as researchers only to help the agency carry
dispositions, such as jail sentences. Statutory out its work, and projects must specifcally advance
Citations: 42 USC 3732 and 3735 (NIBRS), 28 the mission of the Census Bureau irrespective of
USC 534 (criminal records). the broader value that access to data might have for
evidence building. Examples of statistical data from
Federal Student Aid* data are collected by the Principal Statistical Agencies are listed below.
the Department of Education. Data include
information about the Federal assistance given
to students and families under Title IV of the
Statistical Agency Data1
Higher Education Act of 1965, including stu- Bureau of Economic Analysis, in the Depart-
dent aid applications, student loan applications ment of Commerce, collects data on multina-
and receipt, and characteristics of applicants. tional enterprises and international services
Statutory Citation: 20 USC 1092b(d)(2). transactions. Statutory Citations: 22 USC 3101-
3108.
Housing Assistance Data are collected by pub-
lic housing agencies funded by the Department Bureau of Justice Statistics, in the Department
of Housing and Urban Development (HUD) and of Justice, collects data on crime, ofenders,
maintained in shared systems, including the crime victims, and the operations of the crim-
Public and Indian Housing Information Center inal justice system at all levels of government.
and the Tenant Rental Assistance Certifcation Statutory Citations: 42 USC 3732, 42 USC 3735
System. Data collected include characteristics and 3789g.
of properties and units as well as information
about the households receiving subsidized Bureau of Labor Statistics, in the Department
housing assistance. Data regarding partici- of Labor, collects data on employment and un-
pation in homeless assistance programs are employment, projections of the labor force and
collected at the local level through Homeless employment by industry and occupation, prices
Management Information Systems. Statutory and infation, consumer expenditures, wages
Citations: 42 USC 3535 and 3543. and employee benefts, occupational injuries
and illnesses, and productivity and technolog-
Military and Veteran Program data are col- ical change. Statutory Citations: 29 USC 2, 2b, 5,
lected by a variety of sources, including the 6, 8, and 9; 19 USC 2393; 29 USC 673.
Department of Defense and the Department
of Veteran Afairs, among other departments. Bureau of Transportation Statistics, in the
Tese data include demographic, health, and Department of Transportation, collects data
other information about members of the active on transportation and transportation systems,
duty military and their families as well as ser- including freight and travel statistics. Statutory
vices provided to veterans. Citations: 49 USC 6302(c); 49 USC 6306; 49
USC 6307; 49 USC 6313.
Although Federal statistical agencies produce
many statistical data products that are available to Census Bureau, in the Department of Com-
the public without restriction, access to data fles merce, conducts the Decennial Census and
produced by these agencies that contain confden-
tial information about individual people, business-
1. U.S. Congressional Research Service, Selected Information Re-
es, or organizations is, quite appropriately, tightly garding Statistical and Evaluation Entities in the Executive Branch
controlled. Tis is true even when access is sought [Memo] (February 28, 2017), by Clinton T. Brass and Jennifer D.
Williams; https://www.cep.gov/library/testimony.html.
128 The Promise of Evidence-Based Policymaking

administers surveys that collect information on related to education in the United States and
the demographic and economic health of the in other nations. Statutory Citations: 20 USC
United States, such as the American Communi- 9541-9544, 20 USC 9546), 20 USC 9573.
ty Survey and the Survey of Income and Program
Participation. Statutory Citation: 13 USC 9. National Center for Health Statistics, in HHS,
collects data on the extent and nature of illness
Economic Research Service, in the Depart- and disability of the population of the United
ment of Agriculture, collects data on the eco- States and several other specifed metrics. Stat-
nomic and social science aspects of agriculture, utory Citation: 42 USC 242k.
rural development, food, commodity markets,
and the environment. Statutory Citation: 7 USC National Center for Science and Engineering
2276. Statistics, in the National Science Foundation,
conducts periodic surveys and data collections
Energy Information Administration, in the on science, engineering, technology, and re-
Department of Energy, collects data and infor- search and development. Statutory Citation: 42
mation which is relevant to energy resource USC 1862p.
reserves, energy production, demand, and
technology, and related economic and statis- Ofce of Research, Evaluation, and Statistics,
tical information. Statutory Citation: 42 USC in SSA, is responsible for conducting policy
7135(d). research and evaluation, providing statistical
data on SSA programs, sponsoring special-pur-
National Agricultural Statistics Service, in the pose survey data collections and studies, and
Department of Agriculture, conducts the Cen- other activities related to SSAs mission.
sus of Agriculture and collects data on agricul-
tural production and the economic and envi- Statistics of Income Division, in the IRS of the
ronmental status of the farm sector. Statutory Department of the Treasury, collects and dis-
Citations: 7 USC 2276, 7 USC 2204g. seminates income, fnancial, and tax informa-
tion and also provides periodic reports on items
National Center for Education Statistics, in from other returns and schedules. Statutory Ci-
the Department of Education, collects data tation: 26 USC 6108(c).
The Promise of Evidence-Based Policymaking 129

Te Commissions collection of public input,


survey data, and related materials are available
via the National Archives and Records Adminis-
tration. Te materials available from the Archives
include:
Appendix E. Report on the CEP Survey of
Federal Ofces
Appendix F. CEP Public Meeting Materials
and Presentations
Appendix G. CEP Public InputHearing
Testimony and Other Public Comments
Appendix H. Prior Commissions Related to
Evidence Building
This page intentionally left blank
Te Promise of Evidence-Based Policymaking

Report of the Commission on Evidence-Based Policymaking

September 2017