Research is a process used to study materials and sources in order to establish facts and reach new conclusions. In the Code of Federal Regulations (45 CFR 46.102(d), research is defined as: “A systematic investigation (i.e., the gathering and analysis of information) designed to develop or contribute to generalizable knowledge.” The National Academy of Sciences notes the object of research is to extend human knowledge beyond what is already known. Research differs from other forms of discovering knowledge (such as reading a book) because it uses a systematic process, the scientific method.

No matter the specific topic being studied, the value of the research depends on how well it is designed and conducted. This fact is especially pertinent to research related to the nation’s learn-and-work ecosystem and its many areas of investigation (e.g., credentials and providers; employers and workforce; career navigation; quality and value; and verification and recordkeeping of learning).

Many factors affect research conducted in the learn-and-work ecosystem. For example:

• The ecosystem’s interdisciplinary nature attracts scholars with diverse backgrounds, including geography, law, education, business, political science, economics, data systems, and assessment/measurement.
• Researchers face challenges with selection effects when studying credentials. For instance, it can be difficult to determine whether a particular credential leads to higher wages or whether the wage benefits stem from the fact that more qualified individuals self-select into a program that offers that credential.
• The universe of non-degree credentials is large, complex, and ill-defined. No single, complete registry of non-degree credentials exists. The situation is markedly different with degrees. Information about them is standardized and readily available (via the federal Integrated Postsecondary Education Data System [IPEDS] and the National Student Clearinghouse®). 
• Debate continues in the research community on which outcomes matter most; e.g., program completion, wages, total earnings, job quality, and job and life satisfaction.  

The Non-Degree Credential Research Network (NCRN 2021 report) has identified many pressing research questions for the alternative credentialing (non-degree) component of the ecosystem. These include questions such as: (1) Why are so many non-degree credentials emerging? (2) How are employers using the wider array of credentials being earned? (3) What are the implications of non-degree credentials for equity in the labor market? (4) How do we differentiate between high- and low-quality non-degree credentials? (5) How do employers value non-degree credentials relative to degrees? (6) How effective is the public workforce system in supporting the attainment of quality non-degree credentials?

Research in the learn-and-work ecosystem is incomplete and often scattered. Many topics could benefit from study, but due to a lack of grant funding from federal sources, state contracts, and/or foundation grants and contracts, too few studies are conducted. The majority of researchers are employed at grant-funded university centers or institutes, or intermediary research centers and think tanks. These entities, whether located at a university or a for-profit or nonprofit intermediary, typically depend on outside funds to conduct research. This means that priority topics receive uneven attention, and that research often ends too soon—i.e., when grant funds end, not when reliable results are achieved. 

Changing Research Methods & Data Systems 

A growing issue in the research community is the changing “innovation” landscape within the learn-and-work ecosystem. Researchers have some tried-and-true methods to study outcomes, but they are few and increasingly inadequate to fully study a complex, dynamic ecosystem. Many innovations in credentialing, for example, seem sound and necessary. But evidence of actual outcomes—verifiable results, particularly for learners, credential providers, and employers—is needed to warrant full-scale implementation of these innovative approaches.

Education researchers, by necessity, develop new methods to examine outcomes, such as finding ways to match new micro-credential offerings with existing systems of degrees and certificates. Such workarounds are necessary until higher education systems (1) collect data on all credential programs, (2) develop ways to track progress toward completion (rather than simply counting completions), and (3) better link academic outcomes to employment and wage information. 

Further, higher education system data are not typically structured to determine outcomes, even using well-established measures, for learners who pursue non-degree credentials. For example, it may not even be possible to identify in a data system if a student is pursuing an “incremental credential.” And many systems provide no way to track non-credit learning, non-credit to credit pathways, or pathways at all. Degree programs and course catalogs are arguably structured for the efficient delivery of teaching and learning activities en masse. Recognition of course or degree completion rests on a specious assumption of uniformity, an implicit agreement among educators, learners, and the workforce that all students who have completed a given degree are equally prepared, have demonstrated the same outcomes, and are similarly qualified. 

The U.S. credentialing system, which includes degrees and certificates, is moving toward credentialing processes and requirements that are more diverse, flexible, and tailored to the needs of individuals. As this set of innovations becomes more widely institutionalized, research study design and implementation face significant challenges. Four are believed to be especially significant: 

• The treatment being studied is variable by design. In research studies, it is generally assumed that everyone in a given group is getting the same treatment. A one-size-fits-all innovation is easy to study rigorously. However, when a credentialing approach allows greater “flexibility” (often tailoring to the needs of an individual learner), the treatment being assessed is, by its nature, variable. Given the desire to understand what works at scale, education research studies hinge on the assumption that large numbers of students get the same treatment. When the “same treatment” means that every student gets a program that is of the greatest benefit to him or her individually, that flies in the face of the expectation that such studies assess and manage implementation quality and fidelity. In this way, priorities of programming and research (that everyone gets the same thing) confound the effort to ensure equity (that everyone gets what they need), the latter being a primary aim of innovative credentialing approaches.
• Analytic power becomes a problem. Postsecondary programs are innovating at an unrivaled pace, thanks to changing workplace demands and calls for recognition of a wider array of valuable credentials, degrees, and non-degree programs. This will result in many more credential offerings with fewer students potentially in each. This will likely be problematic because quality impact research requires volumes of data large enough to enable satisfactory analysis.
• Innovation and scale-up are in conflict. As we study credentials in a period of rapid innovation, it is difficult to know when a new credential opportunity is “done” and ready for repeatable testing at a greater scale. It is also difficult to scale an intervention that is inherently flexible to the context in which it is being offered. And if innovations cannot be replicated at scale, they cannot be rigorously tested to assess if they “actually work”, the priority long established in education research.
• If innovative credentials work, traditional measures stop working. The purpose and theory behind any educational innovation drive decisions about which outcome measures matter. Traditional credentialing approaches have established a set of measures of higher education success. However, as the array of credentials broadens, those measures miss crucial aspects of new options. As we expand the aims of credentialing, we by necessity redefine what is important to measure. Until now, it has generally been sufficient to track whether a learner completes a degree (and maybe how long that takes). Going forward, it becomes necessary to measure how many different credentials an individual gains, how those credentials are connected, how well they meet educational and employment expectations, and ultimately, how they further learners’ goals.

Background Research & Resources

A Non-Degree Credentials Literature Scan was conducted in 2019 (Kyle Albert, George Washington University). The scan covers documents that describe original research; opinion pieces are excluded, though the scan cites works that analyze the published research of others. Since stakeholders in the world of non-degree credentials differ in their approaches to research and standards of evidence, the scan errs on the side of being inclusive without altering the view of the literature with case studies and industry benchmarking reports. The scan focuses on books and works published in scholarly journals, though it also draws on reports self-published by institutions with a track record for conducting rigorous analyses of educational outcomes.

The scan located literature addressing 5 research questions for each type of credential being considered:

1. How prevalent are non-degree credentials in the U.S. workforce?
2. Who benefits from non-degree credentials in terms of labor market outcomes, and how much?
3. Why have non-degree credentials proliferated?
4. How do employers perceive and use non-degree credentials?
5. What motivates individuals to pursue non-degree credentials?

The scan identified literature by both conducting targeted searches for information on the specific research questions and by searching for work by specific researchers known to be active on the topic of non-degree credentials through the process of identifying nominees for the NCRN. Literature published within the past 10 years was given priority, though influential studies from earlier times were included where appropriate. This scan was not limited to research conducted inside or about the U.S., though the literature was exclusively in English, and the vast majority was conducted by U.S.-based researchers. Studies of non-U.S. credentials (such as British and German apprenticeship programs) were included to the extent that they might inform policy or practice in the U.S..

Also developed in 2019, the Zotero Library on Non-Degree Credentialing Research (NCRN Shared Zotero Library) (Kyle Albert, George Washington University) contains 133 items. They include works cited in the Non-Degree Credentials Literature Scan, plus additional citations that may be helpful to those who study non-degree credentials. The library includes literature relevant to all types of non-degree credentials, including occupational and professional certifications, licenses, certificates, apprenticeships, internships, badges, nano/micro-credentials, bootcamps, and other emergent credentialing phenomena. Most entries (and all abstracts/keywords/tags) have been pulled from library catalogs and databases, which may result in some inconsistencies in formatting.

In 2022, the NCRN Conference: Non-degree Credentials on the Move resulted in a number of presentations on research topics relevant to the learn-and-work ecosystem. Slide decks of these presentations include (PowerPoint downloads unless noted):

• International Developments in Micocredentialing: Margo Griffith
• Data Issues Impacting Researchers and Users of Research: Kyle Albert
• Data Issues Impacting Researchers and Users of Research: Mark D'Amico
• Data Issues Impacting Researchers and Users of Research: Heather McKay
• Data Issues Impacting Researchers and Users of Research: Felicia Perez
• Data Issues Impacting Researchers and Users of Research: Andrew Reamer
• Labor-Market Outcomes of Non-degree Credentials: Matthew Baird
• Labor-Market Outcomes of Non-degree Credentials: Ashley Edwards
• Labor-Market Outcomes of Non-degree Credentials: Daniel Kuehn
• Public Policies Shaping the Non-degree Marketpace: Dana Kunzman, Nick Moore, Cynthia Proctor, Michelle Van Noy
• Credentials As You Go: Kirk Knestis, Nan Travers, Holly Zanville
• Higher Education as a Provider of Non-degree Credentials: Julian Alssid
• Higher Education as a Provider of Non-degree Credentials: Jim Fong (Pdf)
• Higher Education as a Provider of Non-degree Credentials: Christos Makridis
• Summary of NCRN Session on International Developments in Micro-Credentialing contributed by Michael Fung, Tecnologico de Monterrey (Pdf)

Examples of Research & Evaluation Entities

Many entities conduct research and evaluation studies into policies and programs designed to improve the learn-and-work ecosystem. Major areas of investigation focus on apprenticeships, college and career readiness, federal education legislation, and wage growth and employability through credentialing in a variety of industry sectors.

Examples of the many entities that conduct research and evaluation studies include:

• Achieving the Dream
• American Association of Community Colleges (AACC)
• American Council on Education (ACE)
• American Institutes for Research (AIR)
• Brookings Institution
• Burning Glass Institute
• CAEL
• Community College Research Center (CCRC), Columbia University
• Center for the Future of Higher Education and Talent Strategy, Northeastern University
• Corporation for a Skilled Workforce (CSW)
• Credential As You Go
• Credential Engine
• Education Design Lab
• Education Strategy Group (ESG)
• Education & Employment Research Center, Rutgers University
• Federal Reserve Bank
• Georgetown Center for Education and the Workforce, Georgetown University
• Harvard University Project on Workforce; the Harvard Business School Managing the Future of Work Project
• Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA), which led to the 2016 Adult Training and Education Survey (ATES)
• Jobs for the Future (JFF)
• Opportunity@Work
• Manhattan Institute
• Midwestern Higher Education Compact (MHEC)
• National Skills Coalition (NSC)
• National Student Clearinghouse Research Center
• New England Board of Higher Education (NEBHE)
• New America’s Center on Education & Labor
• Program on Skills, Credentials & Workforce Development, George Washington University
• RAND
• State Higher Education Executive Officers (SHEEO)
• SHRM
• UPCEA – The Online and Professional Education Association
• Urban Institute / WorkRise
• U.S. Chamber of Commerce Foundation
• Western Interstate Commission for Higher Education (WICHE)
• Western Governors University (WGU)
• Workcred

Future Research Questions for Non-Degree Credentialing 

New Directions for Non-Degree Credentialing Research: Briefing for Stakeholders of the
Non-Degree Credentials Research Network |
Spring 2021 identified the 15 most pressing research questions believed to be a priority for the research community:

1. Why are so many non-degree credentials emerging?
2. Why do some learners (and not others) choose non-degree credentials?
3. How do individuals choose between non-degree credential fields and programs?
4. Who starts, but does not complete, programs leading to non-degree credential programs?
5. What are the implications of non-degree credentials for equity in the labor market?
6. What barriers exist to the attainment of non-degree credentials?
7. How do we differentiate between high- and low-quality non-degree credentials?
8. What is the value of non-credentialed training and learning experiences?
9. What types of certificates are of greatest value to different subpopulations of learners?
10. How do employers value non-degree credentials relative to degrees?
11. What are the non-wage benefits of non-degree credential attainment?
12. How do the long-term outcomes associated with online non-degree credentials compare to high-contact non-degree credentials?
13. Are more intensive non-degree credentials more valuable than shorter, less rigorous non-degree credentials in the labor market?
14. How effective is the public workforce system in supporting the attainment of quality non-degree credentials?
15. Would innovative credentials gaining traction in the United States be of value in the context of developing countries?

The Education and Employment Research Center (EERC) is a nonprofit research organization based in Rutgers University’s School of Management and Labor Relations. EERC’s research and evaluation examine the intersection between education and work to be an evidence-based resource for policy and practice. Work focuses on issues of equity and quality in education, training for students and workers, the institutions that serve them, and the broader economy in the context of the changing labor market associated with the future of work. Research projects focus on five areas: Student Choices and Pathways; STEM and Technician Education; Noncredit Education and Non-Degree Credentials; Community College Innovation; and Education and Labor Market Connections

The Georgetown Center on Education and the Workforce (CEW) is a research and policy institute that studies the links between education, career qualifications, and workforce demands. Since 2008, CEW has conducted research related to jobs, skills, and equity to better inform students, parents, teachers, and policymakers about the changing relationship between education and careers. Research findings are disseminated widely through a robust website that offers reports, executive summaries, infographics, newsletters, blogs, and videos. The CEW’s Good Jobs project, launched in 2017, investigated the impact structural economic change has had on workers at different education levels. The project includes resources that show the concentration of good jobs by industry and occupation. CEW continues to study the ongoing impact of COVID-19 on the following categories: Jobs (industries, occupations); Workers (race, class, gender, educational attainment); Education Pipeline (higher education, K-12); Students (high school, college); Households (stimulus payments, loss of employment income).

The Hechinger Report (a national nonprofit newsroom) provides in-depth reporting on education. Its October 2022 report on higher education featured What researchers learned about online higher education during the pandemic. The article highlights a unique and major development from the COVID-19 pandemic: Before the pandemic, the scientific research key was often missing from studies of the effectiveness of online higher education, a control group. But during the pandemic, with nearly everyone going online for their education, the result was a randomized trial on a planetary scale with an immense control group.

WorkRise, hosted by the Urban Institute, is a national platform for identifying, testing, and sharing new ideas for transforming the labor market. This is done by funding research on promising practices, policies, and programs across the country, as well as foundational research on labor market trends. Projects generate data and evidence that strengthen employers, inform policymaking, and provide genuine economic mobility and security for workers, especially those who face systemic barriers to opportunity. WorkRise convenes and collaborates with often-siloed groups such as employers, worker advocates, practitioners, policymakers, scholars, and philanthropists to ensure the solutions delivered are relevant and scalable. WorkRise funds scholars and practitioners working to improve economic mobility in the U.S. labor force. Priority topics include: Economic Context, Demographic Disparities, Employer Practices, Worker Power, Skills and Training.

Knestis, K. & Zanville. H. (May 3, 2022). Reflections On Conducting Research in A Changing Credentialing Ecosystem. Credential As You Go

Marcus, J. (October 6, 2022). What researchers learned about online higher education during the pandemic. The Hechinger Report.

What We Know About Non-Degree Credentials: A Literature Scan. A project of the Non-Degree Credentials Research Network at the George Washington Institute of Public Policy, The George Washington University. 6/23/2019  https://gwipp.gwu.edu/sites/g/files/zaxdzs2181/f/downloads/06.23.19%20NCRN%20Literature%20Scan.pdf