Research

Overview

Research is a process used to discover new knowledge. In the Code of Federal Regulations (45 CFR 46.102(d), research is defined as: “A systematic investigation (i.e., the gathering and analysis of information) designed to develop or contribute to generalizable knowledge.” The National Academy of Sciences notes the object of research is to extend human knowledge beyond what is already known. Research differs from other forms of discovering knowledge (such as reading a book) because it uses a systematic process, the scientific method.

No matter what topic is being studied, the value of the research depends on how well it is designed and conducted. This fact is especially pertinent to research related to the nation’s learn-and-work ecosystem and its many areas of investigation (e.g., credentials and providers; employers and workforce; career navigation; quality and value; and verification and recordkeeping of learning).

Many factors affect research conducted in the learn-and-work ecosystem. For example:

• The ecosystem’s interdisciplinary nature attracts scholars with diverse backgrounds, including geography, law, education, business, political science, economics, data systems, and assessment/measurement.
• Researchers face challenges with selection effects when studying credentials. For instance, it can be difficult to determine whether a particular credential actually leads to higher wages or whether the wage benefits stem from the fact that  more qualified individuals self-select into a program that offers that credential.
• The universe of non-degree credentials is large, complex, and ill-defined. No single, complete registry of non-degree credentials exists. The situation is markedly different with degrees. Information about them is standardized and readily available (via the federal Integrated Postsecondary Education Data System (IPEDS) and the National Student Clearinghouse®.) 
• Debate in the research community on which outcomes matter most—(e.g.,  program completion, wages, total earnings, job quality, job and life satisfaction, etc. 

The Non-degree Credential Research Network (NCRN report) has identified the most pressing research questions for the alternative credentialing (non-degree) component of the ecosystem. These include questions such as:

• Why are so many non-degree credentials emerging? 
• How are employers using the wider array of credentials being earned?
• What are the implications of non-degree credentials for equity in the labor market?
• How do we differentiate between high- and low-quality non-degree credentials?
• How do employers value non-degree credentials relative to degrees?
• How effective is the public workforce system in supporting the attainment of quality non-degree credentials?

Research in the learn-and-work ecosystem is incomplete and somewhat scattered. Many topics could benefit from study, but due to lack of grant funding—from federal sources, state contracts, and/or foundation grants and contracts—too few studies are conducted. The majority of researchers are employed at grant-funded university centers or institutes or at intermediary research centers and think tanks. These entities, whether located at a university or a for-profit or nonprofit intermediary, typically depend on outside funds to conduct research. This means that priority topics receive uneven attention, and that research often ends too soon—i.e., when grant funds end, not when reliable results are achieved. 

Many entities conduct research and evaluation studies into the policies and programs designed to improve the learn-and-work ecosystem. Major areas of investigation have focused on apprenticeships; college and career readiness; federal education legislation; and wage growth and employability through credentialing in a variety of industry sectors. Examples of entities conducting research and evaluation studies include: 

• Achieving the Dream
• American Association of Community Colleges 
• American Council on Education (ACE)
• American Institutes for Research (AIR)
• Brookings Institution 
• Community College Research Center at Columbia University 
• Corporation for a Skilled Workforce
• Credential As You Go
• Credential Engine
• Education Design Lab
• Education Strategy Group
• Georgetown Center for Education and the Workforce
• Harvard University Project on Workforce; the Harvard Business School Managing the Future of Work Project
• Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA), which led to the 2016 Adult Training and Education Survey (ATES) 
• Jobs for the Future
• Opportunity@Work
• Manhattan Institute
• MHEC
• National Skills Coalition
• NEBHE
• New America’s Center on Education & Labor 
• Northeastern University’s Center for the Future of Higher Education and Talent Strategy
• Program on Skills, Credentials & Workforce Development, George Washington University
• National Student Clearinghouse Research Center 
• SHEEO
• SHRM
• WICHE
• Northeastern University’s Center for the Future of Higher Education and Talent Strategy,
• RAND
• Rutgers Education and Employment Research Center
• Urban Institute / WorkRise
• U.S. Chamber of Commerce Foundation
• Workcred
  

A growing issue in the research community is the changing “innovation” landscape within the learn-and work ecosystem. Researchers have some tried-and-true methods to study outcomes, but they are few—and increasingly inadequate to fully study a complex, dynamic ecosystem. Many  innovations in credentialing, for example, seem sound and necessary. But evidence of actual outcomes—verifiable results, particularly for learners, credential providers, and employers—is needed to warrant full-scale implementation of these innovative approaches. 

Education researchers by necessity develop new methods to examine outcomes, such as finding ways to match new micro-credential offerings with existing systems of degrees and certificates. Such workarounds are necessary until higher education systems (1) collect data on all credential programs, (2) develop ways to track progress toward completion (rather than simply counting completions), and (3) better link academic outcomes to employment and wage information. 

Further, higher education system data are not typically structured to determine outcomes—even using well-established measures—for learners who pursue non-degree credentials. As example, it may not even be possible to identify in a data system if a student is pursuing an “incremental credential.” And many systems provide no way to track non-credit learning, non-credit to credit pathways, or pathways at all. Degree programs and course catalogs are arguably structured for the efficient delivery of teaching-and-learning activities en masse. Recognition of course or degree completion rests on a specious assumption of uniformity—an implicit agreement among educators, learners, and the workforce that all students who have completed a given degree are equally prepared, have demonstrated the same outcomes, and are similarly qualified. 

The U.S. credentialing system, which includes degrees and certificates, is moving toward credentialing processes and requirements that are more diverse, flexible, and tailored to the needs of individuals. As this set of innovations becomes more widely institutionalized, research study design and implementation face significant challenges. Four will be especially significant: 

• The treatment being studied is variable by design. In research studies, it is generally assumed that everyone in a given group is getting the same treatment. A one-size-fits-all innovation is easy to study rigorously. However, when a credentialing approach allows greater “flexibility” (often tailoring to the needs of an individual learner), the treatment being assessed is, by its nature, variable. Given the desire to understand what works at scale, education research studies hinge on the assumption that large numbers of students get the same treatment. When the “same treatment” means that every student gets a program that is of the greatest benefit to him or her individually, that flies in the face of the expectation that such studies assess and manage implementation quality and fidelity. In this way, priorities of programming and research (that everyone gets the same thing) confound the effort to ensure equity (that everyone gets what they need), the latter being a primary aim of innovative credentialing approaches.
• Analytic power becomes a problem. Postsecondary programs are innovating at an unrivaled pace, thanks to changing workplace demands and calls for recognition of a wider array of valuable credentials—degree and non-degree. This will result in many more credential offerings with fewer students potentially in each. This will likely be problematic, because quality impact research requires volumes of data large enough to enable satisfactory analysis.
• Innovation and scale-up are in conflict. As we study credentials in a period of rapid innovation, it is difficult to know when a new credential opportunity is “done” and ready for repeatable testing at greater scale. It is also difficult to scale an intervention that is inherently flexible to the context in which it is being offered. And if innovations cannot be replicated at scale, they cannot be rigorously tested to assess if they “actually work”—the priority long established in education research.
• If innovative credentials work, traditional measures stop working. The purpose and theory behind any education innovation drives decisions about which outcome measures matter. Traditional credentialing approaches have established a set of measures of higher education success. However, as the array of credentials broadens, those measures miss crucial aspects of new options. As we expand the aims of credentialing, we by necessity redefine what is important to measure. Until now, it has generally been sufficient to track whether a learner completes a degree (and maybe how long that takes). Going forward, it becomes necessary to measure how many different credentials an individual gains, how those credentials are connected, how well they meet educational and employment expectations, and ultimately how they further learners’ goals.

Examples

• The Georgetown Center on Education and the Workforce is a research and policy institute that studies the links between education, career qualifications and workforce demands. Since 2008, CEW has conducted research related to jobs, skills, and equity to better inform students, parents, teachers, and policymakers about the changing relationship between education and careers. Research findings are disseminated widely through a robust website that offers, reports, executive summaries, infographics, newsletters, blogs, and videos. The CEW’s Good Jobs project, launched in 2017, investigated the impact structural economic change has had on workers at different education levels. The project includes resources that show the concentration of good jobs by industry and occupation. CEW continues to study the ongoing impact of COVID-19 on the following categories: Jobs: Industries, occupations; Workers: Race, class, gender, educational attainment; Education pipeline: Higher education, K-12;  Students: High school, college Households: Stimulus payments, loss of employment income.
• The Non-Degree Credentials Research Network (NCRN) is a network of researchers working on various types of non-degree credentialing including licenses, certifications, apprenticeships, certificates, and microcredentials. NCRN includes experts from higher education institutions, research firms, and nonprofit organizations, among others. It was established to address the gap in understanding the role of credentials in contemporary labor markets. 
• WorkRise, hosted by the Urban Institute, is a national platform for identifying, testing, and sharing bold ideas for transforming the labor market. This is done by funding research on promising practices, policies, and programs across the country as well as foundational research on labor market trends. Projects generate data and evidence that strengthens employers, informs policymaking, and provides genuine economic mobility and security for workers—especially those who face systemic barriers to opportunity. WorkRise convenes and collaborates with often-siloed groups—employers, worker advocates, practitioners, policymakers, scholars, and philanthropists—to ensure the solutions delivered are relevant and scalable. WorkRise funds scholars and practitioners working to improve economic mobility in the U.S. labor force. Priority topics include: Economic Context, Demographic Disparities, Employer Practices, Worker Power, Skills and Training.
• The Hechinger Report (a national nonprofit newsroom) provides in-depth reporting on education. Its October 2022 report on higher education featured What researchers learned about online higher education during the pandemic. The article highlights a unique and major development from the COVID-19 pandemic: Prior to the pandemic, scientific research was often missing from studies of the effectiveness of online higher education: a control group. But during the pandemic, with nearly everyone going online for their education, the result was a randomized trial on a planetary scale with an immense control group.