Research
Overview
Research is a process used to discover new knowledge. In the Code of Federal Regulations (45 CFR 46.102(d), research is defined as: “A systematic investigation (i.e., the gathering and analysis of information) designed to develop or contribute to generalizable knowledge.” The National Academy of Sciences notes the object of research is to extend human knowledge beyond what is already known. Research differs from other forms of discovering knowledge (such as reading a book) because it uses a systematic process, the scientific method.
No matter what topic is being studied, the value of the research depends on how well it is designed and conducted. This fact is especially pertinent to research related to the nation’s learn-and-work ecosystem and its many areas of investigation (e.g., credentials and providers; employers and workforce; career navigation; quality and value; and verification and recordkeeping of learning).
Many factors affect research conducted in the learn-and-work ecosystem. For example:
- The ecosystem’s interdisciplinary nature attracts scholars with diverse backgrounds, including geography, law, education, business, political science, economics, data systems, and assessment/measurement.
- Researchers face challenges with selection effects when studying credentials. For instance, it can be difficult to determine whether a particular credential actually leads to higher wages or whether the wage benefits stem from the fact that more qualified individuals self-select into a program that offers that credential.
- The universe of non-degree credentials is large, complex, and ill-defined. No single, complete registry of non-degree credentials exists. The situation is markedly different with degrees. Information about them is standardized and readily available (via the federal Integrated Postsecondary Education Data System (IPEDS) and the National Student Clearinghouse®.)
- Debate in the research community on which outcomes matter most—(e.g., program completion, wages, total earnings, job quality, job and life satisfaction, etc.
The Non-degree Credential Research Network (NCRN report) has identified the most pressing research questions for the alternative credentialing (non-degree) component of the ecosystem. These include questions such as:
- Why are so many non-degree credentials emerging?
- How are employers using the wider array of credentials being earned?
- What are the implications of non-degree credentials for equity in the labor market?
- How do we differentiate between high- and low-quality non-degree credentials?
- How do employers value non-degree credentials relative to degrees?
- How effective is the public workforce system in supporting the attainment of quality non-degree credentials?
Research in the learn-and-work ecosystem is incomplete and somewhat scattered. Many topics could benefit from study, but due to lack of grant funding—from federal sources, state contracts, and/or foundation grants and contracts—too few studies are conducted. The majority of researchers are employed at grant-funded university centers or institutes or at intermediary research centers and think tanks. These entities, whether located at a university or a for-profit or nonprofit intermediary, typically depend on outside funds to conduct research. This means that priority topics receive uneven attention, and that research often ends too soon—i.e., when grant funds end, not when reliable results are achieved.
Many entities conduct research and evaluation studies into the policies and programs designed to improve the learn-and-work ecosystem. Major areas of investigation have focused on apprenticeships; college and career readiness; federal education legislation; and wage growth and employability through credentialing in a variety of industry sectors. Examples of entities conducting research and evaluation studies include:
- Achieving the Dream
- American Association of Community Colleges
- American Council on Education (ACE)
- American Institutes for Research (AIR)
- Brookings Institution
- Community College Research Center at Columbia University
- Corporation for a Skilled Workforce
- Credential As You Go
- Credential Engine
- Education Design Lab
- Education Strategy Group
- Georgetown Center for Education and the Workforce
- Harvard University Project on Workforce; the Harvard Business School Managing the Future of Work Project
- Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA), which led to the 2016 Adult Training and Education Survey (ATES)
- Jobs for the Future
- Opportunity@Work
- Manhattan Institute
- MHEC
- National Skills Coalition
- NEBHE
- New America’s Center on Education & Labor
- Northeastern University’s Center for the Future of Higher Education and Talent Strategy
- Program on Skills, Credentials & Workforce Development, George Washington University
- National Student Clearinghouse Research Center
- SHEEO
- SHRM
- WICHE
- Northeastern University’s Center for the Future of Higher Education and Talent Strategy,
- RAND
- Rutgers Education and Employment Research Center
- Urban Institute / WorkRise
- U.S. Chamber of Commerce Foundation
- Workcred
A growing issue in the research community is the changing “innovation” landscape within the learn-and work ecosystem. Researchers have some tried-and-true methods to study outcomes, but they are few—and increasingly inadequate to fully study a complex, dynamic ecosystem. Many innovations in credentialing, for example, seem sound and necessary. But evidence of actual outcomes—verifiable results, particularly for learners, credential providers, and employers—is needed to warrant full-scale implementation of these innovative approaches.
Education researchers by necessity develop new methods to examine outcomes, such as finding ways to match new micro-credential offerings with existing systems of degrees and certificates. Such workarounds are necessary until higher education systems (1) collect data on all credential programs, (2) develop ways to track progress toward completion (rather than simply counting completions), and (3) better link academic outcomes to employment and wage information.
Further, higher education system data are not typically structured to determine outcomes—even using well-established measures—for learners who pursue non-degree credentials. As example, it may not even be possible to identify in a data system if a student is pursuing an “incremental credential.” And many systems provide no way to track non-credit learning, non-credit to credit pathways, or pathways at all. Degree programs and course catalogs are arguably structured for the efficient delivery of teaching-and-learning activities en masse. Recognition of course or degree completion rests on a specious assumption of uniformity—an implicit agreement among educators, learners, and the workforce that all students who have completed a given degree are equally prepared, have demonstrated the same outcomes, and are similarly qualified.
The U.S. credentialing system, which includes degrees and certificates, is moving toward credentialing processes and requirements that are more diverse, flexible, and tailored to the needs of individuals. As this set of innovations becomes more widely institutionalized, research study design and implementation face significant challenges. Four will be especially significant:
- The treatment being studied is variable by design. In research studies, it is generally assumed that everyone in a given group is getting the same treatment. A one-size-fits-all innovation is easy to study rigorously. However, when a credentialing approach allows greater “flexibility” (often tailoring to the needs of an individual learner), the treatment being assessed is, by its nature, variable. Given the desire to understand what works at scale, education research studies hinge on the assumption that large numbers of students get the same treatment. When the “same treatment” means that every student gets a program that is of the greatest benefit to him or her individually, that flies in the face of the expectation that such studies assess and manage implementation quality and fidelity. In this way, priorities of programming and research (that everyone gets the same thing) confound the effort to ensure equity (that everyone gets what they need), the latter being a primary aim of innovative credentialing approaches.
- Analytic power becomes a problem. Postsecondary programs are innovating at an unrivaled pace, thanks to changing workplace demands and calls for recognition of a wider array of valuable credentials—degree and non-degree. This will result in many more credential offerings with fewer students potentially in each. This will likely be problematic, because quality impact research requires volumes of data large enough to enable satisfactory analysis.
- Innovation and scale-up are in conflict. As we study credentials in a period of rapid innovation, it is difficult to know when a new credential opportunity is “done” and ready for repeatable testing at greater scale. It is also difficult to scale an intervention that is inherently flexible to the context in which it is being offered. And if innovations cannot be replicated at scale, they cannot be rigorously tested to assess if they “actually work”—the priority long established in education research.
- If innovative credentials work, traditional measures stop working. The purpose and theory behind any education innovation drives decisions about which outcome measures matter. Traditional credentialing approaches have established a set of measures of higher education success. However, as the array of credentials broadens, those measures miss crucial aspects of new options. As we expand the aims of credentialing, we by necessity redefine what is important to measure. Until now, it has generally been sufficient to track whether a learner completes a degree (and maybe how long that takes). Going forward, it becomes necessary to measure how many different credentials an individual gains, how those credentials are connected, how well they meet educational and employment expectations, and ultimately how they further learners’ goals.
Examples
- The Georgetown Center on Education and the Workforce is a research and policy institute that studies the links between education, career qualifications and workforce demands. Since 2008, CEW has conducted research related to jobs, skills, and equity to better inform students, parents, teachers, and policymakers about the changing relationship between education and careers. Research findings are disseminated widely through a robust website that offers, reports, executive summaries, infographics, newsletters, blogs, and videos. The CEW’s Good Jobs project, launched in 2017, investigated the impact structural economic change has had on workers at different education levels. The project includes resources that show the concentration of good jobs by industry and occupation. CEW continues to study the ongoing impact of COVID-19 on the following categories: Jobs: Industries, occupations; Workers: Race, class, gender, educational attainment; Education pipeline: Higher education, K-12; Students: High school, college Households: Stimulus payments, loss of employment income.
- The Non-Degree Credentials Research Network (NCRN) is a network of researchers working on various types of non-degree credentialing including licenses, certifications, apprenticeships, certificates, and microcredentials. NCRN includes experts from higher education institutions, research firms, and nonprofit organizations, among others. It was established to address the gap in understanding the role of credentials in contemporary labor markets.
- WorkRise, hosted by the Urban Institute, is a national platform for identifying, testing, and sharing bold ideas for transforming the labor market. This is done by funding research on promising practices, policies, and programs across the country as well as foundational research on labor market trends. Projects generate data and evidence that strengthens employers, informs policymaking, and provides genuine economic mobility and security for workers—especially those who face systemic barriers to opportunity. WorkRise convenes and collaborates with often-siloed groups—employers, worker advocates, practitioners, policymakers, scholars, and philanthropists—to ensure the solutions delivered are relevant and scalable. WorkRise funds scholars and practitioners working to improve economic mobility in the U.S. labor force. Priority topics include: Economic Context, Demographic Disparities, Employer Practices, Worker Power, Skills and Training.
- The Hechinger Report (a national nonprofit newsroom) provides in-depth reporting on education. Its October 2022 report on higher education featured What researchers learned about online higher education during the pandemic. The article highlights a unique and major development from the COVID-19 pandemic: Prior to the pandemic, scientific research was often missing from studies of the effectiveness of online higher education: a control group. But during the pandemic, with nearly everyone going online for their education, the result was a randomized trial on a planetary scale with an immense control group.
References
Knestis, K. & Zanville. H. (May 3, 2022). Reflections On Conducting Research in A Changing Credentialing Ecosystem. Credential As You Go
Marcus, J. (October 6, 2022). What researchers learned about online higher education during the pandemic. The Hechinger Report.
What We Know About Non-Degree Credentials: A Literature Scan. A project of the Non-Degree Credentials Research Network at the George Washington Institute of Public Policy, The George Washington University. 6/23/2019 https://gwipp.gwu.edu/sites/g/files/zaxdzs2181/f/downloads/06.23.19%20NCRN%20Literature%20Scan.pdf
Topics
Certification Competency / Skills Models & Learning Frameworks Credentials Credentials For Young Adults Equity, Equality, Fair/Just Outcomes, Equity-mindedness, Inclusion, Racial Justice Non-degree Credentials Non-degree Credentials Research Network (NCRN) Policy and Practice Work CollegesSee All Topics
Key Initiatives
2U / edX Bootcamp Partnership AACRAO Credit Mobility Initiative American Opportunity Index Building Capacity, Expanding Pathways: Accelerating the Growth of Credential Innovation in Higher Education – UPCEA Career Optimism Index® Cedefop’s Microcredentials & Labor Market Research Centralized DEI (Diversity, Equity & Inclusion) Education Research Platform College-in-Prison Programs Competency Model Clearinghouse, US Department of Labor Credential As You Go CTE Leadership Collaborative Data Collaborative for a Skills-based Economy (Data Collab) – Education Design Lab Delivering on the Degree: The College-to-Jobs Playbook Digital Skills & Knowledge Concepts Labelling Initiative – European Classification of Occupations, Skills & Competences (ESCO) Equity in Digital Learning Student Survey – Every Learner Everywhere Graduate-Level Microcredentials and Workforce Needs Industry Credentials Initiative Job Quality Measurement Initiative & Good Jobs Initiative – U.S. Department of Labor Collaborative Knowledge Center, National Institute for the Study of Transfer Students (NISTS) Kresge Foundation Last Mile Report Launch: Equitable & Accelerated Pathways for All – Education Strategy Group LifeJourney – AI-Powered Skills & Competency Toolkit Manufacturing Credentials: Now and for the Future MicroCreds National Project (Ireland) Open Syllabus OpenStax (OERs) PLA Inside Out: An International Journal on Theory, Research and Practice in Prior Learning Assessment Research Blog: Assessment and evaluation of microcredentials: What success looks like and to whom Research on Work-Based Learning – Studying Coursetaking & Post-Degree Earnings Among Completers Research: National Study of Workplace Equity ResearchDataGov (RDG) Rising Up Through Stronger & More Equitable Transfer: Tracking Transfer & Transfer Playbook 2.0 (Aspen Institute, CCRC) SHEEO RESEARCH SURVEY: State Priorities for Higher Education in 2023 Some College, No Credential Student (SCNC) Outcomes: Annual Progress Report STARS – Skilled Through Alternative Routes – Opportunity@Work State Authorization Reciprocity Agreement (SARA) State Postsecondary Data & Communities of Practice Stronger Nation Report – Lumina Foundation The Good Jobs Project The State of Digital Credentials 2022 Guide (Accredible) The Workforce Almanac The Workforce Futures Initiative XCredit Skills Validation – Education Design LabSee All Key Initiatives
Alliances & Intermediaries
AARP Foundation Achieving the Dream (ATD) ACT Albert Shanker Institute American Association of State Colleges & Universities (AASCU) American Council on Education American Dental Education Association American National Standards Institute (ANSI) Annie E. Casey Foundation Apprenticeships for America (AFA) Ascendium Education Philanthropy Aspen Institute College Excellence Program Association of American Medical Colleges Association of Public and Land-Grant Universities (APLU) Bill & Melinda Gates Foundation Brighthive Brookings Institution Burning Glass Institute Business Roundtable (BRT) Center for Equity and Postsecondary Attainment (CEPA), San Diego State University Center for Postsecondary Readiness and Success (CPRS) Center for the Future of Higher Education & Talent Strategy, Northeastern University Center on Education & Labor at New America Charles Koch Foundation Clayton Christensen Institute Coalition for Career Development Center (CCD) Cognizant US Foundation College in High School Alliance (CHSA) Community College Baccalaureate Association (CCBA) Community College Research Center (CCRC) Competency-based Education Network (C-BEN) Complete College America Corporation for a Skilled Workforce (CSW) Council for Adult and Experiential Learning (CAEL) Council of Graduate Schools (CGS) Credential Engine Data for the American Dream Digital Promise ECMC Foundation Ed2Work Edalex Education & Employment Research Center, School of Management & Labor Relations, Rutgers Education Commission of the States (ECS) Education Design Lab (the Lab) Education Policy Program at New America Education Strategy Group (ESG) Education Trust Educational Testing Service (ETS) Employment & Training Administration, USDOL Excelencia in Education Gardner Institute for Excellence in Undergraduate Education Georgetown Center on Education and the Workforce (CEW) Greater Phoenix Chamber Foundation Groningen Declaration Network (GDN) Harvard Project on Workforce HCM Strategists (HCM) Higher Education Quality Council of Ontario (HEQCO) Hope Center for College, Communications & Justice Hunt Institute idatafy Institute for College Access & Success (TICAS) Irvine Foundation Jobs for the Future Joint Center for Political & Economic Studies JPMorgan Chase & Co. League for the Innovation of Community Colleges Lumina Foundation McKinsey & Company MDRC Midwestern Higher Education Compact (MHEC) National Association of Manufacturing (NAM) National Association of State Workforce Agencies National Association of Student Financial Aid Administrators (NASFAA) National Career Development Association (NCDA) National Center for Higher Educational Management Services (NCHEMS) National Early Care & Education Workforce Center (ECE Workforce Center) National Governors Association (NGA) National Partnership for Women & Families National Skills Coalition National Student Clearinghouse (NSC) New England Board of Higher Education (NEBHE) Office of Career, Technical & Adult Education, USDOE Office of Community College Research and Leadership (OCCRL) Opportunity Nation Opportunity@Work Organisation for Economic Co-operation and Development (OECD) Patrick McGovern Foundation Postsecondary National Policy Institute (PNPI) Rand Corporation SkillsCommons Repository SOLID (Solutions for Information Design), LLC Source for Human Resource Manager Southern Regional Education Board (SREB) SRI International Stand Together State Higher Education Executive Officers (SHEEO) Strada Education Foundation The T3 Innovation Network Third Way U.S. Economic Development Administration UnidosUS University Professional and Continuing Education Association- UPCEA Urban Institute US Chamber of Commerce Foundation WestEd Western Governors University Western Interstate Commission for Higher Education (WICHE) Workcred Young Invincibles (YI)See All Alliances