Importance of Search Engines in Learn-and-Work Ecosystem

Last Updated: 06/08/2024

Overview

Search engines are digital tools that are designed to help users find information on the Internet. They operate by crawling, indexing, and retrieving data from web pages, which they then present in response to a user’s query. The process typically involves five steps that are conducted in near real-time:

Crawling: Bots or spiders visit web pages to collect data. Since it is not possible for any web crawler to crawl the entire Internet, after a certain number of pages crawled, amount of data indexed, or time spent on the website, the spider stops crawling and moves on (crawlers use a crawl policy to determine when the crawling of a site is sufficient).
Indexing: Collected data is organized into an index. Some techniques for indexing are trade secrets by the search engine.
Query Processing: The search engine processes the user query to find relevant data it has collected and indexed.
Ranking: The search engine provides results that are ranked based on relevance and other factors.
Displaying Results: The search engine displays the ranked results to the user. Results commonly include text summaries, hyperlinks to sources of additional information, and sometimes images.

When a user enters a query into a search engine, the query typically contains keywords. Most search engines maintain their own database index (called an inverted index) which contains the names of sites containing the keywords that are stored in search engine working memory. The keywords then can be instantly obtained from the index.

The main processing effort then is to generate the web pages to inform the search results list. The search engine processes the list according to filtering and weighting criteria. The order or rank of artifacts in the search results from this process.

Search engines collect data from a wide variety of sources and users should recognize that search engines vary on the sites they visit:

Websites are the primary source of information for most search engines.
Some search engines index content from academic journals and databases (for example, Google Scholar).
Search engines such as Google Books index content from books, including those in libraries.
Specialized search engines may access specific databases relevant to their focus.

The usefulness of a search engine depends on the relevance of the results of the search. There are often millions of web pages that include a particular word or phrase, but some pages are more relevant, popular, or authoritative than others.

Ranking of the results by most search engines are guided by a criterion to rank the "best" results first. Search engines vary in how they determine the order of results. The methodology of ranking changes as Internet use changes and as technologies evolve.

There are two main types of search engines:

Uses a system of predefined, hierarchically ordered keywords that humans have programmed extensively (relies more on the computer to do the bulk of the work).
Uses a system that generates an inverted index by analyzing texts the search engine locates.

Relationship to Learn-and-Work Ecosystem

Search engines are increasingly indispensable tools to understanding and working effectively in a learn-and-work ecosystem characterized by complex (12) components. While search engines play a critical role in all the components, search engine tools are particularly critical in communication and technology; data ecosystem (data, databases, standards); credentials and providers; employers; career navigation; research, and transparency. They provide:

Rapid Access to Information, providing quick (nearly real-time) access to a vast and growing amount of information. Quickly accessible information can make it easier for:
- Policymakers to consider governmental investments in the many areas of the ecosystem
- Businesses for digital marketing and reaching potential customers through search engine optimization (SEO) and search engine marketing (SEM
- Researchers to conduct timely studies
- Intermediaries and others to learn who else is working on related efforts to leverage resources
- Individuals to learn new skills or find job-related information.
Discovery of Resources
- Students and professionals can more easily find academic papers, articles, books, tutorials, academic and training courses and programs, and other resources.
Job Searching
- Job seekers can use search engines to find job listings, company information, industry trends, and workforce data.
Networking and Collaboration
- Search engines can help users discover professional networks, alliances, forums, and platforms where they can collaborate and share knowledge.
Continuous Learning
- Professionals can use search engines to stay updated on trends, technologies, and best practices in their fields including innovations underway and being tested both in the U.S. and other nations.

Examples of Search Engines

Various search engines have different emphases; for example, they are known for powerful algorithms, user control, transparency, privacy, speed of answering, or a geographical focus.

Google: Google dominates the search engine market, serving nearly 92% of global search engine traffic (Google's market share has rarely moved below 90% since 2014). The search engine is known for powerful algorithms and comprehensive indexing.
Bing: Microsoft's search engine offers similar functionalities to Google’s though it uses different algorithms and user interfaces.
Yahoo! Search: Uses Bing's search technology and offers some unique content.
DuckDuckGo: Emphases a privacy-focused approach, not tracking user data.
Baidu: Serves the Chinese-speaking population and is the leading search engine in China.
Yandex: Serves the Russian-speaking population and is the leading search engine in Russia.
Brave Search: Allows searching the web privately, uses an independent index, does not track searches or clicks by users, does not collect user data.
Ecosia: Uses ad revenue from searches to plant trees where needed the most (a green search site).
You.com: Uses conversational AI in search bar for Chrome, leverages a personal AI search assistant and provides customized recommendations with AI chatbot.
Wayback Machine: Allows users to access archived versions of websites.

Role of Artificial Intelligence (AI) in Changing Search Functions

AI has been revolutionizing search engine functions in several ways:

Natural Language Processing
- NLP can enable search engines to understand and process human language more effectively.
Personalization
- AI algorithms can tailor search results that are based on user preferences and behavior.
Voice Search
- AI-driven voice recognition can allow users to search using spoken commands.
Image and Video Search
- AI can improve the ability to search using images and videos.
Predictive Search
- AI can predict what users are looking for and provides suggestions before they even finish typing the query.
Enhanced Accuracy
- AI can help refine search algorithms in order to deliver more accurate and relevant results. Google’s BERT and MUM models enhance the ability of search engines to comprehend and respond to complex queries.

Monetary Aspects of Search Engines

Most search engines are commercial ventures supported by advertising revenue. Some search engines allow advertisers to have their listings ranked higher in search results for a fee (related terms are paid inclusion / paid placement). Search engines that do not accept money for search results may make money by running search-related ads alongside search engine results. Search engines make money every time someone clicks on one of these ads.

Trustworthiness of Information

The trustworthiness of information found through the use of search engines varies. Although search engines are trained to prioritize content from reputable and authoritative sources, search engines can (and so) return results from less credible sources. Users are advised to evaluate the information they find and consider the credibility of the sources.