Context
Project Overview
I built a command-line search engine in Java that accepts user queries and returns relevant results across a collection of 130,000+ LA Times articles, returning top search results within milliseconds.
I expanded the project's functionality beyond its original scope by developing a data scraping script to obtain my own data (2000 CBC Articles), and utilizing Spring Boot API and Next.js interface to transform it into a dynamic web application.
How it works
Search Engine Build
An index engine was built to be able to process the scraped articles and extract relevant textual content. It tokenizes the text and constructs a lexicon. The engine then builds the inverted index, mapping terms to document identifiers and facilitating efficient query retrieval.
Retrieval Process
Given a user query, the engine consults the inverted index to retrieve the list of candidate documents that contain some or all of the query terms.
The search engine then uses the BM25 scoring algorithm, a popular ranking function used in information retrieval, to assign a relevance score to each document based on the frequency of query terms within the document and other factors such as document length and term frequency within the document collection.
