/Text Summarizer using ML

Text Summarizer Using Machine Learning

1. Introduction

Text Summarization is an important task in Natural Language Processing (NLP) that focuses on automatically creating a shorter version of a large text while preserving its most important information. With the rapid growth of digital content, people have access to a huge amount of information through websites, research papers, news articles, reports, and documents. Reading all of this information can be time-consuming. Text summarization helps solve this problem by automatically identifying and presenting the most important parts of a document.

Text summarization is commonly used in many real-world applications, including:

  • News summarization – Creating short versions of lengthy news articles.

  • Document analysis – Quickly understanding large documents.

  • Research paper summaries – Identifying important information from research papers.

  • Meeting notes generation – Creating summaries of discussions and meetings.

  • AI assistants – Providing concise answers and summaries to users.

There are two main types of text summarization:

1. Extractive Summarization

Extractive summarization selects important sentences directly from the original text. It does not create new sentences. Instead, it analyzes the text, calculates the importance of each sentence, and selects the most relevant sentences to form the summary.

2. Abstractive Summarization

Abstractive summarization generates new sentences that represent the meaning of the original text. This approach is more similar to how humans summarize information because the system can understand the overall meaning and produce new sentences.

In this beginner-level project, Extractive Text Summarization is implemented using basic Natural Language Processing and Machine Learning techniques.

The project uses:

  • NLTK → For text processing and sentence tokenization.

  • Regular Expressions → For cleaning the text.

  • TF-IDF → For identifying important words.

  • Sentence Scoring → For calculating the importance of each sentence.

  • Python → For implementing the complete summarization system.

2. Objective

The main objective of this project is to build a simple text summarization system that can read a long paragraph and automatically generate a shorter summary containing the most important sentences.

The system analyzes the input text, identifies important words using TF-IDF, calculates a score for each sentence, and selects the highest-scoring sentences. The selected sentences are then arranged in their original order to create a meaningful summary.

3. Technologies Used

The following technologies and libraries are used:

  • Python – Main programming language.

  • NLTK (Natural Language Toolkit) – Used for natural language processing.

  • Regular Expressions (re) – Used to remove special characters and clean the text.

  • Scikit-learn – Used to implement TF-IDF vectorization.

  • TF-IDF Vectorizer – Used to measure the importance of words in the text.

4. Working Principle

The summarization system follows several steps.

First, the input paragraph is divided into individual sentences using NLTK's sentence tokenizer. Each sentence is then cleaned by removing unnecessary special characters and converting all words into lowercase.

Next, TF-IDF (Term Frequency-Inverse Document Frequency) is applied to the processed sentences. TF-IDF assigns numerical values to words based on their importance within the text. Words that are more relevant to a particular sentence receive higher scores.

The TF-IDF values of the words in each sentence are then added together to calculate a total score for that sentence. Sentences with higher scores are considered more important.

The system selects the top three highest-scoring sentences. Finally, these sentences are arranged according to their original position in the text so that the generated summary maintains the natural flow of the original document.

5. Python Implementation

The following Python program implements the extractive text summarization system.

# Import required libraries

import nltk
import re

from nltk.tokenize import sent_tokenize, word_tokenize
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer

# Download required NLTK resources

nltk.download("punkt")
nltk.download("stopwords")

# --------------------------------------------
# Step 1: Input Text
# --------------------------------------------

text = """
Artificial Intelligence is one of the fastest growing technologies in the world.
It allows computers to learn from data and perform tasks that normally require
human intelligence. Machine learning and deep learning are important parts of
artificial intelligence. Many industries use AI for healthcare, education,
finance, and automation. AI systems can analyze large amounts of data and help
people make better decisions.
"""

# --------------------------------------------
# Step 2: Split text into sentences
# --------------------------------------------

sentences = sent_tokenize(text)

# --------------------------------------------
# Step 3: Text preprocessing
# --------------------------------------------

stop_words = set(stopwords.words("english"))

processed_sentences = []

for sentence in sentences:

    # Remove special characters
    clean_sentence = re.sub(
        "[^a-zA-Z]",
        " ",
        sentence
    )

    # Convert to lowercase
    clean_sentence = clean_sentence.lower()

    processed_sentences.append(clean_sentence)

# --------------------------------------------
# Step 4: Calculate sentence importance
# --------------------------------------------

vectorizer = TfidfVectorizer(
    stop_words="english"
)

tfidf_matrix = vectorizer.fit_transform(
    processed_sentences
)

# Calculate sentence scores

sentence_scores = {}

for index, sentence in enumerate(tfidf_matrix):
    sentence_scores[index] = sentence.sum()

# --------------------------------------------
# Step 5: Select important sentences
# --------------------------------------------

# Select top 3 sentences

top_sentences = sorted(
    sentence_scores,
    key=sentence_scores.get,
    reverse=True
)[:3]

# Sort sentences in original order

top_sentences.sort()

summary = ""

for index in top_sentences:
    summary += sentences[index] + " "

# --------------------------------------------
# Step 6: Display Summary
# --------------------------------------------

print("Original Text:")
print(text)

print("\nGenerated Summary:")
print(summary)

6. Explanation of the Python Program

The program first imports the required libraries. NLTK is used for natural language processing tasks, while re is used for text cleaning. The TfidfVectorizer from Scikit-learn is used to calculate the importance of words.

The required NLTK resources are downloaded using nltk.download().

In Step 1, a sample paragraph about Artificial Intelligence is stored in the text variable.

In Step 2, the sent_tokenize() function divides the paragraph into individual sentences. This is important because the summarizer needs to evaluate each sentence separately.

In Step 3, the sentences are preprocessed. Special characters are removed using a regular expression, and all text is converted to lowercase. This creates cleaner input for the TF-IDF algorithm.

In Step 4, TfidfVectorizer calculates the TF-IDF value of words in each sentence. Stop words such as common English words are ignored because they generally provide less useful information for identifying important sentences.

The TF-IDF values of each sentence are added together to create a sentence score. A higher score indicates that a sentence contains more important or distinctive words.

In Step 5, the sentences are sorted according to their scores. The top three sentences are selected as the summary. After selection, they are sorted according to their original position in the document. This helps maintain the original flow of information.

Finally, the generated summary is displayed on the screen.

7. Output

Original Text

Artificial Intelligence is one of the fastest growing technologies in the world.
It allows computers to learn from data and perform tasks that normally require
human intelligence. Machine learning and deep learning are important parts of
artificial intelligence. Many industries use AI for healthcare, education,
finance, and automation. AI systems can analyze large amounts of data and help
people make better decisions.

Generated Summary

Artificial Intelligence is one of the fastest growing technologies in the world.

Machine learning and deep learning are important parts of artificial intelligence.

Many industries use AI for healthcare, education, finance, and automation.

8. Advantages

The proposed text summarization system has several advantages:

  • It automatically summarizes lengthy text.

  • It is simple and easy to understand.

  • It reduces the time required to read large documents.

  • It uses commonly available Python libraries.

  • It does not require a large training dataset.

  • It can be used as a basic document analysis tool.

  • It preserves important sentences from the original text.

9. Limitations

Although the system is useful for a beginner-level project, it has some limitations. Since it uses extractive summarization, it can only select sentences from the original text. It cannot create new sentences or rewrite information.

The quality of the summary also depends on the TF-IDF scores. A sentence with a high score is not always the most meaningful sentence. The system also does not have a deep understanding of context, grammar, or the overall meaning of the document.

More advanced systems can use Deep Learning and Transformer-based models to perform abstractive summarization and generate more natural summaries.

10. Conclusion

This project demonstrates how Machine Learning and Natural Language Processing can be used to automatically summarize text. The system uses NLTK for sentence processing and TF-IDF to determine the importance of words and sentences.

By calculating sentence scores and selecting the highest-scoring sentences, the system produces a shorter version of the original text while retaining important information.

Although this is a simple extractive approach, it provides a strong foundation for understanding NLP and text summarization. In the future, the system can be improved by using advanced techniques such as TextRank, LSTM networks, BERT, T5, or other Transformer-based models to generate more accurate and human-like summaries.