In today’s information overload era, text summarization has become an essential tool for navigating the vast sea of content. Transformer models, with their powerful attention mechanisms, have revolutionized text summarization, achieving state-of-the-art results. This blog post delves into the exciting world of Transformer-based text summarization, exploring five different models and providing code examples to get you started.
1. BART: Pre-trained Model for Text Summarization
BART (Bidirectional and Auto-Regressive Transformers) is a powerful pre-trained model for both extractive and abstractive summarization. It uses a BART-large model pre-trained on a massive dataset of text and code, allowing it to generate summaries that are both factually accurate and fluent.
Here’s an example of using BART for extractive summarization in Python with Hugging Face transformers:
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large-cnn")
text = """Over a thousand flights connecting Chennai were cancelled since yesterday after heavy rain due to Cyclone Michaung forced shut the city airport. IndiGo had to cancel 550 flights across India after the airport closed operations yesterday while Vistara suspended 10 flights from Chennai.
The rain has stopped and Chennai now struggles to drain the floodwaters and assess the damages caused due to the heavy rain.
Chennai airport had shut its runway and grounded all planes yesterday after floodwater entered the airport. It resumed operations this morning. Currently, Visakhapatnam airport is shut as the cyclone has moved towards Andhra Pradesh.
"""
inputs = tokenizer(text, return_tensors="pt")
summary_ids = model.generate(inputs["input_ids"])
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(f"Summary: {summary}")
2. FLAN-T5: Text-to-Text Transfer Transformer
FLAN-T5(Text-to-Text Transfer Transformer) is another versatile pre-trained model capable of handling various NLP tasks, including text summarization. Its encoder-decoder architecture allows it to learn complex relationships within the text and generate summaries that capture the main points effectively.
Here’s an example of using FLAN-T5 for abstractive summarization in Python with Hugging Face transformers:
from transformers import T5Tokenizer, TFT5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-base")
model = TFT5ForConditionalGeneration.from_pretrained("google/flan-t5-base")
text = "The Eiffel Tower is an iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
input_ids = tokenizer(text, return_tensors="tf").input_ids
outputs = model.generate(input_ids)
summary = tokenizer.decode(outputs[0],skip_special_tokens=True)
print(f"Summary: {summary}")
3. Pegasus: Abstractive Text Summarization Model
Pegasus is a specifically designed model for abstractive text summarization. It builds upon the success of BART and incorporates additional innovations, allowing it to generate summaries that are more fluent and informative.
Here’s an example of using Pegasus for abstractive summarization in Python with Hugging Face transformers:
from transformers import PegasusForConditionalGeneration, PegasusTokenizer
tokenizer = PegasusTokenizer.from_pretrained("google/pegasus-xsum")
model = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum")
ARTICLE_TO_SUMMARIZE = "Climate change is a long-term shift in temperatures and weather patterns. It is caused by human activities, primarily the burning of fossil fuels."
inputs = inputs = tokenizer(ARTICLE_TO_SUMMARIZE, max_length=1024, truncation=True,return_tensors="pt")
outputs =summary_ids = model.generate(inputs["input_ids"])
summary = tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print(f"Original text: {text}")
print(f"Summary: {summary}")
4. BigBird: Transformer with Long-Range Attention
BigBird addresses the limitations of standard Transformer models in capturing long-range dependencies in text. It utilizes a novel attention mechanism that allows it to efficiently learn relationships between distant words, making it ideal for summarizing long documents.
Here’s an example of using BigBird for extractive summarization in Python with Hugging Face transformers:
from transformers import BigBirdForSequenceClassification, BigBirdTokenizer
tokenizer = BigBirdTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")
model = BigBirdForSequenceClassification.from_pretrained("google/bigbird-pegasus-large-arxiv", num_labels=2)
text = "The United States is a country located in North America. It is the third largest country by total area and the third most populous country by population."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
summary_indices = outputs.logits.argmax(-1)
summary = tokenizer.decode(summary_indices[0])
5. MarianMT: Multilingual Text Summarization Model
MarianMT is a powerful pre-trained model developed by Facebook AI, capable of text summarization in multiple languages. This makes it a valuable tool for anyone working with multilingual content.
Here’s an example of using MarianMT for abstractive summarization in Spanish with Hugging Face transformers:
rom transformers import MarianMTModel, MarianTokenizer
tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-es-en")
model = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-es-en")
text = "El clima es el conjunto de fenómenos meteorológicos que caracterizan una región a lo largo del tiempo."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs)
summary = tokenizer.decode(outputs[0])
print(f"Original text: {text}")
print(f"Summary: {summary}")
Related Posts
Quiz for Embedding-02
Quiz for Embedding-01
