Have you ever wanted to extract similar keywords and turn them into one standard keyword? If so, try FlashText.
FlashText allows you to extract or replace keywords in sentences.
In the code above, we use FlashText to extract the keywords CEO and Python programming language from a sentence that contains only the keywords ceo and Python.
FlashText: Extract and Replace Keywords in Sentences Read More »
If you want to generate plurals, singulars, or indefinite articles from given words, use inflect.
inflect: Generate Plurals, Singulars, and Indefinite Articles Read More »
If you want to parse unstructured names of companies or people into name components, try probablepeople.
probablepeople: Parse Unstructured Names Into Structured Components Read More »
Contraction can cause issues when processing text. If you want to expand contractions using Python, use the library contractions.
Names collected from different sources might have different formats. To convert names into the same format for further processing, use mlxtend’s generalize_names.
Visualizing text data in 2D typically requires several steps: cleaning, encoding, and dimensionality reduction. These processes can be time-consuming.
texthero library simplifies this task, allowing you to perform all these steps efficiently.
The following example demonstrates how to use texthero to visualize CNN news article descriptions from a Kaggle dataset. Each point in the resulting plot represents an article, color-coded by its category.
import pandas as pd
import texthero as hero
import matplotlib.pyplot as plt
# Load the data
df = pd.read_csv("small_CNN.csv")
# Process and reduce dimensionality of the text data
df["pca"] = (df["Description"]
.pipe(hero.clean)
.pipe(hero.tfidf)
.pipe(hero.pca))
# Create the visualization
plt.figure(figsize=(10, 3))
hero.scatterplot(df, col="pca", color="Category", title="CNN News")
plt.show()
This code efficiently cleans the text, applies TF-IDF encoding, performs PCA, and creates a 2D scatter plot of the articles, all in just a few lines of code.
texthero: Reduce Dimension and Visualize Text in One Line of Code Read More »
It can be time-consuming to create data to test edge cases of your NLP model. If you want to quickly create data to test your NLP models, use Checklist.
In the code above, I use Checklist’s Editor to create multiple examples of negation in one line of code.
If you want to get some important statistics from text such as readability score or reading time, use textstat.
In the example above, since the Automated Readability Index is 10.8, the grade level needed to comprehend the text is 10th to 11th grade. The reading time of the text is 2.82 seconds.
If you want to look up the frequency of a certain word in your language, try wordfreq.
wordfreq supports 36 languages. wordfreq even covers words that appear at least once per 10 million words.
Find an example of wordfreq above.
Link to wordfreq.
Link to the source code.
wordfreq: Estimate the Frequency of a Word in 36 Languages Read More »
If you want to summarize text using Python or command line, try sumy.
The great things about sumy compared to other summarization tools are that it is easy to use and it allows you to use 7 different methods to summarize the text.
Above is how sumy summarizes the article How to Learn Data Science (Step-By-Step) in 2020 at DataQuest.
Check out the outputs of other algorithms here.
Link to Sumy.