Basic operations of Natural Language Processing

16503 ワード
NLP Python nltk 自然言語処理 データサイエンス Python テキストリンク

Hello again. Last time I wrote about how to score good in the Kaggle Titanic challenge.
In this article let's look into the basic operations that we need to know when we deal with NLP.
I assume that everyone has a dev environment for data science like Jupyterlab.
Here I’m going to use the nltk library. We can use it to perform NLP tasks like tokenization, stemming etc..

In this article we are going to perform operation of;
1. Tokenization
2. Deal with stop words
3. Stemming
4. Lemmatization
5. wordnet.
6. POS tagging.

So the first step is to install the package.

nlp_oops.ipynb
!pip install nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

1. Tokenization

Dividing text into tokens called the tokenization. Sentences can be tokenized words. Paragraphs can be tokenized into sentences.

nlp_oops.ipynb
import nltk
from  nltk.tokenize import sent_tokenize, word_tokenize
text_ex = "Hello again, it's very dark here in evening due to pre winter season. After work I'm going to play league of legends for fun."

print(word_tokenize(text_ex))
# output
# ['Hello', 'again', ',', 'it', "'s", 'very', 'dark', 'here', 'in', 'evening', 'due', 'to', 'pre', 'winter', 'season', '.', 'After', 'work', 'I', "'m", 'going', 'to', 'play', 'league', 'of', 'legends', 'for', 'fun', '.']

print(sent_tokenize(text_ex))
# output
# ["Hello again, it's very dark here in evening due to pre winter season.", "After work I'm going to play league of legends for fun."]

2. Tokenization

In most projects in NLP, stop words are considered as noise. Because it's too common and we cannot derive much information. But for some cases stop words can be useful.
So, let's take a look, how we can remove the stop words from the corpus. In NLTK we can see 179 stop words for English language.

nlp_oops.ipynb
from nltk.corpus import stopwords
print('Num of stop words: ',  len(stopwords.words('english')))
print(stopwords.words('english'))

# output
# Num of stop words:  179
# ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

And this is the way how we can remove the stop words from texts

nlp_oops.ipynb
from nltk.corpus import stopwords
text = 'The COVID-19 pandemic has completely changed our lives'
text = word_tokenize(text)
text_with_no_stopwords = [word for word in text if word not in stopwords.words('english')]
print(text_with_no_stopwords)

# output
['The', 'COVID-19', 'pandemic', 'completely', 'changed', 'lives']

3. Stemming

In English there are lots of variations for words by adding affixes (suffixes and prefixes).
Stemming is the process that removes those affixes from the words and reduces them to word stems.

Example:
training → train
kids → kid

There are several stemming algorithms available in nltk package and we are going to use PorterStemmer.

nlp_oops.ipynb
from nltk.stem import PorterStemmer 
from nltk.tokenize import word_tokenize 

ps = PorterStemmer() 
text = "Programers program with programing languages"
words = word_tokenize(text) 

for w in words: 
    print(w, " : ", ps.stem(w)) 
# output 
# Programers  :  program
# program  :  program
# with  :  with
# programing  :  program
# languages  :  languag

4. Lemmatizing

In short we can say this is advance version of stemming. While stemming focus on reducing affixes, lemmatization go beyond reducing affixes. For example: The lemma of ‘was’ is ‘be’ and the lemma of ‘mice’ is ‘mouse’. Lemmatization looks at surrounding text to determine a given word’s part of speech, it does not categorize phrases.

nlp_oops.ipynb
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer() 
text = "Programers program with programing languages"
words = word_tokenize(text) 
for w in words:
    print(w, " : ", lemmatizer.lemmatize(w))

# output 
# Programers  :  Programers
# program  :  program
# with  :  with
# programing  :  programing
# languages  :  language

Please check the difference of the output given by the stemming and Lemmatizing.

5. Finding Synonyms and Antonyms

WordNet is a kind of special dictionary built for natural language processing. In terms it is a lexical database. We can utilize the wordnet to get synonyms and antonyms for given words.

nlp_oops.ipynb
from nltk.corpus import wordnet
synonyms = []  
antonyms =[]   
for syns in wordnet.synsets("good"): 
    for i in syns.lemmas():        
        synonyms.append(i.name())       
        if i.antonyms():
            antonyms.append(i.antonyms()[0].name()) 
print(set(synonyms))
# output 
# {'thoroughly', 'just', 'goodness', 'secure', 'safe', 'serious', 'expert', 'sound', 'undecomposed', 'in_force', 'unspoilt', 'dependable', 'adept', 'unspoiled', 'good', 'full', 'ripe', 'near', 'in_effect', 'practiced', 'effective', 'salutary', 'skilful', 'dear', 'trade_good', 'well', 'right', 'skillful', 'respectable', 'soundly', 'proficient', 'estimable', 'commodity', 'honorable', 'beneficial', 'upright', 'honest'}

print(set(antonyms))
# output 
# {'ill', 'evil', 'bad', 'badness', 'evilness'}

6. POS Tagging

Part of speech is very useful most nlp projects It is a process of converting a sentence to forms – list of words, list of tuples (where each tuple is having a form (word, tag)). The tag in case of is a part-of-speech tag, and signifies whether the word is a noun, adjective, verb, and so on.

nlp_oops.ipynb
import nltk
from nltk.tokenize import word_tokenize

text = '''
Programers are program with programing languages
'''
words = word_tokenize(text)
print(nltk.pos_tag(words))

# output
# [('Programers', 'NNS'), ('are', 'VBP'), ('program', 'NN'), ('with', 'IN'), ('programing', 'NN'), ('languages', 'NNS')]

Cheers....

*本記事は @qualitia_cdevの中の一人、@nuwanさんが書いてくれました。
*This article is written by @nuwan a member of @qualitia_cdev.