NLTK stop Words














































NLTK stop Words



Description: 
Natural language processing (nlp) is a research field that presents many challenges such as natural language understanding.

Text may contain stop words like %u2018the%u2019, %u2018is%u2019, %u2018are%u2019. Stop words can be filtered from the text to be processed. There is no universal list of stop words in nlp research, however the nltk module contains a list of stop words.

In this article you will learn how to remove stop words with the nltk module.

Natural Language Processing: remove stop words

from nltk.tokenize import sent_tokenize, word_tokenize data = "All work and no play makes jack dull boy. All work and no play makes jack a dull boy." words = word_tokenize(data) print(words)

//Modified code 

from nltk.tokenize import sent_tokenize, word_tokenize from nltk.corpus import stopwords data = "All work and no play makes jack dull boy. All work and no play makes jack a dull boy." stopWords = set(stopwords.words('english')) words = word_tokenize(data) wordsFiltered = [] for w in words: if w not in stopWords: wordsFiltered.append(w) print(wordsFiltered)

A module has been imported:
from nltk.corpus import stopwords

We get a set of English stop words using the line:

stopWords = set(stopwords.words('english'))

The returned list stopWords contains 153 stop words on my computer.
You can view the length or contents of this array with the lines:

print(len(stopWords)) print(stopWords)

We create a new list called wordsFiltered which contains all words which are not stop words.
To create it we iterate over the list of words and only add it if its not in the stopWords list.

for w in words: if w not in stopWords: wordsFiltered.append(w)


More Articles of Khushboo Singh:

Name Views Likes
Python program to insert an element in binary tree. 867 20
Tokenize text using NLTK in Python. 1251 12
Python Remove multiple elements from list while Iterating. 774 22
Python How to Check if an item exists in list ? 4323 14
Python How to remove multiple elements from list ? 789 26
Python program to check if two trees are mirror of each other without using recursion. 695 19
Python program to find maximum in Binary tree. 974 19
Python Check if all elements are same using Set 753 15
Python program to find diameter of a binary tree. 1127 20
Python program to print root to leaf paths without using recursion. 870 20
Python program to find root of the tree where children id sum for every node is given. 710 23
Introduction of Python NLTK library 1405 25
Categorizing and Tagging Sentences using NLTK in Python . 1063 19
Python program to find height of a tree without using recursion. 690 16
Python program to find sum of all nodes of the given perfect binary tree. 697 19
Python program to find minimum in binary tree. 860 23
Python Check if element exist in list using list.count() function. 724 13
Python program to convert a given binary tree to doubly linked list. 924 20
Python program to find distance between two nodes of a binary tree. 1561 20
NLTK stop Words 1197 13
Python program to find largest binary search tree in a Binary Tree. 973 20
Python program to find inorder successor in binary search tree with recursion. 1246 18
Python program to convert a binary tree into doubly linked list in spiral fashion. 828 15
Python List check if element are same using all() 714 12
Python program to check if two trees are identical using recursion. 739 30
Python Find the occurrence count of an element in the tuple using count() 1001 23
Python Convert two lists to a dictionary 761 19
Python program to construct a complete binary tree from given array. 1480 14
Python program to find diameter of binary tree in O(n). 909 17
Introduction to the AVL tree. 810 15
Python program to check if two trees are identical without using recursion 693 17
Python Convert a list of tuples to dictionary. 1134 24
Python program to convert a binary tree to a circular doubly link list. 688 21
Python Check if element exist in list based on own logic. 787 23
Python program to merge two binary trees by doing node sum using recursion 1043 27
Python program to check whether a given binary tree is perfect or not. 713 17
Python Check if all elements are same using list.count(). 1126 28
Python program to find an element into binary tree 658 12
Python program to find lowest common ancestor in a binary tree 1254 24

Comments