NLTK stop Words














































NLTK stop Words



Description: 
Natural language processing (nlp) is a research field that presents many challenges such as natural language understanding.

Text may contain stop words like %u2018the%u2019, %u2018is%u2019, %u2018are%u2019. Stop words can be filtered from the text to be processed. There is no universal list of stop words in nlp research, however the nltk module contains a list of stop words.

In this article you will learn how to remove stop words with the nltk module.

Natural Language Processing: remove stop words

from nltk.tokenize import sent_tokenize, word_tokenize data = "All work and no play makes jack dull boy. All work and no play makes jack a dull boy." words = word_tokenize(data) print(words)

//Modified code 

from nltk.tokenize import sent_tokenize, word_tokenize from nltk.corpus import stopwords data = "All work and no play makes jack dull boy. All work and no play makes jack a dull boy." stopWords = set(stopwords.words('english')) words = word_tokenize(data) wordsFiltered = [] for w in words: if w not in stopWords: wordsFiltered.append(w) print(wordsFiltered)

A module has been imported:
from nltk.corpus import stopwords

We get a set of English stop words using the line:

stopWords = set(stopwords.words('english'))

The returned list stopWords contains 153 stop words on my computer.
You can view the length or contents of this array with the lines:

print(len(stopWords)) print(stopWords)

We create a new list called wordsFiltered which contains all words which are not stop words.
To create it we iterate over the list of words and only add it if its not in the stopWords list.

for w in words: if w not in stopWords: wordsFiltered.append(w)


More Articles of Khushboo Singh:

Name Views Likes
Python program to insert an element in binary tree. 820 20
Tokenize text using NLTK in Python. 1198 12
Python Remove multiple elements from list while Iterating. 731 22
Python How to Check if an item exists in list ? 4267 14
Python How to remove multiple elements from list ? 737 26
Python program to check if two trees are mirror of each other without using recursion. 660 19
Python program to find maximum in Binary tree. 929 19
Python Check if all elements are same using Set 709 15
Python program to find diameter of a binary tree. 1081 20
Python program to print root to leaf paths without using recursion. 840 20
Python program to find root of the tree where children id sum for every node is given. 669 23
Introduction of Python NLTK library 1355 25
Categorizing and Tagging Sentences using NLTK in Python . 1000 19
Python program to find height of a tree without using recursion. 662 16
Python program to find sum of all nodes of the given perfect binary tree. 656 19
Python program to find minimum in binary tree. 821 23
Python Check if element exist in list using list.count() function. 690 13
Python program to convert a given binary tree to doubly linked list. 883 20
Python program to find distance between two nodes of a binary tree. 1521 20
NLTK stop Words 1135 13
Python program to find largest binary search tree in a Binary Tree. 934 20
Python program to find inorder successor in binary search tree with recursion. 1201 18
Python program to convert a binary tree into doubly linked list in spiral fashion. 795 15
Python List check if element are same using all() 667 12
Python program to check if two trees are identical using recursion. 706 30
Python Find the occurrence count of an element in the tuple using count() 964 23
Python Convert two lists to a dictionary 723 19
Python program to construct a complete binary tree from given array. 1426 14
Python program to find diameter of binary tree in O(n). 870 17
Introduction to the AVL tree. 740 15
Python program to check if two trees are identical without using recursion 660 17
Python Convert a list of tuples to dictionary. 1077 24
Python program to convert a binary tree to a circular doubly link list. 645 21
Python Check if element exist in list based on own logic. 737 23
Python program to merge two binary trees by doing node sum using recursion 999 27
Python program to check whether a given binary tree is perfect or not. 675 17
Python Check if all elements are same using list.count(). 1083 28
Python program to find an element into binary tree 628 12
Python program to find lowest common ancestor in a binary tree 1210 24

Comments