Tokenize text using NLTK in Python.














































Tokenize text using NLTK in Python.



Description:

NLTK is one of the leading platforms for working with human language data and Python, the module NLTK is used for natural language processing. NLTK is literally an acronym for Natural Language Toolkit.

Follow this link if you want to  know more about NLTK.



Tokenize Words:

 A sentence or data can be split into words using the method word_tokenize().

from nltk.tokenize import sent_tokenize,word_tokenize data="Hey! How are You ?" print(word_tokenize(data))

Output:
['Hey', '!', 'How', 'are', 'You', '?']

All of them are words except the comma. Special characters are treated as separate tokens.

Tokenizing Sentences

The same principle can be applied to sentences. We can tokenize the sentences by using method sent_tokenize().

from nltk.tokenize import sent_tokenize data="Hey! How are You ? Hey! How are You ? " print(sent_tokenize(data))

Output:

['Hey!', 'How are You ?', 'Hey!', 'How are You ?']

NLTK and Arrays 

from nltk.tokenize import sent_tokenize,word_tokenize data="Hey! Are you okay? Hey! Are you okay?" sen=sent_tokenize(data) words=word_tokenize(data) print(sen) print(words)
 
Output

['Hey!', 'Are you okay?', 'Hey!', 'Are you okay?']
['Hey', '!', 'Are', 'you', 'okay', '?', 'Hey', '!', 'Are', 'you', 'okay', '?']


More Articles of Khushboo Singh:

Name Views Likes
Python program to insert an element in binary tree. 821 20
Tokenize text using NLTK in Python. 1199 12
Python Remove multiple elements from list while Iterating. 731 22
Python How to Check if an item exists in list ? 4267 14
Python How to remove multiple elements from list ? 737 26
Python program to check if two trees are mirror of each other without using recursion. 660 19
Python program to find maximum in Binary tree. 929 19
Python Check if all elements are same using Set 709 15
Python program to find diameter of a binary tree. 1082 20
Python program to print root to leaf paths without using recursion. 840 20
Python program to find root of the tree where children id sum for every node is given. 669 23
Introduction of Python NLTK library 1355 25
Categorizing and Tagging Sentences using NLTK in Python . 1000 19
Python program to find height of a tree without using recursion. 662 16
Python program to find sum of all nodes of the given perfect binary tree. 656 19
Python program to find minimum in binary tree. 821 23
Python Check if element exist in list using list.count() function. 691 13
Python program to convert a given binary tree to doubly linked list. 883 20
Python program to find distance between two nodes of a binary tree. 1521 20
NLTK stop Words 1135 13
Python program to find largest binary search tree in a Binary Tree. 934 20
Python program to find inorder successor in binary search tree with recursion. 1201 18
Python program to convert a binary tree into doubly linked list in spiral fashion. 795 15
Python List check if element are same using all() 667 12
Python program to check if two trees are identical using recursion. 706 30
Python Find the occurrence count of an element in the tuple using count() 965 23
Python Convert two lists to a dictionary 723 19
Python program to construct a complete binary tree from given array. 1426 14
Python program to find diameter of binary tree in O(n). 870 17
Introduction to the AVL tree. 740 15
Python program to check if two trees are identical without using recursion 660 17
Python Convert a list of tuples to dictionary. 1077 24
Python program to convert a binary tree to a circular doubly link list. 645 21
Python Check if element exist in list based on own logic. 737 23
Python program to merge two binary trees by doing node sum using recursion 999 27
Python program to check whether a given binary tree is perfect or not. 675 17
Python Check if all elements are same using list.count(). 1083 28
Python program to find an element into binary tree 628 12
Python program to find lowest common ancestor in a binary tree 1210 24

Comments