What are some examples of tokenization using NLTK?

What are some examples of tokenization using NLTK?

Tokenization is the process of splitting text into individual units, like words or sentences. In NLTK, you can use functions like word_tokenize and sent_tokenize. Would you like to see some code examples?

Yes, please show me how to use word_tokenize.

Sure! Here's a simple example:

import nltk
from nltk.tokenize import word_tokenize

text = "Hello, world! This is a test."
words = word_tokenize(text)
print(words)

This will output: ['Hello', ',', 'world', '!', 'This', 'is', 'a', 'test', '.'].

What about sentence tokenization?

For sentence tokenization, you can use sent_tokenize. Here's an example:

from nltk.tokenize import sent_tokenize

text = "Hello, world! This is a test. How are you?"
sentences = sent_tokenize(text)
print(sentences)

This will output: ['Hello, world!', 'This is a test.', 'How are you?'].