Token represents text or word in a document with relevant details like its metadata (position, start offset, end offset, token type and its position increment).

TokenStream is an output of the analysis process and it comprises of a series of tokens. It is an abstract class.

This is an abstract base class for each and every type of Analyzer.

This analyzer splits the text in a document based on whitespace.

This analyzer splits the text in a document based on non-letter characters and puts the text in lowercase.

This analyzer works just as the SimpleAnalyzer and removes the common words like ‘a’, ‘an’, ‘the’, etc.

This is the most sophisticated analyzer and is capable of handling names, email addresses, etc. It lowercases each token and removes common words and punctuations, if any.