A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers. More...
#include <zorba/tokenizer.h>
Public Types | |
| typedef Tokenizer::size_type | value_type |
Public Member Functions | |
| State () | |
| Default constructor. More... | |
Public Attributes | |
| value_type | para |
| Paragraph number. More... | |
| value_type | sent |
| Sentence number. More... | |
| value_type | token |
| Token number. More... | |
A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers.
Definition at line 54 of file tokenizer.h.
Definition at line 55 of file tokenizer.h.
| zorba::Tokenizer::State::State | ( | ) |
Default constructor.
| value_type zorba::Tokenizer::State::para |
Paragraph number.
Definition at line 59 of file tokenizer.h.
| value_type zorba::Tokenizer::State::sent |
Sentence number.
Definition at line 58 of file tokenizer.h.
| value_type zorba::Tokenizer::State::token |
Token number.
Definition at line 57 of file tokenizer.h.
