tokenizer.h File Reference

#include <vector>
#include <zorba/config.h>
#include <zorba/locale.h>
#include <zorba/internal/unique_ptr.h>
#include <zorba/internal/ztd.h>

Include dependency graph for tokenizer.h:

Go to the source code of this file.

Classes
class	zorba::Tokenizer::Callback
	A Callback is called once per token. More...
struct	zorba::Tokenizer::Properties
	Various properties of this Tokenizer. More...
struct	zorba::Tokenizer::State
	A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers. More...
class	zorba::Tokenizer
	A Tokenizer breaks a string into a stream of word tokens. More...
class	zorba::TokenizerProvider
	A TokenizerProvider provides a Tokenizer for a given language. More...

Namespaces
namespace	zorba
	Copyright 2006-2011 The FLWOR Foundation.

Macros
#define	ZORBA_TOKENIZER_API_H

Macro Definition Documentation

#define ZORBA_TOKENIZER_API_H

Definition at line 19 of file tokenizer.h.