ERC CISST - cisst software

---

cmnTokenizer Class Reference

#include <cmnTokenizer.h>

Collaboration diagram for cmnTokenizer:

[legend]List of all members.

---

Detailed Description

cmnTokenizer provides a convenient interface for parsing a string into a set of tokens. The parsing uses several sets of control characters:

Delimiters: Separate tokens. A sequence of delimiters which is not quoted or escaped is ignored, and a new token begins after it.

Quote markers: Enclose parts of, or complete, tokens. Anything between a pair of identical quote markers is included in a token.

Escape markers: Quote the character immediately following them.

The default values for delimiters is whitespace (space, tab, CR); for quote markers is the set of double and single quotation marks (",'); for escape markers is backslash (\). But the user can override them by calling the appropriate method.

A cmnTokenizer object maintains an internal copy of the tokenized text, and can return a pointer to an array of pointer, after the fashion of argv.

Note:

It is important to note that the stored tokens have the life span of the tokenizer. If the tokenizer is destroyed, the user cannot access any of the tokens.

Definition at line 66 of file cmnTokenizer.h.

Public Types

• typedef std::vector< const
  char * > TokensContainer
• typedef TokensContainer::size_type size_type

Public Member Functions

• void SetDelimiters (const char *delimiters)
• void SetQuoteMarkers (const char *markers)
• void SetEscapeMarkers (const char *markers)
• const char * GetDelimiters (void) const
• const char * GetQuoteMarkers (void) const
• const char * GetEscapeMarkers (void) const
• void Parse (const char *string) throw (std::runtime_error)
• void Parse (const std::string &string) throw (std::runtime_error)
• size_type GetNumTokens (void) const
• const char *const * GetTokensArray (void) const
• void GetArgvTokens (std::vector< const char * > &argvTokens) const

Static Public Member Functions

• static const char * GetDefaultDelimiters (void)
• static const char * GetDefaultQuoteMarkers (void)
• static const char * GetDefaultEscapeMarkers (void)

---

Member Function Documentation

void cmnTokenizer::Parse

(

const char *

string

)

throw (std::runtime_error)

Parse the input string and store the tokens internally. If there is a syntax error (e.g., unclosed quotes) throw an exception.

Parameters:

string

the text to be parsed.

size_type cmnTokenizer::GetNumTokens

(

void

)

const [inline]

Return the number of tokens stored.

Definition at line 136 of file cmnTokenizer.h.

const char* const* cmnTokenizer::GetTokensArray

(

void

)

const [inline]

Return the array of tokens in an argv fashion.

Note:

This parsing returns exactly the tokens in the input string. For an argv-style set of argument, one needs to have the "name of the program" argument in index 0, and the arguments starting at index 1. Use the method GetArgvTokens() for that.

Definition at line 151 of file cmnTokenizer.h.

void cmnTokenizer::GetArgvTokens

(

std::vector< const char * > &

argvTokens

)

const

This method will fill the input vector with the tokens, but first set the 0-index element to NULL, to follow the argv convention, where argv[0] contains the "name of the program".

---

The documentation for this class was generated from the following file:

• cmnTokenizer.h

---