Tokenizer (ELKI: Environment for DeveLoping KDD-Applications Supported by Index-Structures)

java.lang.Object
- de.lmu.ifi.dbs.elki.utilities.io.Tokenizer

All Implemented Interfaces:

Iter
```
public class Tokenizer
extends Object
implements Iter
```
String tokenizer.

Author:

Erich Schubert

Field Summary

Fields
Modifier and Type	Field and Description
`private int`	`end` Current positions of result and iterator.
`private int`	`index` Current positions of result and iterator.
`private CharSequence`	`input` Data currently processed.
`private static Logging`	`LOG` Class logger.
`private Matcher`	`matcher` Regular expression match helper.
`static String`	`QUOTE_CHAR` Quote characters
`private char[]`	`quoteChars` Stores the quotation character
`private boolean`	`quoted` Whether the current token is a quoted string.
`private int`	`send` Substring to process.
`private int`	`start` Current positions of result and iterator.

Constructor Summary

Constructors
Constructor and Description

Tokenizer(Pattern colSep, String quoteChars)
Constructor.

Constructors
Constructor and Description
`Tokenizer(Pattern colSep, String quoteChars)` Constructor.

Method Summary

Methods
Modifier and Type	Method and Description
`Tokenizer`	`advance()` Moves the iterator forward to the next entry.
`void`	`cleanup()` Perform cleanup.
`double`	`getDouble()` Get current value as double.
`int`	`getEnd()` Get end of token.
`long`	`getLongBase10()` Get current value as long.
`int`	`getStart()` Get start of token.
`String`	`getStrippedSubstring()` Get the current part as substring
`String`	`getSubstring()` Get the current part as substring
`void`	`initialize(CharSequence input, int begin, int end)` Initialize parser with a new string.
`boolean`	`isEmpty()` Test for empty tokens; usually at end of line.
`private char`	`isQuote(int index)` Detect quote characters.
`boolean`	`isQuoted()` Test if the current string was quoted.
`boolean`	`valid()` Returns true if the iterator currently points to a valid object.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
private static final Logging LOG
```
    Class logger.
  - QUOTE_CHAR
```
public static final String QUOTE_CHAR
```
    Quote characters
    
    See Also:
    Constant Field Values
  - quoteChars
```
private char[] quoteChars
```
    Stores the quotation character
  - matcher
```
private Matcher matcher
```
    Regular expression match helper.
  - input
```
private CharSequence input
```
    Data currently processed.
  - send
```
private int send
```
    Substring to process.
  - start
```
private int start
```
    Current positions of result and iterator.
  - end
```
private int end
```
    Current positions of result and iterator.
  - index
```
private int index
```
    Current positions of result and iterator.
  - quoted
```
private boolean quoted
```
    Whether the current token is a quoted string.
- Constructor Detail
  - Tokenizer
```
public Tokenizer(Pattern colSep,
         String quoteChars)
```
    Constructor.
    
    Parameters:
    colSep - Column separator pattern.
    quoteChars - Quotation character.
- Method Detail
  - initialize
```
public void initialize(CharSequence input,
              int begin,
              int end)
```
    Initialize parser with a new string.
    
    Parameters:
    input - New string to parse.
    begin - Begin
    end - End
  - valid
```
public boolean valid()
```
    Description copied from interface: Iter
    
    Returns true if the iterator currently points to a valid object.
    
    Specified by:
    
    valid in interface Iter
    
    Returns:
    a boolean value, whether the position is valid.
  - advance
```
public Tokenizer advance()
```
    Description copied from interface: Iter
    
    Moves the iterator forward to the next entry.
    
    Specified by:
    
    advance in interface Iter
    
    Returns:
    The iterator itself.
  - getSubstring
```
public String getSubstring()
```
    Get the current part as substring
    
    Returns:
    Current value as substring.
  - getStrippedSubstring
```
public String getStrippedSubstring()
```
    Get the current part as substring
    
    Returns:
    Current value as substring.
  - getDouble
```
public double getDouble()
                 throws NumberFormatException
```
    Get current value as double.
    
    Returns:
    double value
    
    Throws:
    
    NumberFormatException - when current value cannot be parsed as double value.
  - getLongBase10
```
public long getLongBase10()
                   throws NumberFormatException
```
    Get current value as long.
    
    Returns:
    double value
    
    Throws:
    
    NumberFormatException - when current value cannot be parsed as long value.
  - isEmpty
```
public boolean isEmpty()
```
    Test for empty tokens; usually at end of line.
    
    Returns:
    Empty
  - isQuote
```
private char isQuote(int index)
```
    Detect quote characters. TODO: support more than one quote character, make sure opening and closing quotes match then.
    
    Parameters:
    index - Position
    
    Returns:
    1 when a quote character, 0 otherwise.
  - isQuoted
```
public boolean isQuoted()
```
    Test if the current string was quoted.
    
    Returns:
    true when quoted.
  - getStart
```
public int getStart()
```
    Get start of token.
    
    Returns:
    Start
  - getEnd
```
public int getEnd()
```
    Get end of token.
    
    Returns:
    End
  - cleanup
```
public void cleanup()
```
    Perform cleanup.

Class Tokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

QUOTE_CHAR

quoteChars

matcher

input

send

start

end

index

quoted

Constructor Detail

Tokenizer

Method Detail

initialize

valid

advance

getSubstring

getStrippedSubstring

getDouble

getLongBase10

isEmpty

isQuote

isQuoted

getStart

getEnd

cleanup