de.lmu.ifi.dbs.elki.datasource.parser

Class Tokenizer

java.lang.Object

de.lmu.ifi.dbs.elki.datasource.parser.Tokenizer

All Implemented Interfaces:

Iter

public class Tokenizer
extends Object
implements Iter

String tokenizer.

Field Summary

Fields

Modifier and Type	Field and Description
private Pattern	colSep Separator pattern.
private int	end Current positions of result and iterator.
private int	index Current positions of result and iterator.
private CharSequence	input Data currently processed.
private static Logging	LOG Class logger.
private Matcher	m Regular expression match helper.
static String	QUOTE_CHAR Quote characters
private char[]	quoteChars Stores the quotation character
private int	send Substring to process.
private int	start Current positions of result and iterator.

Constructor Summary

Constructors

Constructor and Description
Tokenizer(Pattern colSep, String quoteChars) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	advance() Moves the iterator forward to the next entry.
double	getDouble() Get current value as double.
int	getEnd() Get end of token.
long	getLongBase10() Get current value as long.
int	getStart() Get start of token.
String	getSubstring() Get the current part as substring
void	initialize(CharSequence input, int begin, int end) Initialize parser with a new string.
boolean	isEmpty() Test for empty tokens; usually at end of line.
private char	isQuote(int index) Detect quote characters.
boolean	valid() Returns true if the iterator currently points to a valid object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final Logging LOG

Class logger.

colSep

private Pattern colSep

Separator pattern.

QUOTE_CHAR

public static final String QUOTE_CHAR

Quote characters

See Also:

Constant Field Values

quoteChars

private char[] quoteChars

Stores the quotation character

m

private Matcher m

Regular expression match helper.

input

private CharSequence input

Data currently processed.

send

private int send

Substring to process.

start

private int start

Current positions of result and iterator.

end

private int end

Current positions of result and iterator.

index

private int index

Current positions of result and iterator.

Constructor Detail

Tokenizer

public Tokenizer(Pattern colSep,
         String quoteChars)

Constructor.

Parameters:

colSep - Column separator pattern.

quoteChars - Quotation character.

Method Detail

initialize

public void initialize(CharSequence input,
              int begin,
              int end)

Initialize parser with a new string.

Parameters:

input - New string to parse.

begin - Begin

end - End

valid

public boolean valid()

Description copied from interface: Iter

Returns true if the iterator currently points to a valid object.

Specified by:

valid in interface Iter

Returns:

a boolean value, whether the position is valid.

advance

public void advance()

Description copied from interface: Iter

Moves the iterator forward to the next entry.

Specified by:

advance in interface Iter

getSubstring

public String getSubstring()

Get the current part as substring

Returns:

Current value as substring.

getDouble

public double getDouble()
                 throws NumberFormatException

Get current value as double.

Returns:

double value

Throws:

NumberFormatException - when current value cannot be parsed as double value.

getLongBase10

public long getLongBase10()
                   throws NumberFormatException

Get current value as long.

Returns:

double value

Throws:

NumberFormatException - when current value cannot be parsed as long value.

isEmpty

public boolean isEmpty()

Test for empty tokens; usually at end of line.

Returns:

Empty

isQuote

private char isQuote(int index)

Detect quote characters. TODO: support more than one quote character, make sure opening and closing quotes match then.

Parameters:

index - Position

Returns:

1 when a quote character, 0 otherwise.

getStart

public int getStart()

Get start of token.

Returns:

Start

getEnd

public int getEnd()

Get end of token.

Returns:

End