Package org.apache.lucene.classification

Class SimpleNaiveBayesClassifier

java.lang.Object

org.apache.lucene.classification.SimpleNaiveBayesClassifier

All Implemented Interfaces:

Classifier<BytesRef>

Direct Known Subclasses:

CachingNaiveBayesClassifier, SimpleNaiveBayesDocumentClassifier

public class SimpleNaiveBayesClassifier
extends java.lang.Object
implements Classifier<BytesRef>

A simplistic Lucene based NaiveBayes classifier, see http://en.wikipedia.org/wiki/Naive_Bayes_classifier

Field Summary

Fields

Modifier and Type	Field	Description
protected Analyzer	analyzer	Analyzer to be used for tokenizing unseen input text
protected java.lang.String	classFieldName	name of the field to be used as a class / category output
protected IndexReader	indexReader	IndexReader used to access the Classifier's index
protected IndexSearcher	indexSearcher	IndexSearcher to run searches on the index for retrieving frequencies
protected Query	query	Query used to eventually filter the document set to be used to classify
protected java.lang.String[]	textFieldNames	names of the fields to be used as input text

Constructor Summary

Constructors

Constructor	Description
SimpleNaiveBayesClassifier(IndexReader indexReader, Analyzer analyzer, Query query, java.lang.String classFieldName, java.lang.String... textFieldNames)	Creates a new NaiveBayes classifier.

Method Summary

Modifier and Type	Method	Description
ClassificationResult<BytesRef>	assignClass(java.lang.String inputDocument)	Assign a class (with score) to the given text String
protected java.util.List<ClassificationResult<BytesRef>>	assignClassNormalizedList(java.lang.String inputDocument)	Calculate probabilities for all classes for a given input text
private double	calculateLogLikelihood(java.lang.String[] tokenizedText, Term term, int docsWithClass)
private double	calculateLogPrior(Term term, int docsWithClassSize)
protected int	countDocsWithClass()	count the number of documents in the index having at least a value for the 'class' field
private int	docCount(Term term)
java.util.List<ClassificationResult<BytesRef>>	getClasses(java.lang.String text)	Get all the classes (sorted by score, descending) assigned to the given text String.
java.util.List<ClassificationResult<BytesRef>>	getClasses(java.lang.String text, int max)	Get the first max classes (sorted by score, descending) assigned to the given text String.
private double	getTextTermFreqForClass(Term term)	Returns the average number of unique terms times the number of docs belonging to the input class
private int	getWordFreqForClass(java.lang.String word, Term term)	Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)
protected java.util.ArrayList<ClassificationResult<BytesRef>>	normClassificationResults(java.util.List<ClassificationResult<BytesRef>> assignedClasses)	Normalize the classification results based on the max score available
protected java.lang.String[]	tokenize(java.lang.String text)	tokenize a String on this classifier's text fields and analyzer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

indexReader

protected final IndexReader indexReader

IndexReader used to access the Classifier's index

textFieldNames

protected final java.lang.String[] textFieldNames

names of the fields to be used as input text

classFieldName

protected final java.lang.String classFieldName

name of the field to be used as a class / category output

analyzer

protected final Analyzer analyzer

Analyzer to be used for tokenizing unseen input text

indexSearcher

protected final IndexSearcher indexSearcher

IndexSearcher to run searches on the index for retrieving frequencies

query

protected final Query query

Query used to eventually filter the document set to be used to classify

Constructor Detail

SimpleNaiveBayesClassifier

public SimpleNaiveBayesClassifier(IndexReader indexReader,
                                  Analyzer analyzer,
                                  Query query,
                                  java.lang.String classFieldName,
                                  java.lang.String... textFieldNames)

Creates a new NaiveBayes classifier.

Parameters:

indexReader - the reader on the index to be used for classification

analyzer - an Analyzer used to analyze unseen text

query - a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used

classFieldName - the name of the field used as the output for the classifier NOTE: must not be havely analyzed as the returned class will be a token indexed for this field

textFieldNames - the name of the fields used as the inputs for the classifier, NO boosting supported per field

Method Detail

assignClass

public ClassificationResult<BytesRef> assignClass(java.lang.String inputDocument)
                                           throws java.io.IOException

Description copied from interface: Classifier

Assign a class (with score) to the given text String

Specified by:

assignClass in interface Classifier<BytesRef>

Parameters:

inputDocument - a String containing text to be classified

Returns:

a ClassificationResult holding assigned class of type T and score

Throws:

java.io.IOException - If there is a low-level I/O error.

getClasses

public java.util.List<ClassificationResult<BytesRef>> getClasses(java.lang.String text)
                                                          throws java.io.IOException

Description copied from interface: Classifier

Get all the classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

Returns:

the whole list of ClassificationResult, the classes and scores. Returns null if the classifier can't make lists.

Throws:

java.io.IOException - If there is a low-level I/O error.

getClasses

public java.util.List<ClassificationResult<BytesRef>> getClasses(java.lang.String text,
                                                                 int max)
                                                          throws java.io.IOException

Description copied from interface: Classifier

Get the first max classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

max - the number of return list elements

Returns:

the whole list of ClassificationResult, the classes and scores. Cut for "max" number of elements. Returns null if the classifier can't make lists.

Throws:

java.io.IOException - If there is a low-level I/O error.

assignClassNormalizedList

protected java.util.List<ClassificationResult<BytesRef>> assignClassNormalizedList(java.lang.String inputDocument)
                                                                            throws java.io.IOException

Calculate probabilities for all classes for a given input text

Parameters:

inputDocument - the input text as a String

Returns:

a List of ClassificationResult, one for each existing class

Throws:

java.io.IOException - if assigning probabilities fails

countDocsWithClass

protected int countDocsWithClass()
                          throws java.io.IOException

count the number of documents in the index having at least a value for the 'class' field

Returns:

the no. of documents having a value for the 'class' field

Throws:

java.io.IOException - if accessing to term vectors or search fails

tokenize

protected java.lang.String[] tokenize(java.lang.String text)
                               throws java.io.IOException

tokenize a String on this classifier's text fields and analyzer

Parameters:

text - the String representing an input text (to be classified)

Returns:

a String array of the resulting tokens

Throws:

java.io.IOException - if tokenization fails

calculateLogLikelihood

private double calculateLogLikelihood(java.lang.String[] tokenizedText,
                                      Term term,
                                      int docsWithClass)
                               throws java.io.IOException

Throws:

java.io.IOException

getTextTermFreqForClass

private double getTextTermFreqForClass(Term term)
                                throws java.io.IOException

Returns the average number of unique terms times the number of docs belonging to the input class

Parameters:

term - the term representing the class

Returns:

the average number of unique terms

Throws:

java.io.IOException - if a low level I/O problem happens

getWordFreqForClass

private int getWordFreqForClass(java.lang.String word,
                                Term term)
                         throws java.io.IOException

Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)

Parameters:

word - the token produced by the analyzer

term - the term representing the class

Returns:

the number of documents of the input class

Throws:

java.io.IOException - if a low level I/O problem happens

calculateLogPrior

private double calculateLogPrior(Term term,
                                 int docsWithClassSize)
                          throws java.io.IOException

Throws:

java.io.IOException

docCount

private int docCount(Term term)
              throws java.io.IOException

Throws:

java.io.IOException

normClassificationResults

protected java.util.ArrayList<ClassificationResult<BytesRef>> normClassificationResults(java.util.List<ClassificationResult<BytesRef>> assignedClasses)

Normalize the classification results based on the max score available

Parameters:

assignedClasses - the list of assigned classes

Returns:

the normalized results