open-source Chinese TokenizerLibrary PaoDing with Apache Solr 1.4

There are several tokerlizer libraries that comes with Solr 1.4 by default. ChineseTokenlizerFactory is the only one that can analyze and split Chinese text. however, it’s very basic and kind of poor library if you want to do some serious thing with Chinese text.
the following screenshot is the output of the standard ChineseTokerlizerFactory analysis. “Hello World” in Chinese will be “你好世界”, the correct analyzed tokens should be 你好 and 世界,instead of 4 meanness characters 

we expect the following result instead.

 

PaoDing is a popular and open-source libary that is pretty much to do the splitting and analysis job. Here is the step to integrate it with the Solr server.

1. Down paoding bits from googlecode. upzip the bits and setup one system environment variable which points to the dic folder like PAODING_DIC_HOME=c:\solr\paoding\dic
   2. Build one TokerLizerFactory that uses the paoding analyzer.basically two java class.

package test.solr.PaodingTokerFactory; import java.io.IOException; import java.io.Reader; import net.paoding.analysis.analyzer.PaodingAnalyzer; import org.apache.lucene.analysis.Token; import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.analysis.Tokenizer; public class PaoDingTokerLizer extends Tokenizer {     PaodingAnalyzer analyzer = new PaodingAnalyzer();     TokenStream ts;     public PaoDingTokerLizer(Reader reader) {         // TODO Auto-generated constructor stub         super(reader);         ts = analyzer.tokenStream("",reader);     }     @Override     public Token next() {         // TODO Auto-generated method stub         try {             return ts.next();         } catch (IOException e) {             // TODO Auto-generated catch block             e.printStackTrace();         }         return null;     } }

package test.solr.PaodingTokerFactory; import java.io.Reader; import org.apache.lucene.analysis.Tokenizer; import org.apache.solr.analysis.BaseTokenizerFactory; public class PaoDingTokenizerFactory extends BaseTokenizerFactory {          public Tokenizer create(Reader arg0) {         // TODO Auto-generated method stub         return new PaoDingTokerLizer(arg0);     } }

3. Build/Compile the code, and copy the compiled Jar or classes to tomcat lib directory [tomcat\webapps\solr\WEB-INF\lib] where lucen*.jar exists,Also copy the compiled paoding-analysis.jar here
4. Change solr schema.xml file to reference the new library

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">       <analyzer type="index">         <!-- <tokenizer class="solr.ChineseTokenizerFactory"/> -->        <tokenizer class="test.solr.PaodingTokerFactory.PaoDingTokenizerFactory"/>

5. restart the solr server and then you go!

Some references:
Lucene: http://lucene.apache.org/java/docs/
Solr: http://lucene.apache.org/solr/
Lucid Solr: enterprise lelve support for Solr/lucene http://www.lucidimagination.com/blog/
Lucene in Action:http://www.manning.com/hatcher2/
Paoding Library: http://code.google.com/p/paoding/downloads/list

Books:
Solr 1.4 Enterprise Search Server Great start and everything about Solr
Building Search Applications: Lucene, LingPipe, and Gate :the base of Solr, How Indexing works.