Lucene 同义词
项目原来是基于一个挺老的项目基础做的迭代二次开发，原来使用的是庖丁分词，这个分词库已经很久没人更新了，而且和高版本的lucene也有兼容问题,所以处理同义词问题之前，我把原来使用的padding中文分词替换成word中文分词;
同义词处理逻辑实际代码主要参考的这两个帖子:
https://blog.csdn.net/yax405/article/details/43246237
https://blog.csdn.net/winnerspring/article/details/37567739
部分代码如下:
package org.apdplat.word.lucene;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.Analyzer.TokenStreamComponents;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.core.LowerCaseFilterFactory;
import org.apache.lucene.analysis.synonym.SynonymFilterFactory;
import org.apache.lucene.analysis.util.FilesystemResourceLoader;
import org.apache.lucene.util.Version;
import org.apdplat.word.segmentation.Segmentation;
import org.apdplat.word.segmentation.SegmentationAlgorithm;
import org.apdplat.word.segmentation.SegmentationFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class SynonymsAnalyzer
extends Analyzer
{
private static final Logger LOGGER = LoggerFactory.getLogger(ChineseWordAnalyzer.class);
private Segmentation segmentation = null;
public SynonymsAnalyzer()
{
this.segmentation = SegmentationFactory.getSegmentation(SegmentationAlgorithm.BidirectionalMinimumMatching);
}
public SynonymsAnalyzer(String segmentationAlgorithm)
{
try
{
SegmentationAlgorithm sa = SegmentationAlgorithm.valueOf(segmentationAlgorithm);
this.segmentation = SegmentationFactory.getSegmentation(sa);
}
catch (Exception e)
{
this.segmentation = SegmentationFactory.getSegmentation(SegmentationAlgorithm.BidirectionalMinimumMatching);
}
}
public SynonymsAnalyzer(SegmentationAlgorithm segmentationAlgorithm)
{
this.segmentation = SegmentationFactory.getSegmentation(segmentationAlgorithm);
}
public SynonymsAnalyzer(Segmentation segmentation)
{
this.segmentation = segmentation;
}
private static SynonymFilterFactory factory = null;
protected static SynonymFilterFactory getSynonymsFactory()
{
if (factory == null)
{
Map paramsMap = new HashMap();
Version ver = Version.LUCENE_5_5_1;
paramsMap.put("luceneMatchVersion", ver.toString());
paramsMap.put("synonyms", "./synonyms.txt");
paramsMap.put("expand", "true");
factory = new SynonymFilterFactory(paramsMap);
try
{
FilesystemResourceLoader loader = new FilesystemResourceLoader(Paths.get("D:/RobotK/tomcat/synonyms", new String[0]));
factory.inform(loader);
}
catch (IOException e)
{
e.printStackTrace();
}
}
return factory;
}
public static void reloadSynonymFilterFactory()
{
Map paramsMap = new HashMap();
Version ver = Version.LUCENE_5_5_1;
paramsMap.put("luceneMatchVersion", ver.toString());
paramsMap.put("synonyms", "./synonyms.txt");
paramsMap.put("expand", "true");
factory = new SynonymFilterFactory(paramsMap);
try
{
FilesystemResourceLoader loader = new FilesystemResourceLoader(Paths.get("D:/RobotK/tomcat/synonyms", new String[0]));
factory.inform(loader);
}
catch (IOException e)
{
e.printStackTrace();
}
}
private static LowerCaseFilterFactory caseFactory = null;
public static LowerCaseFilterFactory getCaseFilterFactory()
{
if (caseFactory == null)
{
Version ver = Version.LUCENE_5_5_1;
Map<String, String> filterArgs = new HashMap();
filterArgs.put("luceneMatchVersion", ver.toString());
caseFactory = new LowerCaseFilterFactory(filterArgs);
}
return caseFactory;
}
protected Analyzer.TokenStreamComponents createComponents(String fieldName)
{
Tokenizer tokenizer = new ChineseWordTokenizer(this.segmentation);
TokenStream casefilter = getCaseFilterFactory().create(tokenizer);
TokenStream filter = getSynonymsFactory().create(casefilter);
return new Analyzer.TokenStreamComponents(tokenizer, filter);
}
}其他参考的帖子有:
https://blog.csdn.net/u011066470/article/details/60963439
http://www.hankcs.com/program/java/lucene-synonymfilterfactory.html
https://blog.csdn.net/yax405/article/details/43246237
https://blog.csdn.net/liyantianmin/article/details/59485799
https://blog.csdn.net/winnerspring/article/details/37567739
https://my.oschina.net/apdplat/blog/228619
http://www.hankcs.com/program/java/lucene-synonymfilterfactory.html
在Lucene4.6中通过SynonymFilterFactory实现中文同义词非常方便，只需几行代码和一个同义词词典。这个词典还能在Lucene中实现一定程度的拼写纠错，提升搜索体验。在下面这个例子中我们从磁盘载入一个同义词词典，并且对“其实hankcs似好人”这句话进行stream化以供索引，同时还对其中的拼写错误“似->是”做出纠正。
首先是位于./data/synonyms.txt路径下的同义词词典：
可以看出上面有两种词典格式：
通过,分割的可拓展同义词
比如“我,俺,hankcs”代表着这三个词是同义词，并且任何一个词可以被expand（拓展）为其他三个。如果expand设为false的话，则这三个词都会被统一替换为第一个词，也就是“我”。
通过=>收缩的不可拓展同义词
比如“似,is,are => 是”代表这三个词同义，并且无视expand参数，统一会被替换为“是”
然后是加载代码
package com.hankcs.test;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.analysis.synonym.SynonymFilterFactory;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.analysis.util.FilesystemResourceLoader;
import org.apache.lucene.util.Version;
import org.apache.uima.annotator.WhitespaceTokenizer;
import java.io.IOException;import java.io.StringReader;
import java.util.HashMap;import java.util.Map;
/** * @author hankcs */
public class TestSynonyms{
private static void displayTokens(TokenStream ts) throws IOException {
CharTermAttribute termAttr = ts.addAttribute(CharTermAttribute.class);
OffsetAttribute offsetAttribute = ts.addAttribute(OffsetAttribute.class);
ts.reset();
while (ts.incrementToken()) {
String token = termAttr.toString();
System.out.print(offsetAttribute.startOffset() + "-" + offsetAttribute.endOffset() + "[" + token + "] ");
}
System.out.println();
ts.end();
ts.close();
}
public static void main(String[] args) throws Exception {
String testInput = "其实 hankcs 似 好人";
Version ver = Version.LUCENE_46;
Map<String, String> filterArgs = new HashMap<String, String>();
filterArgs.put("luceneMatchVersion", ver.toString());
filterArgs.put("synonyms", "./data/synonyms.txt");
filterArgs.put("expand", "true");
SynonymFilterFactory factory = new SynonymFilterFactory(filterArgs);
factory.inform(new FilesystemResourceLoader());
WhitespaceAnalyzer whitespaceAnalyzer = new WhitespaceAnalyzer(ver);
TokenStream ts = factory.create(whitespaceAnalyzer.tokenStream("someField", testInput));
displayTokens(ts);
}
}输出：
0-2[其实] 3-9[我] 3-9[俺] 3-9[hankcs] 10-11[是] 12-14[好人] 12-14[好心人] 12-14[热心人]
由于 我 俺 hankcs 三个词是同一个意思，所以它们被视为同一个term，并且它们的偏移相同，都是3->9，这个长度取决于原来的词 hankcs 的长度。
https://blog.csdn.net/yax405/article/details/43246237
https://blog.csdn.net/u010366796/article/details/44937025
http://www.voidcn.com/article/p-txqtdabn-bbo.html
http://blog.csdn.net/winnerspring/article/details/37521101
http://www.voidcn.com/article/p-pjrzypvg-bbo.html
http://blog.csdn.net/winnerspring/article/details/37567739
http://blog.csdn.net/hu948162999/article/details/41283597
http://www.voidcn.com/article/p-xrordklc-bah.html
https://iamyida.iteye.com/blog/2197355
https://cloud.tencent.com/info/034aa996312ba4928c57ae831d6acedf.html
lucene 同义词的索引
public interface SynonymEngine {
String[] getSynonyms(String key);
}public class SynonymEngineImpl implements SynonymEngine {
private static HashMap<String,String[]> map = new HashMap<String ,String[]>();
static {
map.put("quick",new String[]{"fast","speedy"});
map.put("jumps",new String[]{"leaps","hops"});
map.put("over",new String[]{"above"});
map.put("lazy",new String[]{"apathetic","sluggish"});
map.put("dog",new String[]{"canine","pooch"});
}
@Override
public String[] getSynonyms(String key) {
// TODO Auto-generated method stub
return map.get(key);
}
}public class SynonymFilter extends TokenFilter {
private SynonymEngine engine;
private CharTermAttribute ct;
private PositionIncrementAttribute pt;
private Stack<String> stack;
private AttributeSource.State current;
protected SynonymFilter(TokenStream input,SynonymEngine engine) {
super(input);
this.engine = engine;
ct = this.addAttribute(CharTermAttribute.class);
pt = this.addAttribute(PositionIncrementAttribute.class);
stack = new Stack<String>();
}
@Override
public boolean incrementToken() throws IOException {
if(stack.size()>0) {
this.restoreState(current);
String p = stack.pop();
ct.setEmpty();
ct.append(p);
pt.setPositionIncrement(0);
return true;
}
System.out.println("++++++"+ct);
if(!input.incrementToken()) return false;
System.out.println("------"+ct);
if(addSynonym(ct.toString())) {
current = this.captureState();
}
return true;
}
private boolean addSynonym(String name) {
String[] sa = engine.getSynonyms(name);
if(sa != null && sa.length>0) {
for(String s:sa) {
stack.push(s);
}
return true;
} else {
return false;
}
}
}public class SynonymAnalyzer extends Analyzer {
private SynonymEngine engine;
public SynonymAnalyzer(SynonymEngine engine) {
this.engine = engine;
}
@Override
public TokenStream tokenStream(String s, Reader reader) {
// TODO Auto-generated method stub
return new SynonymFilter(new StopFilter(Version.LUCENE_35,
new LowerCaseFilter(Version.LUCENE_35,
new StandardFilter(Version.LUCENE_35,
new StandardTokenizer(Version.LUCENE_35,reader)))
,StopAnalyzer.ENGLISH_STOP_WORDS_SET),engine);
}
}public class TestSynonym {
private RAMDirectory directory;
@Test
public void init() {
directory = new RAMDirectory();
SynonymEngine engine = new SynonymEngineImpl();
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_35,new SynonymAnalyzer(engine));
String content = "The quick brown fox jumps over the lazy dog";
try {
IndexWriter writer = new IndexWriter(directory,config);
Document doc = new Document();
doc.add(new Field("content",content,Field.Store.YES,Field.Index.ANALYZED));
writer.addDocument(doc);
writer.close();
IndexReader reader = IndexReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);
TopDocs docs = searcher.search(new TermQuery(new Term("content","pooch")),10);
for(ScoreDoc sd:docs.scoreDocs) {
Document d = searcher.doc(sd.doc);
System.out.println(d.get("content"));
}
} catch (CorruptIndexException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (LockObtainFailedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}http://www.itzk.com/b/1109/581983.shtml
Lucene的同义词分析器讲解
这个分析器用SynonymFilter过滤器对StandardAnalyzer类进行封装,当向这个过滤器输入各个项时,会对这些项进行缓冲,并使用栈存储这些项的同义词[code]
public class SynonymFilter extends TokenFilter{
publicstatic final String TOKEN_TYPE_SYNONYM="SYNONYM";
privateStack synonymStack;
privateSynonynEngine engine;
publicSynonymFilter(TokenStream in,SynonymEngine engine){
super(in);
synonymStack=new Stack();//缓存同义词
this.engine=engine;
}
publicTOken next() throws IOException{
if (synonymStack.size()>0){//如何还有当前词的同义词没有输出,则输出
return (Token) synonymStack.pop();
}
Token token=input.next();//读取新词
if (token==null) {
return null;
}
addAliasesToStack(token);//存储新词的同义词
returntoken;
}
private voidaddAliasesToStack(Token token) throws IOException{
String[] synonyms=engine.getSynonyms(token.termText());
if (synonyms==mull) return;
for (int i=0;i<synonyms.length;i++){
Token synToken=newToken(synonyms[i],token.startOffset(),token.endOffset(),TOKEN_TYPE_SYNONYM);
synToken.setPositionIncrement(0);
synonymStack.push(synToken);
}
}[/code]以下这个接口是关键,可以自由实现,目的是返回s的同义词数组[code]public interface SynonymEngine{
String[] getSynonyms(String s) throws IOException;
}
[/code]对于这个接口要小心使用,在查询时不必列出所有的同义词,如下例[code]Query query=QueryParser.parse("\"foxjumps\"","content",synonymAnalyzer);
Hits hits=searcher.search(query);[/code]是会出错的,找不到任何结果,因为QueryParser不会区别位置增量,所以位置增量为0这一个表明同义的特征无法体现,会将"foxjumps"直接加上同义词解释为"fox jumps hops leaps"
https://stackoverrun.com/cn/q/4705011
我想弄清楚lucene的分析仪是如何工作的？ 我的问题是，lucene如何处理同义词？这里的情况是： 我们有一个词和多词
单：富=酒吧 多的话：富巴= foobar的
对于单个的词：
是否Lucene的扩大索引记录或不？我猜如果一个查询有一个像“foo”这样的词，它也会在查询中添加“bar”。我不知道是否索引或不索引？
对于多话：
是否Lucene的扩大查询和索引？例如，如果我们有“富吧”，它是否将foobar添加到索引/查询？
我的第二个问题是：Lucene使用一个标记流并将它们提供给小写过滤器之类的过滤器。我的问题是lucene如何找到多词？比如它是如何发现“foo bar”是一个多词的？
SynonymFilter可任选，保持原有的单词，并添加同义词到的TokenStream中，通过设置keepOrig =真（见SynonymMap.Builder.add()）。此行为可能会导致PhraseQueries等问题，请参阅SynonymFilter文档中的第注意事项。
如果您使用相同的Analyzer进行查询和编制索引，那么写入索引的查询和文档当然都会以同样的方式处理。 SynonymFilter与keepOrig设置为true是少数几个Analyzers之一，经常在查询和索引之间不合时宜地应用，但这完全取决于您的实现。
至于如何实施，source code可供您使用。
创建 24 6月. 13 femtoRgon
0
它是如何处理多个同义词的？像“纽约”=“纽约” 沃尔玛=沃尔玛=沃尔玛 ，因为它通过令牌执行过滤令牌。我不知道它是如何找到多个单词的同义词 – Mr.Boy 24 6月. 13
0
有没有你对它的行为感到困惑，或者你想知道实现如何处理令牌流？如果是后者，那就是为什么我提供了链接到源代码的原因。如果前者贪婪地搜索最长匹配，它可以从给定的位置（也就是说，如果你有规则'foo' - >'bar'，'foo bar' - >'foobar'，那么'foo bar'会变成'foobar'，而不是'bar bar'）。我不相信它支持'wal mart = wal-mart = walmart'这样的东西（同义词规则有一个输入和一个输出）。如果有什么特别的要问的话，继续。 – femtoRgon 24 6月. 13
0
我的问题是它如何处理令牌流？因为我猜同义词过滤器一个接一个地得到令牌，而且它是无状态的。例如，如果当前令牌为“新”，它如何检查下一个令牌以查看它是否是“约克”？
华为开发者空间发布
让每位开发者拥有一台云主机
- 点赞
- 收藏
- 关注作者
评论（0）