No more cluster attempts left 折腾半天没解决

对redis自动生成数据接口进行压测，压测使用ApiPost进行,并发数50,轮次2000。

图片

图片

起初数据执行是正常的，先是出现了redis集群错误

redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more cluster attempts left.
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:86) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:124) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:124) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:124) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:124) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:124) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:25) ~[jedis-3.0.1.jar:na]
at redis.clients.jedis.JedisCluster.exists(JedisCluster.java:142) ~[jedis-3.0.1.jar:na
查看源码报错位置如下

private T runWithRetries(final int slot, int attempts, boolean tryRandomNode, JedisRedirectionException redirect) {
if (attempts <= 0) {
throw new JedisClusterMaxAttemptsException(“No more cluster attempts left.”);
}
attempts小于等于0时报错,根据日志向上追踪至124行

 // release current connection before recursion
  releaseConnection(connection);
  connection = null;

  if (attempts <= 1) {
    //We need this because if node is not reachable anymore - we need to finally initiate slots
    //renewing, or we can stuck with cluster state without one node in opposite case.
    //But now if maxAttempts = [1 or 2] we will do it too often.
    //TODO make tracking of successful/unsuccessful operations for node - do renewing only
    //if there were no successful responses from this node last few seconds
    this.connectionHandler.renewSlotCache();
  }
  //124行
  return runWithRetries(slot, attempts - 1, tryRandomNode, redirect);

入参之前对attempts 进行-1，也就是说attempts 想要小于等于0，attempts 入参就是小于等于1，根据注释来说当尝试次数是1-2之间时，将会对不成功响应的节点进行初始化。根据压测数量来看，最多数据为2000*50即10万条，排除数据量过大或者版本问题，开发环境redis为集群模式。

最终也没有得到合适的解决方案，当然在压测过程中除了偶发失败之外，用户是无感知的，节点失败重试即可。

getOutputStream() has already been called for this response

response.getOutputStream() 已经用过了不能再次使用。
java.lang.IllegalStateException: getOutputStream() has already been called for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:582) ~[tomcat-embed-core-9.0.14.jar:9.0.14]
at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:227) ~[tomcat-embed-core-9.0.14.jar:9.0.14]
在get请求中，对于token验证和跨域处理，通过实现Filter接口实现，报错位置如下

private void handleException(ServletResponse response, CommonResponse commonResponse) throws IOException {
response.setContentType(“application/json;charset=UTF-8”);
PrintWriter writer = response.getWriter();//此行
writer.write(JSONUtil.tJSON(commonResponse));
writer.flush();
writer.close();
}
图片

图片

针对filter的代码非源码部分，stackoverflow针对此问题有解答

图片

建议执行flush方法后执行close方法，可见在代码中是没有区别的。

知识点

SpringMVC中所有Controller接口进行返回时底层都是用response.getOutputStream() 或 response.getWriter()进行输出的；Tomcat的ServletRequest中, getParameter()方法与getInputStream()/getReader()不兼容, 只能选择一方.调用了一方, 另一方就会是空的(前提：表单的POST请求).

private void handleException(ServletResponse response, CommonResponse commonResponse) throws IOException {
response.setContentType(“application/json;charset=UTF-8”);
response.reset();
PrintWriter writer = response.getWriter();
writer.write(JSONUtil.tJSON(commonResponse));
writer.flush();
writer.close();
}
但是仍没有解决，依旧报错。在HttpServletRequestWrapper中重写getreader和getInputStream方法。

图片这个jodd的StreamUtil真是难找

图片

public class ParameterRequestWrapper extends HttpServletRequestWrapper {

private Map<String,Object> params;

private byte[] body;

public ParameterRequestWrapper(HttpServletRequest request, Map<String,Object> newParams) throws IOException {
super(request);
this.params = newParams;
body = StreamUtil.readBytes(request.getReader(), “UTF-8”);
}

@Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(getInputStream()));
}

@SuppressWarnings(“unchecked”)
public Map getParameterMap() {
return params;
}

@SuppressWarnings(“unchecked”)
public Enumeration getParameterNames() {
Vector l = new Vector(params.keySet());
return l.elements();
}

public String[] getParameterValues(String name) {
Object v = params.get(name);
if (v == null) {
return null;
} else if (v instanceof String[]) {
return (String[]) v;
} else if (v instanceof String) {
return new String[] { (String) v };
} else {
return new String[] { v.toString() };
}
}

public String getParameter(String name) {
Object v = params.get(name);
if (v == null) {
return null;
} else if (v instanceof String[]) {
String[] strArr = (String[]) v;
if (strArr.length > 0) {
return strArr[0];
} else {
return null;
}
} else if (v instanceof String) {
return (String) v;
} else {
return v.toString();
}
}

@Override
public ServletInputStream getInputStream() throws IOException {
final ByteArrayInputStream bais = new ByteArrayInputStream(body);
return new ServletInputStream() {

  @Override
  public boolean isFinished() {
    return bais.available() == 0;
  }

  @Override
  public boolean isReady() {
    return true;
  }

  @Override
  public void setReadListener(ReadListener readListener) {

  }

  @Override
  public int read() throws IOException {
    return bais.read();
  }
};

}

}
仍旧报错图片

在本filter执行dofilterchain之前应将request转为HttpServletRequest然后再进行wrapper初始化。

更改代码如下

HttpServletRequest req = (HttpServletRequest) request;
Map<String,Object> map=new HashMap(request.getParameterMap());
String[] keySet = map.keySet().toArray(new String[0]);

        for(String key : keySet){
            Object value = map.get(key);
            if (value instanceof String[]) {
                String[] valueArray = (String[]) value;
                for (int i = 0; i < valueArray.length; i++) {
                    if(org.apache.commons.lang.StringUtils.isNotEmpty(valueArray[i]) && !"null".equals(valueArray[i])){
                        valueArray[i] = valueArray[i].trim();
                    }else{
                        map.remove(key);
                    }
                }
            }else{
                if(value==null){
                    map.remove(key);
                }
            }
        }
        ParameterRequestWrapper wrapRequest=new ParameterRequestWrapper(req,map);

        chain.doFilter(wrapRequest, response); 

报错仍旧产生，但细心又发现在我这个filter之前还有一个上游组件的filter已经执行了dofilter，但是在上游的dofilter并未执行response的getwriter方法。

在报错的源码中最终定位到org.apache.catalina.connector.Response#getWriter方法，可以看到在if判断中应该是为了保证调用的resposne写方法一致，要不都用getWriter()，要不都用getOutputStream()。而我并没有使用getOutputStream()，那就只能说SpringBoot中底层代码用的getOutputStream。

@Override
public PrintWriter getWriter()
throws IOException {

    if (usingOutputStream) {
        throw new IllegalStateException
            (sm.getString("coyoteResponse.getWriter.ise"));
    }

    if (ENFORCE_ENCODING_IN_GET_WRITER) {
        /*
         * If the response's character encoding has not been specified as
         * described in <code>getCharacterEncoding</code> (i.e., the method
         * just returns the default value <code>ISO-8859-1</code>),
         * <code>getWriter</code> updates it to <code>ISO-8859-1</code>
         * (with the effect that a subsequent call to getContentType() will
         * include a charset=ISO-8859-1 component which will also be
         * reflected in the Content-Type response header, thereby satisfying
         * the Servlet spec requirement that containers must communicate the
         * character encoding used for the servlet response's writer to the
         * client).
         */
        setCharacterEncoding(getCharacterEncoding());
    }

    usingWriter = true;
    outputBuffer.checkConverter();
    if (writer == null) {
        writer = new CoyoteWriter(outputBuffer);
    }
    return writer;
}

遂改造如下

ServletOutputStream outputStream = response.getOutputStream();
outputStream.flush();
outputStream.close();
，尽管如此，我仍觉得流的关闭节点有问题，但我又没什么证据。第一个报错用户虽无法感知，但面临丢数据和偶发报错的情况，如有好的解决方案会及时更新。

强烈推荐参考博文
https://www.cnblogs.com/bencakes/p/14433332.html
https://blog.csdn.net/u012977486/article/details/88821887
https://blog.csdn.net/TimerBin/article/details/90295451