《重新定义Spring Cloud实战》——3.6　Eureka故障演练

3.6　Eureka故障演练

Chaos Engineering，中文翻译为混沌工程，它通过一系列可控的实验模拟真实世界可能出现的故障，来暴露系统的缺陷，从而驱动工程师去构建更具弹性的服务。常见的手段有：随机关闭依赖服务、模拟增加依赖服务的延时、模拟机器故障、网络中断等。对这方面有兴趣的读者可以查看《Chaos Engineering：Building Confidence in System Behavior through Experiments》（https://www.oreilly.com/webops-perf/free/chaos-engineering.csp）这本书。

这里我们手工模拟如下几个故障场景，通过这些故障来阐述Eureka的高可用。

3.6.1　Eureka Server全部不可用

假设Eureka Server全部挂掉，这里我们假设是同时都挂掉，然后分为下面两个场景来讨论。

1.应用服务启动前不可用

如果Eureka Server在应用服务启动之前挂掉或者没有启动的话，那么应用可以正常启动，但是会有报错信息，如下：

2018-05-28 22:16:16.603 ERROR 2048 --- [nfoReplicator-0] c.n.d.s.t.d.RedirectingEurekaHttpClient  : Request execution error

    com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused

    at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.8.8.jar:1.8.8]

    at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.netflix.discovery.shared.transport.jersey.AbstractJerseyEurekaHttpClient.register(AbstractJerseyEurekaHttpClient.java:56) ~[eureka-client-1.8.8.jar:1.8.8]

    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$1.execute(EurekaHttpClientDecorator.java:59) [eureka-client-1.8.8.jar:1.8.8]

    at com.netflix.discovery.shared.transport.decorator.MetricsCollectingEurekaHttpClient.execute(MetricsCollectingEurekaHttpClient.java:73) ~[eureka-client-1.8.8.jar:1.8.8]

由于连不上Eureka Server，自然访问不了service registry的服务注册信息，不能与其他服务交互。针对这种情况，Eureka Server设计了一个eureka.client.backup-registry-impl属性，可以配置在启动时Eureka Server访问不到的情况下，从这个back registry读取服务注册信息，作为fallback。该backup-registry-impl比较适合服务端提供负载均衡或者服务ip地址相对固定的场景。实例如下：

public class StaticBackupServiceRegistry implements BackupRegistry {

    private Applications localRegionApps = new Applications();

    public StaticBackupServiceRegistry() {

        Application orgApplication = new Application("org");

        InstanceInfo orgInstance1 = InstanceInfo.Builder.newBuilder()

            .setAppName("org-service")

            .setVIPAddress("org-service")

            .setSecureVIPAddress("org-service")

            .setInstanceId("org-instance-1")

            .setHostName("192.168.99.100")

            .setIPAddr("192.168.99.100")

            .setPort(9090)

            .setDataCenterInfo(new MyDataCenterInfo(DataCenterInfo.Name.MyOwn))

            .setStatus(InstanceInfo.InstanceStatus.UP)

            .build();

        InstanceInfo orgInstance2 = InstanceInfo.Builder.newBuilder()

            .setAppName("org-service")

            .setVIPAddress("org-service")

            .setSecureVIPAddress("org-service")

            .setInstanceId("org-instance-1")

            .setHostName("192.168.99.100")

            .setIPAddr("192.168.99.100")

            .setPort(9091)

            .setDataCenterInfo(new MyDataCenterInfo(DataCenterInfo.Name.MyOwn))

            .setStatus(InstanceInfo.InstanceStatus.UP)

            .build();

        orgApplication.addInstance(orgInstance1);

        orgApplication.addInstance(orgInstance2);

        localRegionApps.addApplication(orgApplication);

    }

    @Override

    public Applications fetchRegistry() {

        return localRegionApps;

    }

    @Override

    public Applications fetchRegistry(String[] includeRemoteRegions) {

        //ignore remote regions

        return localRegionApps;

    }

}

2.应用服务运行时不可用

Eureka Client在本地内存中有个AtomicReference<Applications>类型的localRegionApps变量，来维护从Eureka Server拉取回来的注册信息。Client端有个定时任务CacheRefreshThread，会定时从Server端拉取注册信息更新到本地，如果Eureka Server在应用服务运行时挂掉的话，本地的CacheRefreshThread会抛出异常，本地的localRegionApps变量不会得到更新。

异常输出实例如下：

2018-05-29 23:12:59.888 ERROR 1908 --- [freshExecutor-0] c.n.d.s.t.d.RedirectingEurekaHttpClient  : Request execution error

com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused (Connection refused)

    at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.8.8.jar:1.8.8]

    at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.1.jar:1.19.1]

    at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) ~[jersey-client-1.19.1.jar:1.19.1]

可以看到名为freshExecutor-0的线程请求Eureka Server时抛出了异常。