하루y

[Load Balancing & Session Clustering 작업결과]

 

1.현상
- Load Balancing 은 되나, Sessoin Clustering은 동작하지 않음.
- 운영 WAS #1와 WAS#2의 Sessoin 이 공유되지 않음.
- 테스트 중에, 스테이징과 운영 WAS1/2 와는 통신이 되었으나, WAS1/2사이엔 통신이 되지 않았음.
- 오류로그

    Mar 04, 2015 9:35:45 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
   INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 0, 160}:5000,{192, 168, 0, 160},5000, alive=90360857, secu
   rePort=-1, UDP Port=-1, id={82 -47 -59 -123 -70 31 75 28 -107 -5 92 -126 66 64 -42 -105 }, payload={}, command={}, domain={}, ]]
   Mar 04, 2015 9:35:45 AM org.apache.catalina.tribes.tipis.LazyReplicatedMap publishEntryInfo
   SEVERE: Unable to replicate backup key:9F4800800260348B471306CA12F6963A.tomcat-ims1 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 0, 160}:5000,
   {192, 168, 0, 160},5000, alive=90360857, securePort=-1, UDP Port=-1, id={82 -47 -59 -123 -70 31 75 28 -107 -5 92 -126 66 64 -42 -105 }, payload={}, command={}, domain={}, ].
   Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{192, 168, 0, 160}:5000;
   org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{192, 168, 0, 160}:5000;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)
        at org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:64)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)
        at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:166)
        at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1006)
        at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:991)
        at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:592)
        at org.apache.catalina.session.StandardSession.setId(StandardSession.java:391)
        at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:288)
        at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:653)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2946)
        at org.apache.catalina.connector.Request.getSession(Request.java:2310)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:897)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:909)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at org.springframework.security.web.savedrequest.HttpSessionRequestCache.saveRequest(HttpSessionRequestCache.java:40)
        (중략...)

2. 조치결과
2.1 Session Manager 설정 변경
- server.xml 에서 <Manager/>태그를 context.xml 태그로 이동 함. tomcat 문서상으로 <Manager/>태그는 server.xml 또는 context.xml 에 올 수 있지만, 운영환경 때문인지 Manager 가 정상적으로 작동하지 않음.
- server.xml 에서는 <Manager/> 태그를 주석처리하고, context.xml 에 아래와 같이 <Manager/>추가

          <Manager className="org.apache.catalina.ha.session.BackupManager" 
                   expireSessionsOnShutdown="false" 
                   notifyListenersOnReplication="true" 
                   mapSendOptions="6"/>   
  

- 서버 재시작 후, 운영 WAS1/2의 catalina 로그를 보면 세션 Cluster가 구성되어 있는지 확인이 됨.

   Mar 10, 2015 9:53:11 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
   INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 0, 160}:6001,{192, 168, 0, 160},6001, alive=1011, 
         securePort=-1, UDP Port=-1, id={75 3 -118 -110 -29 94 64 3 -96 67 -86 92 68 43 -23 7 }, payload={}, command={}, domain={}, ]
   [localhostPWAS1 application]$
   [localhostPWAS1 application]$
   
 

2.2 일부 서버에 대해서 사용자 요청이 Was1/2로 분산되는 현상에 대해서 아래 추가 작업 수행함.
   1) apache 의 workers.properties 에 설정된 session_cookie 키와 Tomcat의 context.xml에 정의된 <Context  sessionCookieName=""> 의 
      session_id가 다르면, sticky_session 이 되지 않음. (Specifies whether requests with SESSION ID's should be routed back to the same Tomcat worker.)

   2) apache 의 workers.properties 에 설정된 balance_workers 의 work명과 Tomcat의 server.xml 에 정의된 
      <Engine ... jvmRoute=""> 명이 다른 경우 요청이 분산됨. apache는 session뒤에 있는 was명을 보고 route 된다.
     - 변경시 workers.properties 의 lb_work명은 "JkMount /* lb_work명" 과 같아야 함으로, 관련정보를 같이 수정해야 한다. (중요)
     - 작업대상서버: svc1, svc2, svc3, ...

   ps. 참조 : http://tomcat.apache.org/connectors-doc/reference/workers.html
   
 
9.1 WEB 추가 작업
 * worker 메뉴얼
   https://tomcat.apache.org/connectors-doc/reference/workers.html

 - ping 작업 : fail Over를 빠르게 하기 위해서.
 - retries 조정 : worker가 tomcat으로 중복 요청 방지 하기 위해서.
 
 1) normal workers 설정
    ping_mode=CI                  <- P(prepost), C(connect), I(Interval), A(All)
    ping_timeout=1000             <- ms
    connection_ping_interval=5    <- seconds, ping_mode=I 기능에서 사용.
    retries=1                     <- failover 등 오류 시 retry (기본:2회) / 간혹, 2로 할 경우, 간혹 was에서 중복처리될 수 있다.
    route=route_name     <-- tomcat의 jvmRoute 명과 같아야 한다. 설정하지 않으면, work명이 route 명이 된다.
    
 2) load balancer worker 설정
 
 PS. load balancer workers 에 설정 중 retries 횟수 조정.
     : load balancer workers 의 retries 수는 normal workers 수만큼 설정해야 한다. 
       # balance_workers 수만큼 설정해야 fail-over 됨.
       worker.lb-service.retries=2 <-- normal worker의 retries가 아닌, load balancer의 worker 설정이다.
       총 3개 normal worker가 있는데, retries가 2면 1,2번 서버가 내려갔을 때, 3으로 failOver 되지 않는다.
    retries=2                     <- failover 등 오류 시 다른 worker로 retry 횟수 (기본:2회)
worker.lbwkr.type=lb
worker.lbwrk.method=S                                      # 로드밸런싱 방법 지정. R[equest], S[ession], N[ext], T[raffic], B[usyness]
worker.lbwrk.session_cookie=JSESSIONID
worker.lbwrk.balance_workers=tomcat1,tomcat2
worker.lbwrk.sticky_session=1                              # 세션ID있을 때, 동일한 WAS로 라우팅함. 
worker.lbwrk.sticky_session_force=0
worker.lbwrk.retries=2                                     # balance_workers 수만큼 설정해야 fail-over 됨.
 ps. tomcat7.0.x는 다른 was로 라우팅되어도 세션ID에 붙어있는 jvmroute 명이 변경되지 않고, 추후 이전 was가 살아나면 다시 그쪽으로 라우팅 되는 듯 함. 예전에, 다른 was(weblogic이든가?ㅋ)는 라우팅된 명으로 변경된 것 같은데...