Uploaded image for project: 'Couchbase Java Client'
  1. Couchbase Java Client
  2. JCBC-573

Sell role went down after 24 to 32 hours with Illegal state exception :Timed out waiting to add Cmd

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • None
    • 1.4.4
    • Core
    • Security Level: Public
    • None

    Description

      Current release, we had enabled write to couchbase turned ON. Couchbase regression was fine in QA and PE env. It was fine even in Production intially, but after 24 to 30 hours of deployemnt, we started seeing below exceptions and blade went down. Sell flow was redirecting to error page.
      Need to know why is this happening. Once we restart the application server exception doesnt show up.
      But we are sure, it will start happening soon

      2014-10-02 00:26:34,000 [c49a#1061c/https:///sell/] priority=ERROR app_name=stubhubSLXRole thread=ajp-0.0.0.0-8009-41 location=NewAccessControlHandler line=175 Exception
      java.lang.IllegalStateException: Timed out waiting to add Cmd: 1 Opaque: 1147840 Key: com.stubhub.user.business.entity.UserSession|63C21A07311389A1EC361D834BF46E72 Cas: 0 Exp: 7200 Flags: 1 Data Length: 1748(max wait=10000ms)
      at net.spy.memcached.protocol.TCPMemcachedNodeImpl.addOp(TCPMemcachedNodeImpl.java:362)
      at net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:1267)
      at com.couchbase.client.CouchbaseConnection.addOperation(CouchbaseConnection.java:277)
      at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:1185)
      at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:328)
      at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:929)
      at com.stubhub.common.cache.store.couchbase.CouchbaseStore.put(CouchbaseStore.java:148)
      at com.stubhub.common.cache.store.CompositeCacheStore.put(CompositeCacheStore.java:69)
      at com.stubhub.common.session.impl.UserSessionCacheManagerImpl.putCrossModuleValue(UserSessionCacheManagerImpl.java:48)
      at com.stubhub.user.business.manager.impl.UserSessionCacheMgrImpl.putUserSessionToStore(UserSessionCacheMgrImpl.java:269)
      at com.stubhub.user.business.manager.impl.UserSessionCacheMgrImpl.createUserSession(UserSessionCacheMgrImpl.java:806)
      at sun.reflect.GeneratedMethodAccessor1305.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at com.stubhub.common.util.StubHubPerformanceMonitorInterceptor.invoke(StubHubPerformanceMonitorInterceptor.java:28)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at $Proxy725.createUserSession(Unknown Source)
      at com.stubhub.user.business.facade.impl.UserSessionFacadeImpl.createUnidentifiedSession(UserSessionFacadeImpl.java:564)
      at com.stubhub.user.business.facade.impl.UserSessionFacadeImpl.createUnidentifiedSession(UserSessionFacadeImpl.java:94)
      at sun.reflect.GeneratedMethodAccessor1297.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            james.mauss James Mauss (Inactive)
            rnagarajaiah@stubhub.com Rukmini Nagarajaiah
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty