Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62684

Ingestion Halt: Failure when trying to connect to create dcp connection

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • Columnar 1.0.0
    • Columnar 1.0.0
    • analytics

    Description

      Cluster Config: 8vCPU, 32GB
      Steps to reproduce:
      1. Create a columnar instance.
      2. Create a capella provisioned cluster
      3. Make 2 collections on provisioned cluster
      4. Make 5 links randomly on provisioned collections and 50 collections randomly on the links.
      5. Scale node to 8 and wait for ingestion complete.
      6. When ingestion completes scale the nodes to 2
      7. Scale back to 8 nodes
      8. Add more data in the provisioned collections

      Note: Queries are always running regardless the state of instance

      These we saw in file: cbcollect_info_ns_1@svc-da-node-008.nxiirohw1zljp9d.sandbox.nonprod-project-avengers.com_20240711-051224

      2024-07-10T21:33:32.636+00:00 WARN CBAS.adapter.CouchbaseConnector [SAO:JID:5.609:TAID:TID:ANID:ODID:14:0:17:0:(ZdqlyALlbSa6/DHNF)[17]:BO] failure when trying to connect to create dcp connection
      java.lang.InterruptedException: null
      	at java.base/java.lang.Object.wait(Native Method) ~[?:?]
      	at java.base/java.lang.Object.wait(Object.java:472) ~[?:?]
      	at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:699) ~[core-io-2.5.2.jar:?]
      	at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:300) ~[core-io-2.5.2.jar:?]
      	at com.couchbase.client.dcp.conductor.NonStreamingConfigProvider.tryConnectHost(NonStreamingConfigProvider.java:116) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.client.dcp.conductor.NonStreamingConfigProvider.tryConnectHosts(NonStreamingConfigProvider.java:91) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.client.dcp.conductor.NonStreamingConfigProvider.refresh(NonStreamingConfigProvider.java:79) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.client.dcp.conductor.NonStreamingConfigProvider.refresh(NonStreamingConfigProvider.java:67) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.client.dcp.conductor.Conductor.connect(Conductor.java:83) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.client.dcp.Client.connect(Client.java:289) ~[columnar-dcp-client.jar:1.0.0-2203]
      	at com.couchbase.analytics.adapter.CouchbaseConnector.init(CouchbaseConnector.java:678) ~[columnar-connector.jar:1.0.0-2203]
      	at com.couchbase.analytics.adapter.CouchbaseConnector.hasNext(CouchbaseConnector.java:663) ~[columnar-connector.jar:1.0.0-2203]
      	at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.hasNext(FeedRecordDataFlowController.java:156) ~[asterix-external-data.jar:1.0.0-2203]
      	at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:87) ~[asterix-external-data.jar:1.0.0-2203]
      	at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:41) ~[asterix-external-data.jar:1.0.0-2203]
      	at org.apache.asterix.common.external.IDataSourceAdapter.start(IDataSourceAdapter.java:75) ~[asterix-common.jar:1.0.0-2203]
      	at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:50) ~[columnar-connector.jar:1.0.0-2203]
      	at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:101) ~[asterix-active.jar:1.0.0-2203]
      	at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:233) ~[hyracks-api.jar:1.0.0-2203]
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
      	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
      

      The cluster also got few high_cpu_usage warnings
      Dataset template:

      {
        "product_id": "1720689673445047358",
        "product_name": "Seth Abshire",
        "category": "Beauty",
        "subcategory": "Skincare",
        "brand": "IJK",
        "model": "cheap",
        "description": "Molestias ipsum ipsum iure odit. Ducimus non doloremque eius rerum.",
        "specifications": "Reiciendis illo perferendis voluptatem tempore. Et ut rerum quo et.",
        "price": 486.59,
        "currency": "USD",
        "stock_quantity": 44,
        "availability_status": "Coming Soon",
        "sku": "1498d306-e07e-48d2-9eb2-6024fdab877c",
        "ean": "17",
        "upc": "17",
        "isbn": "17",
        "manufacturer": "Gerhold, Boehm and Williamson",
        "supplier": "Sawayn-Lakin",
        "warranty": "1 Year",
        "warranty_period": "2 Years",
        "color": "LightSteelBlue",
        "size": "Medium",
        "weight": 1.856102231752582,
        "dimensions": "31x8x99 cm",
        "material": "Wood",
        "origin_country": "Australia",
        "release_date": "2004-03-30T06:18:22.272574963Z",
        "expiry_date": "1908-01-15T12:09:55.327810417Z",
        "rating": 3.4840514548982586,
        "reviews_count": 729,
        "images": [
          "http://pipsum.com/300x300.jpg"
        ],
        "videos": [
          "https://www.seniorreinvent.io/reinvent/synergistic"
        ],
        "manual": "https://www.customerenable.org/facilitate/proactive/wireless",
        "tags": [
          "officia"
        ],
        "shipping_weight": 5.925161322614086,
        "shipping_dimensions": "39x75x35 cm",
        "shipping_cost": 14.34,
        "shipping_time": "5-7 days",
        "return_policy": "Possimus labore quia tenetur ea omnis expedita dolore consequatur eius perferendis voluptas officiis cumque ea veritatis eligendi laudantium facere sint.",
        "related_products": [
          "e516ed04-0f53-4e9b-89d1-6144fdaa5596",
          "c05776f8-3b62-4eab-a148-9cf5b2ea83d2"
        ],
        "featured": false,
        "bestseller": true,
        "new_arrival": false,
        "discount": 55.28592863531945,
        "discount_start_date": "1938-12-04T19:25:51.461226988Z",
        "discount_end_date": "2016-08-12T02:21:55.791264992Z",
        "tax_rate": 15.51577240269022,
        "tax_included": true,
        "eco_friendly": true,
        "certifications": "perferendis",
        "barcode": "aec6a179-1f7c-40c0-ab38-848751fdadf8",
        "sales": 9184,
        "views": 28204,
        "clicks": 1411,
        "conversions": 74200,
        "meta_title": "Voluptatem omnis dolorem.",
        "meta_description": "Et ducimus ut vel quia. Nostrum culpa officia voluptatem qui.",
        "meta_keywords": "Sit quia magni.",
        "social_sharing_enabled": false,
        "affiliate_links": [
          "https://www.chiefvalue-added.io/viral/revolutionary/enterprise/empower",
          "http://www.dynamictransition.name/users"
        ],
        "bundle_offers": [
          "Voluptatibus non nam.",
          "Cum repellendus nostrum."
        ],
        "customizable": false,
        "personalization_options": "Odit alias distinctio.",
        "preorder": true,
        "preorder_date": "1931-08-13T23:12:07.96009226Z",
        "limited_edition": true,
        "serial_number": "caa3fa2a-643f-4a46-85aa-5cbaf3adcaf5",
        "subscription_available": true,
        "subscription_frequency": "vitae",
        "user_manual": "http://www.customerembrace.name/architect",
        "care_instructions": "In ex id.",
        "age_recommendation": "earum",
        "safety_warnings": "Expedita provident commodi.",
        "legal_disclaimer": "Excepturi aliquid aut.",
        "customer_support_contact": "1134737575",
        "faq": "Distinctio et quia.",
        "live_chat_support": true,
        "email_support": true,
        "phone_support": true,
        "wholesale_available": false,
        "minimum_order_quantity": 82,
        "bulk_discount": 34.38590144517376,
        "sample_available": false,
        "free_shipping": false,
        "international_shipping": false,
        "drop_shipping": true,
        "gift_wrapping_available": true,
        "loyalty_points": 241,
        "reward_points": 894,
        "part_number": "f7e0f6a6-1122-4d25-b027-527743fb0e4c",
        "battery_included": true,
        "battery_required": true,
        "battery_type": "velit",
        "battery_life": "Laudantium rerum et.",
        "power_source": "velit",
        "voltage": "perferendis",
        "wattage": "sint",
        "amperage": "ut",
        "energy_efficiency_rating": "aliquid",
        "compatibility": "sit",
        "software_included": "dicta",
        "license_key": "sequi"
      }
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhay.aggrawal Abhay Aggrawal
            abhay.aggrawal Abhay Aggrawal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty