Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59486

Ingestion is hung at 6191999 while ingesting 10M items from mongo onPrem

Details

    Description

      Ingestion through kafa link from an onPrem mongo deployment is hung at 6191999 items.

      Create link:

      CREATE LINK Default.onPremMongo TYPE KAFKA WITH
      { "sourceDetails": {
       "source": "MONGODB",
       "connectionFields":
        { "connectionUri": "mongodb://Administrator1:password@mongo.cbqeoc.com:27017/" }
      }};
      

      Create Collection:

      CREATE COLLECTION `volCollection` PRIMARY KEY (`_id`: string) ON myMongo.volCollection AT onPremMongo;
      

      Connection Link:

      CONNECT LINK onPremMongo;

      Wait for ingestion and keep checking for items count:

      select count(*) from volCollection;
      

      Results in 6191999 items on checking after 3 days.

      Link state:

      ritesh.agarwal@MBP-2 create_user % curl -u couchbase-cloud-admin:'HAjanAAxeoaJ2F0r#euW1QeV' -X GET https://r-stmgmpftyc.instance-i8fs3nnga.us-east-1.aws.omnistrate.cloud:18001/analytics/link/Default%2FDefault/onPremMongo | jq
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   710  100   710    0     0   2053      0 --:--:-- --:--:-- --:--:--  2094
      [
        {
          "associations": {
            "Default.Default.volCollection": "myMongo.volCollection"
          },
          "database": "Default",
          "kafkaClusterDetails": "<redacted sensitive entry>",
          "linkState": "CONNECTED",
          "name": "onPremMongo",
          "properties": {
            "SINK-IDENTIFIER": "arn:aws:kafkaconnect:us-east-2:768250113943:connector/couchbase-analytics-47b4c762-5ea6-4565-bfa6-0e7855425eb0/d8aa1eef-b02a-4a57-b372-2cf531a33f76-4",
            "SOURCE-IDENTIFIER": "arn:aws:kafkaconnect:us-east-2:768250113943:connector/mongodb-47b4c762-5ea6-4565-bfa6-0e7855425eb0/90659982-5f16-48fc-859a-f7710c8c397b-4",
            "connectionId": "47b4c762-5ea6-4565-bfa6-0e7855425eb0"
          },
          "scope": "Default",
          "sourceDetails": {
            "connectionFields": "<redacted sensitive entry>",
            "source": "MONGODB"
          },
          "type": "kafka"
        }
      ]
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritik.raj Ritik Raj
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty