Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48146

[BP 7.0.2 MB-48110] - [IPV6] no index available when switching to ipv6 or ipv6only IP family

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      To repro (require node with ipv4/ipv6 support)

      1. setup cluster with kv,index and n1ql (use hostname), use default ipv4 ip family
      2. disable autofailover
      3. load travel-sample
      4. run simple query: select * from `travel-sample` limit 10;
      5. switch to ipv6 family: /opt/couchbase/bin/couchbase-cli ip-family -c localhost -u Administrator -p password --set --ipv6
      1. run query again

      cbq> select * from `travel-sample` limit 10;
      {
          "requestID": "a98244a2-b0e7-4865-95d4-4371a8c50648",
          "errors": [
              {
                  "code": 4000,
                  "msg": "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online."
              }
          ],
          "status": "fatal", 

      Note indexer process:

      [root@cluster logs]# /opt/couchbase/bin/couchbase-cli ip-family -c localhost -u Administrator -p password --set --ipv4
      Switched IP family for node: http://cluster.couchbase.com:8091
      SUCCESS: Switched IP family of the cluster
       
      [root@cluster logs]# lsof -i -P -n  | grep LISTEN | grep couchbase| grep -i index
      indexer   10900 couchbase   15u  IPv4 185082073      0t0  TCP *:9101 (LISTEN)
      indexer   10900 couchbase   22u  IPv4 185078777      0t0  TCP *:9100 (LISTEN)
      indexer   10900 couchbase   24u  IPv4 185078780      0t0  TCP *:9102 (LISTEN)
      indexer   10900 couchbase   25u  IPv4 185078781      0t0  TCP *:19102 (LISTEN)
      indexer   10900 couchbase  124u  IPv4 185082939      0t0  TCP *:9105 (LISTEN)
       
      [root@cluster logs]# /opt/couchbase/bin/couchbase-cli ip-family -c localhost -u Administrator -p password --set --ipv6
      Switched IP family for node: http://cluster.couchbase.com:8091
      SUCCESS: Switched IP family of the cluster
       
      [root@cluster logs]# lsof -i -P -n  | grep LISTEN | grep couchbase| grep -i index
      indexer   11354 couchbase   15u  IPv6 185084236      0t0  TCP *:9101 (LISTEN)
      indexer   11354 couchbase   22u  IPv6 185084263      0t0  TCP *:9100 (LISTEN)
      indexer   11354 couchbase   24u  IPv6 185084935      0t0  TCP *:9102 (LISTEN)
      indexer   11354 couchbase   25u  IPv6 185084936      0t0  TCP *:19102 (LISTEN) 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jeelan.poola Jeelan Poola created issue -
            jeelan.poola Jeelan Poola made changes -
            Field Original Value New Value
            Link This issue Clones MB-48110 [ MB-48110 ]
            jeelan.poola Jeelan Poola made changes -
            Fix Version/s Neo [ 17615 ]
            jeelan.poola Jeelan Poola made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]
            jeelan.poola Jeelan Poola made changes -
            Labels ipv6 approved-for-7.0.2 ipv6

            Marking this as a test blocker as it blocks the testing for N1QL/GSI support for Enforce IPv4-only/IPv6-only feature

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Marking this as a test blocker as it blocks the testing for N1QL/GSI support for Enforce IPv4-only/IPv6-only feature
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Priority Critical [ 2 ] Test Blocker [ 6 ]

            Mihir Kamdar,  The fix is in cheshire-cat branch of gometa. Please use the latest build. Resolving this issue as done.

            varun.velamuri Varun Velamuri added a comment - Mihir Kamdar ,  The fix is in cheshire-cat branch of gometa. Please use the latest build. Resolving this issue as done.
            varun.velamuri Varun Velamuri made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Build couchbase-server-7.1.0-1202 contains gometa commit eb82f1e with commit message:
            MB-48146 Do not resolve TCP address when creating connection

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1202 contains gometa commit eb82f1e with commit message: MB-48146 Do not resolve TCP address when creating connection

            Build couchbase-server-7.0.2-6591 contains gometa commit 08bf2cd with commit message:
            MB-48146 Do not resolve TCP address when creating connection

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6591 contains gometa commit 08bf2cd with commit message: MB-48146 Do not resolve TCP address when creating connection

            This is still an issue on 7.0.2-6591

            # cat /opt/couchbase/VERSION.txt 
            7.0.2-6591
             
            #  /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com:8091 -u Administrator -p password --get
            Cluster using ipv6
             
            # /opt/couchbase/bin/cbq -u Administrator -p password
             Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit.
             
             
             Path to history file for the shell : /root/.cbq_history 
            cbq> select * from `travel-sample` limit 10;
            {
                "requestID": "c4a5fa18-98d0-4d1c-9cd7-c9d0df806973",
                "errors": [
                    {
                        "code": 4000,
                        "msg": "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online."
                    }
                ],
                "status": "fatal",
                "metrics": {
                    "elapsedTime": "2.267583366s",
                    "executionTime": "2.267338645s",
                    "resultCount": 0,
                    "resultSize": 0,
                    "serviceLoad": 6,
                    "errorCount": 1
                }
            }
            cbq>  

            pierre.regazzoni Pierre Regazzoni added a comment - This is still an issue on 7.0.2-6591 # cat /opt/couchbase/VERSION.txt  7.0 . 2 - 6591   #  /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com: 8091 -u Administrator -p password --get Cluster using ipv6   # /opt/couchbase/bin/cbq -u Administrator -p password  Connected to : http: //localhost:8091/. Type Ctrl-D or \QUIT to exit.      Path to history file for the shell : /root/.cbq_history  cbq> select * from `travel-sample` limit 10 ; {     "requestID" : "c4a5fa18-98d0-4d1c-9cd7-c9d0df806973" ,     "errors" : [         {             "code" : 4000 ,             "msg" : "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online."         }     ],     "status" : "fatal" ,     "metrics" : {         "elapsedTime" : "2.267583366s" ,         "executionTime" : "2.267338645s" ,         "resultCount" : 0 ,         "resultSize" : 0 ,         "serviceLoad" : 6 ,         "errorCount" : 1     } } cbq> 
            pierre.regazzoni Pierre Regazzoni made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]

            Pierre Regazzoni, I think it is because cbq-engine is using github.com/couchbase/gometa v0.0.0-20200717102231-b0e38b71d711 version of gometa in their go.mod file. So, the fix many not take effect in the latest build. I faced this issue while testing the patch but I did not expect production builds to contain this issue (I literally spent one entire day trying to figure this issue)

            Kamini Jagtiani, Is there any reason why cbq-engine uses a local gometa directory rather than point to the gometa package it self? GSI client in cbq-engine relies on gometa and we don't want it to use an older version. Can you please remove the dependency on local gometa package and use the actual gometa package.

            varun.velamuri Varun Velamuri added a comment - Pierre Regazzoni , I think it is because cbq-engine is using  github.com/couchbase/gometa v0.0.0-20200717102231-b0e38b71d711 version of gometa in their go.mod file. So, the fix many not take effect in the latest build. I faced this issue while testing the patch but I did not expect production builds to contain this issue (I literally spent one entire day trying to figure this issue) Kamini Jagtiani , Is there any reason why cbq-engine uses a local gometa directory rather than point to the gometa package it self? GSI client in cbq-engine relies on gometa and we don't want it to use an older version. Can you please remove the dependency on local gometa package and use the actual gometa package.
            varun.velamuri Varun Velamuri made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Kamini Jagtiani [ kamini.jagtiani ]
            varun.velamuri Varun Velamuri made changes -
            Component/s query [ 10718 ]
            Component/s secondary-index [ 11211 ]

            This is how go modules works. query uses go module.
            If you need to get the fixes you have two options.
            1) update go.mod/go.sum with specific version. This will maintenance issue.
            2) use replace directory on top (prefered)

            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - This is how go modules works. query uses go module. If you need to get the fixes you have two options. 1) update go.mod/go.sum with specific version. This will maintenance issue. 2) use replace directory on top (prefered)
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 23075 ]
            varun.velamuri Varun Velamuri made changes -
            Component/s secondary-index [ 11211 ]
            Component/s query [ 10718 ]
            varun.velamuri Varun Velamuri made changes -
            Assignee Kamini Jagtiani [ kamini.jagtiani ] Varun Velamuri [ varun.velamuri ]
            varun.velamuri Varun Velamuri made changes -
            Component/s query [ 10718 ]
            Component/s secondary-index [ 11211 ]

            Build couchbase-server-7.0.2-6623 contains gometa commit aaea903 with commit message:
            MB-48146 Initialise go.mod in gometa repo

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6623 contains gometa commit aaea903 with commit message: MB-48146 Initialise go.mod in gometa repo

            Build couchbase-server-7.0.2-6623 contains n1fty commit 1f5554b with commit message:
            MB-48146 Update n1fty go.mod to "replace" gometa

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6623 contains n1fty commit 1f5554b with commit message: MB-48146 Update n1fty go.mod to "replace" gometa

            Build couchbase-server-7.0.2-6623 contains query commit 07f3cad with commit message:
            MB-48146 Update query go.mod to "replace" gometa

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6623 contains query commit 07f3cad with commit message: MB-48146 Update query go.mod to "replace" gometa

            Pierre Regazzoni, Please try with 7.0.2-6623 build. 

            varun.velamuri Varun Velamuri added a comment - Pierre Regazzoni , Please try with 7.0.2-6623 build. 
            varun.velamuri Varun Velamuri made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            pierre.regazzoni Pierre Regazzoni made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Pierre Regazzoni [ JIRAUSER25157 ]

            Verified on 7.0.2-6623

             # /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com:8091 -u Administrator -p password --get
            Cluster using ipv6only
             
            # /opt/couchbase/bin/cbq -u Administrator -p password
             Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit.
             
             
             Path to history file for the shell : /root/.cbq_history 
            cbq> select airportname from `travel-sample`.inventory.airport where city = 'Lyon';
            {
                "requestID": "fefbc4bc-921c-45d6-b570-0d56d8c7e058",
                "signature": {
                    "airportname": "json"
                },
                "results": [
                {
                    "airportname": "Saint Exupery"
                },
                {
                    "airportname": "Bron"
                },
                {
                    "airportname": "Lyon Part-Dieu Railway"
                }
                ],
                "status": "success",
                "metrics": {
                    "elapsedTime": "8.129148ms",
                    "executionTime": "7.843986ms",
                    "resultCount": 3,
                    "resultSize": 138,
                    "serviceLoad": 6
                }
            }

            and

            # /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com:8091 -u Administrator -p password --get
            Cluster using ipv6
             
            # /opt/couchbase/bin/cbq -u Administrator -p password
             Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit.
             
             
             Path to history file for the shell : /root/.cbq_history 
            cbq> select airportname from `travel-sample`.inventory.airport where city = 'Lyon';
            {
                "requestID": "449ee755-7c62-4959-a361-8bfe6e96a0e6",
                "signature": {
                    "airportname": "json"
                },
                "results": [
                {
                    "airportname": "Saint Exupery"
                },
                {
                    "airportname": "Bron"
                },
                {
                    "airportname": "Lyon Part-Dieu Railway"
                }
                ],
                "status": "success",
                "metrics": {
                    "elapsedTime": "351.605003ms",
                    "executionTime": "351.204667ms",
                    "resultCount": 3,
                    "resultSize": 138,
                    "serviceLoad": 6
                }
            } 

            pierre.regazzoni Pierre Regazzoni added a comment - Verified on 7.0.2-6623 # /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com: 8091 -u Administrator -p password --get Cluster using ipv6only   # /opt/couchbase/bin/cbq -u Administrator -p password  Connected to : http: //localhost:8091/. Type Ctrl-D or \QUIT to exit.      Path to history file for the shell : /root/.cbq_history  cbq> select airportname from `travel-sample`.inventory.airport where city = 'Lyon' ; {     "requestID" : "fefbc4bc-921c-45d6-b570-0d56d8c7e058" ,     "signature" : {         "airportname" : "json"     },     "results" : [     {         "airportname" : "Saint Exupery"     },     {         "airportname" : "Bron"     },     {         "airportname" : "Lyon Part-Dieu Railway"     }     ],     "status" : "success" ,     "metrics" : {         "elapsedTime" : "8.129148ms" ,         "executionTime" : "7.843986ms" ,         "resultCount" : 3 ,         "resultSize" : 138 ,         "serviceLoad" : 6     } } and # /opt/couchbase/bin/couchbase-cli ip-family -c cluster.couchbase.com: 8091 -u Administrator -p password --get Cluster using ipv6   # /opt/couchbase/bin/cbq -u Administrator -p password  Connected to : http: //localhost:8091/. Type Ctrl-D or \QUIT to exit.      Path to history file for the shell : /root/.cbq_history  cbq> select airportname from `travel-sample`.inventory.airport where city = 'Lyon' ; {     "requestID" : "449ee755-7c62-4959-a361-8bfe6e96a0e6" ,     "signature" : {         "airportname" : "json"     },     "results" : [     {         "airportname" : "Saint Exupery"     },     {         "airportname" : "Bron"     },     {         "airportname" : "Lyon Part-Dieu Railway"     }     ],     "status" : "success" ,     "metrics" : {         "elapsedTime" : "351.605003ms" ,         "executionTime" : "351.204667ms" ,         "resultCount" : 3 ,         "resultSize" : 138 ,         "serviceLoad" : 6     } }
            pierre.regazzoni Pierre Regazzoni made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Sitaram.Vemulapalli Sitaram Vemulapalli made changes -
            Component/s secondary-index [ 11211 ]
            Sitaram.Vemulapalli Sitaram Vemulapalli made changes -
            Component/s query [ 10718 ]
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

            It is 2 parts.
            1) You changed the gometa this is indexer change to fix the issue. http://review.couchbase.org/c/gometa/+/160077
            2) Query doesn't need the gometa change. Indexer client is need the change. If that need to build in cbq-engine that component owned by secondary-index. cc Kamini Jagtiani

            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited It is 2 parts. 1) You changed the gometa this is indexer change to fix the issue. http://review.couchbase.org/c/gometa/+/160077 2) Query doesn't need the gometa change. Indexer client is need the change. If that need to build in cbq-engine that component owned by secondary-index. cc Kamini Jagtiani
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 23075 ]

            People

              pierre.regazzoni Pierre Regazzoni
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty