Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8153

[Doc'd] cbworkloadgen shows error import sqlite3 module

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None
    • Environment:
      centos 5.7 64 bit
    • Sprint:
      PCI Team - Sprint 6

      Description

      Install couchbase server 2.0.2-773
      cbworkloadge does not work

      Error:

      root@cen-0408 thuan]# /opt/couchbase/bin/cbworkloadgen -h
      Error: could not import sqlite3 module
      [root@cen-0408 thuan]# cd /opt/couchbase/bin
      [root@cen-0408 bin]# ./cbworkloadgen -h
      Error: could not import sqlite3 module

      [root@cen-0408 bin]# ./cbworkloadgen
      Error: could not import sqlite3 module

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        thuan Thuan Nguyen created issue -
        maria Maria McDuff (Inactive) made changes -
        Field Original Value New Value
        Assignee Bin Cui [ bcui ] Anil Kumar [ anil ]
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Interesting:

        1. cat /etc/redhat-release
          CentOS release 5.8 (Final)
        1. cat /opt/couchbase/VERSION.txt
          2.0.2-774-rel
        1. /opt/couchbase/bin/cbworkloadgen -h
          Usage: cbworkloadgen [options]

        Generate workload to destination.

        Examples:
        cbworkloadgen -n localhost:8091
        cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \
        -s 100 -b my-other-bucket --threads=10

        Options:
        -h, --help show this help message and exit
        -r .95, --ratio-sets=.95
        set/get operation ratio
        -n 127.0.0.1:8091, --node=127.0.0.1:8091
        node's ns_server ip:port
        -b default, --bucket=default
        insert data to a different bucket other than default
        -i 10000, --max-items=10000
        number of items to be inserted
        -s 10, --size=10 minimum value size
        --prefix=pymc prefix to use for memcached keys or json ids
        -j, --json insert json data
        -l, --loop loop forever until interrupted by users
        -u USERNAME, --username=USERNAME
        REST username for cluster or server node
        -p PASSWORD, --password=PASSWORD
        REST password for cluster or server node
        -t 1, --threads=1 number of concurrent workers
        -v, --verbose verbose logging; more -v's provide more verbosity

        Show
        pavelpaulau Pavel Paulau added a comment - Interesting: cat /etc/redhat-release CentOS release 5.8 (Final) cat /opt/couchbase/VERSION.txt 2.0.2-774-rel /opt/couchbase/bin/cbworkloadgen -h Usage: cbworkloadgen [options] Generate workload to destination. Examples: cbworkloadgen -n localhost:8091 cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \ -s 100 -b my-other-bucket --threads=10 Options: -h, --help show this help message and exit -r .95, --ratio-sets=.95 set/get operation ratio -n 127.0.0.1:8091, --node=127.0.0.1:8091 node's ns_server ip:port -b default, --bucket=default insert data to a different bucket other than default -i 10000, --max-items=10000 number of items to be inserted -s 10, --size=10 minimum value size --prefix=pymc prefix to use for memcached keys or json ids -j, --json insert json data -l, --loop loop forever until interrupted by users -u USERNAME, --username=USERNAME REST username for cluster or server node -p PASSWORD, --password=PASSWORD REST password for cluster or server node -t 1, --threads=1 number of concurrent workers -v, --verbose verbose logging; more -v's provide more verbosity
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Thuan, could you provide output of:

        1. ls -l /opt/couchbase/lib/python/
        Show
        pavelpaulau Pavel Paulau added a comment - Thuan, could you provide output of: ls -l /opt/couchbase/lib/python/
        anil Anil Kumar made changes -
        Assignee Anil Kumar [ anil ] Bin Cui [ bcui ]
        maria Maria McDuff (Inactive) made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        thuan Thuan Nguyen added a comment -

        this vm is available at 10.1.3.140 using key to login
        python version 2.7

        Show
        thuan Thuan Nguyen added a comment - this vm is available at 10.1.3.140 using key to login python version 2.7
        Hide
        bcui Bin Cui added a comment -

        Looks like we cannot load sqlite again for this python version.

        Show
        bcui Bin Cui added a comment - Looks like we cannot load sqlite again for this python version.
        Hide
        bcui Bin Cui added a comment -

        -bash-3.2$ python
        Python 2.7 (r27:82500, Jul 29 2012, 09:49:59)
        [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import sqlite3
        Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/opt/python27/lib/python2.7/sqlite3/_init_.py", line 24, in <module>
        from dbapi2 import *
        File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
        from _sqlite3 import *
        ImportError: No module named _sqlite3

        -bash-3.2$ cd /opt/couchbase/lib/python
        -bash-3.2$ python
        Python 2.7 (r27:82500, Jul 29 2012, 09:49:59)
        [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> from pysqlite2 import dbapi2 as sqlite3
        Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "pysqlite2/dbapi2.py", line 27, in <module>
        from pysqlite2._sqlite import *
        ImportError: pysqlite2/_sqlite.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
        >>>

        Show
        bcui Bin Cui added a comment - -bash-3.2$ python Python 2.7 (r27:82500, Jul 29 2012, 09:49:59) [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sqlite3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/python27/lib/python2.7/sqlite3/_ init _.py", line 24, in <module> from dbapi2 import * File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module> from _sqlite3 import * ImportError: No module named _sqlite3 -bash-3.2$ cd /opt/couchbase/lib/python -bash-3.2$ python Python 2.7 (r27:82500, Jul 29 2012, 09:49:59) [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pysqlite2 import dbapi2 as sqlite3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pysqlite2/dbapi2.py", line 27, in <module> from pysqlite2._sqlite import * ImportError: pysqlite2/_sqlite.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8 >>>
        Hide
        bcui Bin Cui added a comment -

        Quote:
        This is usually caused by a mismatch in the Unicode mode of the python
        interpreter and the extension module.

        Python can be built to use either 2-byte or 4-byte Unicode code points.
        If you build an extension on a Python interpreter that uses one, but use
        it on another, this error is the most common result.

        Show
        bcui Bin Cui added a comment - Quote: This is usually caused by a mismatch in the Unicode mode of the python interpreter and the extension module. Python can be built to use either 2-byte or 4-byte Unicode code points. If you build an extension on a Python interpreter that uses one, but use it on another, this error is the most common result.
        pavelpaulau Pavel Paulau made changes -
        Assignee Bin Cui [ bcui ] Pavel Paulau [ pavelpaulau ]
        pavelpaulau Pavel Paulau made changes -
        Sprint PCI Team - Sprint 6 [ 11 ]
        pavelpaulau Pavel Paulau made changes -
        Rank Ranked higher
        pavelpaulau Pavel Paulau made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Per Bin, this is a release blocker... he's working with Pavel on this issue.

        Show
        maria Maria McDuff (Inactive) added a comment - Per Bin, this is a release blocker... he's working with Pavel on this issue.
        maria Maria McDuff (Inactive) made changes -
        Priority Critical [ 2 ] Blocker [ 1 ]
        Hide
        pavelpaulau Pavel Paulau added a comment - - edited

        Thuan, just a quick question. How did you install Python 2.7 on this machine? Was it installed after Couchbase Server?

        Show
        pavelpaulau Pavel Paulau added a comment - - edited Thuan, just a quick question. How did you install Python 2.7 on this machine? Was it installed after Couchbase Server?
        Show
        pavelpaulau Pavel Paulau added a comment - http://review.couchbase.org/#/c/25938/
        Hide
        thuan Thuan Nguyen added a comment -

        On this vm, I just install couchbase server only. So I think python 2.7 is pre-installed before

        Show
        thuan Thuan Nguyen added a comment - On this vm, I just install couchbase server only. So I think python 2.7 is pre-installed before
        pavelpaulau Pavel Paulau made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        maria Maria McDuff (Inactive) made changes -
        Assignee Pavel Paulau [ pavelpaulau ] Thuan Nguyen [ thuan ]
        Hide
        maria Maria McDuff (Inactive) added a comment -

        pls verify / close.

        Show
        maria Maria McDuff (Inactive) added a comment - pls verify / close.
        Hide
        thuan Thuan Nguyen added a comment -

        I still repro this bug in build 2.0.2-793 on centos 5.7 64 bit server 10.1.3.140 (same vm)

        couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm
        [root@cen-0408 thuan]# rpm -i couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm
        Minimum RAM required : 4 GB
        System RAM configured : 3945388 kB

        Minimum number of processors required : 4 cores
        Number of processors on the system : 4 cores

        Starting couchbase-server[ OK ]

        You have successfully installed Couchbase Server.
        Please browse to http://cen-0408:8091/ to configure your server.
        Please refer to http://couchbase.com for additional resources.

        Please note that you have to update your firewall configuration to
        allow connections to the following ports: 11211, 11210, 11209, 4369,
        8091, 8092 and from 21100 to 21299.

        By using this software you agree to the End User License Agreement.
        See /opt/couchbase/LICENSE.txt.

        [root@cen-0408 thuan]# /opt/couchbase/bin/cbworkloadgen -h
        Error: could not import sqlite3 module
        [[root@cen-0408 thuan]# /opt/couchbase/bin/sqlite3
        SQLite version 3.7.2
        Enter ".help" for instructions
        Enter SQL statements terminated with a ";"
        sqlite>
        [1]+ Stopped /opt/couchbase/bin/sqlite3
        [root@cen-0408 thuan]# cd /opt/couchbase/bin/
        [root@cen-0408 bin]# ./cbworkloadgen -h
        Error: could not import sqlite3 module
        [root@cen-0408 bin]# ./sqlite3
        SQLite version 3.7.2
        Enter ".help" for instructions
        Enter SQL statements terminated with a ";"
        sqlite>
        [2]+ Stopped ./sqlite3
        [root@cen-0408 bin]# python -V
        Python 2.7

        Show
        thuan Thuan Nguyen added a comment - I still repro this bug in build 2.0.2-793 on centos 5.7 64 bit server 10.1.3.140 (same vm) couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm [root@cen-0408 thuan] # rpm -i couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm Minimum RAM required : 4 GB System RAM configured : 3945388 kB Minimum number of processors required : 4 cores Number of processors on the system : 4 cores Starting couchbase-server[ OK ] You have successfully installed Couchbase Server. Please browse to http://cen-0408:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091, 8092 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. [root@cen-0408 thuan] # /opt/couchbase/bin/cbworkloadgen -h Error: could not import sqlite3 module [ [root@cen-0408 thuan] # /opt/couchbase/bin/sqlite3 SQLite version 3.7.2 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> [1] + Stopped /opt/couchbase/bin/sqlite3 [root@cen-0408 thuan] # cd /opt/couchbase/bin/ [root@cen-0408 bin] # ./cbworkloadgen -h Error: could not import sqlite3 module [root@cen-0408 bin] # ./sqlite3 SQLite version 3.7.2 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> [2] + Stopped ./sqlite3 [root@cen-0408 bin] # python -V Python 2.7
        thuan Thuan Nguyen made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        thuan Thuan Nguyen made changes -
        Assignee Thuan Nguyen [ thuan ] Maria McDuff [ maria ]
        thuan Thuan Nguyen made changes -
        Assignee Maria McDuff [ maria ] Bin Cui [ bcui ]
        Hide
        pavelpaulau Pavel Paulau added a comment - - edited

        Normally Python 2.7 includes sqlite3, however you have custom installation that was compiled without sqlite support. So it obviously fails.

        Expected:
        $ python2.7 -c "import sqlite3"
        $ echo $?
        0

        Your machine:
        $ python2.7 -c "import sqlite3"
        Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/opt/python27/lib/python2.7/sqlite3/_init_.py", line 24, in <module>
        from dbapi2 import *
        File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
        from _sqlite3 import *
        ImportError: No module named _sqlite3

        Normal recommendation in such cases is to install sqlite-devel and rebuild Python. Or to use default OS setup.

        Addressing such edge cases is too expensive effort IMHO. This is my input, PMs may have other suggestions.

        Show
        pavelpaulau Pavel Paulau added a comment - - edited Normally Python 2.7 includes sqlite3, however you have custom installation that was compiled without sqlite support. So it obviously fails. Expected: $ python2.7 -c "import sqlite3" $ echo $? 0 Your machine: $ python2.7 -c "import sqlite3" Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/python27/lib/python2.7/sqlite3/_ init _.py", line 24, in <module> from dbapi2 import * File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module> from _sqlite3 import * ImportError: No module named _sqlite3 Normal recommendation in such cases is to install sqlite-devel and rebuild Python. Or to use default OS setup. Addressing such edge cases is too expensive effort IMHO. This is my input, PMs may have other suggestions.
        anil Anil Kumar made changes -
        Assignee Bin Cui [ bcui ] Anil Kumar [ anil ]
        Hide
        bcui Bin Cui added a comment -

        The current assumption is that we support the following python environments:

        1. python 2.4 which doesn't have sqlite3 bundled. we will install our bundled version.
        2. python 2.5 and above. python will have its own version of sqlite3 installed. But if it doesn't meet our sqlite3 version requirement, we will install our bundled version.
        3. This QA setup is something that we never meet before and it is not a standard environment, to say the least.

        Show
        bcui Bin Cui added a comment - The current assumption is that we support the following python environments: 1. python 2.4 which doesn't have sqlite3 bundled. we will install our bundled version. 2. python 2.5 and above. python will have its own version of sqlite3 installed. But if it doesn't meet our sqlite3 version requirement, we will install our bundled version. 3. This QA setup is something that we never meet before and it is not a standard environment, to say the least.
        Hide
        anil Anil Kumar added a comment -

        tony to test this on clean VM to verify if this repro.

        1. installing python 2.7 on clean vm
        2. check if it comes with sqllite3 already

        Show
        anil Anil Kumar added a comment - tony to test this on clean VM to verify if this repro. 1. installing python 2.7 on clean vm 2. check if it comes with sqllite3 already
        anil Anil Kumar made changes -
        Assignee Anil Kumar [ anil ] Thuan Nguyen [ thuan ]
        anil Anil Kumar made changes -
        Priority Blocker [ 1 ] Critical [ 2 ]
        Hide
        pavelpaulau Pavel Paulau added a comment -

        0. Package "sqlite-devel" must be installed before Python 2.7 installation.

        Show
        pavelpaulau Pavel Paulau added a comment - 0. Package "sqlite-devel" must be installed before Python 2.7 installation.
        maria Maria McDuff (Inactive) made changes -
        Summary cbworkloadgen shows error import sqlite3 module [RN 2.0.2] cbworkloadgen shows error import sqlite3 module
        maria Maria McDuff (Inactive) made changes -
        Component/s documentation [ 10012 ]
        Hide
        maria Maria McDuff (Inactive) added a comment -

        karen, pls doc sqlite-devel need to be installed first prior to python 2.7. thanks.

        tony, pls verify / close. thanks.

        Show
        maria Maria McDuff (Inactive) added a comment - karen, pls doc sqlite-devel need to be installed first prior to python 2.7. thanks. tony, pls verify / close. thanks.
        kzeller kzeller made changes -
        Rank Ranked higher
        kzeller kzeller made changes -
        Rank Ranked lower
        kzeller kzeller made changes -
        Rank Ranked higher
        Hide
        kzeller kzeller added a comment -

        Added to RN 2.0.2:

        <rnentry>

        <version ver="2.0.0m"/>

        <class id="fix"/>

        <issue type="cb" ref="MB-8153"/>

        <rntext>

        <para>
        In the past when you used <command>cbworkloadgen</command> you see this error <literal>ImportError: No module named _sqlite3</literal>.
        This has been fixed.</para>

        </rntext>

        </rnentry>

        Show
        kzeller kzeller added a comment - Added to RN 2.0.2: <rnentry> <version ver="2.0.0m"/> <class id="fix"/> <issue type="cb" ref=" MB-8153 "/> <rntext> <para> In the past when you used <command>cbworkloadgen</command> you see this error <literal>ImportError: No module named _sqlite3</literal>. This has been fixed.</para> </rntext> </rnentry>
        kzeller kzeller made changes -
        Summary [RN 2.0.2] cbworkloadgen shows error import sqlite3 module [Doc'd] cbworkloadgen shows error import sqlite3 module
        Hide
        thuan Thuan Nguyen added a comment -

        Test on build 2.0.2-804 with 3 vms with python 2.4.3, 2.6.5 and 2.7.1
        cbworkloadgen works as expected

        /opt/couchbase/bin/cbworkloadgen -h
        Usage: cbworkloadgen [options]

        Generate workload to destination.

        Examples:
        cbworkloadgen -n localhost:8091
        cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \
        -s 100 -b my-other-bucket --threads=10

        Options:
        -h, --help show this help message and exit
        -r .95, --ratio-sets=.95
        set/get operation ratio
        -n 127.0.0.1:8091, --node=127.0.0.1:8091
        node's ns_server ip:port
        -b default, --bucket=default
        insert data to a different bucket other than default
        -i 10000, --max-items=10000
        number of items to be inserted
        -s 10, --size=10 minimum value size
        --prefix=pymc prefix to use for memcached keys or json ids
        -j, --json insert json data
        -l, --loop loop forever until interrupted by users
        -u USERNAME, --username=USERNAME
        REST username for cluster or server node
        -p PASSWORD, --password=PASSWORD
        REST password for cluster or server node
        -t 1, --threads=1 number of concurrent workers
        -v, --verbose verbose logging; more -v's provide more verbosity

        Show
        thuan Thuan Nguyen added a comment - Test on build 2.0.2-804 with 3 vms with python 2.4.3, 2.6.5 and 2.7.1 cbworkloadgen works as expected /opt/couchbase/bin/cbworkloadgen -h Usage: cbworkloadgen [options] Generate workload to destination. Examples: cbworkloadgen -n localhost:8091 cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \ -s 100 -b my-other-bucket --threads=10 Options: -h, --help show this help message and exit -r .95, --ratio-sets=.95 set/get operation ratio -n 127.0.0.1:8091, --node=127.0.0.1:8091 node's ns_server ip:port -b default, --bucket=default insert data to a different bucket other than default -i 10000, --max-items=10000 number of items to be inserted -s 10, --size=10 minimum value size --prefix=pymc prefix to use for memcached keys or json ids -j, --json insert json data -l, --loop loop forever until interrupted by users -u USERNAME, --username=USERNAME REST username for cluster or server node -p PASSWORD, --password=PASSWORD REST password for cluster or server node -t 1, --threads=1 number of concurrent workers -v, --verbose verbose logging; more -v's provide more verbosity
        thuan Thuan Nguyen made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        thuan Thuan Nguyen made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Sprint Status Current Sprint [ 10027 ]
        wayne Wayne Siu made changes -
        Component/s documentation-don't-use-put-in-doc-project [ 10012 ]

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes