Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-3682

[3.1.4 backport] sgcollect can't upload a zip more than 2GB



    • Improvement
    • Resolution: Fixed
    • Major
    • 3.1.4
    • 3.1.2
    • None
    • Security Level: Public
    • None
    • CBG Sprint 145
    • 1


      This can cause an error like:

      Zipfile built: /tmp/sync-gateway-747784f75c-2mckq-2023-11-02T14:33:52+01:00.zip
      Traceback (most recent call last):
        File "tasks.py", line 1090, in do_upload
        File "urllib/request.py", line 517, in open
        File "urllib/request.py", line 534, in _open
        File "urllib/request.py", line 494, in _call_chain
        File "urllib/request.py", line 1389, in https_open
        File "urllib/request.py", line 1346, in do_open
        File "http/client.py", line 1285, in request
        File "http/client.py", line 1331, in _send_request
        File "http/client.py", line 1280, in endheaders
        File "http/client.py", line 1079, in _send_output
        File "http/client.py", line 1001, in send
        File "ssl.py", line 1205, in sendall
        File "ssl.py", line 1174, in send
      OverflowError: string longer than 2147483647 bytes
      Zipfile deleted: /tmp/sync-gateway-747784f75c-2mckq-2023-11-02T14:33:52+01:00.zip

      This code was introduced in https://github.com/couchbase/sync_gateway/pull/1991 / https://github.com/couchbase/sync_gateway/pull/1990

      At this time, python3 APIs are available and we shouldn't mmap a file, but pass the open file object directly directly to urllib.request.Request without reading the entire file.

      Curiosly, cbcollect_info does still use curl, presumably a version of curl is bundled on windows in the couchbase server distribution? https://github.com/couchbase/ns_server/blob/1e82181a201f58995638769c3a3fc54751bc1fea/cbcollect_info#L2342

      It isn't correct to say that it loads the entire file into memory, it mmaps a the size of the on disk file into virtual memory and a page fault should add and evict the data. I hypothesize there isn't enough memory on a machine to do this or the eviction doesn't happen fast enough. Regardless, it's easy enough to change in python3.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              gregory.newmansmith Gregory Newman-Smith
              adamf Adam Fraser
              0 Vote for this issue
              1 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes