Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28274

Audit logs for operations on data may require a lot of disk space

    XMLWordPrintable

Details

    Description

      This is a cross-component issue that could affect both memcached and N1QL events.

      Imagine a query node that performs 1K queries/sec on average. Each query creates a record like this:

      {
      		 
        "timestamp": "2018-02-20T14:56:38.816-08:00",
      		 
        "real_userid": {
      		 
          "source": "local",
      		 
          "user": "bucket-1"
      		 
        },
      		 
        "requestId": "ac18bce0-faca-4f96-bdd4-ecad3cfc895c",
      		 
        "statement": "",
      		 
        "positionalArgs": [
      		 
          "\"n1ql-000007815562\""
      		 
        ],
      		 
        "isAdHoc": false,
      		 
        "userAgent": "libcouchbase/2.8.4 (Linux-4.14.6-300.fc27.x86_64; x86_64; GNU 5.3.1) PYCBC/2.2.6",
      		 
        "node": "172.23.100.55:8091",
      		 
        "status": "success",
      		 
        "metrics": {
      		 
          "elapsedTime": "1.68499ms",
      		 
          "executionTime": "1.643921ms",
      		 
          "resultCount": 1,
      		 
          "resultSize": 988
      		 
        },
      		 
        "id": 28672,
      		 
        "name": "SELECT statement",
      		 
        "description": "A N1QL SELECT statement was executed"
      		 
      }

      That is about 550 bytes of information. The complete audit log will require about 44GB of disk space per day or almost 16TB of disk space per year. As far as I can see, audit logs never get compressed nor they ever expire.

      Of course, only a few users/customers will want to audit all read and update operations. I hope this ticket set the expectations.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. MB-41352 Audit log lifecycle management

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-28300 Only important admin events should be audited by default

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-28276 Auditing requires up to one CPU core per 20K events/sec

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. DOC-3534 Add info to documentation to give some guidance and expectation for using the auditing feature

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-28274
          # Subject Branch Project Status CR V
          89775,2 Do not enable auditing by default master perfrunner Status: ABANDONED 0 +1
          90366,2 MB-28274 add audit log file size warning master ns_server Status: MERGED +2 +1

          Activity

            People

              ianmccloy Ian McCloy (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty