Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Duplicate
Priority: Critical
Fix Version/s: Columnar 1.0.0
Affects Version/s: Columnar 1.0.0
Component/s: analytics
Labels:
- system-test
Environment:
1.0.0-2203

Triage:
Untriaged
Story Points:
0
Is this a Regression?:
Unknown

Description

The workload at the time of OOM is as follows -

Type	Number of collections	Number of items in millions	Total count in millions
Remote	160	75	12000
Standalone	100	8	6400*
Kafka	5	10	50

*Some standalone collections have 8 mil and some have multiples of 8 million items. The total doc count is 6400 million ( 6.4 billion) items.
Number of links = 6 ( 2 remote + 2 external + 2 kafka). 1 remote link and 1 kafka link is active.

It went through scaling operations. From 4 to 8 to 16 to 32 back to 8 to 4 nodes.

During the second cycle copy from s3 phase of loading data, the test has run into OOM error -

2024-07-11T17:42:00.993+00:00 INFO CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:5.3182:TAID:TID:ANID:ODID:322:0:34:0:(linkIUWEhdXs/default1)[34]:BO] controller is being set from CREATED to STARTED

2024-07-11T17:42:00.993+00:00 INFO CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:5.3182:TAID:TID:ANID:ODID:322:0:37:0:(linkIUWEhdXs/default1)[37]:BO] controller is being set from CREATED to STARTED

2024-07-11T17:42:00.993+00:00 INFO CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:5.3182:TAID:TID:ANID:ODID:322:0:45:0:(linkIUWEhdXs/default1)[45]:BO] controller is being set from CREATED to STARTED

2024-07-11T17:42:00.994+00:00 INFO CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:5.3182:TAID:TID:ANID:ODID:322:0:47:0:(linkIUWEhdXs/default1)[47]:BO] controller is being set from CREATED to STARTED

Terminating due to java.lang.OutOfMemoryError: Java heap space

2024-07-11T17:43:01.655+00:00 WARN CBAS.cbas analytics driver has exited w/ exit status 3

2024-07-11T17:43:02.314+00:00 INFO CBAS.cbas analytics driver exited; restarting

2024-07-11T17:43:02.320+00:00 INFO CBAS.cbas setting max number of CPUs to: 16

2024-07-11T17:43:02.320+00:00 INFO CBAS.cbas nodeConfigMap: map[additionalCcNodes:[] address:0.0.0.0 analyticsCbHome:/opt/couchbase analyticsCcHttpPort:9111 analyticsHttpAdminListenPort:9110 analyticsHttpListenAddress:[127.0.0.1 10.1.4.17 10.1.4.104] analyticsHttpListenPort:8095 analyticsHttpsListenAddress:[127.0.0.1 10.1.4.17 10.1.4.104] analyticsHttpsListenPort:18095 analyticsNodeName:svc-da-node-016.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com:8091 analyticsSslEnabled:true cbasAddress:127.0.0.1 cbasPort:9122 ccAddress:0.0.0.0 ccAnalyticsHttpListenPort:9111 ccAnalyticsMetadataPartitionId:-1 ccClientListenPort:9113 ccClusterListenPort:9112 ccControllerId:15 ccHeartbeatMaxMisses:20 ccHeartbeatPeriod:30000 ccPublicAddress:svc-da-node-016.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com clusterAddress:svc-da-node-006.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com clusterCompatibility:458758 clusterListenPort:9115 clusterPort:9112 dataListenPort:9116 defaultDir:/var/cb-cache/@analytics/v_iodevice_0 developerPreview:false initialRun:false iodevices:[/var/cb-cache/@analytics/v_iodevice_0 /var/cb-cache/@analytics/v_iodevice_1 /var/cb-cache/@analytics/v_iodevice_2 /var/cb-cache/@analytics/v_iodevice_3 /var/cb-cache/@analytics/v_iodevice_4 /var/cb-cache/@analytics/v_iodevice_5 /var/cb-cache/@analytics/v_iodevice_6 /var/cb-cache/@analytics/v_iodevice_7 /var/cb-cache/@analytics/v_iodevice_8 /var/cb-cache/@analytics/v_iodevice_9 /var/cb-cache/@analytics/v_iodevice_10 /var/cb-cache/@analytics/v_iodevice_11 /var/cb-cache/@analytics/v_iodevice_12 /var/cb-cache/@analytics/v_iodevice_13 /var/cb-cache/@analytics/v_iodevice_14 /var/cb-cache/@analytics/v_iodevice_15] logDir:/opt/couchbase/var/lib/couchbase/logs messagingListenPort:9118 metadataNode:f8bbb0059527fb8c59160733f2baae59 ncservicePort:-1 nsServerAddress:127.0.0.1 nsServerPort:8091 nsServerSslPort:18091 numReplicas:0 productVersion:1.0.0-2203 publicAddress:svc-da-node-016.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com replicationEnabled:false replicationFactor:4096 replicationListenPort:9120 replicationLogBufferPagesize:262144 replicationStrategy:all resultListenPort:9117 sslEnabled:true startingPartitionId:240 txnLogBufferPagesize:262144]

2024-07-11T17:43:02.324+00:00 INFO CBAS.cbas running /opt/couchbase/lib/columnar/runtime/bin/java args:[-XX:+ExitOnOutOfMemoryError -XX:MaxGCPauseMillis=60000 -XX:+UseParallelGC -XX:ActiveProcessorCount=16 -Xms18846m -Xmx37692m -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/couchbase/var/lib/couchbase/tmp -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -Dproduct.version=1.0.0-2203 -DlogRolloverThresholdBytes=41943040 -DlogRolloverMaxFiles=10 -Dnetworkaddress.cache.ttl=15 -Dnetworkaddress.cache.negative.ttl=10 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dcom.couchbase.analytics.bootstrapLogLevel=DEBUG -DlogDir=/opt/couchbase/var/lib/couchbase/logs -cp /opt/couchbase/lib/columnar/repo/columnar-install-1.0.0.jar com.couchbase.columnar.ColumnarDriver --jsonconfig /var/cb-cache/@analytics/v_iodevice_0/config.json --nc -node-id 13a9dd40a55e2ff4810c0ac726d94185]

2024-07-11T17:43:02.325+00:00 INFO CBAS.cbas analytics driver is running; pid: 29273

2024-07-11T17:43:04.493+00:00 WARN CBAS.util.PreferSystemPropertiesPropertySource [main] System properties is the preferred source of log4j properties

Info for QE ->

Corresponding test logs ->

2024-07-11T17:41:29.895Z	debug	api call success	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 600, "status": 200, "requestId": "e7f35721-d85a-4264-a100-e65060cba25a"}

2024-07-11T17:41:29.895Z	info	random remote link chosen  tsvExternalLinkemgMmJmxKP	{"env": "other", "action": "Starting Copy from S3", "InstanceID": "a5f72537-6565-4271-a901-3ce41026a8fc", "ProjectID": "fc226955-6229-41ea-9300-7eada8cbdab9", "TypeDef": [], "SourceBucket": "columnar-functional-sanity-test-data", "ExternalLinkRegion": "us-east-1", "ExternalLinkType": "S3", "Timeout": 10800, "Count": 100}

2024-07-11T17:41:29.895Z	info	Starting Copy from S3 to collection Database16DDWvvtAd.scope1NBIAbpGN.standalonehnieaVHJ	{"env": "other", "action": "Starting Copy from S3", "InstanceID": "a5f72537-6565-4271-a901-3ce41026a8fc", "ProjectID": "fc226955-6229-41ea-9300-7eada8cbdab9", "TypeDef": [], "SourceBucket": "columnar-functional-sanity-test-data", "ExternalLinkRegion": "us-east-1", "ExternalLinkType": "S3", "Timeout": 10800, "Count": 100}

Running Columnar query COPY INTO `Database16DDWvvtAd`.`scope1NBIAbpGN`.`standalonehnieaVHJ` FROM `columnar-functional-sanity-test-data` AT `tsvExternalLinkemgMmJmxKP` PATH "" WITH {"format":"avro","include":["*file_4*.avro"]}

2024-07-11T17:43:07.296Z	warn	api call failed	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 10800, "attempt": 1, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "error": "unexpected status code 500", "status_code": 500, "body": "{\n\t\"requestID\": \"ee7724fa-f4bd-4147-857b-2a7e63ab64be\",\n\t\"errors\": [{ \n\t\t\"code\": 25000,\t\t\"msg\": \"Internal error\"\t} \n\t],\n\t\"status\": \"fatal\",\n\t\"metrics\": {\n\t\t\"elapsedTime\": \"97.271348576s\",\n\t\t\"executionTime\": \"97.270691498s\",\n\t\t\"compileTime\": \"0ns\",\n\t\t\"queueWaitTime\": \"0ns\",\n\t\t\"resultCount\": 0,\n\t\t\"resultSize\": 0,\n\t\t\"processedObjects\": 0,\n\t\t\"bufferCacheHitRatio\": \"0.00%\",\n\t\t\"bufferCachePageReadCount\": 0,\n\t\t\"errorCount\": 1\n\t}\n}\n", "requestId": "cc5640c0-96eb-4d1d-b6d4-aa2bd22758fc"}

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI.func3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:210

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].do

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:66

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].DoWithContext

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:46

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:188

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.MustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:254

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).ColumnarExecuteAnalyticsQuery

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:318

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).CopyFromS3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:734

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/actions/columnar.CopyFromS3.Do

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/actions/columnar/copy_from_s3.go:109

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:77

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

2024-07-11T17:43:07.797Z	warn	retrying api call	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 10800, "attempt": 1, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "error": "unexpected status code 500", "status_code": 500, "requestId": "cc5640c0-96eb-4d1d-b6d4-aa2bd22758fc"}

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI.func1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:176

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].DoWithContext

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:53

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:188

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.MustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:254

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).ColumnarExecuteAnalyticsQuery

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:318

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).CopyFromS3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:734

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/actions/columnar.CopyFromS3.Do

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/actions/columnar/copy_from_s3.go:109

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:77

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

2024-07-11T17:43:07.969Z	warn	api call failed	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 10800, "attempt": 2, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "error": "unexpected status code 503", "status_code": 503, "body": "{\n\t\"requestID\": \"25e56629-b09e-4a79-b464-8e194f23d5b7\",\n\t\"errors\": [{ \n\t\t\"code\": 23000,\t\t\"msg\": \"Analytics Service is temporarily unavailable\",\t\t\"retriable\": true\t} \n\t],\n\t\"status\": \"fatal\",\n\t\"metrics\": {\n\t\t\"elapsedTime\": \"2.82872ms\",\n\t\t\"executionTime\": \"2.294152ms\",\n\t\t\"compileTime\": \"0ns\",\n\t\t\"queueWaitTime\": \"0ns\",\n\t\t\"resultCount\": 0,\n\t\t\"resultSize\": 0,\n\t\t\"processedObjects\": 0,\n\t\t\"bufferCacheHitRatio\": \"0.00%\",\n\t\t\"bufferCachePageReadCount\": 0,\n\t\t\"errorCount\": 1\n\t}\n}\n", "requestId": "3464bcd3-b28c-4f12-9238-5a1624d7f1fe"}

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI.func3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:210

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].do

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:66

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].DoWithContext

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:46

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:188

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.MustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:254

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).ColumnarExecuteAnalyticsQuery

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:318

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).CopyFromS3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:734

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/actions/columnar.CopyFromS3.Do

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/actions/columnar/copy_from_s3.go:109

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:77

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

2024-07-11T17:43:08.969Z	warn	retrying api call	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 10800, "attempt": 2, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "error": "unexpected status code 503", "status_code": 503, "requestId": "3464bcd3-b28c-4f12-9238-5a1624d7f1fe"}

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI.func1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:176

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].DoWithContext

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:53

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:188

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.MustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:254

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).ColumnarExecuteAnalyticsQuery

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:318

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).CopyFromS3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:734

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/actions/columnar.CopyFromS3.Do

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/actions/columnar/copy_from_s3.go:109

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:77

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

2024-07-11T17:43:09.141Z	warn	api call failed	{"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "timeoutSec": 10800, "attempt": 3, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/c7dc1b2c-51ce-4d2c-a93a-51006931d6f7/projects/fc226955-6229-41ea-9300-7eada8cbdab9/instance/a5f72537-6565-4271-a901-3ce41026a8fc/proxy/query/service", "error": "unexpected status code 503", "status_code": 503, "body": "{\n\t\"requestID\": \"cc0e7486-35fd-4bb2-953a-8b5fdde2ba77\",\n\t\"errors\": [{ \n\t\t\"code\": 23000,\t\t\"msg\": \"Analytics Service is temporarily unavailable\",\t\t\"retriable\": true\t} \n\t],\n\t\"status\": \"fatal\",\n\t\"metrics\": {\n\t\t\"elapsedTime\": \"2.404822ms\",\n\t\t\"executionTime\": \"1.934106ms\",\n\t\t\"compileTime\": \"0ns\",\n\t\t\"queueWaitTime\": \"0ns\",\n\t\t\"resultCount\": 0,\n\t\t\"resultSize\": 0,\n\t\t\"processedObjects\": 0,\n\t\t\"bufferCacheHitRatio\": \"0.00%\",\n\t\t\"bufferCachePageReadCount\": 0,\n\t\t\"errorCount\": 1\n\t}\n}\n", "requestId": "86bb7b9a-2bcd-46da-9f14-8e30d8c79f23"}

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI.func3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:210

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].do

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:66

github.com/couchbase/tools-common/utils/v3/retry.Retryer[...].DoWithContext

	/go/pkg/mod/github.com/couchbase/tools-common/utils/v3@v3.0.0/retry/retryer.go:46

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.mustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:188

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.MustCallAPI

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/http_client.go:254

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).ColumnarExecuteAnalyticsQuery

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:318

github.com/couchbasecloud/couchbase-cloud/internal/test/tdk2.(*Session).CopyFromS3

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/internal/test/tdk2/v2_columnar_utils.go:734

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/actions/columnar.CopyFromS3.Do

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/actions/columnar/copy_from_s3.go:109

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:77

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

2024-07-11T17:43:11.143Z	error	canceling due to unexpected error:	{"env": "other", "error": "exhausted retry count after 3 attempts: unexpected status code 503", "action": "Starting Copy from S3"}

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).Execute

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:112

github.com/couchbasecloud/couchbase-cloud/cmd/cp-cli/task.(*Tree).do.func1.1

	/data/workspace/cp-cli-gsi-system-test/couchbase-cloud/cmd/cp-cli/task/task.go:161

Please let me know if the number of collections is causing this OOM issue and I can scale the test down to have fewer collections.

cbcollect ->

https://cb-engineering.s3.amazonaws.com/SysTestColumnarJul9/collectinfo-2024-07-11T180705-ns_1%40svc-da-node-006.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJul9/collectinfo-2024-07-11T180705-ns_1%40svc-da-node-008.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJul9/collectinfo-2024-07-11T180705-ns_1%40svc-da-node-016.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJul9/collectinfo-2024-07-11T180705-ns_1%40svc-da-node-022.adewm3olqtoa4vfw.sandbox.nonprod-project-avengers.com.zip

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Pavan PB

Reporter:: Pavan PB

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 12/Jul/24 2:36 AM

Updated:: 12/Jul/24 8:00 AM

Resolved:: 12/Jul/24 6:29 AM

Gerrit Reviews

There are no open Gerrit changes

[System Test]java.lang.OutOfMemoryError: Java heap space error seen

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty