Adding security groups via annotations to CB Cluster Service (ELB)

For a couchbase cloud cluster(all services enabled) deployed via operator. We are hitting the AWS Limit of 60 rules per Security group if we add to LoadBalancerSourceRanges (4 are added by us, a couple of user rules) Each IP in these source ranges adds 8 rules to the security group.
To ensure a user gets more rules to add before increasing the limit. I created a separate security group for the 4 that we added. I am trying to patch the service.
kubectl patch services cb-0000-exposed-ports -n a21272b3-7312-4925-a589-1854e85dec45 -p '{"metadata":{ "annotations":

{"service.beta.kubernetes.io/aws-load-balancer-extra-security-groups":"sg-066009b95eb52636d"}

}}'
However the annotation disappears on the service after a few seconds.
I am wondering if i am doing something wrong?
Would like some help on this issue?