Collect all perishable information (on restart) before we need to restart a process

Most of the time, when customers raise an issue, they generally solve the issue by a restart of the problematic service.

This hinders the process of providing an RCA. A restart of certain services purges all in-memory stored information.

A simple example will be the system:completed_requests of the query service. The results of this query generally help us greatly with our analysis but if the customer restarts the cbq-engine before we can collect the output of system:completed_requests it is lost.

Another situation that we observe is a few issues that aren't generally reproducible. The major concern is with getting the customer to a stable state and we direct them to restart the concerned processes and don't take all the necessary files.

A simple solution that I can think off is to have a tool, that is designed especially to kill processes. But before it kills a process, it run's all the required commands to collect the necessary information.

For example:

./couchbase-kill cbq -n localhost:8091 -u Administrator -p password

This will first generate outputs for queries like 

select * from system:completed_requests;

Collect {{Go Routine Dumps}}

pkill cbq

A simple tool like this, would help collect all the necessary information and restart the specific service.

A further analysis can be done on all the different commands that should be run before we kill each service.