Skip to main content

Aerospike Debugging

1) for read latency
asloglatency -h {NAMESPACE}-read -f -7200
asloglatency -h {NAMESPACE}-write -f 'Apr 17 2019 16:00:00' -l aerospike.log-20190418 

 2) for cluster network conection error
grep CLUSTER /var/log/aerospike/aerospike.log|grep -v "CLUSTER-SIZE 12"
grep "departed node" /var/log/aerospike/aerospike.log
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "tsvc-q 0"
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "rw-hash 0 proxy-hash 0"

3)for expiry,eviction
grep "(nsup)" /var/log/aerospike/aerospike.log|grep "Total time"|grep -E "[0-9]{4,9} ms"
If this kind of warning is found in log:
WARNING (nsup): (thr_nsup.c:1044) {NAMESPACE} no records below eviction void-time 299051849 - threshold bucket 0, width 260 sec, count 51354 > target 14699 (0.5 pct)
then  evict-hist-buckets should be increased and also increase evict-tenths-pct. default value is 5( 0.5%).
verify the bucket size: asinfo -v namespace/NAMESPACE -l|grep bucket
then increase evict-hist-buckets number
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-hist-buckets=100000'"
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-tenths-pct=10'"

4)for error
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log
grep "hwm" /var/log/aerospike/aerospike.log
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log|grep -oE "GMT:.*"|sort -n|uniq -c|sort -n
4) config
asinfo -v "get-config:" -l|grep "proto"
asinfo -v namespace/NAMESPACE -l
asinfo -v "set-config:context=service;proto-fd-max=100000"
asinfo -v "set-config:context=namespace;id=test;high-water-memory-pct=50"
or asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-memory-pct=50'"
for disk hwm:
asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-disk-pct=50'"
note: "id" is name of namespace here.

5) network connections
netstat -an|grep tcp|wc -l
netstat -an|grep tcp|awk '{print $5}'|grep -oE ".*:"|sort -n|uniq -c|sort -n
netstat -an|grep tcp|awk '{print $6}'|sort -n|uniq -c|sort -n
ss|awk '{print $2}'|sort -n|uniq -c|sort -n

Comments

Post a Comment

Popular posts from this blog

On Divisibility of Numbers

There are some rules to check that a given number is divisible by a number without actual division. e.g. to check if a given number is divisible by 2, you have just check the last digit is even or not. Similarly for 3 , if the sum of digit of any given number is divisible by 3 then the number will also be divisible by 3. One day I was thinking that if there is some generalized rule for every number. Here is some interesting thing I found out. Suppose we have to test divisibility by 17. Let us check whether 4913 is divisible by 17 or not. Here is the rule:- Step 1) :- found out multiple of the number( in this case 17) which ends with 1 or 9. 17 X 3 = 51 ends with 1. Step 2) :- strip the last digit . so we got 5 here. Step 3) :- take the last digit of 4913 which is 3. multiply it by 5 3X5=15. Remove the last digit from 4913 and subtract 15 from it. 491-15 = 476 again repeat this on 476 47 -...
Post a Comment
Read more

Network delay simulation

"tc" and "iptable" both can be used for network control. "iptable" is dangerous as one can accidently block ssh port. "tc" provide extra level of customization for traffic control. 1) for adding delay: tc qdisc add dev eth0 root netem delay 100ms   if "tc" command not found, then you can install it:   yum -y install tc   Note:name of device "eth0" can be found running "ifconfig" command The name eth0 can differ in various system.   for checking if delay is introduced or not, run: tc -s qdisc   for removing all "tc" rules, run this command: tc qdisc del dev eth0 root netem  and again verfiy : tc -s qdisc     2) for adding delay in statistically distributed fashion: tc qdisc add dev eth0 root netem delay 150ms 50ms distribution normal note: if it gives error "RTNETLINK answers: File exists" then run:  tc qdisc change dev eth0 r...
Post a Comment
Read more