1) for read latency
asloglatency -h {NAMESPACE}-read -f -7200
asloglatency -h {NAMESPACE}-write -f 'Apr 17 2019 16:00:00' -l aerospike.log-20190418
2) for cluster network conection error
grep CLUSTER /var/log/aerospike/aerospike.log|grep -v "CLUSTER-SIZE 12"
grep "departed node" /var/log/aerospike/aerospike.log
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "tsvc-q 0"
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "rw-hash 0 proxy-hash 0"
3)for expiry,eviction
grep "(nsup)" /var/log/aerospike/aerospike.log|grep "Total time"|grep -E "[0-9]{4,9} ms"
If this kind of warning is found in log:
WARNING (nsup): (thr_nsup.c:1044) {NAMESPACE} no records below eviction void-time 299051849 - threshold bucket 0, width 260 sec, count 51354 > target 14699 (0.5 pct)
then evict-hist-buckets should be increased and also increase evict-tenths-pct. default value is 5( 0.5%).
verify the bucket size: asinfo -v namespace/NAMESPACE -l|grep bucket
then increase evict-hist-buckets number
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-hist-buckets=100000'"
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-tenths-pct=10'"
4)for error
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log
grep "hwm" /var/log/aerospike/aerospike.log
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log|grep -oE "GMT:.*"|sort -n|uniq -c|sort -n
4) config
asinfo -v "get-config:" -l|grep "proto"
asinfo -v namespace/NAMESPACE -l
asinfo -v "set-config:context=service;proto-fd-max=100000"
asinfo -v "set-config:context=namespace;id=test;high-water-memory-pct=50"
or asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-memory-pct=50'"
for disk hwm:
asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-disk-pct=50'"
note: "id" is name of namespace here.
5) network connections
netstat -an|grep tcp|wc -l
netstat -an|grep tcp|awk '{print $5}'|grep -oE ".*:"|sort -n|uniq -c|sort -n
netstat -an|grep tcp|awk '{print $6}'|sort -n|uniq -c|sort -n
ss|awk '{print $2}'|sort -n|uniq -c|sort -n
asloglatency -h {NAMESPACE}-read -f -7200
asloglatency -h {NAMESPACE}-write -f 'Apr 17 2019 16:00:00' -l aerospike.log-20190418
2) for cluster network conection error
grep CLUSTER /var/log/aerospike/aerospike.log|grep -v "CLUSTER-SIZE 12"
grep "departed node" /var/log/aerospike/aerospike.log
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "tsvc-q 0"
grep "in-progress" /var/log/aerospike/aerospike.log|grep -v "rw-hash 0 proxy-hash 0"
3)for expiry,eviction
grep "(nsup)" /var/log/aerospike/aerospike.log|grep "Total time"|grep -E "[0-9]{4,9} ms"
If this kind of warning is found in log:
WARNING (nsup): (thr_nsup.c:1044) {NAMESPACE} no records below eviction void-time 299051849 - threshold bucket 0, width 260 sec, count 51354 > target 14699 (0.5 pct)
then evict-hist-buckets should be increased and also increase evict-tenths-pct. default value is 5( 0.5%).
verify the bucket size: asinfo -v namespace/NAMESPACE -l|grep bucket
then increase evict-hist-buckets number
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-hist-buckets=100000'"
asadm -e "asinfo -v 'set-config:context=namespace;id=NAMESPACE;evict-tenths-pct=10'"
4)for error
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log
grep "hwm" /var/log/aerospike/aerospike.log
grep -E "ERROR|WARN" /var/log/aerospike/aerospike.log|grep -oE "GMT:.*"|sort -n|uniq -c|sort -n
4) config
asinfo -v "get-config:" -l|grep "proto"
asinfo -v namespace/NAMESPACE -l
asinfo -v "set-config:context=service;proto-fd-max=100000"
asinfo -v "set-config:context=namespace;id=test;high-water-memory-pct=50"
or asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-memory-pct=50'"
for disk hwm:
asadm -e "asinfo -v 'set-config:context=namespace;id=test;high-water-disk-pct=50'"
note: "id" is name of namespace here.
5) network connections
netstat -an|grep tcp|wc -l
netstat -an|grep tcp|awk '{print $5}'|grep -oE ".*:"|sort -n|uniq -c|sort -n
netstat -an|grep tcp|awk '{print $6}'|sort -n|uniq -c|sort -n
ss|awk '{print $2}'|sort -n|uniq -c|sort -n
Comments
Post a Comment