no data in table metrics_locator #791
Description
Hi
In my test project,I have ingested metrics for 7 days.The number of metrics is 130 000 in a minute.
After 7 days,the ingest maybe ok, but there is no data in table metrics_locator.The metric_full is still ingested data.Why is no data in table metrics_locator?
I find the thread in my process
ps -mp 7678 -o THREAD,tid,time
USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME
bf 12.6 - - - - - - 1-00:28:55
bf 0.0 19 - futex_ - - 7678 00:00:00
bf 0.0 19 - futex_ - - 7680 00:00:11
bf 0.0 19 - futex_ - - 7681 00:11:00
bf 0.0 19 - futex_ - - 7682 00:11:00
bf 0.0 19 - futex_ - - 7683 00:11:02
bf 0.0 19 - futex_ - - 7684 00:11:03
bf 0.0 19 - futex_ - - 7685 00:11:01
bf 0.0 19 - futex_ - - 7686 00:10:59
bf 0.0 19 - futex_ - - 7687 00:10:59
bf 0.0 19 - futex_ - - 7688 00:11:00
bf 0.0 19 - futex_ - - 7689 00:01:20
bf 0.0 19 - futex_ - - 7690 00:00:00
bf 0.0 19 - futex_ - - 7691 00:00:00
bf 0.0 19 - futex_ - - 7692 00:00:00
bf 0.0 19 - futex_ - - 7693 00:00:16
bf 0.0 19 - futex_ - - 7694 00:00:14
bf 0.0 19 - futex_ - - 7695 00:00:14
bf 0.0 19 - futex_ - - 7696 00:00:04
bf 0.0 19 - futex_ - - 7697 00:00:00
bf 0.0 19 - futex_ - - 7698 00:04:09
bf 0.0 19 - futex_ - - 7699 00:00:07
bf 0.0 19 - futex_ - - 7702 00:06:01
bf 0.0 19 - futex_ - - 7761 00:00:00
bf 0.0 19 - futex_ - - 7762 00:00:00
bf 0.0 19 - futex_ - - 7768 00:02:03
bf 0.9 19 - futex_ - - 7769 01:55:38
bf 0.0 19 - futex_ - - 7770 00:00:00
bf 0.0 19 - futex_ - - 7771 00:00:00
bf 0.0 19 - - - - 7772 00:00:24
bf 0.0 19 - - - - 7773 00:00:24
bf 0.0 19 - futex_ - - 7774 00:00:29
bf 0.2 19 - - - - 7776 00:26:39
the thread 7769 's cpu is more higher,and
then
jstack 7678 |grep 1e59 -A 30
"Shard state writer" #15 prio=5 os_prio=0 tid=0x00007fb69433c000 nid=0x1e59 waiting on condition [0x00007fb692181000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.rackspacecloud.blueflood.service.ShardStateWorker.run(ShardStateWorker.java:91)
at java.lang.Thread.run(Thread.java:745)
"Shard state reader" #16 prio=5 os_prio=0 tid=0x00007fb69433a800 nid=0x1e58 sleeping[0x00007fb692282000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.rackspacecloud.blueflood.service.ShardStateWorker.run(ShardStateWorker.java:91)
at java.lang.Thread.run(Thread.java:745)
"MetadataBatchedWrites" #14 prio=5 os_prio=0 tid=0x00007fb694323800 nid=0x1e52 in Object.wait() [0x00007fb692383000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c1379e28> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c1379e28> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"MetadataBatchedReads" #13 prio=5 os_prio=0 tid=0x00007fb694322000 nid=0x1e51 in Object.wait() [0x00007fb692484000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c137a2c0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c137a2c0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)