Skip to content

Incorrect heap usage reporting #270

Open
@wenyuzhao

Description

@wenyuzhao

MMTk GCs could not run hbase due to the incorrect or missing implementation of heap stat hooks.

Probably related to these two methods: https://github.com/mmtk/mmtk-openjdk/blob/master/openjdk/mmtkHeap.hpp#L93-L94

Steps to run hbase and reproduce the error:

wget https://archive.apache.org/dist/hbase/2.4.14/hbase-2.4.14-bin.tar.gz
tar -zxvf hbase-2.4.14-bin.tar.gz

export MMTK_PLAN=Immix 
export JAVA_HOME="MY-MMTK-JDK-HOME"
export HBASE_OPTS="-Xms1650m -Xmx1650m  -XX:+UseThirdPartyHeap "

bash ./hbase-2.4.14/bin/start-hbase.sh

After the script execution is complete, HBase did not start correctly. By examining the logs at ./hbase-2.4.14/log, the following error messages can be observed.

2024-03-01 02:05:27,911 ERROR [main] impl.MetricsSourceAdapter: Error getting metrics from source JvmMetrics
java.lang.IllegalArgumentException: committed = 9223372036854775807 should be < max = 1730150400
    at java.management/java.lang.management.MemoryUsage.<init>(MemoryUsage.java:166)
    at java.management/sun.management.MemoryImpl.getMemoryUsage0(Native Method)
    at java.management/sun.management.MemoryImpl.getHeapMemoryUsage(MemoryImpl.java:71)
    at org.apache.hadoop.metrics2.source.JvmMetrics.getMemoryUsage(JvmMetrics.java:146)
    at org.apache.hadoop.metrics2.source.JvmMetrics.getMetrics(JvmMetrics.java:138)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
    at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:329)
    at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:315)
    at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:66)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:223)
    at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
    at org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:111)
    at org.apache.hadoop.metrics2.source.JvmMetrics$Singleton.init(JvmMetrics.java:61)
    at org.apache.hadoop.metrics2.source.JvmMetrics.initSingleton(JvmMetrics.java:120)
    at org.apache.hadoop.hbase.metrics.BaseSourceImpl$DefaultMetricsSystemInitializer.init(BaseSourceImpl.java:54)
    at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:111)
    at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:42)
    at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:37)
    at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:95)
    at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:32)
    at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:172)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:591)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:422)
    at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:318)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:122)
    at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:222)
    at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:169)
    at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:112)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:240)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:144)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2934)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions