我正在尝试以独立模式安装HBase v1.1.2(与Hadoop v2.7.2),并且正在尽可能地关注Apache HBase文档,但是配置似乎有些棘手。不知道这是否与我使用自制软件安装HBase相对,而不是通过其镜像站点之一直接下载有关。我的目标是达到可以执行创建表,列出现有表等的目的。
我已经安装了Java 1.7.0,并且已经运行“ brew install hbase”来安装hbase。在Homebrew的hbase目录中,我打开了/libexec/conf/hbase-env.sh,在第27行看到以下内容:
export JAVA_HOME="$(/usr/libexec/java_home)"
当我/usr/libexec/java_home
在命令行中执行时,它输出/Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home
。因此,似乎设置了有效的$ JAVA_HOME env变量。
在/libexec/conf/hbase-site.xml
配置文件中,我设置了以下键/值对:
hbase.rootdir file:///home/richiethomas/hbase hbase.zookeeper.property.dataDir /home/richiethomas/zookeeper
换句话说,我已经复制了HBase文档中显示的配置,只是将“ testuser”替换为“ richiethomas”,因为这是我在命令行中使用的用户名。
然后在Homebrew hbase
目录中,运行`./bin/start-hbase.sh'。我看到以下内容:
starting master, logging to /usr/local/Cellar/hbase/1.1.2/libexec/bin/../logs/hbase-richiethomas-master-MacBook-Pro-5.local.out
然后,我跑步hbase shell
,看到以下内容:
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.1.2/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2017-01-03 17:27:52,056 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable HBase Shell; enter 'help' for list of supported commands. Type "exit " to leave the HBase Shell Version 1.1.2, rcc2b70cf03e3378800661ec5cab11eb43fafe0fc, Wed Aug 26 20:11:27 PDT 2015 >>
完成此操作后,在新的CLI选项卡中,我cat
从(/usr/local/Cellar/hbase/1.1.2/libexec/bin/../logs/hbase-richiethomas-master-MacBook-Pro-5.local.out
)上方看到的路径中获取日志,并且看到以下内容:
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.1.2/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
然后,回到我的HBase shell中,我运行以下命令查看是否一切正常,并且看到以下内容:
>> list TABLE 2017-01-03 17:28:05,108 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:05,320 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:05,625 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:06,128 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:07,134 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:09,144 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2017-01-03 17:28:13,158 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. ERROR: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. Here is some help for this command: List all tables in hbase. Optional regular expression parameter could be used to filter the output. Examples: hbase> list hbase> list 'abc.*' hbase> list 'ns:abc.*' hbase> list 'ns:.*'
根据Apache文档:
您不需要创建HBase数据目录。HBase将为您完成此任务。如果创建目录,则HBase将尝试进行迁移,这不是您想要的。
我的解释是,我不需要在中设置以下属性hbase-site.xml
,特别是因为“ hbase”是“ zookeeper.znode.parent”的默认设置:
zookeeper.znode.parent /hbase
的输出jps
如下:
32146 ZooKeeperMain 1364 QuorumPeerMain 83568 34120 Jps
我的zoo.cfg
文件位于中/usr/local/etc/zookeeper/zoo.cfg
,如下所示:
# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/usr/local/var/run/zookeeper/data # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1
我尝试过的解决方案:
一世。
我试图取代的价值zookeeper.property.dataDir
中hbase-site.xml
有文件/usr/local/var/run/zookeeper/data
,但是这并没有解决问题。
二。
当我zkcli
在命令行中运行并运行时ls /
,我只会看到zookeeper
,没有hbase
目录。我尝试在zookeeper shell中运行以下命令:
create /hbase my_data
然后,我看到以下内容:
[zk: localhost:2181(CONNECTED) 4] ls / [hbase, zookeeper]
然后,当我退出并重新运行HBase shell并尝试运行时list
,我看到以下内容:
>> list TABLE ERROR: Can't get master address from ZooKeeper; znode data == null
三,
根据Quora的这篇文章,HBase 1.1.2仅支持Hadoop 2.5.x以下的版本,因此我brew uninstall hadoop
依次brew install hadoop25
运行,停止和启动hbase脚本,然后重新启动hbase shell。仍然没有运气。
IV。
我尝试卸载Homebrew版本并从Apache镜像站点下载HBase,并在hbase-site.xml中重复配置过程,但是我遇到了相同的错误(The node /hbase is not in ZooKeeper. It should have been written by the master.
)。
V.
我通过Homebrew卸载/重新安装了HBase,并按照“注意事项”部分中的说明进行操作:
==> Caveats To have launchd start hbase now and restart at login: brew services start hbase Or, if you don't want/need a background service you can just run: /usr/local/opt/hbase/bin/start-hbase.sh
我注意到,在运行上述命令后,该hbase-site.xml
文件已被填充了合理的前瞻性值hbase.rootdir
和hbase.zookeeper.property.dataDir
(特别是file:///usr/local/var/hbase
和/usr/local/var/zookeeper
,分别)。我还注意到,当我运行brew services start hbase
然后又运行时jps
,我没有看到正在运行的“ HMaster”进程,但是我确实在运行时看到了HMaster /usr/local/opt/hbase/bin/start-hbase.sh
。但是,然后我尝试启动hbase shell
并输入status
,但出现了与以前相同的错误(The node /hbase is not in ZooKeeper.
)。
VI。
我已验证我的Hadoop安装运行正常:
MacBook-Pro-5:2.5.2 richiethomas$ hadoop jar /usr/local/Cellar/hadoop25/2.5.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 2 5 Number of Maps = 2 Samples per Map = 5 17/01/03 20:09:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Wrote input for Map #0 Wrote input for Map #1 Starting Job ... Job Finished in 2.156 seconds Estimated value of Pi is 3.60000000000000000000
我的HBase配置似乎出了点问题,但是我不确定在此还可以尝试什么。
当您要建立独立版本时hbase
,无需构建hdfs
或构建zk
。
您出现问题的原因是,hbase
将尝试启动内部zk
服务,但是由于已有Java进程QuorumPeerMain
占用了port 2181
,因此hbase
将无法启动。如果hbase
成功启动,将有一个名为的Java进程HMaster
。
要验证这一点,请检查log的最后一行hbase-richiethomas-master-MacBook-Pro-5.local.out
,您可能会看到类似以下的错误消息:
Could not start ZK at requested port of 2181. ZK was started at port: 2182. Aborting as clients (e.g. shell) will not be able to find this ZK quorum.
因此,解决方案是:
杀死现有的 QuorumPeerMain
再跑start-hbase.sh
一次