我正在使用spark-shell
带有mesos和zookeeper的Apache Spark ,这似乎运行良好,除了我从动物园管理员那里得到的记录比我想要的多,这有点让人分心:
Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.5.0 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60) Type in expressions to have them evaluated. Type :help for more information. 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.6 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@716: Client environment:host.name=myhost 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-34-generic 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@725: Client environment:os.version=#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@733: Client environment:user.name=myname 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:9999 sessionTimeout=10000 watcher=0xffffffffff sessionId=0 sessionPasswd=context=0x777777777777 flags=0 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:9999] 2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:9999], sessionId=0x11111111111111, negotiated timeout=10000 Spark context available as sc. SQL context available as sqlContext. scala>
我试过改变 $SPARK_CONF_DIR/log4j.properties
log4j.logger.org.apache.zookeeper=WARN
哪个不起作用,但我并不感到惊讶,因为mesos文档提到zookeeper记录直接转到stderr:
注意:第三方日志消息(例如ZooKeeper)仅写入stderr!
添加一个jar到spark的类路径有一个带有外部函数接口的静态块调用zookeeper C api函数来改变日志级别似乎可能有用,但我不确定,我没有任何经验从JVM调用C代码.