kafka connect test.sink.txt并没有更新

参考https://stackoverflow.com/questions/35049287/kafka-connect-test-sink-txt-is-not-getting-updated-after-running-connectors-an

执行bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties，发现test.sink.txt并没有更新。

因为之前手动删了test.sink.txt文件，offset没有重置，所以读不出内容。将/tmp/connect.offsets删除即可。

kafka bootstrap servers配置

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

一组host和port用于初始化连接. 不管这里配置了多少台server, 都只是用作发现整个集群全部server信息. 这个配置不需要包含集群所有的机器信息. 但是最好多于一个, 以防服务器挂掉.

如果只配置了一个的话，这个server挂掉了kafka集群就无法工作。

具体点说，如果producer.properties中的bootstrap.servers都挂掉，producer无法工作；如果consumer.properties中的bootstrap.servers都挂掉，consumer无法工作。

kafka启动不断刷屏

bin/kafka-server-start.sh config/server.properties 启动kafka时被下面两行日志刷频：
  INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
  INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

把zookeeper的dataDir：/tmp/zookeeper文件夹删掉重启了zookeeper和kafka之后就正常了。

彻底删除kafka的topic

参考https://stackoverflow.com/questions/33537950/how-to-delete-a-topic-in-apache-kafka/33538299

1. 停掉 kafka
2. 删除 kafka 配置中 /tmp/kafka-logs 中 topic 的目录
3. bin/zookeeper-shell.sh localhost:2181
4. ls /brokers/topics
5. rmr /brokers/topics/yourtopic
6. 重启 kafka

intellij开发javascript设置

在设置中的Language & Frameworks中JavaScript设置Javascript language version。

设置node.js library：Language & Frameworks中Node.js and NPM，设置enable Node.js Core library。
参考https://intellij-support.jetbrains.com/hc/en-us/community/posts/206324989-Unresolved-function-or-method-require-at-line-1

Idea license 过期

参考 http://blog.csdn.net/u013400939/article/details/55223631

设置 license 地址 http://idea.iteblog.com/key.php 即可

pyspark java.lang.OutOfMemoryError: GC overhead limit exceeded

参考https://stackoverflow.com/questions/32336915/pyspark-java-lang-outofmemoryerror-java-heap-space

在 $SPARK_HOME/conf/spark-defaults.conf 中调大 spark.driver.memory 的值，如果没有配的话，默认是 1g