Read Python code in Source Insight

First, download the Python.CLF file in this website: link.

In Source Insight, click Options -> Preferences... and import Python.CLF.

Then in Options -> Document Options..., click 'Add Type...' and add Python Source File. Settings are shown below:

Finish.

生成应用商店的用户数据文件

Select with concat:
hive> select concat(device_id, ',', appid, ',', apk_url) from download_log where device_id is not NULL;
hive> select concat(device_id, ',', package_name) from detail_log where device_id is not NULL;
Above data is too large to save.

For the huge number of detail log, I group them first:
hive> select device_id, package_name, count(package_name) rank from detail_log where length(device_id) > 0 and length(package_name) > 0 group by device_id, package_name order by rank desc;

If you want to split each entry by comma instead of tab, you can use awk (assume that your filename is test):
$ cat test | awk '{print $1“,"$2","$3}' > test_commma

For the download log, also I group them:
hive> select device_id, appid, count(appid) rank from download_log where length(device_id) > 0 group by device_id, appid order by rank desc;

Save results into local file:
insert overwrite local directory 'your_path'
row format delimited
fields terminated by '\t'
stored as textfile
select * from table where ...;

Spark MLlib中的数据类型

Please see this doc.

Spark

Please see my doc.

The official document is a good reference for you.

Maybe these links are useful: link1, link2, link3, link4.

Here is a blog about big data.

Sublime Text使用教程

Please see thie doc.

IntelliJ IDEA分别搭建基于maven和sbt的Scala项目

About the settings and keyboard shortcuts in IntelliJ, please see this doc.

About how to build Spark apps in IntelliJ, please see this doc.

Scala中this.type的使用

一直有一个疑问没搞懂，有必要在这里发上一帖，参见我提的问题和回答：链接。

问题如下：

I defined two classes:

class A { def method1 = this }
class B extends A { def method2 = this }

val b = new B

Then I checked the type of b.method1:

scala> b.method1.getClass
res29: Class[_ <: A] = class B

In this case, I cannot use b.method1.method2:

scala> b.method1.method2
<console>:11: error: value method2 is not a member of A
              b.method1.method2

So I have to define A and B like this:

class A { def method1: this.type = this } 
class B extends A { def method2: this.type = this } 

val b = new B

Now I check the type of b.method1:

scala> b.method1.getClass
res31: Class[_ <: B] = class B

Here b.method1.method2 works:

scala> b.method1.method2
res32: b.type = B@4ff3ac

My question here is what does it mean by saying Class[_ <: A] = class B and Class[_ <: B] = class B? And why does the first doesn't work as Class[_ <: A] = class B seems to say that it's also class B?

Reactormonk很好地解答了我的疑问：

Let's split the expression Class[_ <: A] = class B. The first part, Class[_ <: A] tells you what the compiler knows at compile time, that b.method1 returns something of type Class[_ <: A]. The second part, class B mentions that you have a class B as value, but the compiler doesn't know that. That's runtime information.

this.type specializes the method for subclasses, where the type inferred from a plain thisdoesn't.