pyspark之創建SparkSession

1.from pyspark.sql import SparkSession

2.spark = SparkSession.builder.master("spark://master:7077") \
.appName('compute_customer_age') \
.config('spark.executor.memory','2g') \
.enableHiveSupport() \
.getOrCreate()

3.創建完畢

4.可以用於構建DataFrame|用於訪問hive

4.1DataFrame

documentDF = spark.createDataFrame([
("Hi I heard about Spark".split(" "), ),
("I wish Java could use case classes".split(" "), ),
("Logistic regression models are neat".split(" "), )
], ["text"])

4.2訪問hive

sql = """
"""
df = spark.sql(sql)
df.show()