If I execute these commands in spark-shell it correctly returns the data in the “people” table:
val dataframe_mysql = spark.sqlContext.read.format("jdbc").option("url", "jdbc:mysql://localhost/db_spark").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "people").option("user", "root").option("password", "****").load()
dataframe_mysql.show
The problem is if I close spark-shell and return it open, the connection to the database is not maintained.
Advertisement
Answer
As per Spark’s documentation, SparkContext and HiveContext get created inside the spark shell (when spark-shell command is executed) with HiveContext defined as SQLContext. As the connection is mapped to SQLContext, closing the spark shell will mean you won’t be able to access SQLContext and hence, you won’t be able to connect.
Here’s another reference:
When you run spark-shell, which is your interactive driver application, it automatically creates a SparkContext defined as sc and a HiveContext defined as sqlContext.