Hadoop集群升级NoSuchMethodError: org.apache.thrift.transport.TSocket.<init>(Lorg/apache/thrift/TConfig
·
Spark查询/保存Hive数据,在Hadoop集群升级后遇到如下异常
- NoSuchMethodError: org.apache.thrift.transport.TSocket.(Lorg/apache/thrift/TConfiguration;Ljava/lang/String;II)V
- RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
原因Spark应用中使用了org.apache.thrift libthrift依赖包新版本不兼容老版本。比如老版本是0.9.3,Hadoop版本升级后,libthrift版本为0.16.0,需要调整pom中的依赖。示例如下:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.16.0</version>
</dependency>
更多推荐



所有评论(0)