建议使用以下浏览器,以获得最佳体验。 IE 9.0+以上版本 Chrome 31+ 谷歌浏览器 Firefox 30+ 火狐浏览器
设置昵称

在此一键设置昵称,即可参与社区互动!

确定
我再想想
选择版块
鲲鹏BoostKit 主题:1022帖子:2725

【大数据】

Presto 0.221移植指导书

黄龙吐翠 2020/5/11 5625


1      前置条件

类别

子项

版本

硬件

CPU

鲲鹏920

OS

CentOS

7.6

软件

GCC

4.8.5

Maven

3.5.4

JDK

OpenJDK 1.8

Ant

1.9.4

编译环境需要连接至外网。

 

2      移植

2.1      netty-4.1.29/4.1.13/4.0.37移植

参考相应指导书进行移植。

2.2      jline-2.14.6移植

参考相应指导书进行移植。


2.3      libhadoop.so移植

presto-0.221依赖hadoop-2.7.4的本地动态库libhadoop.so,需要使用hadoop-2.7.4的源码编译后获得。


安装依赖包:

yum install snappy snappy-devel protobuf protobuf-devel -y


获取hadoop-2.7.4源码:

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz

tar -zxf hadoop-2.7.4-src.tar.gz


cd hadoop-2.7.4-src/hadoop-common-project/hadoop-common

执行编译:

mvn package -Pnative -Drequire.snappy=true -DskipTests -Dmaven.javadoc.skip=true

可从target/native/target/usr/local/lib下找到后继步骤所需的libhadoop.so.1.0.0


2.4      presto-hadoop-apache2-2.7.4-5移植

上一步安装snappy后可在/lib64下找到所需的libsnappy.so.1.1.4

 

wget https://github.com/prestodb/presto-hadoop-apache2/archive/2.7.4-5.tar.gz

tar -zxf 2.7.4-5.tar.gz

cd presto-hadoop-apache2-2.7.4-5

mkdir -p src/main/resources/nativelib/Linux-aarch64

将上文获取的libhadoop.so.1.0.0复制到Linux-aarch64文件夹下,改名为libhadoop.so

将上文获取的libsnappy.1.1.4复制到Linux-aarch64文件夹下,改名为libsnappy.so

 

mvn install

编译安装到本地仓库。

2.5      presto-cassandra-driver移植

2.5.1        jffi-1.2.10移植

wget https://github.com/jnr/jffi/archive/1.2.10.tar.gz

tar -zxf 1.2.10.tar.gz

cd jffi-1.2.10

ant

编译好的jffidist目录下。

 

需要把jffi.jar复制并更名为/root/.m2/repository/com/github/jnr/jffi/1.2.10/jffi-1.2.10.jarjffi-aarch64-Linux.jar复制并更名为/root/.m2/repository/com/github/jnr/jffi/1.2.10/jffi-1.2.10-native.jar

2.5.2        presto-cassandra-driver移植

wget https://github.com/prestodb/presto-cassandra-driver/archive/3.1.4-1.tar.gz

tar -zxf 3.1.4-1.tar.gz

cd presto-cassandra-driver-3.1.4-1

mvn install -DskipTests

 

2.6      airlift移植

airliftfacebook开发的命令行框架。

2.6.1        procname移植

git clone http://github.com/airlift/procname.git

cd procname

make

在目录下可找到编译好的libprocname.so

2.6.2        airlift移植

git clone http://github.com/airlift/airlift.git

cd airlift

git checkout 0.178

cd launcher/src/main/resources/procname

mkdir Linux-aarch64

把上文编译得到的libprocname.so放进Linux-aarch64目录。

 

mvn install -DskipTests

airlift安装进本地仓库。

2.7      presto移植

2.7.1        获取源码

git clone https://github.com/prestodb/presto.git


cd presto

git checkout 0.221

2.7.2        修改源码

presto-0.221默认不允许在amd64ppc64le以外的架构上运行,需要进行修改。

vi presto-main/src/main/java/com/facebook/presto/server/PrestoSystemRequirements.java

找到如下代码:

image.png

加入aarch64的代码,允许在aarch64上运行。修改好的效果如图:

image.png

 

vi pom.xml

注释第121presto-docs模块标签。

image.png

 

vi presto-server-rpm/pom.xml

修改77x86_64aarch64,方便标识,注释83architecture标签

image.png

2.7.3        编译打包

执行编译:

mvn package -DskipTests

image.png

可在presto-server/target下找到打包好的presto-server-0.221.tar.gz,在presto-server-rpm/target下找到打包好的presto-server-rpm-0.221.aarch64.rpm

3      presto验证

复制presto-server-rpm-0.221.aarch64.rpmpresto-cli/presto-cli-0.221-executable.jar到测试机。

 

执行yum install presto-server-rpm-0.221.aarch64.rpm -y

安装presto-server

 

mkdir -p /etc/presto/catalog

创建/etc/presto/catalog/hive.properties,输入以下内容

connector.name=hive-hadoop2

hive.metastore.uri=thrift://localhost:9083

hive.config.resources=/etc/hadoop/core-site.xml,/etc/hadoop/hdfs-site.xml

 

Hive配置及部署过程略。

执行service presto start启动服务。

可在默认日志路径/var/log/presto/server.log中看到服务启动:

image.png

浏览器访问测试主机8080端口:

image.png


chmod 777 presto-cli-0.221-executable.jar


./presto-cli-0221-executable.jar --server localhost:8080 --catalog hive --schema default

presto-cli中输入show tables;

image.png

任务可正常完成。

4      FAQ

4.1      .git directory not found报错

image.png

presto不能通过下载releasetar.gz源码包编译,必须git clonecheckout

 

4.2      maven报错failed to run the report

image.png

按步骤2.5.2禁用presto-docs模组再重新编译。presto-docs为开发文档,禁用不影响使用。

 

4.3      运行presto日志报错failed to load Hadoop native library

image.png

presto依赖hadoop-2.7.4,报错是因为编译presto时没有按步骤解决hadoop依赖。需要按照2.2-2.3步骤解决依赖,再编译presto


4.4      连接Hive报错This version of libhadoop is built without snappy support.

这是由于编译hadoop的机器没有snappy,也没有指定编译参数-Drequire.snappy=true,导致snappy压缩模块被默认关闭,需要安装snappysnappy-devel两个库,并在编译hadoop时使用-Drequire.snappy=true确保snappy模块被启用。

    Presto 0.221移植指导书.docx 657.35 KB,下载次数:11

回复11

2020/5/11 22:32

感谢分享  真棒。。

2020/5/12 09:10

lemon-_
0 0
2020/5/13 12:18

火灵
0 0
2020/5/15 19:06

内容很多,点个赞,回去继续看

ZN
1 0
2020/6/16 10:57

楼主请教 

2.2.2 编译hadoop-2.7.4源码 为什么只有hadoop-common-project/hadoop-common/target目录,没有hadoop-common-2.7.4/lib/native/libhadoop.so.1.0.0 

黄龙吐翠

2020-6-17 15:00

你好,指导书编译hadoop的参数和路径有误,已更新。谢谢指出!
... 展开
2020/7/9 09:15

"2.3      presto-hadoop-apache2-2.7.4-5移植"步骤,mvn install报错如下:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNGMapConfigurator@156643d4
Tests run: 11, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.808 sec <<< FAILURE!
testCodecRoundTrip(com.facebook.presto.hadoop.TestHadoopNative)  Time elapsed: 0.002 sec  <<< FAILURE!
java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:58)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:53)
Caused by: java.lang.RuntimeException: library not found: /nativelib/Linux-aarch64/libhadoop.so
        at com.facebook.presto.hadoop.HadoopNative.loadLibrary(HadoopNative.java:88)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
        ... 35 more

testNative(com.facebook.presto.hadoop.TestHadoopNative)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:44)
        at com.facebook.presto.hadoop.TestHadoopNative.testNative(TestHadoopNative.java:41)
Caused by: java.lang.RuntimeException: library not found: /nativelib/Linux-aarch64/libhadoop.so
        at com.facebook.presto.hadoop.HadoopNative.loadLibrary(HadoopNative.java:88)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:53)
        ... 34 more


Results :

Failed tests:   testCodecRoundTrip(com.facebook.presto.hadoop.TestHadoopNative): failed to load Hadoop native library
  testNative(com.facebook.presto.hadoop.TestHadoopNative): failed to load Hadoop native library

Tests run: 11, Failures: 2, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23.002 s
[INFO] Finished at: 2020-07-09T08:35:40+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project hadoop-apache2: There are test failures.
[ERROR]
[ERROR] Please refer to /root/presto-hadoop-apache2-2.7.4-5/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

使用mvn install -DskipTests编译成功,这样是否有问题?

2020/7/9 10:30

安装rpm包时,报如下错误,要怎么解决,按照文件的步骤编译的是0.211版本的。

[root@hdp102 ~]# yum -y install presto-server-rpm-0.211.aarch64.rpm
Loaded plugins: fastestmirror
Examining presto-server-rpm-0.211.aarch64.rpm: presto-server-rpm-0.211-1.noarch
Marking presto-server-rpm-0.211.aarch64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package presto-server-rpm.noarch 0:0.211-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================
 Package                           Arch                   Version                 Repository                                        Size
=========================================================================================================================================
Installing:
 presto-server-rpm                 noarch                 0.211-1                 /presto-server-rpm-0.211.aarch64                 596 M

Transaction Summary
=========================================================================================================================================
Install  1 Package

Total size: 596 M
Installed size: 596 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
+======================================================================+
|      Error: Required Java version could not be found                 |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK/JRE from the Java web site     |
|       > http://www.oracle.com/technetwork/java/javase/downloads <    |
|                                                                      |
| Presto requires Java 1.8 update 151 (8u151)                            |
| NOTE: This script will attempt to find Java whether you install      |
|       using the binary or the RPM based installer.                   |
+======================================================================+
error: %pre(presto-server-rpm-0:0.211-1.noarch) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package presto-server-rpm-0.211-1.noarch
  Verifying  : presto-server-rpm-0.211-1.noarch                                                                                      1/1

Failed:
  presto-server-rpm.noarch 0:0.211-1                                                                                                     

Complete!

黄龙吐翠

2020-7-9 14:05

你好,这个问题是由于presto需要java环境,安装presto的机器上缺少java环境导致的。需要安装openjdk 1.8.0并配置JAVA_HOME和PATH环境变量,使用java -version命令可以检查java是否正确安装。
... 展开

全棧工程师-lsj

2020-7-9 14:52

java命令可执行,这个问题是因为rpm的preinstall脚本强制作安装环境检查导致的,必须是要oracle jdk,脚本内容如下: # Pre installation script # Ensure that the proper version of Java exists on the system java_version() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME. JAVA="$1"/bin/java "$JAVA" -version 2>&1 | grep "\(java\|openjdk\) version" | awk '{ print substr($3, 2, length($3)-2); }' } java_vendor() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME). # Returns the java vendor name. eg: Oracle Corporation JAVA="$1"/bin/java "$JAVA" -XshowSettings:properties -version 2>&1 | grep "java.vendor =" | awk '{ print $3 " " $4; }' } ava_version() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME. JAVA="$1"/bin/java "$JAVA" -version 2>&1 | grep "\(java\|openjdk\) version" | awk '...
... 展开
2020/7/9 11:34

是否可以把使用分析工具扫描的结果,以及分析的x86安装包下载链接也一并附上?

2020/7/9 12:40

编译的presto版本是0.211

运行presto连接hive时报错如下:

2020-07-09T11:24:22.036+0800    ERROR   main    com.facebook.presto.server.PrestoServer null
java.lang.ExceptionInInitializerError
        at com.facebook.presto.hive.HdfsEnvironment$$FastClassByGuice$$a9f55374.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240)
        at com.facebook.presto.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:120)
        at com.facebook.presto.connector.ConnectorManager.createConnector(ConnectorManager.java:319)
        at com.facebook.presto.connector.ConnectorManager.addCatalogConnector(ConnectorManager.java:195)
        at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:187)
        at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:173)
        at com.facebook.presto.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:96)
        at com.facebook.presto.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:74)
        at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:119)
        at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)
Caused by: java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:58)
        at com.facebook.presto.hive.HdfsEnvironment.<clinit>(HdfsEnvironment.java:37)
        ... 31 more
Caused by: java.lang.RuntimeException: library not found: /nativelib/Linux-aarch64/libhadoop.so
        at com.facebook.presto.hadoop.HadoopNative.loadLibrary(HadoopNative.java:88)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
        ... 32 more


2020-07-09T11:24:22.037+0800    INFO    Thread-92       io.airlift.bootstrap.LifeCycleManager   Life cycle stopping...

"2.3      presto-hadoop-apache2-2.7.4-5移植"步骤,在本地仓库中/root/.m2/repository/com/facebook/presto/hadoop/hadoop-apache2/2.7.4-5/hadoop-apache2-2.7.4-5.jar的resources目录中包含/nativelib/Linux-aarch64/libhadoop.so文件 

发现编译完的/usr/lib/presto/lib/plugin/hive-hadoop2目录中使用的是hadoop-apache2-2.7.4-3.jar

总结:

编译presto 0.211版本,需要移植presto-hadoop-apache2-2.7.4-3版本

重新编译presto-hadoop-apache2-2.7.4-3后,再次重新编译presto 0.211后问题解决


上划加载中
直达楼层
标签
您还可以添加5个标签
  • 没有搜索到和“关键字”相关的标签
  • 云产品
  • 解决方案
  • 技术领域
  • 通用技术
  • 平台功能
取消

采纳成功

您已采纳当前回复为最佳回复

黄龙吐翠

发帖: 15粉丝: 4

发消息 + 关注

发表于2020年05月11日 18:58:45 5625 11
直达本楼层的链接
楼主
显示全部楼层
[大数据] Presto 0.221移植指导书


1      前置条件

类别

子项

版本

硬件

CPU

鲲鹏920

OS

CentOS

7.6

软件

GCC

4.8.5

Maven

3.5.4

JDK

OpenJDK 1.8

Ant

1.9.4

编译环境需要连接至外网。

 

2      移植

2.1      netty-4.1.29/4.1.13/4.0.37移植

参考相应指导书进行移植。

2.2      jline-2.14.6移植

参考相应指导书进行移植。


2.3      libhadoop.so移植

presto-0.221依赖hadoop-2.7.4的本地动态库libhadoop.so,需要使用hadoop-2.7.4的源码编译后获得。


安装依赖包:

yum install snappy snappy-devel protobuf protobuf-devel -y


获取hadoop-2.7.4源码:

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz

tar -zxf hadoop-2.7.4-src.tar.gz


cd hadoop-2.7.4-src/hadoop-common-project/hadoop-common

执行编译:

mvn package -Pnative -Drequire.snappy=true -DskipTests -Dmaven.javadoc.skip=true

可从target/native/target/usr/local/lib下找到后继步骤所需的libhadoop.so.1.0.0


2.4      presto-hadoop-apache2-2.7.4-5移植

上一步安装snappy后可在/lib64下找到所需的libsnappy.so.1.1.4

 

wget https://github.com/prestodb/presto-hadoop-apache2/archive/2.7.4-5.tar.gz

tar -zxf 2.7.4-5.tar.gz

cd presto-hadoop-apache2-2.7.4-5

mkdir -p src/main/resources/nativelib/Linux-aarch64

将上文获取的libhadoop.so.1.0.0复制到Linux-aarch64文件夹下,改名为libhadoop.so

将上文获取的libsnappy.1.1.4复制到Linux-aarch64文件夹下,改名为libsnappy.so

 

mvn install

编译安装到本地仓库。

2.5      presto-cassandra-driver移植

2.5.1        jffi-1.2.10移植

wget https://github.com/jnr/jffi/archive/1.2.10.tar.gz

tar -zxf 1.2.10.tar.gz

cd jffi-1.2.10

ant

编译好的jffidist目录下。

 

需要把jffi.jar复制并更名为/root/.m2/repository/com/github/jnr/jffi/1.2.10/jffi-1.2.10.jarjffi-aarch64-Linux.jar复制并更名为/root/.m2/repository/com/github/jnr/jffi/1.2.10/jffi-1.2.10-native.jar

2.5.2        presto-cassandra-driver移植

wget https://github.com/prestodb/presto-cassandra-driver/archive/3.1.4-1.tar.gz

tar -zxf 3.1.4-1.tar.gz

cd presto-cassandra-driver-3.1.4-1

mvn install -DskipTests

 

2.6      airlift移植

airliftfacebook开发的命令行框架。

2.6.1        procname移植

git clone http://github.com/airlift/procname.git

cd procname

make

在目录下可找到编译好的libprocname.so

2.6.2        airlift移植

git clone http://github.com/airlift/airlift.git

cd airlift

git checkout 0.178

cd launcher/src/main/resources/procname

mkdir Linux-aarch64

把上文编译得到的libprocname.so放进Linux-aarch64目录。

 

mvn install -DskipTests

airlift安装进本地仓库。

2.7      presto移植

2.7.1        获取源码

git clone https://github.com/prestodb/presto.git


cd presto

git checkout 0.221

2.7.2        修改源码

presto-0.221默认不允许在amd64ppc64le以外的架构上运行,需要进行修改。

vi presto-main/src/main/java/com/facebook/presto/server/PrestoSystemRequirements.java

找到如下代码:

image.png

加入aarch64的代码,允许在aarch64上运行。修改好的效果如图:

image.png

 

vi pom.xml

注释第121presto-docs模块标签。

image.png

 

vi presto-server-rpm/pom.xml

修改77x86_64aarch64,方便标识,注释83architecture标签

image.png

2.7.3        编译打包

执行编译:

mvn package -DskipTests

image.png

可在presto-server/target下找到打包好的presto-server-0.221.tar.gz,在presto-server-rpm/target下找到打包好的presto-server-rpm-0.221.aarch64.rpm

3      presto验证

复制presto-server-rpm-0.221.aarch64.rpmpresto-cli/presto-cli-0.221-executable.jar到测试机。

 

执行yum install presto-server-rpm-0.221.aarch64.rpm -y

安装presto-server

 

mkdir -p /etc/presto/catalog

创建/etc/presto/catalog/hive.properties,输入以下内容

connector.name=hive-hadoop2

hive.metastore.uri=thrift://localhost:9083

hive.config.resources=/etc/hadoop/core-site.xml,/etc/hadoop/hdfs-site.xml

 

Hive配置及部署过程略。

执行service presto start启动服务。

可在默认日志路径/var/log/presto/server.log中看到服务启动:

image.png

浏览器访问测试主机8080端口:

image.png


chmod 777 presto-cli-0.221-executable.jar


./presto-cli-0221-executable.jar --server localhost:8080 --catalog hive --schema default

presto-cli中输入show tables;

image.png

任务可正常完成。

4      FAQ

4.1      .git directory not found报错

image.png

presto不能通过下载releasetar.gz源码包编译,必须git clonecheckout

 

4.2      maven报错failed to run the report

image.png

按步骤2.5.2禁用presto-docs模组再重新编译。presto-docs为开发文档,禁用不影响使用。

 

4.3      运行presto日志报错failed to load Hadoop native library

image.png

presto依赖hadoop-2.7.4,报错是因为编译presto时没有按步骤解决hadoop依赖。需要按照2.2-2.3步骤解决依赖,再编译presto


4.4      连接Hive报错This version of libhadoop is built without snappy support.

这是由于编译hadoop的机器没有snappy,也没有指定编译参数-Drequire.snappy=true,导致snappy压缩模块被默认关闭,需要安装snappysnappy-devel两个库,并在编译hadoop时使用-Drequire.snappy=true确保snappy模块被启用。

Presto 0.221移植指导书.docx 657.35 KB,下载次数:11

举报
分享

分享文章到朋友圈

分享文章到微博

采纳成功

您已采纳当前回复为最佳回复
发表于2020年05月11日 22:32:35
直达本楼层的链接
沙发
显示全部楼层

感谢分享  真棒。。

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复

五湖四海@银河护卫队

发帖: 188粉丝: 77

级别 : 版主,版块专家

发消息 + 关注

发表于2020年05月12日 09:10:50
直达本楼层的链接
板凳
显示全部楼层

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复

lemon-_

发帖: 46粉丝: 6

发消息 + 关注

发表于2020年05月13日 12:18:37
直达本楼层的链接
地板
显示全部楼层

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复

火灵

发帖: 172粉丝: 11

发消息 + 关注

发表于2020年05月15日 19:06:45
直达本楼层的链接
5#
显示全部楼层

内容很多,点个赞,回去继续看

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复

ZN

发帖: 0粉丝: 0

发消息 + 关注

发表于2020年06月16日 10:57:34
直达本楼层的链接
6#
显示全部楼层

楼主请教 

2.2.2 编译hadoop-2.7.4源码 为什么只有hadoop-common-project/hadoop-common/target目录,没有hadoop-common-2.7.4/lib/native/libhadoop.so.1.0.0 

评论
黄龙吐翠 2020-6-17 15:00 评论

你好,指导书编译hadoop的参数和路径有误,已更新。谢谢指出!

... 查看全部
点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复
发表于2020年07月09日 09:15:05
直达本楼层的链接
7#
显示全部楼层

"2.3      presto-hadoop-apache2-2.7.4-5移植"步骤,mvn install报错如下:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNGMapConfigurator@156643d4
Tests run: 11, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.808 sec <<< FAILURE!
testCodecRoundTrip(com.facebook.presto.hadoop.TestHadoopNative)  Time elapsed: 0.002 sec  <<< FAILURE!
java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:58)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:53)
Caused by: java.lang.RuntimeException: library not found: /nativelib/Linux-aarch64/libhadoop.so
        at com.facebook.presto.hadoop.HadoopNative.loadLibrary(HadoopNative.java:88)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
        ... 35 more

testNative(com.facebook.presto.hadoop.TestHadoopNative)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:44)
        at com.facebook.presto.hadoop.TestHadoopNative.testNative(TestHadoopNative.java:41)
Caused by: java.lang.RuntimeException: library not found: /nativelib/Linux-aarch64/libhadoop.so
        at com.facebook.presto.hadoop.HadoopNative.loadLibrary(HadoopNative.java:88)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:53)
        ... 34 more


Results :

Failed tests:   testCodecRoundTrip(com.facebook.presto.hadoop.TestHadoopNative): failed to load Hadoop native library
  testNative(com.facebook.presto.hadoop.TestHadoopNative): failed to load Hadoop native library

Tests run: 11, Failures: 2, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23.002 s
[INFO] Finished at: 2020-07-09T08:35:40+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project hadoop-apache2: There are test failures.
[ERROR]
[ERROR] Please refer to /root/presto-hadoop-apache2-2.7.4-5/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

使用mvn install -DskipTests编译成功,这样是否有问题?

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复
发表于2020年07月09日 10:30:21
直达本楼层的链接
8#
显示全部楼层

安装rpm包时,报如下错误,要怎么解决,按照文件的步骤编译的是0.211版本的。

[root@hdp102 ~]# yum -y install presto-server-rpm-0.211.aarch64.rpm
Loaded plugins: fastestmirror
Examining presto-server-rpm-0.211.aarch64.rpm: presto-server-rpm-0.211-1.noarch
Marking presto-server-rpm-0.211.aarch64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package presto-server-rpm.noarch 0:0.211-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================
 Package                           Arch                   Version                 Repository                                        Size
=========================================================================================================================================
Installing:
 presto-server-rpm                 noarch                 0.211-1                 /presto-server-rpm-0.211.aarch64                 596 M

Transaction Summary
=========================================================================================================================================
Install  1 Package

Total size: 596 M
Installed size: 596 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
+======================================================================+
|      Error: Required Java version could not be found                 |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK/JRE from the Java web site     |
|       > http://www.oracle.com/technetwork/java/javase/downloads <    |
|                                                                      |
| Presto requires Java 1.8 update 151 (8u151)                            |
| NOTE: This script will attempt to find Java whether you install      |
|       using the binary or the RPM based installer.                   |
+======================================================================+
error: %pre(presto-server-rpm-0:0.211-1.noarch) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package presto-server-rpm-0.211-1.noarch
  Verifying  : presto-server-rpm-0.211-1.noarch                                                                                      1/1

Failed:
  presto-server-rpm.noarch 0:0.211-1                                                                                                     

Complete!

评论
黄龙吐翠 2020-7-9 14:05 评论

你好,这个问题是由于presto需要java环境,安装presto的机器上缺少java环境导致的。需要安装openjdk 1.8.0并配置JAVA_HOME和PATH环境变量,使用java -version命令可以检查java是否正确安装。

... 查看全部
全棧工程师-lsj 2020-7-9 14:52 评论

java命令可执行,这个问题是因为rpm的preinstall脚本强制作安装环境检查导致的,必须是要oracle jdk,脚本内容如下: # Pre installation script # Ensure that the proper version of Java exists on the system java_version() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME. JAVA="$1"/bin/java "$JAVA" -version 2>&1 | grep "\(java\|openjdk\) version" | awk '{ print substr($3, 2, length($3)-2); }' } java_vendor() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME). # Returns the java vendor name. eg: Oracle Corporation JAVA="$1"/bin/java "$JAVA" -XshowSettings:properties -version 2>&1 | grep "java.vendor =" | awk '{ print $3 " " $4; }' } ava_version() { # The one argument is the location of java (either $JAVA_HOME or a potential # candidate for JAVA_HOME. JAVA="$1"/bin/java "$JAVA" -version 2>&1 | grep "\(java\|openjdk\) version" | awk '...

... 查看全部
点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复

一只笨熊喵

发帖: 9粉丝: 3

发消息 + 关注

发表于2020年07月09日 11:34:53
直达本楼层的链接
9#
显示全部楼层

是否可以把使用分析工具扫描的结果,以及分析的x86安装包下载链接也一并附上?

点赞 评论 引用 举报

采纳成功

您已采纳当前回复为最佳回复
发表于2020年07月09日 12:40:15
直达本楼层的链接
10#