通过dockercompose快速部署Hive详细教程

　　一、概述
　　其实通过dockercompose部署hive是在继上篇文章Hadoop部署的基础之上叠加的，Hive做为最常用的数仓服务，所以是有必要进行集成的，感兴趣的小伙伴请认真阅读我以下内容，通过dockercompose部署的服务主要是用最少的资源和时间成本快速部署服务，方便小伙伴学习、测试、验证功能等等
　　关于Hadoop部署可以查阅我以下几篇文章：通过dockercompose快速部署Hadoop集群详细教程
　　通过dockercompose快速部署Hadoop集群极简教程
　　最好是先浏览一下Hadoop部署的文章，如果不care详细过程，就可以只查阅通过dockercompose快速部署Hadoop集群极简教程这篇文章即可
　　关于Hive的介绍可以查阅我以下文章：大数据Hadoop之数据仓库Hive
　　二、前期准备1）部署docker安装yumconfigmanager配置工具yumyinstallyumutils建议使用阿里云yum源：（推荐）yumconfigmanageraddrepohttps：download。docker。comlinuxcentosdockerce。repoyumconfigmanageraddrepohttp：mirrors。aliyun。comdockercelinuxcentosdockerce。repo安装dockerce版本yuminstallydockerce启动并开机启动systemctlenablenowdockerdockerversion2）部署dockercomposecurlSLhttps：github。comdockercomposereleasesdownloadv2。16。0dockercomposelinuxx8664ousrlocalbindockercomposechmodxusrlocalbindockercomposedockercomposeversion三、创建网络创建，注意不能使用hadoopnetwork，要不然启动hs2服务的时候会有问题！！！dockernetworkcreatehadoopnetwork查看dockernetworkls四、MySQL部署1）mysql镜像dockerpullmysql：5。7dockertagmysql：5。7registry。cnhangzhou。aliyuncs。combigdatacloudnativemysql：5。72）配置mkdirpconfdatadbcatconfmy。cnfEOF〔mysqld〕charactersetserverutf8logbinmysqlbinserverid1pidfilevarrunmysqldmysqld。pidsocketvarrunmysqldmysqld。sockdatadirvarlibmysqlsqlmodeSTRICTTRANSTABLES，NOZEROINDATE，NOZERODATE，ERRORFORDIVISIONBYZERO，NOAUTOCREATEUSER，NOENGINESUBSTITUTIONsymboliclinks0securefileprivwaittimeout120interactivetimeout120defaulttimezone8：00skipexternallockingskipnameresolveopenfileslimit10240maxconnections1000maxconnecterrors6000tableopencache800maxallowedpacket40msortbuffersize2Mjoinbuffersize1Mthreadcachesize32querycachesize64MtransactionisolationREADCOMMITTEDtmptablesize128Mmaxheaptablesize128Mlogbinmysqlbinsyncbinlog1binlogformatROWbinlogcachesize1Mkeybuffersize128Mreadbuffersize2Mreadrndbuffersize4Mbulkinsertbuffersize64Mlowercasetablenames1explicitdefaultsfortimestamptrueskipnameresolveONeventschedulerONlogbintrustfunctioncreators1innodbbufferpoolsize512Minnodbflushlogattrxcommit1innodbfilepertable1innodblogbuffersize4Minnodblogfilesize256Minnodbmaxdirtypagespct90innodbreadiothreads4innodbwriteiothreads4EOF3）编排version：3services：db：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativemysql：5。7mysql版本containername：mysqlhostname：mysqlvolumes：。datadb：varlibmysql。confmy。cnf：etcmysqlmysql。conf。dmysqld。cnfrestart：alwaysports：13306：3306networks：hadoopnetworkenvironment：MYSQLROOTPASSWORD：123456访问密码securefilepriv：healthcheck：test：〔CMDSHELL，curlIlocalhost：3306exit1〕interval：10stimeout：5sretries：3连接外部网络networks：hadoopnetwork：external：true4）部署mysqldockercomposefmysqlcompose。yamlupddockercomposefmysqlcompose。yamlps登录容器mysqlurootp123456
　　四、Hive部署1）下载hive
　　下载地址：http：archive。apache。orgdisthive下载wgethttp：archive。apache。orgdisthivehive3。1。3apachehive3。1。3bin。tar。gz解压tarzxvfapachehive3。1。3bin。tar。gz2）配置
　　imageshiveconfighivesite。xmllt；？xmlversion1。0？lt；？xmlstylesheettypetextxslhrefconfiguration。xsl？configuration！配置hdfs存储目录propertynamehive。metastore。warehouse。dirnamevalueuserhiveremotewarehousevaluepropertypropertynamehive。metastore。localnamevaluefalsevalueproperty！所连接的MySQL数据库的地址，hivelocal是数据库，程序会自动创建，自定义就行propertynamejavax。jdo。option。ConnectionURLnamevaluejdbc：mysql：mysql：3306hivemetastore？createDatabaseIfNotExisttrueuseSSLfalseserverTimezoneAsiaShanghaivalueproperty！MySQL驱动propertynamejavax。jdo。option。ConnectionDriverNamename！valuecom。mysql。cj。jdbc。Drivervaluevaluecom。mysql。jdbc。Drivervalueproperty！mysql连接用户propertynamejavax。jdo。option。ConnectionUserNamenamevaluerootvalueproperty！mysql连接密码propertynamejavax。jdo。option。ConnectionPasswordnamevalue123456valueproperty！元数据是否校验propertynamehive。metastore。schema。verificationnamevaluefalsevaluepropertypropertynamesystem：user。namenamevaluerootvaluedescriptionusernamedescriptionpropertypropertynamehive。metastore。urisnamevaluethrift：hivemetastore：9083valueproperty！hostpropertynamehive。server2。thrift。bind。hostnamevalue0。0。0。0valuedescriptionBindhostonwhichtoruntheHiveServer2Thriftservice。descriptionproperty！hs2端口默认是10000propertynamehive。server2。thrift。portnamevalue10000valuepropertypropertynamehive。server2。active。passive。ha。enablenamevaluetruevaluepropertyconfiguration3）启动脚本！usrbinenvshwaitfor（）｛echoWaitingfor1tolistenon2。。。while！ncz12；doechowaiting。。。；sleep1s；done｝starthdfsnamenode（）｛if〔！ftmpnamenodeformated〕；then｛HADOOPHOME｝binhdfsnamenodeformattmpnamenodeformatedfi｛HADOOPHOME｝binhdfsloglevelINFOdaemonstartnamenodetailf｛HADOOPHOME｝logsnamenode。log｝starthdfsdatanode（）｛waitfor12｛HADOOPHOME｝binhdfsloglevelINFOdaemonstartdatanodetailf｛HADOOPHOME｝logsdatanode。log｝startyarnresourcemanager（）｛｛HADOOPHOME｝binyarnloglevelINFOdaemonstartresourcemanagertailf｛HADOOPHOME｝logsresourcemanager。log｝startyarnnodemanager（）｛waitfor12｛HADOOPHOME｝binyarnloglevelINFOdaemonstartnodemanagertailf｛HADOOPHOME｝logsnodemanager。log｝startyarnproxyserver（）｛waitfor12｛HADOOPHOME｝binyarnloglevelINFOdaemonstartproxyservertailf｛HADOOPHOME｝logsproxyserver。log｝startmrhistoryserver（）｛waitfor12｛HADOOPHOME｝binmapredloglevelINFOdaemonstarthistoryservertailf｛HADOOPHOME｝logshistoryserver。log｝starthivemetastore（）｛if〔！f｛HIVEHOME｝formated〕；thenschematoolinitSchemadbTypemysqlverbose｛HIVEHOME｝formatedfiHIVEHOMEbinhiveservicemetastore｝starthivehiveserver2（）｛HIVEHOMEbinhiveservicehiveserver2｝case1inhadoophdfsnn）starthdfsnamenode；；hadoophdfsdn）starthdfsdatanode23；；hadoopyarnrm）startyarnresourcemanager；；hadoopyarnnm）startyarnnodemanager23；；hadoopyarnproxyserver）startyarnproxyserver23；；hadoopmrhistoryserver）startmrhistoryserver23；；hivemetastore）starthivemetastore23；；hivehiveserver2）starthivehiveserver223；；）echo请输入正确的服务启动命令；；esac4）构建镜像DockerfileFROMregistry。cnhangzhou。aliyuncs。combigdatacloudnativehadoop：v1COPYhiveconfig｛HIVEHOME｝confCOPYbootstrap。shoptapacheCOPYmysqlconnectorjava5。1。49mysqlconnectorjava5。1。49bin。jar｛HIVEHOME｝libRUNsudomkdirphomehadoopsudochownRhadoop：hadoophomehadoopRUNyumyinstallwhich
　　开始构建镜像构建镜像dockerbuildtregistry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1。nocache推送镜像（可选）dockerpushregistry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1参数解释t：指定镜像名称。：当前目录Dockerfilef：指定Dockerfile路径nocache：不缓存5）编排version：3services：hadoophdfsnn：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoophdfsnnhostname：hadoophdfsnnrestart：alwaysprivileged：trueenvfile：。envports：30070：｛HADOOPHDFSNNPORT｝command：〔sh，c，optapachebootstrap。shhadoophdfsnn〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPHDFSNNPORT｝exit1〕interval：20stimeout：20sretries：3hadoophdfsdn0：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoophdfsdn0hostname：hadoophdfsdn0restart：alwaysdependson：hadoophdfsnnenvfile：。envports：30864：｛HADOOPHDFSDNPORT｝command：〔sh，c，optapachebootstrap。shhadoophdfsdnhadoophdfsnn｛HADOOPHDFSNNPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPHDFSDNPORT｝exit1〕interval：30stimeout：30sretries：3hadoophdfsdn1：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoophdfsdn1hostname：hadoophdfsdn1restart：alwaysdependson：hadoophdfsnnenvfile：。envports：30865：｛HADOOPHDFSDNPORT｝command：〔sh，c，optapachebootstrap。shhadoophdfsdnhadoophdfsnn｛HADOOPHDFSNNPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPHDFSDNPORT｝exit1〕interval：30stimeout：30sretries：3hadoophdfsdn2：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoophdfsdn2hostname：hadoophdfsdn2restart：alwaysdependson：hadoophdfsnnenvfile：。envports：30866：｛HADOOPHDFSDNPORT｝command：〔sh，c，optapachebootstrap。shhadoophdfsdnhadoophdfsnn｛HADOOPHDFSNNPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPHDFSDNPORT｝exit1〕interval：30stimeout：30sretries：3hadoopyarnrm：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopyarnrmhostname：hadoopyarnrmrestart：alwaysenvfile：。envports：30888：｛HADOOPYARNRMPORT｝command：〔sh，c，optapachebootstrap。shhadoopyarnrm〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，netstattnlpgrep：｛HADOOPYARNRMPORT｝exit1〕interval：20stimeout：20sretries：3hadoopyarnnm0：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopyarnnm0hostname：hadoopyarnnm0restart：alwaysdependson：hadoopyarnrmenvfile：。envports：30042：｛HADOOPYARNNMPORT｝command：〔sh，c，optapachebootstrap。shhadoopyarnnmhadoopyarnrm｛HADOOPYARNRMPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPYARNNMPORT｝exit1〕interval：30stimeout：30sretries：3hadoopyarnnm1：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopyarnnm1hostname：hadoopyarnnm1restart：alwaysdependson：hadoopyarnrmenvfile：。envports：30043：｛HADOOPYARNNMPORT｝command：〔sh，c，optapachebootstrap。shhadoopyarnnmhadoopyarnrm｛HADOOPYARNRMPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPYARNNMPORT｝exit1〕interval：30stimeout：30sretries：3hadoopyarnnm2：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopyarnnm2hostname：hadoopyarnnm2restart：alwaysdependson：hadoopyarnrmenvfile：。envports：30044：｛HADOOPYARNNMPORT｝command：〔sh，c，optapachebootstrap。shhadoopyarnnmhadoopyarnrm｛HADOOPYARNRMPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，curlfailhttp：localhost：｛HADOOPYARNNMPORT｝exit1〕interval：30stimeout：30sretries：3hadoopyarnproxyserver：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopyarnproxyserverhostname：hadoopyarnproxyserverrestart：alwaysdependson：hadoopyarnrmenvfile：。envports：30911：｛HADOOPYARNPROXYSERVERPORT｝command：〔sh，c，optapachebootstrap。shhadoopyarnproxyserverhadoopyarnrm｛HADOOPYARNRMPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，netstattnlpgrep：｛HADOOPYARNPROXYSERVERPORT｝exit1〕interval：30stimeout：30sretries：3hadoopmrhistoryserver：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hadoopmrhistoryserverhostname：hadoopmrhistoryserverrestart：alwaysdependson：hadoopyarnrmenvfile：。envports：31988：｛HADOOPMRHISTORYSERVERPORT｝command：〔sh，c，optapachebootstrap。shhadoopmrhistoryserverhadoopyarnrm｛HADOOPYARNRMPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，netstattnlpgrep：｛HADOOPMRHISTORYSERVERPORT｝exit1〕interval：30stimeout：30sretries：3hivemetastore：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hivemetastorehostname：hivemetastorerestart：alwaysdependson：hadoophdfsdn2envfile：。envports：30983：｛HIVEMETASTOREPORT｝command：〔sh，c，optapachebootstrap。shhivemetastorehadoophdfsdn2｛HADOOPHDFSDNPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，netstattnlpgrep：｛HIVEMETASTOREPORT｝exit1〕interval：30stimeout：30sretries：5hivehiveserver2：image：registry。cnhangzhou。aliyuncs。combigdatacloudnativehadoophive：v1user：hadoop：hadoopcontainername：hivehiveserver2hostname：hivehiveserver2restart：alwaysdependson：hivemetastoreenvfile：。envports：31000：｛HIVEHIVESERVER2PORT｝command：〔sh，c，optapachebootstrap。shhivehiveserver2hivemetastore｛HIVEMETASTOREPORT｝〕networks：hadoopnetworkhealthcheck：test：〔CMDSHELL，netstattnlpgrep：｛HIVEHIVESERVER2PORT｝exit1〕interval：30stimeout：30sretries：5连接外部网络networks：hadoopnetwork：external：true6）开始部署dockercomposefdockercompose。yamlupd查看dockercomposefdockercompose。yamlps
　　简单测试验证
　　【问题】如果出现以下类似的错误，是因为多次启动，之前的数据还在，但是datanode的IP是已经变了的（宿主机部署就不会有这样的问题，因为宿主机的IP是固定的），所以需要刷新节点，当然也可清理之前的旧数据，不推荐清理旧数据，推荐使用刷新节点的方式（如果有对外挂载的情况下，像我这里没有对外挂载，是因为之前旧容器还在，下面有几种解决方案）：
　　org。apache。hadoop。ipc。RemoteException（org。apache。hadoop。hdfs。server。protocol。DisallowedDatanodeException）：Datanodedeniedcommunicationwithnamenodebecausethehostisnotintheincludelist：DatanodeRegistration（172。30。0。12：9866，datanodeUuidf81884764a884cd6836f769d510929e4，infoPort9864，infoSecurePort0，ipcPort9867，storageInfolv57；cidCIDf998d368222c4a9a88a585497a82dcac；nsid1840040096；c1680661390829）
　　【解决方案】删除旧容器重启启动清理旧容器dockerrmdockerpsagrepExitedawk｛print1｝重启启动服务dockercomposefdockercompose。yamlupd查看dockercomposefdockercompose。yamlps登录namenode刷新datanodedockerexecithadoophdfsnnhdfsdfsadminrefreshNodes登录任意节点刷新datanode这里以hadoophdfsdn0为例dockerexecithadoophdfsdn0hdfsdfsadminfshdfs：hadoophdfsnn：9000refreshNodes
　　到此，Hive的容器化部署就完成了，有任何疑问的小伙伴欢迎给我留言，后续会持续更新相关技术文章，也可关注我的公众号【大数据与云原生技术分享】深入交流技术或私信咨询问题