K8s监控方案kubestack（prometheusope

　　写在前面学习K8s监控涉及网上的教程大都不全或者有些旧，所以整理分享给小伙伴。博文内容为K8s集群通过helm方式创建kubeprometheusstack监控平台教程折腾了一晚上，搞定了，一开始一直用prometheusoperator这个chart来装，报错各种找问题，后来才发现我的集群版本太高了，1。22的版本，而且prometheusoperator之后的版本改变了名字kubeprometheusstack，旧的版本可能不兼容。
　　人生不尽美好，追求自我注定孤独，而这就是生命的意义黑塞《彼得卡门青》环境版本
　　我的K8s集群版本〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕kubectlgetnodesNAMESTATUSROLESAGEVERSIONvms81。liruilongs。github。ioReadycontrolplane，master34dv1。22。2vms82。liruilongs。github。ioReadynone34dv1。22。2vms83。liruilongs。github。ioReadynone34dv1。22。2
　　hrlm版本〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕helmversionversion。BuildInfo｛Version：v3。2。1，GitCommit：fe51cd1e31e6a202cba7dead9552a6d418ded79a，GitTreeState：clean，GoVersion：go1。13。10｝prometheusoperator（旧名字）安装出现的问题〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕helmsearchrepoprometheusoperatorNAMECHARTVERSIONAPPVERSIONDESCRIPTIONaliprometheusoperator8。7。00。35。0ProvideseasymonitoringdefinitionsforKubern。。。azureprometheusoperator9。3。20。38。1DEPRECATEDProvideseasymonitoringdefinitions。。。〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕helminstallliruilongaliprometheusoperatorError：failedtoinstallCRDcrdscrdalertmanager。yaml：unabletorecognize：nomatchesforkindCustomResourceDefinitioninversionapiextensions。k8s。iov1beta1〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕helmpullaliprometheusoperator〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕解决办法：新版本安装
　　直接下载kubeprometheusstack（新）的chart包，通过命令安装：
　　https：github。comprometheuscommunityhelmchartsreleasesdownloadkubeprometheusstack30。0。1kubeprometheusstack30。0。1。tgz〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕lsindex。yamlkubeprometheusstack30。0。1。tgzliruilonghelmliruilonghelm0。1。0。tgzmysqlmysql1。6。4。tgz〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕helmlistNAMENAMESPACEREVISIONUPDATEDSTATUSCHARTAPPVERSION
　　解压chart包kubeprometheusstack30。0。1。tgz〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕tarzxfkubeprometheusstack30。0。1。tgz
　　创建新的命名空间〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕cdkubeprometheusstack〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectlcreatensmonitoringnamespacemonitoringcreated〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectlconfigsetcontext（kubectlconfigcurrentcontext）namespacemonitoringContextkubernetesadminkubernetesmodified。
　　进入文件夹，直接通过helminstallliruilong。安装〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕lsChart。lockchartsChart。yamlCONTRIBUTING。mdcrdsREADME。mdtemplatesvalues。yaml〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕helminstallliruilong。
　　kubeprometheusadmissioncreate对应Pod的相关镜像下载不下来问题〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕kubectlgetpodsNAMEREADYSTATUSRESTARTSAGEliruilongkubeprometheusadmissioncreate1bn7x201ImagePullBackOff033s
　　查看pod详细信息，发现是谷歌的一个镜像国内无法下载〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕kubectldescribepodliruilongkubeprometheusadmissioncreate1bn7x2Name：liruilongkubeprometheusadmissioncreate1bn7x2Namespace：monitoringPriority：0Node：vms83。liruilongs。github。io192。168。26。83StartTime：Sun，16Jan202202：43：070800Labels：appkubeprometheusstackadmissioncreateapp。kubernetes。ioinstanceliruilongapp。kubernetes。iomanagedbyHelmapp。kubernetes。iopartofkubeprometheusstackapp。kubernetes。ioversion30。0。1chartkubeprometheusstack30。0。1controlleruid2ce48cd2a1184e23a27f0228ef6c45e7heritageHelmjobnameliruilongkubeprometheusadmissioncreatereleaseliruilongAnnotations：cni。projectcalico。orgpodIP：10。244。70。832cni。projectcalico。orgpodIPs：10。244。70。832Status：PendingIP：10。244。70。8IPs：IP：10。244。70。8ControlledBy：JobliruilongkubeprometheusadmissioncreateContainers：create：ContainerID：Image：k8s。gcr。ioingressnginxkubewebhookcertgen：v1。0sha256：f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068ImageID：Port：noneHostPort：
　　在dokcer仓库里找了一个类似的，通过kubectledit修改image：k8s。gcr。ioingressnginxkubewebhookcertgen：v1。0替换为：docker。ioliangjwkubewebhookcertgen：v1。1。1
　　或者也可以修改配置文件从新install（记得要把sha注释掉）〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕lsindex。yamlkubeprometheusstackkubeprometheusstack30。0。1。tgzliruilonghelmliruilonghelm0。1。0。tgzmysqlmysql1。6。4。tgz〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreate〕cdkubeprometheusstack〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕lsChart。lockchartsChart。yamlCONTRIBUTING。mdcrdsREADME。mdtemplatesvalues。yaml〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕catvalues。yamlgrepA3B2kubewebhookcertgenenabled：trueimage：repository：docker。ioliangjwkubewebhookcertgentag：v1。1。1sha：f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068pullPolicy：IfNotPresent〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕helmdelliruilong；helminstallliruilong。
　　之后其他的相关pod正常创建中〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectlgetpodsNAMEREADYSTATUSRESTARTSAGEliruilonggrafana5955564c75zpbjq03ContainerCreating027sliruilongkubeprometheusoperator5cb699b469fbkw501ContainerCreating027sliruilongkubestatemetrics5dcf758c47bbwt401ContainerCreating027sliruilongprometheusnodeexporterrfsc501ContainerCreating028sliruilongprometheusnodeexportervm7s901ContainerCreating028sliruilongprometheusnodeexporterz9j8b01ContainerCreating028s
　　kubestatemetrics这个pod的镜像也没有拉取下来。应该也是相同的原因〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectlgetpodsNAMEREADYSTATUSRESTARTSAGEalertmanagerliruilongkubeprometheusalertmanager022Running03m35sliruilonggrafana5955564c75zpbjq33Running04m46sliruilongkubeprometheusoperator5cb699b469fbkw511Running04m46sliruilongkubestatemetrics5dcf758c47bbwt401ImagePullBackOff04m46sliruilongprometheusnodeexporterrfsc511Running04m47sliruilongprometheusnodeexportervm7s911Running04m47sliruilongprometheusnodeexporterz9j8b11Running04m47sprometheusliruilongkubeprometheusprometheus022Running03m34s
　　同样k8s。gcr。iokubestatemetricskubestatemetrics：v2。3。0这个镜像没办法拉取〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectldescribepodliruilongkubestatemetrics5dcf758c47bbwt4Name：liruilongkubestatemetrics5dcf758c47bbwt4Namespace：monitoringPriority：0Node：vms82。liruilongs。github。io192。168。26。82StartTime：Sun，16Jan202202：59：530800Labels：app。kubernetes。iocomponentmetricsapp。kubernetes。ioinstanceliruilongapp。kubernetes。iomanagedbyHelmapp。kubernetes。ionamekubestatemetricsapp。kubernetes。iopartofkubestatemetricsapp。kubernetes。ioversion2。3。0helm。shchartkubestatemetrics4。3。0podtemplatehash5dcf758c47releaseliruilongAnnotations：cni。projectcalico。orgpodIP：10。244。171。15332cni。projectcalico。orgpodIPs：10。244。171。15332Status：PendingIP：10。244。171。153IPs：IP：10。244。171。153ControlledBy：ReplicaSetliruilongkubestatemetrics5dcf758c47Containers：kubestatemetrics：ContainerID：Image：k8s。gcr。iokubestatemetricskubestatemetrics：v2。3。0ImageID：Port：8080TCP
　　同样的，我们通过docker仓库找一下相同的，然后通过kubectleditpod修改一下k8s。gcr。iokubestatemetricskubestatemetrics替换为：docker。iodyrnqkubestatemetrics：v2。3。0
　　可以先在节点机上拉取一下〔rootvms81。liruilongs。github。io〕〔ansible〕ansiblenodemshelladockerpulldyrnqkubestatemetrics：v2。3。0192。168。26。82CHANGEDrc0v2。3。0：Pullingfromdyrnqkubestatemetricse8614d09b7be：Pullingfslayer53ccb90bafd7：Pullingfslayere8614d09b7be：VerifyingChecksume8614d09b7be：Downloadcompletee8614d09b7be：Pullcomplete53ccb90bafd7：VerifyingChecksum53ccb90bafd7：Downloadcomplete53ccb90bafd7：PullcompleteDigest：sha256：c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status：Downloadednewerimagefordyrnqkubestatemetrics：v2。3。0docker。iodyrnqkubestatemetrics：v2。3。0192。168。26。83CHANGEDrc0v2。3。0：Pullingfromdyrnqkubestatemetricse8614d09b7be：Pullingfslayer53ccb90bafd7：Pullingfslayere8614d09b7be：VerifyingChecksume8614d09b7be：Downloadcompletee8614d09b7be：Pullcomplete53ccb90bafd7：VerifyingChecksum53ccb90bafd7：Downloadcomplete53ccb90bafd7：PullcompleteDigest：sha256：c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status：Downloadednewerimagefordyrnqkubestatemetrics：v2。3。0docker。iodyrnqkubestatemetrics：v2。3。0
　　修改完之后，会发现所有的pod都创建成功〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectlgetpodsNAMEREADYSTATUSRESTARTSAGEalertmanagerliruilongkubeprometheusalertmanager022Running061mliruilonggrafana5955564c75zpbjq33Running062mliruilongkubeprometheusoperator5cb699b469fbkw511Running062mliruilongkubestatemetrics5dcf758c47bbwt411Running7（32mago）62mliruilongprometheusnodeexporterrfsc511Running062mliruilongprometheusnodeexportervm7s911Running062mliruilongprometheusnodeexporterz9j8b11Running062mprometheusliruilongkubeprometheusprometheus022Running061m〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕
　　然后我们需要修改liruilonggrafanaSVC的类型为NodePort，这样，物理机就可以访问了〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsvcNAMETYPECLUSTERIPEXTERNALIPPORT（S）AGEalertmanageroperatedClusterIPNonenone9093TCP，9094TCP，9094UDP33mliruilonggrafanaClusterIP10。99。220。121none80TCP34mliruilongkubeprometheusalertmanagerClusterIP10。97。193。228none9093TCP34mliruilongkubeprometheusoperatorClusterIP10。101。106。93none443TCP34mliruilongkubeprometheusprometheusClusterIP10。105。176。19none9090TCP34mliruilongkubestatemetricsClusterIP10。98。94。55none8080TCP34mliruilongprometheusnodeexporterClusterIP10。110。216。215none9100TCP34mprometheusoperatedClusterIPNonenone9090TCP33m〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstack〕kubectleditsvcliruilonggrafanaserviceliruilonggrafanaedited〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsvcNAMETYPECLUSTERIPEXTERNALIPPORT（S）AGEalertmanageroperatedClusterIPNonenone9093TCP，9094TCP，9094UDP35mliruilonggrafanaNodePort10。99。220。121none80：30443TCP36mliruilongkubeprometheusalertmanagerClusterIP10。97。193。228none9093TCP36mliruilongkubeprometheusoperatorClusterIP10。101。106。93none443TCP36mliruilongkubeprometheusprometheusClusterIP10。105。176。19none9090TCP36mliruilongkubestatemetricsClusterIP10。98。94。55none8080TCP36mliruilongprometheusnodeexporterClusterIP10。110。216。215none9100TCP36mprometheusoperatedClusterIPNonenone9090TCP35m
　　物理机访问
　　通过secrets解密获取用户名密码〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsecretsgrepgrafanaliruilonggrafanaOpaque338mliruilonggrafanatesttokenq8z8jkubernetes。ioserviceaccounttoken338mliruilonggrafanatokenj94p8kubernetes。ioserviceaccounttoken338m〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsecretsliruilonggrafanaoyamlapiVersion：v1data：adminpassword：cHJvbS1vcGVyYXRvcgadminuser：YWRtaW4ldaptoml：kind：Secretmetadata：annotations：meta。helm。shreleasename：liruilongmeta。helm。shreleasenamespace：monitoringcreationTimestamp：20220115T18：59：40Zlabels：app。kubernetes。ioinstance：liruilongapp。kubernetes。iomanagedby：Helmapp。kubernetes。ioname：grafanaapp。kubernetes。ioversion：8。3。3helm。shchart：grafana6。20。5name：liruilonggrafananamespace：monitoringresourceVersion：1105663uid：c03ff5f3deb5458c8583787f41034469type：Opaque〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsecretsliruilonggrafanaojsonpath｛。data。adminuser｝｝base64dadminbase64：输入无效〔rootvms81。liruilongs。github。io〕〔ansiblek8shelmcreatekubeprometheusstacktemplates〕kubectlgetsecretsliruilonggrafanaojsonpath｛。data。adminpassword｝｝base64dpromoperatorbase64：输入无效
　　得到用户名密码：adminpromoperator
　　正常登录，查看监控信息
　　生活加油