调试 Puppet 和 PuppetDB 之间的连接(在单独的容器中)

调试 Puppet 和 PuppetDB 之间的连接(在单独的容器中)

最近,托管我们的 Puppet 服务器的服务器瘫痪了。

重新部署容器后似乎出现 SSL 问题。

2018-01-16T14:36:49.770274413Z Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

2018-01-16T14:36:49.770278010Z  at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

2018-01-16T14:36:49.770281700Z  at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)

2018-01-16T14:36:49.770285230Z  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)

2018-01-16T14:36:49.770288860Z  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)

2018-01-16T14:36:49.770292535Z  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)

2018-01-16T14:36:49.770296037Z  at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)

2018-01-16T14:36:49.770299517Z  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)

2018-01-16T14:36:49.770303285Z  at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)

2018-01-16T14:36:49.770306850Z  at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)

2018-01-16T14:36:49.770310430Z  at java.security.AccessController.doPrivileged(Native Method)

2018-01-16T14:36:49.770314068Z  at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)

2018-01-16T14:36:49.770317603Z  at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)

2018-01-16T14:36:49.770321175Z  at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)

2018-01-16T14:36:49.770324797Z  ... 9 common frames omitted

2018-01-16T14:36:49.770328925Z Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed

2018-01-16T14:36:49.770336317Z  at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352)

2018-01-16T14:36:49.770340178Z  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)

2018-01-16T14:36:49.770344615Z  at sun.security.validator.Validator.validate(Validator.java:260)

2018-01-16T14:36:49.770350867Z  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)

2018-01-16T14:36:49.770355767Z  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)

2018-01-16T14:36:49.770359543Z  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)

2018-01-16T14:36:49.770363103Z  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)

2018-01-16T14:36:49.770366760Z  ... 17 common frames omitted

2018-01-16T14:36:49.770370253Z Caused by: java.security.cert.CertPathValidatorException: timestamp check failed

2018-01-16T14:36:49.770373823Z  at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)

2018-01-16T14:36:49.770377522Z  at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:219)

2018-01-16T14:36:49.770381140Z  at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)

2018-01-16T14:36:49.770384758Z  at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)

2018-01-16T14:36:49.770388458Z  at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)

2018-01-16T14:36:49.770392038Z  at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347)

2018-01-16T14:36:49.770395575Z  ... 23 common frames omitted

2018-01-16T14:36:49.770399060Z Caused by: java.security.cert.CertificateExpiredException: NotAfter: Mon Jan 15 18:12:18 UTC 2018

2018-01-16T14:36:49.770402708Z  at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274)

2018-01-16T14:36:49.770408587Z  at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629)

2018-01-16T14:36:49.770413647Z  at sun.security.provider.certpath.BasicChecker.verifyTimestamp(BasicChecker.java:190)

2018-01-16T14:36:49.770419840Z  at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)

2018-01-16T14:36:49.770429403Z  at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)

2018-01-16T14:36:49.770443412Z  ... 28 common frames omitted

2018-01-16T14:36:49.774570269Z 2018-01-16 14:36:49,774 WARN  [puppetserver] Puppet Error connecting to puppetdb on 8081 at route /pdb/cmd/v1?checksum=6a40b1127a0e8c1dee4fdd40cd45c9a9b4478dc6&version=8&certname=2klic-dev-596e89d2fe5e08410003f2e6&command=store_report&producer-timestamp=1516113409, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list

2018-01-16T14:36:49.776385101Z 2018-01-16 14:36:49,776 ERROR [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=6a40b1127a0e8c1dee4fdd40cd45c9a9b4478dc6&version=8&certname=2klic-dev-596e89d2fe5e08410003f2e6&command=store_report&producer-timestamp=1516113409' on at least 1 of the following 'server_urls': https://puppetdb:8081

2018-01-16T14:36:49.777516859Z 74.57.127.213 - - - 16/Jan/2018:14:36:49 +0000 "PUT /puppet/v3/report/2klic-dev-596e89d2fe5e08410003f2e6?environment=2klic_smart_controller_ws1_2_beta& HTTP/1.1" 200 12 74.57.127.213 74.57.127.213 8140 246

我删除了 puppetdb 作为节点,然后让它重新签入,SSL 过程顺利进行。但 PuppetDB 仍然没有记录签入数据。

我还查看了 PuppetDb 上的 ca 文件,检查它是否已过期:

openssl x509 -enddate -noout -in /etc/puppetlabs/puppetdb/ssl/ca

输出notAfter=Jun 25 20:16:09 2022 GMT

puppet 服务器 ca 上的类似输出:

openssl x509 -enddate -noout -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem

项目结构基于: https://github.com/puppetlabs/puppet-in-docker-examples/blob/master/compose/docker-compose.yml

答案1

我要检查的第一件事是您是否已通过 ntp/chrony 在所有服务器上同步时间,因为堆栈跟踪显示:“时间戳检查失败”

查看 openssl s_client 在尝试建立连接时返回的错误信息也可能会有所帮助:

openssl s_client -connect YOUR-PUPPET-DB-HOST:8081

相关内容