PuppetDB：无法提交“替换事实”命令

Question

我最近撤销/清理了 Puppet 代理证书，这似乎对 PuppetDB 产生了负面影响。我看到有人提交了一个错误这里并提供了修复该问题的一些说明。一位用户遇到了类似的问题这里，但这些对我都不起作用。

服务器运行的是 CentOS 6.2、Puppet 2.7.13 和 Puppet DB 0.9。错误如下：

root@harp:/etc/puppetdb/ssl> puppet agent --test
err: Cached facts for harp failed: Failed to find facts from PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
info: Loading facts in /etc/puppet/modules/dns/lib/facter/datacenter.rb
info: Caching facts for harp
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
err: Could not run Puppet configuration client: Could not retrieve local facts: Failed to submit 'replace facts' command for harp to PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

据我所见，NTP 运行正常，日期时间看起来也不错。“harp”实际上是 puppet master 服务器，所以这里的代理和服务器之间不应该存在时间问题，因为它们是相同的。

旧证书：

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (DF:8F:65:36:58:4C:DE:66:2B:65:D1:E6:18:B7:F2:33)

清理并为代理生成新的证书：

root@harp:/etc/puppetdb/ssl> puppet cert clean harp
notice: Revoked certificate with serial 18
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/ca/signed/harp.pem'
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/certs/harp.pem'
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/certificate_requests/harp.pem'
notice: Removing file Puppet::SSL::Key harp at '/var/lib/puppet/ssl/private_keys/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet agent --test
info: Creating a new SSL key for harp
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for harp
info: Certificate Request fingerprint (md5): 72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

root@harp:/etc/puppetdb/ssl> puppet cert list
  harp (72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD)

root@harp:/etc/puppetdb/ssl> puppet cert sign harp
notice: Signed certificate request for harp
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/ca/requests/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79)

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb: /etc/init.d/puppetdb: line 77: kill: (8623) - No such process
                                                           [FAILED]
Starting puppetdb:                                         [  OK  ]

好的，请重新启动以确保万无一失：

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb:                                         [  OK  ]
Starting puppetdb:                                         [  OK  ]

跑过SSL 配置脚本

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
cp: cannot stat `/var/lib/puppet/ssl/certs/harp.pem': No such file or directory

root@harp:/etc/puppetdb/ssl> ls -la /var/lib/puppet/ssl/certs
total 12
drwxr-xr-x 2 puppet root 4096 Jun 19 07:19 ./
drwxrwx--x 8 puppet root 4096 Apr 24 10:04 ../
-rw-r--r-- 1 puppet root 1854 Apr 24 10:04 ca.pem

好的，请再试一次：

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
Certificate was added to keystore
Usage: pkcs12 [options]
where options are
-export       output PKCS12 file
-chain        add certificate chain
-inkey file   private key if not infile
-certfile f   add all certs in f
-CApath arg   - PEM format directory of CA's
-CAfile arg   - PEM format file of CA's
-name "name"  use name as friendly name
-caname "nm"  use nm as CA friendly name (can be used more than once).
-in  infile   input filename
...snip...
-CSP name     Microsoft CSP name
-LMK          Add local machine keyset attribute to private key

/etc/puppetdb/ssl 中的密钥库似乎没有更改/重新生成。此时，运行puppet agent --test会导致相同的错误，重新启动 puppet 和 puppetdb 也无济于事。

密钥库信息：

root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

harp.mydomain.com, May 25, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 06:A8:D3:2A:70:F3:6D:34:62:91:45:22:8A:C4:A8:86
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

puppetdb ca, May 25, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp.mydomain.com
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
err: Could not call fingerprint: Could not find a certificate or csr for harp.mydomain.com

root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
harp 4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79

我怎样才能真正重新生成 puppetdb 密钥库？我尝试删除 /etc/puppetdb/ssl/ 中的文件，但没有成功。

Answer 1

我已经开始了，但无法确切地说出哪些步骤是必要的，哪些步骤不是必要的。

这个问题的出现是因为多个主机上的身份验证速度很慢或挂起，似乎与域控制器/DNS 缓存问题有关。domain mydomain.com从 Puppet Master 和 Agent 上删除条目/etc/resolv.conf解决了这个问题，但这会给现有的 Puppet 证书带来问题。我puppet cert clean --all在 Master 上运行并尝试重新创建所有证书，但这与 PuppetDB 配合不佳。

解决方案

清除主服务器上的旧证书：

puppet cert clean --all

清除所有代理上的旧证书：

rm -rf /var/lib/puppet/ssl

重新创建 PuppetDB 密钥库：

facter fqdndomain foo.com从删除后不可用/etc/resolv.conf。这会导致puppetdb-ssl-setup静默失败。

编辑/usr/sbin/puppetdb-ssl-setup，添加一段代码以在为空facter hostname时使用facter fqdn：

# near line 10
fqdn=`facter fqdn`
# add this "if" section
if [ ! -n "$fqdn" ] ; then
  fqdn=`facter hostname`
fi

权限修复：

chown -R puppetdb:puppetdb /etc/puppetdb/ssl

使用新的密钥库/信任库密码（相同密码）更新 /etc/puppetdb/conf.d/jetty.ini 中的密码，您可以从以下位置获取：

cat /etc/puppetdb/ssl/puppetdb_keystore_pw.txt

重新启动puppetdb

service puppetdb restart

然后去每个代理并请求新的证书并在主服务器上签署每个证书。

Answer 2

当 puppetdb 的内存设置太低时，也会发生这种情况。

vim /etc/default/puppetdb

编辑行

JAVA_ARGS="-Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

应该成为

JAVA_ARGS="-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

并重新启动 puppetdb

sudo service puppetdb restart

Answer 3

遇到了类似的问题。解决方案：

1.) 删除主机上的 pe-puppetdb pid 文件 2.) 停止主机上的 pe-puppetdb 服务 3.) 启动主机上的 pe-puppetdb 服务等待 30 秒。

Answer 4

将 puppet master（包括 puppetdb 从 1.6.3 到 2.3.8）从 3.7.x 升级到 3.8.x 后，我遇到了类似的问题，并收到以下错误消息：

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for puppet-client to PuppetDB at puppetmaster:8081: Connection refused - connect(2)

解决方法是，一方面重启 puppetdb，另一方面重启 puppet 代理客户端。之后代理就可以继续工作了。