☰

服务器反复崩溃（错误 500）（在 EC2 实例上运行 nginx + gunicorn）

tag-icon • tag-icon nginx mysql python gunicorn 500-error
服务器反复崩溃（错误 500）（在 EC2 实例上运行 nginx + gunicorn）

好吧，我真的很困惑这里发生了什么。我的服务器已经运行了大约一个月，一切都很好。服务器的流量很少（每天大约 50-100 个请求），所以我认为这应该是我最不担心的问题。然后，有一天我注意到服务器给出了内部错误 500 输出。我刚刚重启了 Gunicorn 和 Nginx，一切都很好。现在，大约一周后，同样的问题出现了。我不明白发生了什么。

在某些情况下，我正在使用 Gunicorn 运行 Python 应用程序，并将套接字传递给 Nginx。作为应用程序的一部分，我使用 pymysql 作为全局变量初始化了一个数据库连接，以避免为每个请求重新打开数据库连接（我知道这对于每天几十个用户来说不是问题，但我试图让这个解决方案推广到更高的数量）。

这是上次成功的 gunicorn 记录消息以及第一条错误消息的输出（该消息随后在后续的每个请求中都会重复出现）。似乎发生的事情是数据库连接变量以某种方式被删除了，但我不明白为什么会这样，或者为什么这种情况偶尔会发生。

[] (sys) Ready to run...
[2020-03-31 19:21:03 +0000] [8162] [INFO] Starting gunicorn 20.0.4
[2020-03-31 19:21:03 +0000] [8162] [INFO] Listening at: unix:APP_NAME.sock (8162)
[2020-03-31 19:21:03 +0000] [8162] [INFO] Using worker: sync
[2020-03-31 19:21:03 +0000] [8166] [INFO] Booting worker with pid: 8166
[2020-03-31 19:21:03 +0000] [8167] [INFO] Booting worker with pid: 8167
[2020-03-31 19:21:03 +0000] [8169] [INFO] Booting worker with pid: 8169
[2020-04-12 04:16:06 +0000] [8169] [INFO] Worker exiting (pid: 8169)
Loaded
[2020-04-12 04:16:06 +0000] [8167] [INFO] Worker exiting (pid: 8167)
Loaded
[2020-04-12 04:16:06 +0000] [8166] [INFO] Worker exiting (pid: 8166)
Loaded
[2020-04-12 04:16:06 +0000] [8162] [INFO] Handling signal: term
[2020-04-12 04:16:06 +0000] [8162] [INFO] Shutting down: Master    
[] (sys) Starting
[] (sys) Ready to run...
[2020-04-12 04:16:14 +0000] [29732] [INFO] Starting gunicorn 20.0.4
[2020-04-12 04:16:14 +0000] [29732] [INFO] Listening at: unix:APP_NAME.sock (29732)
[2020-04-12 04:16:14 +0000] [29732] [INFO] Using worker: sync
[2020-04-12 04:16:14 +0000] [29736] [INFO] Booting worker with pid: 29736
[2020-04-12 04:16:14 +0000] [29737] [INFO] Booting worker with pid: 29737
[2020-04-12 04:16:14 +0000] [29738] [INFO] Booting worker with pid: 29738
[2020-04-19 00:41:18,150] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ec2-user/APP_NAME/APP_NAME.py", line 26, in main_page
    add_record("index")
  File "/home/ec2-user/APP_NAME/APP_NAME.py", line 48, in add_record
    db.conn.cursor().execute(qstr)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
[2020-04-19 00:41:18,882] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ec2-user/APP_NAME/APP_NAME.py", line 26, in main_page
    add_record("index")
  File "/home/ec2-user/APP_NAME/APP_NAME.py", line 48, in add_record
    db.conn.cursor().execute(qstr)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 516, in query
    self._execute_command(COMMAND.COM_QUERY, sql)
  File "/home/ec2-user/APP_NAME/lib64/python3.6/site-packages/pymysql/connections.py", line 750, in _execute_command
    raise err.InterfaceError("(0, '')")
pymysql.err.InterfaceError: (0, '')
[2020-04-19 04:15:22,832] ERROR in app: Exception on / [GET]

以下是相关的 Python 代码：

from flask import Flask, redirect, render_template, url_for, request
import pymysql
from datetime import datetime
import os, random
app = Flask(__name__)
db = None

class db_connector():
    def __init__(self):
        self.conn = pymysql.connect(<redacted>)

db = db_connector()

@app.route('/')
@app.route('/index.html')
def main_page():
    add_record("index")
    return render_template("index.html")

def add_record(page_name):
   ip = request.environ.get('HTTP_X_REAL_IP', request.remote_addr)
   dtstr = datetime.strftime(datetime.now(), "%Y-%m-%d %H:%M:%S")
   qstr = "<SQL insert string redacted>"
   db.conn.cursor().execute(qstr)

相关内容