我们有一个在 WebSphere 上运行的应用程序(不幸的是它是 6.1,不再受支持,它还没有在生产中迁移到更高版本),由于线程挂起,它变得完全没有响应。
据我所知,我们已经完全耗尽了一个线程池。
我已经激活了挂起线程检测,并且当检测到挂起线程时,我会得到核心/线程转储。
该服务器可以连续运行几天而不会出现问题,但本周已崩溃两次。
在“IBM Thread and Monitor Dump Analyzer for Java”中加载核心/线程转储时,它告诉我有一定数量的挂起线程(这次是 2 个,上次是 11 个)和多个(通常大约 40 个)线程“等待条件”以及一些正在运行的线程。我相信其中一个线程池的大小大约是这个数(50)。
现在我看到的是正在等待锁、已经拥有锁或处于等待状态的线程。
它们中的大多数都显示一个总是像这样结束的堆栈轨迹:
在 java/lang/Object.wait(本机方法)在 java/lang/Object.wait(Object.java:231)
现在,我该如何将其追溯到服务器配置问题、应用程序问题、WebSphere 问题还是其他问题?当其中几乎所有内容都涉及 IBM 代码时,这如何帮助我追踪问题?
我无法寻求 IBM 的帮助,因为 6.1 现在是不受支持的 WebSphere 版本,尽管已经完成了使其在 WebSphere 7 下运行的工作,但我们尚未准备好在生产中切换到它。
这些是被阻塞的线程的堆栈跟踪:
HAManager.thread.pool : 0
Blocked
Waiting for Monitor Lock on java/lang/Object@12648F78/12648F84
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:203)
at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:564)
at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:869)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1617)
WebContainer : 5
Blocked
Waiting for Monitor Lock on com/ibm/ejs/ras/TraceLogger@11265878/11265884
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:299)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.debug(Tr.java:610)
at com/ibm/ws/rsadapter/spi/WSRdbManagedConnectionImpl.inGlobalTransaction(WSRdbManagedConnectionImpl.java:1045)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.commit(WSJdbcConnection.java:961)
...
“等待条件”的线程包括以下几种:
at java/lang/Thread.sleep(Native Method)
at java/lang/Thread.sleep(Thread.java:941)
....
还有很多
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
...
我有一个非常奇怪的问题,它为什么要等这个?:
at java/util/Calendar.setFieldsComputed(Calendar.java:1402)
at java/util/Calendar.complete(Calendar.java:1349)
at java/util/Calendar.get(Calendar.java:1120)
at java/text/SimpleDateFormat.subFormat(SimpleDateFormat.java:930)
at java/text/SimpleDateFormat.format(SimpleDateFormat.java:837)
at java/text/SimpleDateFormat.format(SimpleDateFormat.java:809)
at com/ibm/ws/logging/TraceLogFormatter.formatTimeAndThread(TraceLogFormatter.java:125)
at com/ibm/ws/logging/TraceLogFormatter.formatHeaderBasic(TraceLogFormatter.java:223)
at com/ibm/ws/logging/TraceLogFormatter.formatBasic(TraceLogFormatter.java:439)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:295)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
...
我也有这个:
WebContainer : 6
Owns Monitor Lock on com/ibm/ejs/ras/TraceLogger@11265878/11265884
at sun/io/CharToByteSingleByte.convert(CharToByteSingleByte.java:329)
at com/ibm/misc/IOConverter.convert(IOConverter.java:111)
at sun/io/CharToByteConverter.convertAny(CharToByteConverter.java:191)
at sun/nio/cs/StreamEncoder$ConverterSE.implWrite(StreamEncoder.java:259)
at sun/nio/cs/StreamEncoder.write(StreamEncoder.java:186)
at java/io/OutputStreamWriter.write(OutputStreamWriter.java:214)
at java/io/BufferedWriter.flushBuffer(BufferedWriter.java:132)
at java/io/PrintStream.write(PrintStream.java:492)
at java/io/PrintStream.print(PrintStream.java:636)
at java/io/PrintStream.println(PrintStream.java:773)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:300)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.entry(Tr.java:776)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2056)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2039)
...
最后是这个:
WebContainer : 8
at java/lang/StringBuffer.append(StringBuffer.java:239)
at com/ibm/ws/logging/TraceLogFormatter.formatTimeAndThread(TraceLogFormatter.java:137)
at com/ibm/ws/logging/TraceLogFormatter.formatHeaderBasic(TraceLogFormatter.java:223)
at com/ibm/ws/logging/TraceLogFormatter.formatBasic(TraceLogFormatter.java:439)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:295)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.exit(Tr.java:796)
at com/ibm/ejs/j2c/PoolManager.processPoolRequestStats(PoolManager.java:6879)
at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:2026)
at com/ibm/ejs/j2c/ConnectionManager.allocateMCWrapper(ConnectionManager.java:940)
at com/ibm/ejs/j2c/ConnectionManager.allocateConnection(ConnectionManager.java:614)
at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:449)
at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:418)
...