OpsCenter - DSE Cassandra - 无法达到一致性级别 LOCAL_ONE

OpsCenter - DSE Cassandra - 无法达到一致性级别 LOCAL_ONE

我在本地安装了 DSE Cassandra 的 OpsCenter 节点上遇到了此错误。它位于自己的 DC 中,并且与另一个 DC 中的其他三个节点一起属于一个集群。

使用 org.apache.cassandra.auth.PasswordAuthenticator 和 org.apache.cassandra.auth.CassandraAuthorizer 在集群上启用身份验证。system_auth 表在集群中的所有节点上复制,使用简单复制策略,复制因子为 4。

[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL 规范 3.2.0 | 本机协议 v3]

从 OpsCenter 节点上的 Cassandra system.log:

WARN  [MessagingService-Incoming-/192.168.33.152] 2015-05-29 16:52:42,646  IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Got range slice command for nonexistent table dse_system.real_leaders.  If the table was just created, this is likely due to the schema not being fully propagated.  Please wait for schema agreement on table creation.
        at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:193) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:151) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]






INFO  [MemtableFlushWriter:3] 2015-05-29 16:53:49,817  Memtable.java:380 - Completed flushing /usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db (164 bytes) for commitlog position ReplayPosition(segmentId=1432918359985, position=2480593)
INFO  [CompactionExecutor:3] 2015-05-29 16:53:49,818  CompactionTask.java:140 - Compacting [SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-18-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-17-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-19-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db')]
INFO  [CompactionExecutor:3] 2015-05-29 16:53:49,908  CompactionTask.java:270 - Compacted 4 sstables to [/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-21,].  1,556 bytes to 498 (~32% of original) in 88ms = 0.005397MB/s.  23 total partitions merged to 8.  Partition merge counts were {1:1, 3:6, 4:1, }
WARN  [pool-5-thread-1] 2015-05-29 16:54:39,512  LeaderManagerWatcher.java:379 - OH GOD
java.lang.NullPointerException: null
        at com.datastax.bdp.util.LeaderManagerWatcher.checkNetwork(LeaderManagerWatcher.java:223) ~[dse-4.7.0.jar:4.7.0]
        at com.datastax.bdp.util.LeaderManagerWatcher.getNextView(LeaderManagerWatcher.java:456) ~[dse-4.7.0.jar:4.7.0]
        at com.datastax.bdp.util.LeaderManagerWatcher.run(LeaderManagerWatcher.java:326) ~[dse-4.7.0.jar:4.7.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
WARN  [pool-5-thread-1] 2015-05-29 16:54:44,526  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
INFO  [pool-5-thread-1] 2015-05-29 16:54:44,527  LeaderManagerWatcher.java:290 - /192.168.33.158: Leader HadoopJT.DC_ANALYTICS_1 changed from /192.168.33.156 to <null> [unregistered] [notified 0 listeners]
WARN  [pool-5-thread-1] 2015-05-29 16:54:49,529  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:54:54,532  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:54:59,534  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:04,536  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:09,538  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:14,539  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE

集群详细信息:

nodetool status
Datacenter: DC_DATA_1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID                               Rack
UN  192.168.33.151  854.95 KB  256     ?       2af17bd4-e454-4cd0-8621-f42481f52398  RAC1
UN  192.168.33.153  857.32 KB  256     ?       712a892a-44f0-46fd-b28b-04eae418ebde  RAC1
UN  192.168.33.152  870.35 KB  256     ?       ce08a377-8b71-4789-beae-e80a70b3b685  RAC1
Datacenter: DC_OPSCENTER_1
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID                               Rack
UN  192.168.33.158  512.37 KB  256     ?       22468c08-cd3e-4fc2-8ff8-fa4aeff12caa  RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless




[root@cassandra-opscenter-1 ~]# cqlsh -u cassandra_admin -p password 192.168.33.151 9042
Connected to Cluster1 at 192.168.33.151:9042.
[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra_admin@cqlsh> select * from system.schema_keyspaces;

 keyspace_name   | durable_writes | strategy_class                                       | strategy_options
-----------------+----------------+------------------------------------------------------+----------------------------
     system_auth |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"4"}
      dse_system |           True |      org.apache.cassandra.locator.EverywhereStrategy |                         {}
       OpsCenter |           True | org.apache.cassandra.locator.NetworkTopologyStrategy |     {"DC_OPSCENTER_1":"1"}
          system |           True |           org.apache.cassandra.locator.LocalStrategy |                         {}
 sample_keyspace |          False |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}
   system_traces |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}

(6 rows)
cassandra_admin@cqlsh> desc keyspaces;

sample_keyspace  system_auth  system  "OpsCenter"  system_traces  dse_system

相关内容