Я столкнулся с этой ошибкой на узле OpsCenter с локальной установкой DSE Cassandra. Он находится в собственном DC и является частью кластера с тремя другими узлами в другом DC.
В кластере включена аутентификация с помощью org.apache.cassandra.auth.PasswordAuthenticator и org.apache.cassandra.auth.CassandraAuthorizer. Таблица system_auth реплицируется на всех узлах кластера с простой стратегией репликации и коэффициентом репликации 4.
[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL spec 3.2.0 | Собственный протокол v3]
Из журнала Cassandra system.log на узле OpsCenter:
WARN [MessagingService-Incoming-/192.168.33.152] 2015-05-29 16:52:42,646 IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Got range slice command for nonexistent table dse_system.real_leaders. If the table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation.
at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:193) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:151) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
INFO [MemtableFlushWriter:3] 2015-05-29 16:53:49,817 Memtable.java:380 - Completed flushing /usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db (164 bytes) for commitlog position ReplayPosition(segmentId=1432918359985, position=2480593)
INFO [CompactionExecutor:3] 2015-05-29 16:53:49,818 CompactionTask.java:140 - Compacting [SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-18-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-17-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-19-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db')]
INFO [CompactionExecutor:3] 2015-05-29 16:53:49,908 CompactionTask.java:270 - Compacted 4 sstables to [/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-21,]. 1,556 bytes to 498 (~32% of original) in 88ms = 0.005397MB/s. 23 total partitions merged to 8. Partition merge counts were {1:1, 3:6, 4:1, }
WARN [pool-5-thread-1] 2015-05-29 16:54:39,512 LeaderManagerWatcher.java:379 - OH GOD
java.lang.NullPointerException: null
at com.datastax.bdp.util.LeaderManagerWatcher.checkNetwork(LeaderManagerWatcher.java:223) ~[dse-4.7.0.jar:4.7.0]
at com.datastax.bdp.util.LeaderManagerWatcher.getNextView(LeaderManagerWatcher.java:456) ~[dse-4.7.0.jar:4.7.0]
at com.datastax.bdp.util.LeaderManagerWatcher.run(LeaderManagerWatcher.java:326) ~[dse-4.7.0.jar:4.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_67]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_67]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
WARN [pool-5-thread-1] 2015-05-29 16:54:44,526 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
INFO [pool-5-thread-1] 2015-05-29 16:54:44,527 LeaderManagerWatcher.java:290 - /192.168.33.158: Leader HadoopJT.DC_ANALYTICS_1 changed from /192.168.33.156 to <null> [unregistered] [notified 0 listeners]
WARN [pool-5-thread-1] 2015-05-29 16:54:49,529 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN [pool-5-thread-1] 2015-05-29 16:54:54,532 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN [pool-5-thread-1] 2015-05-29 16:54:59,534 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN [pool-5-thread-1] 2015-05-29 16:55:04,536 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN [pool-5-thread-1] 2015-05-29 16:55:09,538 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN [pool-5-thread-1] 2015-05-29 16:55:14,539 LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up? - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
Детали кластера:
nodetool status
Datacenter: DC_DATA_1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.33.151 854.95 KB 256 ? 2af17bd4-e454-4cd0-8621-f42481f52398 RAC1
UN 192.168.33.153 857.32 KB 256 ? 712a892a-44f0-46fd-b28b-04eae418ebde RAC1
UN 192.168.33.152 870.35 KB 256 ? ce08a377-8b71-4789-beae-e80a70b3b685 RAC1
Datacenter: DC_OPSCENTER_1
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.33.158 512.37 KB 256 ? 22468c08-cd3e-4fc2-8ff8-fa4aeff12caa RAC1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
[root@cassandra-opscenter-1 ~]# cqlsh -u cassandra_admin -p password 192.168.33.151 9042
Connected to Cluster1 at 192.168.33.151:9042.
[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra_admin@cqlsh> select * from system.schema_keyspaces;
keyspace_name | durable_writes | strategy_class | strategy_options
-----------------+----------------+------------------------------------------------------+----------------------------
system_auth | True | org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"4"}
dse_system | True | org.apache.cassandra.locator.EverywhereStrategy | {}
OpsCenter | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"DC_OPSCENTER_1":"1"}
system | True | org.apache.cassandra.locator.LocalStrategy | {}
sample_keyspace | False | org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}
system_traces | True | org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}
(6 rows)
cassandra_admin@cqlsh> desc keyspaces;
sample_keyspace system_auth system "OpsCenter" system_traces dse_system