high availability – mongodb replica set No primary choice with 7 of 8 nodes UP

I have 8 instances running on Scaleway, 5 for Data and 3 Arbiter.

And the mongo versions in them are such:

mongodb2 (mongodb 3.4.9)
mongodb4 (mongodb 3.4.15)
mongodb5 (mongodb 3.4.15)
mongodb6 (mongodb 3.4.15)
mongodb-arb1 (mongodb 3.4.18)
mongodb-arb2 (mongodb 3.4.18)
mongodb-arb3 (mongodb 3.4.18)

I need to close mongodb2 For example, I am trying to make one of the other DATA members become PRIMARY. I'm trying to step down or turn off mongodb2, but nothing works …

Below is the output of rs.stepDown () .

{
"ok": 0,
"errmsg": "No eligible children were detected as of 2018-12-07T16: 36: 04.546 + 0300. Use {force: true} to force the node to pass".
"code": 50,
"codeName": "ExceededTimeLimit"
}

And when I use force: true, as db.adminCommand ({replSetStepDown: 120, secondaryCatchUpPeriodSecs: 15, force: true}) The exit:

QUERY    [thread1] Error: error when making the query: error: network error when trying to execute the command & # 39; replSetStepDown & # 39; on the host & # 39; 127.0.0.1: 27017 & # 39;
DB.prototype.runCommand@src/mongo/shell/db.js: 132: 1
DB.prototype.adminCommand@src/mongo/shell/db.js: 150: 16
@ (shell): 1: 1
2018-12-07T16: 37: 29.322 + 0300 I RED  [thread1] when trying to reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2018-12-07T16: 37: 29.324 + 0300 I RED  [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok

and the mongodb2 becomes SECONDARY, but becomes PRIMARY after a few seconds.

I will provide the configuration below when I just turn OFF the primary Node, mongodb2.

[
            {
                    "_id" : 1,
                    "name" : "mongodb2:27017",
                    "health" : 0,
                    "state" : 8,
                    "stateStr" : "(not reachable/healthy)",
                    "uptime" : 0,
                    "optime" : {
                            "ts" : Timestamp(0, 0),
                            "t" : NumberLong(-1)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(0, 0),
                            "t" : NumberLong(-1)
                    },
                    "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                    "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.187Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:22.796Z"),
                    "pingMs" : NumberLong(0),
                    "lastHeartbeatMessage" : "Connection refused",
                    "configVersion" : -1
            },
            {
                    "_id" : 3,
                    "name" : "mongodb4:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 11771707,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "infoMessage" : "could not find member to sync from",
                    "configVersion" : 11,
                    "self" : true
            },
            {
                    "_id" : 4,
                    "name" : "mongodb5:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 11771703,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "optimeDurableDate" : ISODate("2018-12-07T13:25:17Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.161Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:53.141Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            },
            {
                    "_id" : 5,
                    "name" : "mongodb6:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 4982,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "optimeDurableDate" : ISODate("2018-12-07T13:25:17Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:53.139Z"),
                    "pingMs" : NumberLong(0),
                    "configVersion" : 11
            },
            {
                    "_id" : 6,
                    "name" : "mongodb-arb:27017",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 5462,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.730Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            },
            {
                    "_id" : 7,
                    "name" : "mongodb-arb:27018",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 295,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.726Z"),
                    "pingMs" : NumberLong(0),
                    "configVersion" : 11
            },
            {
                    "_id" : 8,
                    "name" : "mongodb-arb:27019",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 162,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.773Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            }
    ]

In this state, nothing changes, no matter how long you wait. The healthy members of DATA just keep spitting the log:

2018-12-07T13: 25: 48.166 + 0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb2: 27017
2018-12-07T13: 25: 48.167 + 0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Error connecting to mongodb2: 27017 - HostUnreachable: connection refused
2018-12-07T13: 25: 48.167 + 0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Removing all connections grouped to mongodb2: 27017 due to a failed operation on a connection
2018-12-07T13: 25: 48.167 + 0000 I REPL     [ReplicationExecutor] Error in the request of heartbeat to mongodb2: 27017; HostUnreachable: connection rejected

and there is no choice of a PRIMARY node