Again, I can log in as nutanix user, run the faillock clear and I can log in as root again.

I have nothing running that should be generating logins with root.

This is extremely odd, do you see anything in the auth.log of the AHV host/hosts, or any errors in the genesis.out log found in /home/nutanix/data/logs on the CVM.   

This is particularly troubling:
There were 11031 failed login attempts since the last successful login.

What version of CE are you running on this cluster and what version of NCC is running?   If you go to LCM and run the inventory, can you provide a screenshot of the output?

I am running wireshark right now and there is nothing attempting connections to that box.

You may be hitting a known issue where the CVM is making SSH connections via the internal network between the CVM and the AHV host (over the 192.168.5.0/24) network which never leaves the box.   This applies with specific versions of AOS/AHV/NCC, so that’s what i want to validate.  

Nutanix 6.5.2 LTS

Nothing of note in genesis, but the audit log shows the CVM is attempting root logins to the AHV many times per second. The secure log is 8596079 in size at the moment. If it is connecting in such rapid fire, whatever password that is out of synch must really hammer the root account. The secure log is still growing, it is at 8683311 right now.
 

Sample from secure log

2024-08-13T13:01:13.947528+00:00 NTNX-cdb56c24-A sshdd15731]: Close session: user root from nutanixcvm port 47916 id 0
2024-08-13T13:01:13.986385+00:00 NTNX-cdb56c24-A sshdd15731]: Starting session: command for root from nutanixcvm port 47916 id 0
2024-08-13T13:01:14.004319+00:00 NTNX-cdb56c24-A sshdh15731]: Close session: user root from nutanixcvm port 47916 id 0
2024-08-13T13:01:14.043010+00:00 NTNX-cdb56c24-A sshds15731]: Starting session: command for root from nutanixcvm port 47916 id 0

It may very well be the ‘known issue’ you are referencing.

If I mistakenly locked the account because of password fails, it makes sense it would log a lot of attempts before I unlocked the account again.

It doesn’t really explain why ssh logins continue to work. I have seen that now on WebGUI as well as the server console where they are locked but SSH continues to function. Seems like that would be a security issue if a hacker were truly trying to gain access.

Here’s the screencap you wanted.

Can you check to see if when you run an LCM inventory if there is an NCC update available for you?
 

You should at least have NCC 4.6.4+ available to you.  Just update that component and see if after your resets things stick.  

There are 5 updates waiting. I’ll apply them and post back.

Any luck on this?   I imagine you may need to do the reset of passwords one last time depending on how many failed attempts were occuring, but after that it should be good.

This morning the root account was again locked, so I unlocked it.

Today I applied all the updates. I just went for it, started out with NCC, then did foundation updates, AOS and finally AHV itself. All went pretty smoothly except it couldn’t ‘evacuate’ the VM’s so I manually shut them down and re-ran it. Worked fine. I will monitor it post updates and see how it performs.

I was able to successfully login this morning without having to reset the root password so that is an improvement. There are still attempts at logging in but apparently it is able to avoid the lockout window.

Last login: Wed Aug 14 17:38:54 UTC 2024 from userpc on pts/0
Last failed login: Wed Aug 14 18:04:19 UTC 2024 from nutanixcvm on ssh:notty
There were 27 failed login attempts since the last successful login.

 

Good, glad to see you’re not getting locked out anymore.