File Analytics Troubleshooting Guide

  • 19 August 2020
  • 0 replies
  • 3053 views

Userlevel 3
Badge +2

File Analytics deployment & teardown is done via Prism UI. Logs for both - deployment & teardown process are in prism logs - /home/nutanix/data/logs/prism_gateway.log on the CVM leader. Errors are updated in ergon tasks as well.

 

 

Most common errors during deployment -

Error Message

Cause and fix

Cannot connect to File Analytics VM from Prism. Please verify that network details for the VM are correct and the IP is reachable from Prism.

The network details provided during deployment were incorrect [either wrong IP/subnet/gateway or wrong VLAN selected] or there was a genuine network connectivity issue. Please try deploying again with correct network settings.

One or more services are not running, please check logs for more details

During deployment, one or more services failed to start. Logs would be collected from File Analytics VM on CVM at /home/nutanix/data/logbay/bundles/NTNX-Log-***.zip. Need to check logs for root cause.

Error creating volume group, please check logs for more details

During deployment, Volume Group creation/discovery failed. Logs would be collected from File Analytics VM on CVM at /home/nutanix/data/logbay/bundles/NTNX-Log-***.zip. Need to check logs for root cause.

User Creation steps failed after AVM teardown and redeployment

This is an intermittent issue with AOS v5.10.2. Please remove the file_analytics from prism user list manually and re-trigger the deployment.

 

 

Most common errors during teardown -

Error

Cause and fix

Error fetching subscribed file servers list from File Analytics

Failed to get list of file servers which are subscribed for analytics. During teardown, if those file servers are still enabled, teardown process will try to disable them. Manual fix is to delete Notification Policy, Partner Server & REST user from file server.

Disable failed for file servers with uuid <file server UUIDs> 

Disable/unsubscription failed for the mentioned file servers. Need manual cleanup as mentioned above.

 

 

 

Subscription errors -

Error 

Cause/fix

One or more services are down, failed to subscribe file server

Services are not running in File Analytics VM, need to check which service has error, start it & then try again

File Server is already subscribed

File server is already subscribed for Analytics

File server is not reachable, please check file                      server state in Prism

File server is marked as unreachable in prism, please check whether file server is in good state & then try again

Missing required inputs: <AD or LDAP credentials/particular field input>

File server is configured with the specified protocol [AD/LDAP] and we need credentials for communicating with file server over that protocol. Please provide required inputs & try again.

Can’t connect to LDAP server/provided Domain

Cannot contact the AD/LDAP server. Please check whether the DNS configured on File Analytics can resolve the AD/LDAP hostname & try again. If needed, change DNS server. Steps to change DNS covered later in the document.

Invalid Credentials

Invalid credentials for AD/LDAP

User <username> is not configured or mapped to file server admin role. Please select the File server in Prism and go to 'Manage roles' option and add user / roles in 'Add admins' section.

The AD user provided as input needs to be added in Manage Roles page for the file server as an Admin user with Full Admin Privileges. Please update file server  configuration & try again.

Failed to create service in Analytics

Failed to create Kafka Topic. Please check that kafka server is running & that kafka settings in API server config file are updated properly, then try again.

Cannot contact Prism - Invalid Credentials

Prism credentials are file_analytics & Nutanix/4u990 [applicable only for Tech Preview]. Please set prism user credentials to these & try again.

Error creating File Server user

Exception occurred while creating a REST user for the file server. Please check API logs for more details about the exception.

Partner Server with same IP already exists

Partner server with same IP/hostname already exists on the file server. Please delete the same from file server & try again.

Error creating Partner Server/Notification policy

File Server returned error while creating the entity. Please check API logs.

Timed out waiting for Partner Server/Notification Policy creation

Request was accepted by File Server to create a partner server/notification policy, but the entity was not created.

Failed to save File Server

Failed to add file server record in ElasticSearch index, exception details can be seen in API logs

Failed to update service in Analytics

Failed to update consumer and full scan config, please check API logs for exception

 

 

How to check if the container is running fine?

Docker commands -

List running containers

# docker ps

List all containers [running as well as stopped]

# docker ps -a

To gain access to container shell

# docker exec -it <container_name_or_ID> /bin/bash

To quit container shell

Ctrl + shift + p +q

 

 

How to collect log for AVM?

Log collection command -

From Prism CLI, execute the following command to collect logs from File Analytics VM -

# logbay collect -s <cvm_ip> -t avm_log

  In case the logbay command has issues, please execute the following command on File          

            Analytics VM CLI  - 

# /opt/nutanix/analytics/log_collector/bin/collect_logs

 

 

How to manually collect logs from each of the components?

Log locations -

Gateway/API  - /mnt/containers/logs/gateway/analytics_gateway.log.INFO & /mnt/containers/logs/gateway/uwsgi.out

Anomalies - /mnt/containers/logs/gateway/anomaliesdetection.INFO, /mnt/containers/logs/gateway/anomaliesdetection.out, /mnt/containers/logs/gateway/anomaliesdetection.localhost*

Event Processor/Meta data collector - /mnt/containers/logs/gateway/event_processor.log.INFO, /mnt/containers/logs/gateway/metadata_collector.log.INFO, /mnt/containers/logs/gateway/main.out

Elastic Search - /mnt/containers/logs/elasticsearch/

Kafka & ZK – /mnt/containers/logs/kafka/

Bootstrap log - /var/log/nutanix/

Monitoring Service - /mnt/containers/logs/monitoring.log.INFO

 

If you have any issues, please do open a case with us at portal.nutanix.com

 

For more on deploying File analytics:

https://portal.nutanix.com/page/documents/details?targetId=File-Analytics-v2_0:ana-fs-analytics-c.html


This topic has been closed for comments