File system permissions in Linux, the suggested operating system for Splunk, can be worrisome for Splunk administrators. Often times, a Splunk Administrator needs correct permissions to read system logs; however, by default in Linux, these permissions are not available, or only available to the one Linux superuser account.
To work around this, a common configuration in the field for Linux based systems is where root is configured as the user to run the Splunk system. This configuration alleviates the complaints that the Splunk non-root user does not have the proper permissions to access and view the data. Naturally, running the program as the root user will allow access to the files that are being read.
The drawback in doing this is that running Splunk as the root user creates a security vulnerability for these systems. Due to this, best practices from Splunk Professional Services suggest not running Splunk as the root user. Surprisingly, a large number of Splunk consulting engagements that I have been on have ignored this security risk. This article suggests ways to reduce the Splunk security risks associated with root access, stay in bounds on Splunk best practices, while simultaneously providing Splunk Administrators the access to the data they need access to.
The traditional fix that does not involve running the Splunk software as root is to chmod the files or directories with permissions that can be set for the owner, group and everyone. It is not uncommon to manipulate the group ownership of a file so that the non-root Splunk user can view and subsequently index its contents. While that solution works, there is a large risk of causing other issues on the system.
Thankfully, the getfacl and setfacl commands are available and allow directory/file access control for file access. The command runs as root or as an authorized pseudo-user. Individual users and/or groups can be given read, write and/or execute permissions.
File permissions fix
The setfacl command however only solves the issue if you are aware which files needed to have their permissions modified before deploying your configuration. To fix the file access permissions after the fact can be done with a search of the splunk.log using Splunk for the host in question or a script can be run like the one below.
The script below is a great example that will help to alleviate the permissions concerns. The script works by reading the splunkd.log and picking out the files that have been reported with insufficient permissions. The script will then process the command above with the spunk user as the intended recipient of the new file permissions. Bundled into the script are a couple of options that allow setting:
- Splunk location (Enterprise or Universal Forwarder)
- Debug (logged output)
- Test Run (additional verification step if needed)
The ideal location to run the script is from the /tmp directory and as the root user account. Logic has been added into the script where each file is tested for existing permissions. If the permissions already exist a message is logged with that information.
If you’d like to copy and use this script, download the text version here.
File permissions can be a big problem when trying to index data into Splunk. The steps to mediate the issues can range from simple to difficult. Using chmod may open other issues with the system and its overall security. Using the getfacl and setfacl commands expand on the chmod command in Linux allowing per-user access, while also allowing greater control over file permissions and access. These commands make them the preferred method of balancing Splunk Administration and system controls.
Using the script above automates the repair process that will allow files with permissions errors to be indexed into the Splunk environment. The script goes a little further by providing output logs that can be audited if the script is to run on a schedule or on-demand.
As always, Splunk Answers can the go-to source for help and further information on this topic can be found on Splunk Answers (https://answers.splunk.com/index.html). Splunk professional services firms such as Aditum are also always here to help Splunk customers with these and many other issues, to make sure that your Splunk instance is optimized and driving significant business value for your organization.
This article was authored by Cedric Milan, a senior Splunk Professional Services consultant with Aditum. Cedric is a Splunk Certified Consultant II, the highest level of certification in the Splunk domain. Aditum is a leading Splunk business partner, assisting customers with software acquisition and Splunk consulting services.
Aditum (Latin: “to access”) is a professional services firm specializing in next-generation Analytics solutions based upon the Splunk platform. We’re quickly building one of North America’s largest and most accomplished Splunk consulting teams with extensive experience across all of Splunk’s major use cases including IT Ops, Security, DevOps, Business Analytics and IoT.
Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk. Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.
Contact us directly to learn more.