Crumpled paper & lightbulb

Splunk Deployment Best Practices- Things I Wish I’d Known

One of the best ways to ensure that you’re getting the most out of any tool is to get a handle on others’ mistakes and lessons learned. Our team of Splunk Professional Services engineers is continuously gaining insight into how to best deploy Splunk, including how to overcome technical challenges as well as internal hurdles. We’re sharing these insights to help your organization gain faster value from your Splunk deployment.

This post is the second in a series of articles on “Things I Wish I’d Known” about Splunk. In today’s post, Roman LopezSplunk Professional Services Consultant with Aditum and Jon PappAditum’s Professional Services Manager, explore lessons learned that can improve Splunk deployment and adoption. For more best practices and lessons learned, you can also view the first post in this series, “Splunk Data Management – Things I Wish I’d Known.”

Have a Good Plan to Drive User Adoption

Once Splunk has caught on with a few key users, it’s use, functionality, and value-add start to take off rapidly. But how do you get those key users to adopt Splunk?
Many organizations will plan internal demos or provide official Splunk training to employees in hopes of encouraging them to take advantage of Splunk. This is a great starting point – but it doesn’t go far enough. When bringing Splunk into a new environment, you need a plan that connects the specific pains of your users with the solutions Splunk can provide.

Start by gathering requirements – what kinds of problems does a team have that can be solved by Splunk? Then solve some of those specific problems – build a dashboard to show user account lockouts, or build an email alert for unexpected server shutdowns.

Now that users see Splunk can solve their problems, they’ll be more open to attending something like an internal hack-a-thon – book a conference room for 4 hours and work with interested users in solving their problems – by writing their own Splunk queries – in Splunk! Plans for user adoption will vary by organization, but if you want to drive increased value with Splunk, you need your users taking full advantage of it – have a plan!

Build a Test Environment

The reality is that clients (even big corporate clients) ignore the test or dev environment requirement. If you have a multi-site, multi-cluster environment in production you should have something similar in test. I have seen this at a client where it was implemented after an ES upgrade went very badly south. A test/dev environment is critical for testing upgrades, replicating and fixing errors in your prod Search Head cluster between sites, etc.

The majority of clients do not have a test environment but it is so easy to set up, even on a local laptop using VirtuBox or VMware.

Utilize Splunk Docs (Correctly!)

Use Splunk docs, and when doing so, remember to change the version in the top right corner. On my last engagement with a Medical equipment provider, they were running 6.5.0 and we were running some commands on the SHC. These differed significantly between 6.5.0 and 6.6.2

Once you select a topic, the next page comes up:

The Splunk wiki and blog is another excellent resource. Topics include:

  • Troubleshooting Your Splunk Installation
  • Deploying Splunk
  • Getting Data Into Splunk
  • Searching, Alerting and Reporting
  • Much More!

Measure Twice, Cut Once

This is something that comes naturally after your first handful of deployments once you’ve been bitten by inefficiencies that you unknowingly built into your platform!

Every time you develop a new component you need to ask yourself: How will this scale? How likely fragile is it (think dependencies and assumptions) and how gracefully will this fail? How easy would it be for another person to understand and fix? Can I say this is “Enterprise” level? What will the maintenance overhead be?

Aditum – Expertise for a Successful Splunk Deployment

Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk. Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.

Contact us directly to learn more.

Aditum Partners
Share this Article