Last Updated on February 5, 2021
When considering deploying Splunk within your environment, there are many questions to be answered starting with how you can immediately make an impact on your organization while planning for the future of your Splunk environment. Splunk can scale from the smallest very large-scale environments; it can be purchased to run using on premise hardware utilizing and local staffing or purchased as a service to be run within the Splunk Cloud. Server requirements are defined, but the platform you utilize can be determined by you, the customer. In this article, we’ll walk you through some of the items to consider.
A common question to be answered is whether you should purchase Splunk Enterprise or Splunk Cloud. The three biggest factors in your decision are resources, training budget, and intended size of your environment. Keeping these three items in mind through our discussion will help guide you in that decision.
In order to assist with design requirements, Splunk lays out a very clear architecture roadmap: the Splunk Validated Architecture (a link to a descriptive document is listed at the end of this article). Whether your environment simply requires a single server deployment which encapsulates all Splunk functions or your environment requires a level of availability and data redundancy that leads to a more complex architecture, the roadmap is available to help you find your starting place. If you adhere to Splunk’s architectural direction, your initial deployment will be designed for scalability. This is an important point, because once the utility of Splunk is made evident to an organization, the ingest and number of use cases can increase dramatically.
A few items to consider when you’re starting to plan your deployment:
- Daily Ingest How much uncompressed data do you plan to ingest daily?
- Concurrent Searches Many people mistake this for concurrent users, keep in mind that a small number of users may be able to run many searches at the same time.
- Search Availability How flexible is your environment in regard to the ability to recover from outages at the search layer?
- Data Ingest Availability Does your environment require that data may be consistently ingested at any time?
- Data Availability Does your environment require reliable access to all of the data at any time?
Deployments can be made up of a single server (Single-instance deployment) or multiple servers (Distributed-deployment). A single instance can serve up to 300GB/day of data ingestion. Anything over that amount will require you to distribute the load across multiple servers. Within a distributed deployment, customer availability requirements will dictate whether servers should be clustered at the search layer, indexing layer, or both. In more complex environments, Splunk deployments may also be spread across geographical locations as well.
A typical deployment would be made up of a Single Search Head with a distributed collection of indexers. A minimum of three indexers are required in order to create a Splunk Indexer Cluster, if availability needs require it. With additional complexity, an on-premise environment will require additional servers to support the following roles: Cluster Master, License Master, SHC Deployer, Management Console, and Deployment Server.
Considerations that may be more customer-dependent include:
- Platform What platform do you intend to deploy: Linux or Windows? This is usually dictated by the staffing resources available to you internally and what expertise you have on-hand. Platform-specific requirements can be found within the system requirements doc linked at the end of this article. Note the cautionary guidance regarding degraded performance when incorporating VMs. Likewise, you will find specific guidance about the use of containers.
- Maintenance and Operations As with any enterprise-level application, Splunk builds have specific server requirements that include user-level settings and server-level settings, as well as local-firewalling tweaks that need to be made to allow for inter-server communication. Maintaining consistent build processes for these servers through the life of the application will enable you to prevent unneeded outages.
- Universal Forwarders A universal forwarder is the small Splunk installation that is loaded on host machines that ‘forward’ their data to the Splunk environment. You will need a process to install, manage and distribute this software. Splunk provides a Deployment Server functionality which helps to manage the configurations of the Splunk software; however, you may need to utilize customer automation tools (such as Puppet, Chef, etc.) for distribution of the software.
- Firewalling You should have appropriate firewalling rules in place to allow for UF-to-Indexer communication and user-to-search head communication, while allowing for appropriate security of the search/indexing layer, as well as all utility functions.
- Training Splunk is not an application that can be stood up and left alone. The administration of the environment through daily user requests, server operations and application upgrades require specific Splunk knowledge – more so if Splunk is running in a distributed environment. The development and maintenance of in-house Splunk skills is required as the environment grows.
Splunk Cloud Deployment
Another option that is available to customers is Splunk Cloud. For most users, the Splunk Cloud product looks and feels like Splunk Enterprise. The difference is in the details and may only be recognized by those who have had experience administering their own on-premise Splunk environment.
Splunk Cloud has many advantages and for a customer just starting out with Splunk, it may be the ideal solution.
Topology and architectural details are determined and implemented by Splunk in order to maintain contractual SLAs. You will not have to worry about many of issues related to planning an on premise deployment. Further, much of the ongoing operational aspects of maintaining the environment are the responsibility of the Splunk Cloud operations team. However, on premise functionality is not monitored or managed by Splunk. The architectural details of a Splunk Cloud deployment are discussed in the Splunk Validated Architectures document.
- A dedicated single-tenant cloud environment that includes: a single search head (plus one for each additional premium application), a clustered indexing tier, and an Inputs Data Manager
- Availability is enhanced via the use of AWS and GCP infrastructure layer(s) within multiple regions
- 90 Day retention + archive options (see Dynamic Data Active Archive)
- Multi-Factor Authentication is available if using a SAML provider
- Secure options are available for PCI and HIPAA data.
- Connectivity for data ingest to Splunk Cloud requires one or more of the following: HTTP(889), Universal Forwarder (9997), AWS Kinesis Data Firehose. The customer’s Universal Forwarders will communicate directly with the Splunk Cloud indexers.
- Scalability: The environment is scalable with little effort from the customer, and gives flexibility to scale with additional ingest requirements as well as additional functional requirements such as additional Splunk Applications, as well as the adoption of Splunk Premium applications such as Enterprise Security and ITSI
- Concurrency: Search concurrency scales with additional ingest licensing
What seems different in Splunk Cloud vs Splunk Enterprise?
- All applications and changes loaded into Splunk Cloud are required to be vetted and changes approved. You will have less freedom to load configurations into Splunk Cloud. Splunk has these policies to assure the systems consistency and availability.
- Splunk Cloud provides a limited administrative role. You may notice this if you have worked in a Splunk environment before. However, Splunk Cloud still enables you to manage users, apps, indexes, and knowledge objects. Though there should be a pool of knowledgeable resources in your operation to act as administrators,, the server-related tasks are done behind the scenes by the Splunk Cloud operations team. So, the required depth of Splunk server decreased.
- No CLI, No direct access to indexers for TCP,UDP,syslog outputs Again, this is most likely going to be noticed by previous Splunk administrators. However, data ingestion can be done via a forwarder without direct access.
What’s required by the customer?
- Universal Forwarders A Splunk Cloud still requires that data is ingested into the environment in some way. On-premise data requires that a Universal Forwarder be loaded on host(s) in order to forward their data into the cloud. Like on premises, the management of the universal forwarders’ configurations may be done via a Splunk Deployment Server, while the distribution of the software should be handled by your existing processes.
- Firewalling As already mentioned, specific connectivity is required between Splunk Cloud and the customer REST (443), HTTP (889), Forwarder(9997), AWS Kinesis Data Firehose. Obviously, all of these connectivity configurations are the customer’s responsibility.
- Other On Premises Functionality Your Splunk Cloud deployment can gain a level of flexibility and control by using an on premises Inputs Data Manager combined with a Heavy Forwarder, if needed. You can use a Hybrid Search Head to enable searches across both environments if there is an existing Splunk services on site.
Most organizations have traditionally deployed Splunk on premises. You would own all of the hardware for the environment, as well as all of the operational responsibility. This option gives you increased control and flexibility within the environment, but with of hardware, training, staffing, and operational planning costs. You will need to manage a greater number of hardware resources.
On the other hand, in its simplest form, Splunk Cloud takes all of the operational tasks of maintaining the Splunk search/indexing environment out of your hands. You have the simple task of pushing data into the Splunk Cloud via a set of Universal Forwarders managed by a Splunk Deployment Server.
An appropriate approach toward a Splunk deployment is based on your needs. By no means is this article an all-inclusive discussion of the pros/cons of hosting your environment within Splunk Cloud vs On-Premise; and this article certainly can’t assume all of the variables that are present in your environment. However, given the cost, effort, and need for customization/management, this document should shed some light on a direction that is right for your organization.
Splunk Validated Architectures
Splunk System Requirements on-premises