Splunk has multiple methods in regards to Getting Data In (GDI). One very popular method is the Http Event Collector (HEC). The use of the HEC allows data ingestion into Splunk via HTTP POST messages. Two popular methods that send POST messages out of AWS into Splunk are the AWS services: Lambda and Firehose. A common question is when to use one versus the other to get data into Splunk.
Below is a high level overview of each service as it relates to Splunk. Lastly, a small decision table illustrates which features each AWS service can utilize.
Lambda and why do I care?
Lambda is an AWS service that allows serverless functions. The functions that Lambda can perform are up to the coding and the constraints of time and size that are given as timeout and memory, respectively. When using Lambda to send data, additional services and logic are needed for storage and error handling.
An advantage for Lambda is that the throughput for sparse or very low volume sources is higher than for Firehose.
Firehose and why do I care?
Kinesis Firehose allows data to be streamed natively or altered to configurable endpoints. The altered data format is performed by a LAMBDA function which passes the data back to the Firehose. The Firehose has Storage and built error handling as part of its base service.
One additional feature that can help to meet requirements is the option to backup all data or just errors to an AWS S3 bucket.
When should I choose one or the other?
Decision table below:
|Feature/Function||AWS Lambda||AWS FireHose|
|Splunk HEC EndPoint||x||x|
|Auto Error Handling||x|
|Navtive AWS Bifurcation to S3||x|
|Ease of Administration||x|
|Large Data Set Efficiency||x||x|
|Small Data Set Efficiency||x|
|Cost||Dataset Dependent||Dataset Dependent|
If you are looking for ease of administration, then AWS Firehose is the easiest way to reliably load data into Splunk. Its fully managed service automatically scales and requires no ongoing administration. Most importantly it’s considered Splunk best practices, a good rule to live by would be, “Firehose first, Firehose Last, and Lambda only when required.”
If you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly. One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.
Here are a couple AWS/Splunk articles you mind find helpful as well.
Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk.
Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.
Contact us directly to learn more.