Crystal ball with question mark inside

AWS Lambda vs AWS Firehose

Splunk has multiple methods in regards to Getting Data In (GDI).  One very popular method is the Http Event Collector (HEC).  The use of the HEC allows data ingestion into Splunk via HTTP POST messages.  Two popular methods that send POST messages out of AWS into Splunk are the AWS services: Lambda and Firehose.  A common question is when to use one versus the other to get data into Splunk.

Below is a high level overview of each service as it relates to Splunk.  Lastly, a small decision table illustrates which features each AWS service can utilize.

Lambda and why do I care?

Lambda is an AWS service that allows serverless functions.  The functions that Lambda can perform are up to the coding and the constraints of time and size that are given as timeout and memory, respectively. When using Lambda to send data, additional services and logic are needed for storage and error handling. 

An advantage for Lambda is that the throughput for sparse or very low volume sources is higher than for Firehose.

Firehose and why do I care?

Kinesis Firehose allows data to be streamed natively or altered to configurable endpoints.  The altered data format is performed by a LAMBDA function which passes the data back to the Firehose. The Firehose has Storage and built error handling as part of its base service.

One additional feature that can help to meet requirements is the option to backup all data or just errors to an AWS S3 bucket.

When should I choose one or the other?

Decision table below:

Feature/Function AWS LambdaAWS FireHose
Splunk HEC EndPoint xx
Transform Eventsxx
Auto Error Handlingx
Navtive AWS Bifurcation to S3x
Ease of Administration x
Large Data Set Efficiencyxx
Small Data Set Efficiencyx
Cost Dataset DependentDataset Dependent

In Summary 

If you are looking for ease of administration, then AWS Firehose is the easiest way to reliably load data into Splunk. Its fully managed service automatically scales and requires no ongoing administration.  Most importantly it’s considered Splunk best practices, a good rule to live by would be, “Firehose first, Firehose Last, and Lambda only when required.” 

If  you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly.  One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.

Here are a couple AWS/Splunk articles you mind find helpful as well.

About Aditum

Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk.

Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.

Contact us directly to learn more.

Cedric Milan
Latest posts by Cedric Milan (see all)
Share this Article