Logging for Cloudwatch Events using Splunk HEC
Welcome to another installment on how to log multiple AWS accounts into Splunk, when the recommended method is not feasible. In this article we will take a look at Splunking AWS Cloudwatch Event data using an HTTP Event Collector (HEC) input. The most common method to retrieve Cloudwatch Events data is to use the Cloudwatch Logs input configuration through the AWS Add-On.
Pull (near real-time)
The pull method involves using the AWS Add-on for Splunk and initiating the API request from a primary Splunk system. While the method is straight forward, there is sometimes the desire to use a push method for AWS logging into Splunk. The process is further complicated when a single point of log consolidation is required or desired for multiple AWS accounts. We are going to examine the ‘push’ method that involves Splunk HEC as our data receiver.
Options available to send messages involve setting a subscription to AWS Lambda. This step is offered as an action once the associated Cloudwatch Logging group is selected. The data is processed by an AWS Lambda step function that processes the data and lastly sends it as an HTTP Post message to the Splunk HEC endpoint, configured in the Lambda config. As with other Lambda functions, care should be taken when planning this method due to the high volume of messages that will be traversing the function.
The processing of the Cloudwatch Events should still utilize the existing sourcetype of aws:cloudwatch logs provided by the AWS Add-on for Splunk. Therefore, the AWS Add-on should still be utilized however no inputs need to be created via the add-on as done with the traditional pull configurations.
As with nearly all AWS services there will be a cost associated with using Lambda.
AWS Account consolidation
There are references available in the AWS environment that detail the options and processes needed to allow account consolidation for the purpose of logging. One solution is to create a specialized account with the purpose of logging to external or internal software for further analysis and processing. The steps and methodology to centralize AWS logging are in documents/blogs/articles on the AWS site.
A couple of great links from AWS about AWS Account Consolidated Logging are:
AWS Cloudwatch Events help organizations onboard data into Splunk from AWS Services such as Lambda, Batch, Kinesis and so forth. The ability to reliably log the information in a reliable and highly available implementation establishes the value of the data as well as the information that it contains. Using Splunk HEC to log the data gives customers a solution that can withstand growth and failures. There are multiple methods that can be used to get the data into Splunk HEC, the method above allows organizations without AWS external endpoints and CA signed certs to still utilize the power of Splunk HEC.
If you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly. One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.
Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk.
Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.
Contact us directly to learn more.