Did you know that Splunk has the capability to ingest non-log based data through multiple onboarding methods? Two of those methods are HTTP Event Collector (HEC) and polling via an API. In this blog, we will touch on the Splunk API data ingestion, as it is traditionally the most common method utilized with the HEC(push) method catching up very rapidly. Data is available within all electronic systems, in the past it was seen as best practice to “lock” in customers to a custom bus and language to drive platform adoption. There has been a change, with most systems going forward as it is considered a large advantage for a system to enable easy integration with multiple tools throughout the enterprise.
As mentioned above API based data ingestion is typically used for ingesting non-log based data into Splunk. There are those situations when API data ingestion is the best method and can be chosen over traditional log ingestion depending on enterprise. To get the ingestion process started careful planning should be executed with attention to:
- Data Volume
- Data Type
- Data Security
- Data Storage Requirements
From a configuration viewpoint, you will need Spunk equipment with associated code. The equipment can include a search head, HF or the stand-alone Splunk system as well. The system is equipped with a piece of code, usually a TA or Add-on. The Add-on will utilze the API language(protocol) that can be a proprietary custom syntax or one of the industry standards, such as the REST interface. The configuration when polling data from the targeted device or system is usually given as an IP Address or URL and a associated port. Most if not all, API utilize some form of authentication and will allow SSL/TLS traffic encryption if desired.
The caveats of API data ingestion through an endpoint:
- Pull/poll method of data collection through the usage of polling intervals
- Near real-time ingestion may be achieved using short intervals, increasing the load and demand on the equipment involved with pulling the data
- Complex or complicated redundancy methodology and failover planning schemes
- Different resultant formats can be utilized dependent upon the endpoints API implementation. The most common are JSON or CSV formatted output.
The tools in Splunk to help with API ingestion are listed below:
- Push Messaging
- HEC (built-in) (Pushed based, HTTP POST message)
- Pull/Poll Retrieval Messaging
- REST API Modular Input: https://splunkbase.splunk.com/app/1546/#/overview
- Custom/Specific Add-On/Technology Add-Ons: https://splunkbase.splunk.com/
- Custom scripts utilized through modular inputs
Aditum’s Splunk Professional Services consultants can assist your team with best practices to optimize your Splunk deployment and get more from Splunk.
Our certified Splunk Architects and Splunk Consultants manage successful Splunk deployments, environment upgrades and scaling, dashboard, search, and report creation, and Splunk Health Checks. Aditum also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.
Contact us directly to learn more.