Huge amounts of Log Data generated day by day, organizations are ending up investing huge amounts of money to store and process Log data. With efficient and proper reduction techniques of log data, one can reduce the amount of cost and time spending on log data. This whitepaper describes about reducing the cost, space and log management.
Who should invest?
- The companies, who use Splunk environment
- Who require real time log events, data from application or system software
- Who have issues in search performance, installation, indexing, cluster-based, reporting
- Who require development of solutions for big data and text mining business problems, will automatically have to deal with huge expenses while using Splunk. The challenge for these companies arises, when they have to analyze humungous data, like more than 100 GB every day and as a result end up in incurring more costs.
- Taking longer querying time to fetch the records
- Expensive in future to store large volumes of data
- Huge licensing cost
- Huge infrastructure is required for processing and storing
- Vast amount of data
- Performance will be slow
- Longer time to run longer queries
- Storing only the field values in the database.
- Store only relevant info in the log management tool.
- Removing duplicate events
- Removing redundant fields
- Converting to formats that occupy less space
- Efficient licensing model
- The above recommendations will help to reduce the time, cost.
1. Licensing models
Companies are spending huge amount of money to monitor their logs; it will be more beneficial when we reduce their expenditure by optimizing the log management tools.
To buy licensing for log management tools, we need to approach any one of the licensing models, each licensing model has their own features.
Types of licensing models:
It is subscription of a software or software as a service (SaaS), we should pay the less Upfront Cost and have to pay annually.
It is buying license upfront by paying large amount for single time and owning it.
- Volume-based licensing model.
- It is like paying as you go; you will never own the software even if you use it forever. Low upfront costs and will get charged only the amount of volume you have used.
- Vendor is responsible for software and hosting infrastructure and maintenance.
Reference to above figure:
Daily volume of data results in the cost
The following are the different types of Service providers and their pricing is listed daily Volume wise.
Perpetual and Term Licensing are two options for licensing Splunk Enterprise.
Perpetual license: This includes the full functionality of Splunk Enterprise and starts as low as shown in the table with only annual support charges.
Term license: this provides the option of paying a yearly fee instead of the one-time perpetual license fee. Term licenses start at $1,800 per year*, which includes annual support fees
Reference to above shown data:
Sumo Logic Provides Four types of services:
Primarily focused on monitoring and troubleshooting use cases.
The price is starting at $2.25 Per GB of logs
- Enterprise Operations
Designed for best-in-class monitoring & troubleshooting, backed by full enterprise level 24×7 support.
The price is starting at $4.95 per GB of Logs
- Enterprise Security
Designed for teams with SaaS SIEM, threat analytics & reporting, and audit compliance use cases, backed by full enterprise level 24x7support.
- Enterprise Suit
All in one platform – most economical plan for organizations with multiple use cases. Hyper-scale on-demand or continuous log analytics, monitoring& troubleshooting, SIEM and much more.
- $ 0.11/ GB log search on-demand
- $2.475/ GB log search unlimited
- $5.50/ GB log search, alerts, dashboard
Elastic search is not giving service as volume based, it will charge monthly.
2.Reducing the volume of log data sent to Splunk to process or store
In the industry, Log analysis is mainly based on two key factors:
- Volume of data
- Retention of data
The expansion of any organization is directly proportional to the expansion of the log data, which in turn is directly proportional to the expense. In multiple cases a huge amount of the data does not require long-term retention.
Today, this plain fact is dismissed while estimating the costs by the log analysis solutions and offer an either fully or not at all operative models that compel companies to pay for retention schemes that do not consider this difference and so are gratuitously costly and ineffectual.
Benefits of reducing log data:
- Minimizes cost effectively
- Reduces log noise
As Splunk is licensed by daily indexed data volume, you need to pay for the total amount of data you send to Splunk per day. So, it is in every customer’s scrutiny to keep the data volume generated as low as possible.
It is important to estimate the cost based on data generated from the Splunk add-on and data stored.
How can we reduce the volume of log data?
- Filtering logs that are not needed:
- Once we know what is amount of generated data, having an idea, of how much and what are the details that can be reduced without actually affecting the reliability of data is important.
- Move the log statement to a lower level, to such as
debug, so that we don’t actually lose the log, when we need it.
- Drop logs containing specific key words
- Sampling logs:
- Logs are sampled into subsets of logs
- This can shrink the volume and also provide excessive perceptibility.
- Reduce log size
- Minimizing the log size can free space in a log file.
- Minimizing the log size can free space in a log file.
REDUCE LOG(Decreases the Assigned Capacity of the Recovery log)
- This command allows to decrease the space that can be used by the recovery log.
- REDUCE LOG command can be used when users are accessing the server.
- QUERY LOG command can be used to know to what extent you can minimize the assigned capacity of the recovery log.
- This command can generate a background process that can be canceled with the CANCEL PROCESS command.
- If a REDUCE LOG background process is canceled, the assigned capacity of the recovery log may be partially reduced.
- To display information on background processes, use the QUERY PROCESS command.
How agents can be used to process the raw events before ingestion into the Splunk?
- From the agent’s perspective, the data volume per host depends on the various dependencies like the types of applications, the system configurations, types of browsers, background processes running etc.
- Agents could be using dashboards, displaying the data volume details through the dashboards is very much beneficial.
- The steps involved in the processing of raw events depends on the agent’s architecture.
- Aggregation of the raw events and analyzing the structure of the raw events is done.
3. How to Reduce the Event size?
- Log Events are generated based on the actions done by the user or system on machine
- These actions may or may not be repetitive, but most of them are repetitive
- For each action done on a machine, log events are stored regarding that specific action
- Since most of the actions are repetitive, events regarding to those actions will have same structure but field values may differ.
How the flow looks like?
- The flow starts with the collecting all the log events from the virtual machine.
- Create Patterns for the events which are having similar structure and name the fields which are varying across the events.
- Store the field values in the database.
- When we want to recreate the events, we will reverse the process what we had done, fetch the pattern that which event belongs and fill the fields with their respective values from the database.
Explaining the steps with few log events:
These are the sample syslog events that are collected from a machine. If we analyze them carefully events 1, 5, 6 from top have similar structure but with different field values. So let us group and form them as a pattern.
The above red marked values are the varying values from this events.Now name and extract the fields from the above pattern.
So from the above pattern we can extract EVENTIME, PID, USER and these values are stored in database key value storage) rather than events. We will store this pattern and use to recreate the events. A single pattern covers multiple events.
4. Reducing the event size
Retaining key details:
How to use the key details in ad hoc queries or reports?
- A non-standard inquiry: An ad hoc query is created to obtain information as the need arises. Contrast with a query that is predefined and routinely processed. See query and ad hoc
- These queries can be created based on the key data we have stored. To create ad-hoc queries; you first specify the source view for the query to determine the type of records to include. Then you can specify output fields and filter criteria for the query. You can use categories to group your queries. When you view ad-hoc queries, you can use filters such as Type or Category to limit how many queries are shown.
Examples of how to do a raw event comparison with key detail storage
- Windows account login attempts generate events containing copious metadata. See below for an authentication event in Snare Syslog format.
- However, only important ingress authentication details are needed. Procedure can decrease the data size. Leaving only the important event details required by the system.
- The difference in size between the two is 2,360 characters (2,917 vs only 557 characters), which is a reduction of 80.9%.
Full event sample of a Windows Failed Authentication Event in Syslog
Windows Failed Authentication Event after log enrichment
5. How to reduce the amount of data stored?
Explaining Patterns for logevents. Each type of action is an event, same actions – similar patterns
- Creating database tables for similar patterns to analyze
- Similar events will be considered as patterns
- If similarity is not found then that will be considered as separate patterns
- Below are the different types of log events
- Application logs
- Database logs
- Network data
- Configuration files
- Performance data
- Time-based data
- More data captured = more visibility
- In Database store key once and values for same patterns
Formatting data before uploading into Splunk
- Timestamp key=value key=value key=value key=value key=value
- May 26 18:14:15 myhostname DBIP=10.5.10.2 Service=Oracle ClientIP=18.104.22.168 SrcPort=80 DestPort=8080 UID=10534 Sql_Text=Select * from Table1 where uname=”dummy”.
- keys and values are faster readable to Splunk
- Keys are same. So, we will retain keys and pass only values.
- Splunk will index and search as fast as we can write it out.
- The time stamp can be in ISO 8601 form – i.e. YYYY-MM-DD HH:MM:SS.mmm TZ DST.
- Example: 2011-10-24 14:04:02 +0200 DST
- If you do not want or need the time zone then that can be omitted
How to reduce the amount of data stored?
Converting json data into csv, that occupies less space and also to extract the information.
- First, we need to convert the raw data into JSON, and then export a “CSV with one field containing the JSON.
- Splunk can export events in JSON via the web interface and when queried via the REST api can return JSON output.
Benefits of LogMiner tool
Reduction of Splunk Costs can be achieved by LogMiner using the following methods:
- Pattern generation
- Retaining key details
- Duplicate removal
The essence of this whitepaper is to endow solution to the challenges mentioned earlier in the paper, especially to reduce the substantial Splunk costs, where in LogMiner tool gives the best quick-fix, with its methodical and efficient approach.
Link to BDSR website: