LogTrend's DataBase Consolidation Manual

[logo]

LogTrend

Copyright  2001 by LogTrend http://www.logtrend.org

This software and all affiliated files are Copyright (C) 2001 by Atrid Systmes
under the terms of the GNU General Public License. A copy of this license
entitled "GNU General Public License" is included with the software. The
original text can be found on http://www.gnu.org/copyleft/gpl.html

Permission is granted to copy, distribute and/or modify this document under the
terms of the GNU Free Documentation License, Version 1.1 or any later version
published by the Free Software Foundation; with no Invariant Sections , with no
Front-Cover Texts , and with no Back-Cover Texts. A copy of the license
entitled "GNU Free Documentation License" can be found with this software. The
original text can be found on http://www.gnu.org/copyleft/fdl.html

-------------------------------------------------------------------------------

Table of Contents
1. LogTrend DataBase Consolidation
   
    What is Consolidation ?
    How does it work ?
    Configuration file
       
        DataBase (required)
        Consolidation (required)
        Default rules (optional)
       
    How to run Consolidation ?
    Example of consolidation
   
List of Examples
1-1.  
1-2.  

-------------------------------------------------------------------------------

Chapter 1. LogTrend DataBase Consolidation

What is Consolidation ?

Consolidation is used to keep useful data for trend analysis while reducing
sample number to limit the data base size increase.

For real time visulisation, it's useful to have a lot of values for recent
data. But it's not necessary to keep all these data after on a long time span.
To keep a representative sample of these data (mean, min max, ...), can reduce
the data base size increase, speed up data analysis without harming the quality
and the precision of the results.

-------------------------------------------------------------------------------

How does it work ?

The LogTrend's database consolidation is based on rules applied to time
intervals. These rules defined how to consolidate the database :

For the data named cpu_system on agent 3 source 14 :

  * For data older than 7 days, do average 5 per 5.
   
  * For data older than 15 days, do average 10 per 10.
   
With such rules, if the agent collects data each 100 seconds, after
consolidation the database look like :

  * Data younger than 7 days : No consolidation. Data Frequency equal to data
    collection frequency, a data each 100 seconds.
   
  * Data between 7 and 15 days : a data each 500 seconds.
   
  * Data older than 15 days : a data each 1000 seconds.
   
-------------------------------------------------------------------------------

Configuration file

Like all LogTrend config files, consolidation config files are in XML :
          
<Consolidations>
  <DataBase Name="logtrend" Host="myserver.mydomain.com" Port="5432" 
    User="logtrend" Password="logtrend" />
  <!-- Default rules (applied to all data which are 
       not in Consolidation tags -->
  <DefaultRules>
    <!-- For data older than 30 day, do average 10 per 10 -->
    <Rule>
      <Offset>30</Offset>
      <Factor>10</Factor>
    </Rule>
  </DefaultRules>
  <Consolidation Active="Yes">
    <!-- Agent 5 and 6 on source 5  -->
    <Source Number="14">
      <Agent>3</Agent>
      <Agent>6</Agent>
    </Source>
    <!-- All LinuxAgent of source 6 -->
    <Source Number="2"> -->
      <AgentType>LinuxAgent</AgentType> -->
    </Source> -->
    <!-- Name of data -->
    <Data>cpu_user</Data>
    <Data>cpu_system</Data>
    <!-- For data older tha 7 days,
         keep the max 5 per 5 -->
    <Rule Method="Max>
      <Offset>7</Offset>
      <Factor>5</Factor>
    </Rule>
    <!-- For data older tha 14 days,
         keep the max 10 per 10 -->
    <Rule Method="Max">
      <Offset>14</Offset>
      <Factor>10</Factor>
    </Rule>
    <!-- Delete data older than 1 year -->
    <Rule>
      <Offset>365</Offset>
      <Factor>0</Factor>
    </Rule>
  </Consolidation>
  <!-- Inactive Consolidation -->
  <Consolidation Active="No">
    <!-- For All data of HttAgent on source 3 -->
    <Source Number="3">
      <Agent>1</Agent>
    </Source>
    <Source Number="5">
      <AgentType>HttpAgent</AgentType>
    </Source>
    <AllData/>
    <!-- No rule, data are not consolidated -->
  </Consolidation>
</Consolidations>

          
        

-------------------------------------------------------------------------------

DataBase (required)

          
 <DataBase Name="logtrend" Host="myserver.mydomain.com" Port="5432" 
           User="logtrend" Password="logtrend" />
          
        

The DataBase tag is used to set up LogTrend database connection parameters.
This tag has 5 attributes and contains no data.

Attributes (all is required, no default values) :

  * Name : Name of the database
   
  * Host : Name of the host where is the database
   
  * Port : Port of the database on the server
   
  * User : User name to connect to the database
   
  * Password : Password of this user
   
-------------------------------------------------------------------------------

Consolidation (required)

          
<Consolidation Active="Yes">
        .
        .
        .
</Consolidation>
          
        

This tag contains all Consolidation parameters. The Active attribute indicate
if the consolidation is active or not. The default value of Active is Yes.

Config file can have several Consolidation tags.

-------------------------------------------------------------------------------

Configuration of data to consolidate

LogTrend's Agents data are defined by a source number, an agent number and a
data name. To select data in the consolidation config file you should indicate
:

  * The source number
   
  * Agent numbers on this source concerned by this consoolidation
   
  * The data name
   
Sources and agents. To select agents number 3 and 6 on source 14, you must
write :
                
<Source Number="14">
  <Agent>3</Agent>
  <Agent>6</Agent>
</Source> 
              
It possible to select all agents of a type. If you want to select all HttpAgent
of source 5, you must right :
                
<Source Number="5">
  <AgentType>HttpAgent</AgentType>
</Source> 
              
Several sources can be concerned by the consolidation :
                
<Source Number="7">
  <Agent>1</Agent>
  <Agent>3</Agent>
</Source>
<Source Number="13">
  <AgentType>LinuxAgent</AgentType>
</Source> 
              

Data name. After having selected sources and agents, you must select data name
:
                
<Data>cpu_user</Data>
<Data>cpu_system</Data>
<Data>memory_used</Data>
<Data>memory_total</Data> 
              
You can select all data of the agent :
                
<AllData/> 
              

-------------------------------------------------------------------------------

Consolidation Rules

Now, you must declare rules to apply to data :
              
<Rule Method="Mean">
   <Offset>30</Offset>
   <Factor>10</Factor>
</Rule> 
            
This rule stand for : "For data older than 30 days, do average 10 per 10".

The Method attribute specify the methd to use for consolidation. Possible
values :

  * Mean
   
  * Max
   
  * Min
   
By default, Mean is used.

Several rules can by applied :
              
<Rule Method="Max">
  <Offset>7</Offset>
  <Factor>5</Factor>
</Rule>
<Rule Method="Max">
  <Offset>30</Offset>
  <Factor>10</Factor>
</Rule> 
            
These rules stand for : "For data alder than 7 days, do the max 5 per 5 and for
data older 30 days, do the max 10 per 10.

    Note: For more detail to understand how works consolidation and how rules
    are applied, see the section called Example of consolidation.
   
-------------------------------------------------------------------------------

Default rules (optional)

            
<DefaultRules>
  <Rule>
    <Offset>30</Offset>
    <Factor>10</Factor>
  </Rule>
</DefaultRules> 
          
Default rules are applied to all data which was not selected in Consolidation
tags. Several Rules tag can be present in DefaultRule.

-------------------------------------------------------------------------------

How to run Consolidation ?

You can run consoldation with the command :
          # Consolidation my_conf_file.xml
        
You must run this command periodically (once per day) with cron (see
conso_crontab.example for examples).

-------------------------------------------------------------------------------

Example of consolidation

In this section, I would like to show you further example to help you to
understand how work the consolidation. Consider rules :

  * For data older than 7 days, do average 5 per 5.
   
  * For data older than 15 days, do average 10 per 10.
   
Example 1-1.

[example1]

In this example, one day is passed since the last consolidation. You can see
that the consolidation module can detect where a previous consolidation has
stopped and consolidate only new data.

In addition, you can see that the rule is applied by report to the current data
collection frequency. For data between 15 and 16 days, the rule said : "For
data older than 15 days, do average 10 per 10." But these data were already
consolidated 5 per 5. Consequently, these data will be consolidated only 2 per
2 (because 5*2 = 10).

+-----------------------------------------------------------------------------+
|                                   Warning                                   |
+-----------------------------------------------------------------------------+
|Because of that, each factor must be a multiple of the precedents.           |
|                                                                             |
|(5 10 20) is a good suite because 10 = 5*2, 20 = 4*5 and 20 = 2*10.          |
|                                                                             |
|(5 10 15) is NOT a good suite because it's not possible to consolidate data  |
|from 10 to 15.                                                               |
+-----------------------------------------------------------------------------+

Example 1-2.

[example2]

In this example, 17 days are passed since the last consolidation. But data
between 16 and 17 days were already consolidated 5 per 5. Consequently, data
between 16 and 17 days will be consolidated 2 per 2, data between 15 and 16
days (which have never been consolidated) will be consolidated 10 per 10 and
data between 7 and 15 will be consolidated 5 per 5.

