Debian Codegraph @

examples/ratod/README

raw
1 /*
2 * Gargoyle Client Software. Tools to read, analyze and manage Argus data.
3 * Copyright (c) 2000-2024 QoSient, LLC
4 * All rights reserved.
5 *
6 * THE ACCOMPANYING PROGRAM IS PROPRIETARY SOFTWARE OF QoSIENT, LLC,
7 * AND CANNOT BE USED, DISTRIBUTED, COPIED OR MODIFIED WITHOUT
8 * EXPRESS PERMISSION OF QoSIENT, LLC.
9 *
10 * QOSIENT, LLC DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
11 * SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
12 * AND FITNESS, IN NO EVENT SHALL QOSIENT, LLC BE LIABLE FOR ANY
13 * SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
14 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER
15 * IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
16 * ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
17 * THIS SOFTWARE.
18 *
19 */
20
21 /*
22 * ratod - tools used to develop time of day behavioral flow data
23 * The essence of the effort is to generate the canonical day,
24 * through statistical process of 1,000s of days, and comparing
25 * data presentations.
26 *
27 * The first concept is to remove the time from the flows, so
28 * for daily processing, we'll set the starting time to be
29 * 0 by subtracting the starting second for the specific day,
30 * and all other time is offset by this value.
31 *
32 * Then, we'll bin the data to generate 5m inventories.
33 * While this maybe a lot of data, we'll do some experiments to
34 * see what works ...
35 *
36 *
37 *
38 * Author: Carter Bullard [email protected]
39 */
40
41 NOTES
42
43 OK, so currently the implementation provides rabins() behavior
44 with time normalization to Jan 1, 1970. This provides the
45 greatest flexibility to generate aggregated or unaggregated
46 data. Right now we're doing Time of Whatever normalization.
47 Using the time bins command line configuration data to
48 specify arbitrary time period normalization.
49
50 The easiest support for ToD processing is to stream data
51 through a tool that only does time normalization, but
52 putting the rabins() aggregation does provide most of
53 what we need in a tool like this.
54
55 Simple normalization of the time is not enough to provide usable
56 data, as direction information is critical to making useful TOD
57 flow run data.
58
59 The concept to date is to provide time normalized 300sec resolved
60 L2/L3 service oriented flow run data. This will be read in using
61 the " -M rtime " options, so that it represents a TOD cache of
62 expected network behavior. The flow records can span the entire
63 day, if, for example, there are no idle times, or it can be broken
64 down into 300 sec bins.
65
66 NOTE - Racurses() will need to manage its timeout values based on
67 " ltime " for this to work.
68
69
70 Service oriented flow run data
71 Normalizing a set of flows into a daily canonical flow run dataset,
72 involes removing the ephemeral data, such as source port values.
73 This demands that service direction be a part of the process, so
74 as to minimize ephemeral collision (where ephemeral data is inadvertently
75 introduced into the comparison sets).
76
77 The goal is identify new flows that are outside the services box,
78 as well as identify when expected flows do not show up.
79 The primary motivator in the ratod() analytics.
80