Debian Codegraph @

notes/NEWIN_v5

raw
1 This is the v5.5.3 stable version
2
3 Note that large sections of the code has been re-written and there are
4 significant changes to the configuration files since v4.
5
6 V5 is written in c++11 and so to compile it you will need gcc v5.4
7 or later. (or another complier that supports the full c++11).
8 Note that the target systems may also need an c++11 library update.
9
10 See notes/NEWIN_v4 for details of the major process model changes
11 incorporated from version 4.
12
13 REVISED LIST and STORYBOARDING MODEL
14
15 Version 5 has a completely revised model for defining and using lists.
16
17 List definition is now separated from list application. Lists are no longer
18 hard-coded, but mapped to a function using a storyboard.
19
20 Filtering logic flow is simplified and made more consistent. Requests are
21 analysed first and flags set (exception, grey, blocked etc) and once this
22 checking is complete actions are taken. Large sections of duplicate logic
23 has been removed from ConnectionHandler and large sections are now
24 separate functions.
25
26 Storyboarding is a simple scripting language that maps lists to functions
27 and allows flags to be set.
28
29 This means that new lists can be added without changing the code, by adding
30 a new list definition and then applying it in a revised storyboard.
31
32 A different storyboard can be applied to each filtergroup, so if required,
33 each filtergroup can have a different logic flow.
34
35 Please read notes/V5_list_definition.pdf & notes/V5_Storyboard.pdf for
36 details.
37
38 TRANSPARENT HTTPS
39
40 Detects SNI and flags whether traffic is TLS.
41 Currently limited to port 443 traffic on BSD.
42 On Linux multiple ports can be redirected.
43
44 ICAP SERVER
45
46 REQMOD and RESPMOD mode supported.
47 See notes/icap.
48
49 DIRECT UPSTREAM ACCESS I.e. not via proxy.
50
51 To implement globaly comment out 'proxyip =' in e2guardian.conf.
52
53 The storyboard action setgodirect can be used within checkrequest functions
54 to enable selected protocols/site/urls to godirect.
55 e.g. to send all connect requests directly add
56 if(connect) setgodirect
57 to a requestchecks function.
58 This can be also useful to by-pass squid for some requests (e.g. os update
59 sites) when squid authentication methods are being used.
60
61 STORYBOARD TRACING
62
63 New option 'storyboardtrace' to enable tracing output -
64 for storyboard bug-fixing
65
66 READABLE THREAD_ID FOR LOGS & DEBUG
67
68 Most debug and syslog messages are now prefixed with a thread ID as
69 follows:-
70
71 master: for master thread
72
73 listen1_proxy: normal proxy listener
74 listen1_thttps: tranparent https listener
75 listen1_icap: icap listener
76
77 where '1' is index
78
79 hw10: for http_worker threads where '10' is the thread number
80
81 log: for logging thread for access.log
82
83 HTTP/1.1
84
85 Support for HTTP v1.1 completed - including Chunked encoding
86
87 ANTI-VIRUS PLUGINS
88
89 Anti-virus plugins implemented for proxy, trans and ICAP
90
91 INCLUDE FILES IN e2guardianf1.conf
92
93 Filtergroup configuration files may now include other files, enabling
94 a more DRY approach to configuration. So configuration common to several
95 filtergroups can be placed in a file which is included in the filtergroup
96 config file.
97
98 Syntax is same as list includes -
99 .include<full_path_to_file>
100
101 OPTION processing
102
103 Where single options and list defines with the same name are
104 repeated only the last one read will be actioned. This differs
105 from pre-v5 versions where the first single option was actioned and any
106 repeats ignored. This allows the overwriting of single options and
107 re-definition of lists in a structured way.
108
109 LIST INPUT VIA STDIN
110
111 This replaces the totalblocklist in previous versions allowing multiple
112 lists to be loaded via stdin. See notes/lists_via_stdin.
113
114 OPENSSL v1.1 SUPPORT
115
116 Supports OpenSSL v1.1
117 Note: Support for pre v1.1 OpenSSL versions is withdrawn in v5.5.
118
119 % OPTION FOR PHRASE LISTS
120
121 New % weighting option for phrase lists
122
123 Code amended so that -Wall compile option gives no warnings
124
125 BETTER HANDLING OF TRANSPARENT HTTPS REQUESTS
126
127 Requests without SNI and those not TLS no longer tried for MITM.
128
129 New option to use original ip & port (currently Linux only)
130
131 RELATIVE PATHS IN .Includes and list definition
132
133 The directory of the current .conf (or list file) is taken as the prefix
134 path.
135
136 LIST DIRECTORY VARIABLE
137
138 A list directory variable (LISTDIR) can be defined in config files.
139 This works like an exported shell variable and the contents will replace
140 __LISTDIR__ wherever it occurs in a .conf file or list file .Include.
141
142 This feature will allow the development of templates and easier separation
143 between boilerplate and user configuration.
144
145 REORGANIZED e2guardian.conf files and list directories.
146
147 Both example e2guardian.conf files are reorganized into sections and more
148 usage notes added to make it easier for new users to understand what changes
149 they will need.
150
151 List directory reorganized into sub-directories and READMEs added to help
152 users to know which list(s) to change.
153 The example.group sub-directory should be copied in a groupn directory for
154 each group.
155 Common lists are now in the common sub-directory.
156 As each group's lists are now in a different directory
157 the e2guardianf1.conf files will be able to be copied into a new group .conf
158 with minimal changes.
159
160 V5.5
161
162 NEW LOGGER and DEBUG system for logging, messages and debugging.
163
164 Thanks to KDGunermann for all his work on this!
165
166 This replaces the legacy and 'NEWDEBUG' with a unified logging system.
167
168 Note that the --with-e2debug configure option is replaced by two debug 'levels'
169 --with-debug_high intended for high level debugging (similar to NEWDEBUG, but covering the whole
170 product)
171 and --with-debug_low for low level code debugging
172
173 See notes/HOWTO_Logger.md and example e2guardian.conf for more details
174
175 LOG ROTATION
176
177 New command line option -t for log rotation
178
179 See notes/rotating_logs for more details
180
181 SECURE PROXY
182
183 Secure (TLS) Proxy is now supported.
184
185 See NETWORK section in e2guardian.conf for more information.
186
187 OTHER enhancements
188
189 Made sockets non-blocking so time-out works with MITM - completed but using blocking i/o
190
191 Reduce unnecessary memory usage on in DataBuffer & DMplugins
192
193 Remove broken mapauthtoports option/logic.
194
195 New Bearer over basic token-based auth plugin.
196
197 Integrate DMplugins with storyboards for plugins.
198
199 Option for output file for storyboard trace (part of new logging system)
200
201 Storyboard action by category added
202
203 alert.log option to enable reporting of key category violations by external email alert and reporting systems
204
205 'Auto' MITM certificate generation implimented
206
207
208 TESTING -
209 ---------
210
211 Please prefix all bug reports with "v5.5: "
212
213
214 TO DO IN THIS RELEASE
215 ---------------------
216
217
218 TODO - for later releases.
219 -----------------------
220
221 Create 'output-filter' plug-in system for content modification
222
223 POST content filtering
224
225 Implement a "learning" mode, test the rules effects without block
226
227 Remove duplicate list/storyboard warnings with a DRY system.
228
229
230 WISHLIST
231 --------
232
233 Adding a 'control' port (something similar to REST API) to pass commands
234 (eg: reload, update lists, etc) and check status
235
236 Multi-tenancy option.
237
238 Support for authenticaion helpers to allow ntlm, basic etc auth
239 without using squid proxy.
240
241 Per-user exceptions/blocks.
242
243 Per-room blocks.
244
245 Rate-limiting feature
246
247 HTTP/2 support (v6)
248
249 IPv6 support (v6)
250
251 Philip Pearce 4th April 2025
252
253