Debian Codegraph @

dev/patchbot/README

raw
1 Patchbot: AI bot making use of Natural Language Processing to suggest backports
2 =============================================================== 2023-12-18 ====
3
4
5 Background
6 ----------
7
8 Selecting patches to backport from the development branch is a tedious task, in
9 part due to the abundance of patches and the fact that many bug fixes are for
10 that same version and not for backporting. The more it gets delayed, the harder
11 it becomes, and the harder it is to start, the less likely it gets started. The
12 urban legend along which one "just" has to do that periodically doesn't work
13 because certain patches need to be left hanging for a while under observation,
14 others need to be merged urgently, and for some, the person in charge of the
15 backport might simply need an opinion from the patch's author or the affected
16 subsystem maintainer, and this cannot make the whole backport process stall.
17
18 The information needed to figure if a patch needs to be backported is present
19 in the commit message, with varying nuances such as "may", "may not", "should",
20 "probably", "shouldn't unless", "keep under observation" etc. One particularly
21 that is specific to backports is that the opinion on a patch may change over
22 time, either because it was later found to be wrong or insufficient, or because
23 the former analysis mistakenly suggested to backport or not to.
24
25 This means that the person in charge of the backports has to read the whole
26 commit message for each patch, to figure the backporting instructions, and this
27 takes a while.
28
29 Several attempts were made over the years to try to partially automate this
30 task, including the cherry-pick mode of the "git-show-backports" utility that
31 eases navigation back-and-forth between commits.
32
33 Lately, a lot of progress was made in the domain of Natural Language
34 Understanding (NLU) and more generally Natural Language Processing (NLP). Since
35 the first attempts in early 2023 involving successive layers of the Roberta
36 model, called from totally unreliable Python code, and December 2023, the
37 situation evolved from promising but unusable to mostly autonomous.
38
39 For those interested in history, the first attempts in early 2023 involved
40 successive layers of the Roberta model, but these were relying on totally
41 unreliable Python code that broke all the time and could barely be transferred
42 to another machine without upgrading or downgrading the installed modules, and
43 it used to use huge amounts of resources for a somewhat disappointing result:
44 the verdicts were correct roughly 60-70% of the time, it was not possible to
45 get hints such as "wait" nor even "uncertain". It could just be qualified as
46 promising. Another big limitation was the limit to 256 tokens, forcing the
47 script to select only the last few lines of the commit message to take the
48 decision. Roughly at the same time, in March 2023 Meta issued their much larger
49 LLaMa model, and Georgi Gerganov released "llama.cpp", an open-source C++
50 engine that loads and runs such large models without all the usual problems
51 inherent to the Python ecosystem. New attempts were made with LLaMa and it was
52 already much better than Roberta, but the output was difficult to parse, and it
53 required to be combined with the final decision layer of Roberta. Then new
54 variants of LLaMa appeared such as Alpaca, which follows instructions, but
55 tends to forget them if given before the patch, then Vicuna which was pretty
56 reliable but very slow at 33B size and difficult to tune, then Airoboros,
57 which was the first one to give very satisfying results in a reasonable time,
58 following instructions reasonably closely with a stable output, but with
59 sometimes surprising analysis and contradictions. It was already about 90%
60 reliable and considered as a time saver in 13B size. Other models were later
61 tried as they appeared such as OpenChat-3.5, Juna, OpenInstruct, Orca-2,
62 Mistral-0.1 and it variants Neural and OpenHermes-2.5. Mistral showed an
63 unrivaled understanding despite being smaller and much faster than other ones,
64 but was a bit freewheeling regarding instructions. Dolphin-2.1 rebased on top
65 of it gave extremely satisfying results, with less variations in the output
66 format, but still the script had difficulties trying to catch its conclusion
67 from time to time, though it was pretty much readable for the human in charge
68 of the task. And finally just before releasing, Mistral-0.2 was released and
69 addressed all issues, with a human-like understanding and perfectly obeying
70 instructions, providing an extremely stable output format that is easy to parse
71 from simple scripts. The decisions now match the human's ones in close to 100%
72 of the patches, unless the human is aware of extra context, of course.
73
74
75 Architecture
76 ------------
77
78 The current solution relies on the llama.cpp engine, which is a simple, fast,
79 reliable and portable engine to load models and run inference, and the
80 Mistral-0.2 LLM.
81
82 A collection of patches is built from the development branch since the -dev0
83 tag, and for each of them, the engine is called to evaluate the developer's
84 intent based on the commit message. A detailed context explaining the haproxy
85 maintenance model and what the user wants is passed, then the LLM is invited to
86 provide its opinion on the need for a backport and an explanation of the reason
87 for its choice. This often helps the user to find a quick summary about the
88 patch. All these outputs are then converted to a long HTML page with colors and
89 radio buttons, where patches are pre-selected based on this classification,
90 that the user can consult and adjust, read the commits if needed, and the
91 selected patches finally provide some copy-pastable commands in a text-area to
92 select commit IDs to work on, typically in a form that's suitable for a simple
93 "git cherry-pick -sx".
94
95 The scripts are designed to be able to run on a headless machine, called from a
96 crontab and with the output served from a static HTTP server.
97
98 The code is currently found from Georgi Gerganov's repository:
99
100 https://github.com/ggerganov/llama.cpp
101
102 Tag b1505 is known to work fine, and uses the GGUF file format.
103
104 The model(s) can be found on Hugging Face user "TheBloke"'s collection of
105 models:
106
107 https://huggingface.co/TheBloke
108
109 Model Mistral-7B-Instruct-v0.2-GGUF quantized at Q5K_M is known to work well
110 with the llama.cpp version above.
111
112
113 Deployment
114 ----------
115
116 Note: it is a good idea to start to download the model(s) in the background as
117 such files are typically 5 GB or more and can take some time to download
118 depending on the internet bandwidth.
119
120 It seems reasonable to create a dedicated user to periodically run this task.
121 Let's call it "patchbot". Developers should be able to easily run a shell from
122 this user to perform some maintenance or testing (e.g. "sudo").
123
124 All paths are specified in the example "update-3.0.sh" script, and assume a
125 deployment in the user's home, so this is what is being described here. The
126 proposed deployment layout is the following:
127
128 $HOME (e.g. /home/patchbot)
129 |
130 +- data
131 | |
132 | +-- models # GGUF files from TheBloke's collection
133 | |
134 | +-- prompts # prompt*-pfx*, prompt*-sfx*, cache
135 | |
136 | +-- in
137 | | |
138 | | +-- haproxy # haproxy Git repo
139 | | |
140 | | +-- patches-3.0 # patches from development branch 3.0
141 | |
142 | +-- out # report directory (HTML)
143 |
144 +- prog
145 | |
146 | +-- bin # program(s)
147 | |
148 | +-- scripts # processing scripts
149 | |
150 | +-- llama.cpp # llama Git repository
151
152
153 - Let's first create the structure:
154
155 mkdir -p ~/data/{in,models,prompts} ~/prog/{bin,scripts}
156
157 - data/in/haproxy must contain a clone of the haproxy development tree that
158 will periodically be pulled from:
159
160 cd ~/data/in
161 git clone https://github.com/haproxy/haproxy
162 cd ~
163
164 - The prompt files are a copy of haproxy's "dev/patchbot/prompt/" subdirectory.
165 The prompt files are per-version because they contain references to the
166 haproxy development version number. For each prompt, there is a prefix
167 ("-pfx"), that is loaded before the patch, and a suffix ("-sfx") that
168 precises the user's expectations after reading the patch. For best efficiency
169 it's useful to place most of the explanation in the prefix and the least
170 possible in the suffix, because the prefix is cacheable. Different models
171 will use different instructions formats and different explanations, so it's
172 fine to keep a collection of prompts and use only one. Different instruction
173 formats are commonly used, "llama-2", "alpaca", "vicuna", "chatml" being
174 common. When experimenting with a new model, just copy-paste the closest one
175 and tune it for best results. Since we already cloned haproxy above, we'll
176 take the files from there:
177
178 cp ~/data/in/haproxy/dev/patchbot/prompt/*txt ~/data/prompts/
179
180 Upon first run, a cache file will be produced in this directory by parsing
181 an empty file and saving the current model's context. The cache file will
182 automatically be deleted and rebuilt if it is absent or older than the prefix
183 or suffix file. The cache files are specific to a model so when experimenting
184 with other models, be sure not to reuse the same cache file, or in doubt,
185 just delete them. Rebuilding the cache file typically takes around 2 minutes
186 of processing on a 8-core machine.
187
188 - The model(s) from TheBloke's Hugging Face account have to be downloaded in
189 GGUF file format, quantized at Q5K_M, and stored as-is into data/models/.
190
191 - data/in/patches-3.0/ is where the "mk-patch-list.sh" script will emit the
192 patches corresponding to new commits in the development branch. Its suffix
193 must match the name of the current development branch for patches to be found
194 there. In addition, the classification of the patches will be emitted there
195 next to the input patches, with the same name as the original file with a
196 suffix indicating what model/prompt combination was used.
197
198 mkdir -p ~/data/in/patches-3.0
199
200 - data/out is where the final report will be emitted. If running on a headless
201 machine, it is worth making sure that this directory is accessible from a
202 static web server. Thus either create a directory and place a symlink or
203 configuration somewhere in the web server's settings to reference this
204 location, or make it a symlink to another place already exported by the web
205 server and make sure the user has the permissions to write there.
206
207 mkdir -p ~/data/out
208
209 On Ubuntu-20.04 it was found that the package "micro-httpd" works out of the
210 box serving /var/www/html and follows symlinks. As such this is sufficient to
211 expose the reports:
212
213 sudo ln -s ~patchbot/data/out /var/www/html/patchbot
214
215 - prog/bin will contain the executable(s) needed to operate, namely "main" from
216 llama.cpp:
217
218 mkdir -p ~/prog/bin
219
220 - prog/llama.cpp is a clone of the "llama.cpp" GitHub repository. As of
221 december 2023, the project has improved its forward compatibility and it's
222 generally both safe and recommended to stay on the last version, hence to
223 just clone the master branch. In case of difficulties, tag b1505 was proven
224 to work well with the aforementioned model. Building is done by default for
225 the local platform, optimised for speed with native CPU.
226
227 mkdir -p ~/prog
228 cd ~/prog
229 git clone https://github.com/ggerganov/llama.cpp
230 [ only in case of problems: cd llama.cpp && git checkout b1505 ]
231
232 make -j$(nproc) main LLAMA_FAST=1
233 cp main ~/prog/bin/
234 cd ~
235
236 - prog/scripts needs the following scripts:
237 - mk-patch-list.sh from haproxy's scripts/ subdirectory
238 - submit-ai.sh, process-*.sh, post-ai.sh, update-*.sh
239
240 cp ~/data/in/haproxy/scripts/mk-patch-list.sh ~/prog/scripts/
241 cp ~/data/in/haproxy/dev/patchbot/scripts/*.sh ~/prog/scripts/
242
243 - verify that the various paths in update-3.0.sh match your choices, or
244 adjust them:
245
246 vi ~/prog/scripts/update-3.0.sh
247
248 - the tool is memory-bound, so a machine with more memory channels and/or
249 very fast memory will usually be faster than a higher CPU count with a
250 lower memory bandwidth. In addition, the performance is not linear with
251 the number of cores and experimentation shows that efficiency drops above
252 8 threads. For this reason the script integrates a "PARALLEL_RUNS" variable
253 indicating how many instances to run in parallel, each on its own patch.
254 This allows to make better use of the CPUs and memory bandwidth. Setting
255 2 instances for 8 cores / 16 threads gives optimal results on dual memory
256 channel systems.
257
258 From this point, executing this update script manually should work and produce
259 the result. Count around 0.5-2 mn per patch on a 8-core machine, so it can be
260 reasonably fast during the early development stages (before -dev1) but
261 unbearably long later, where it can make more sense to run it at night. It
262 should not report any error and should only report the total execution time.
263
264 If interrupted (Ctrl-C, logout, out of memory etc), check for incomplete .txt
265 files in ~/data/in/patches*/ that can result from this interruption, and delete
266 them because they will not be reproduced:
267
268 ls -lart ~/data/in/patches-3.0/*.txt
269 ls -lS ~/data/in/patches-3.0/*.txt
270
271 Once the output is produced, visit ~/data/out/ using a web browser and check
272 that the table loads correctly. Note that after a new release or a series of
273 backports, the table may appear empty, it's just because all known patches are
274 already backported and collapsed by default. Clicking on "All" at the top left
275 will unhide them.
276
277 Finally when satisfied, place it in a crontab, for example, run every hour:
278
279 crontab -e
280
281 # m h dom mon dow command
282 # run every hour at minute 02
283 2 * * * * /home/patchbot/update-3.0.sh
284
285
286 Usage
287 -----
288
289 Using the HTML output is a bit rustic but efficient. The interface is split in
290 5 columns from left to right:
291
292 - first column: patch number from 1 to N, just to ease navigation. Below the
293 number appears a radio button which allows to mark this patch as the start
294 of the review. When clicked, all prior patches disappear and are not listed
295 anymore. This can be undone by clicking on the radio button under the "All"
296 word in this column's header.
297
298
299 - second column: commit ID (abbreviated "CID" in the header). It's a 8-digit
300 shortened representation of the commit ID. It's presented as a link, which,
301 if clicked, will directly show that commit from the haproxy public
302 repository. Below the commit ID is the patch's author date in condensed
303 format "DD-MmmYY", e.g. "18-Dec23" for "18th December 2023". It was found
304 that having a date indication sometimes helps differentiate certain related
305 patches.
306
307 - third column: "Subject", this is the subject of the patch, prefixed with
308 the 4-digit number matching the file name in the directory (e.g. helps to
309 remove or reprocess one if needed). This is also a link to the same commit
310 in the haproxy's public repository. At the lower right under the subject
311 is the shortened e-mail address (only user@domain keeping only the first
312 part of the domain, e.g. "foo@haproxy"). Just like with the date, it helps
313 figuring what to expect after a recent discussion with a developer.
314
315 - fourth column: "Verdict". This column contains 4 radio buttons prefiguring
316 the choice for this patch between "N" for "No", represented in gray (this
317 patch should not be backported, let's drop it), "U" for "Uncertain" in
318 green (still unsure about it, most likely the author should be contacted),
319 "W" for "Wait" in blue (this patch should be backported but not
320 immediately, only after it has spent some time in the development branch),
321 and "Y" for "Yes" in red (this patch must be backported, let's pick it).
322 The choice is preselected by the scripts above, and since these are radio
323 buttons, the user is free to change this selection. Reloading will lose the
324 user's choices. When changing a selection, the line's background changes to
325 match a similar color tone, allowing to visually spot preselected patches.
326
327 - fifth column: reason for the choice. The scripts try to provide an
328 explanation for the choice of the preselection, and try to always end with
329 a conclusion among "yes", "no", "wait", "uncertain". The explanation
330 usually fits in 2-4 lines and is faster to read than a whole commit message
331 and very often pretty accurate. It's also been noticed that Mistral-v0.2
332 shows much less hallucinations than others (it doesn't seem to invent
333 information that was not part of its input), so seeing certain topics being
334 discussed there generally indicate that they were in the original commit
335 message. The scripts try to emphasize the sensitive parts of the commit
336 message such as risks, dependencies, referenced issues, oldest version to
337 backport to, etc. Elements that look like issues numbers and commit IDs are
338 turned to links to ease navigation.
339
340 In addition, in order to improve readability, the top of the table shows 4
341 buttons allowing to show/hide each category. For example, when trying to focus
342 only on "uncertain" and "wait", it can make sense to hide "N" and "Y" and click
343 "Y" or "N" on the displayed ones until there is none anymore.
344
345 In order to reduce the risk of missing a misqualified patch, those marked "BUG"
346 or "DOC" are displayed in bold even if tagged "No". It has been shown to be
347 sufficient to catch the eye when scrolling and encouraging to re-visit them.
348
349 More importantly, the script will try to also check which patches were already
350 backported to the previous stable version. Those that were backported will have
351 the first two columns colored gray, and by default, the review will start from
352 the first patch after the last backported one. This explains why just after a
353 backport, the table may appear empty with only the footer "New" checked.
354
355 Finally, at the bottom of the table is an editable, copy-pastable text area
356 that is redrawn at each click. It contains a series of 4 shell commands that
357 can be copy-pasted at once and assign commit IDs to 4 variables, one per
358 category. Most often only "y" will be of interest, so for example if the
359 review process ends with:
360
361 cid_y=( 7dab3e82 456ba6e9 75f5977f 917f7c74 )
362
363 Then copy-pasting it in a terminal already in the haproxy-2.9 directory and
364 issuing:
365
366 git cherry-pick -sx ${cid_y[@]}
367
368 Will result in all these patches to be backported to that version.
369
370
371 Criticisms
372 ----------
373
374 The interface is absolutely ugly but gets the job done. Proposals to revamp it
375 are welcome, provided that they do not alter usability and portability (e.g.
376 the ability to open the locally produced file without requiring access to an
377 external server).
378
379
380 Thanks
381 ------
382
383 This utility is the proof that boringly repetitive tasks that can be offloaded
384 from humans can save their time to do more productive things. This work which
385 started with extremely limited tools was made possible thanks to Meta, for
386 opening their models after leaking it, Georgi Gerganov and the community that
387 developed around llama.cpp, for creating the first really open engine that
388 builds out of the box and just works, contrary to the previous crippled Python-
389 only ecosystem, Tom Jobbins (aka TheBloke) for making it so easy to discover
390 new models every day by simply quantizing all of them and making them available
391 from a single location, MistralAI for producing an exceptionally good model
392 that surpasses all others, is the first one to feel as smart and accurate as a
393 real human on such tasks, is fast, and totally free, and of course, HAProxy
394 Technologies for investing some time on this and for the available hardware
395 that permits a lot of experimentation.