| 1 |
<?xml version="1.0" encoding="utf-8"?> |
| 2 |
<!DOCTYPE chapter SYSTEM "docbook-dtd-45/docbookx.dtd"> |
| 3 |
|
| 4 |
<chapter id="externals"> |
| 5 |
<title>External Callbacks and Processors</title> |
| 6 |
|
| 7 |
<para> |
| 8 |
The <link linkend="external">EXTERNAL rule types</link> spawn and pass a window to an external process, |
| 9 |
and expect a modified window in return. The external process is spawned only once the first time the rule hits, |
| 10 |
is then sent an initial message of the current protocol version, and subseqently passed only windows. |
| 11 |
It is expected that the reply from a window is another window, or a null reply if there are no changes. |
| 12 |
</para> |
| 13 |
|
| 14 |
<para> |
| 15 |
What follows is a description of the protocol using C++ structs. For those who want a more hands-on example, |
| 16 |
the source tree scripts/CG3_External.pm and scripts/external.pl and test/T_External/* is a working example of |
| 17 |
calling a Perl program that simply adds a tag to every reading. |
| 18 |
</para> |
| 19 |
|
| 20 |
<para> |
| 21 |
All datatypes correspond to the C and C++ <stdint.h> types, and no endianness conversion is performed; |
| 22 |
it is assumed that the external program is native to the same arch as CG-3 is running on. |
| 23 |
Fields marked 'const' denote they may not be changed by the external. |
| 24 |
Notice that you may not change the number of cohorts or readings, but you may change the number of tags per reading. |
| 25 |
</para> |
| 26 |
|
| 27 |
<section id="external-datatypes"> |
| 28 |
<title>Protocol Datatypes</title> |
| 29 |
|
| 30 |
<screen> |
| 31 |
uint32_t protocol_version = 7226; |
| 32 |
uint32_t null_response = 0; |
| 33 |
|
| 34 |
struct Text { |
| 35 |
uint16_t length; |
| 36 |
char *utf8_bytes; |
| 37 |
}; |
| 38 |
|
| 39 |
enum READING_FLAGS { |
| 40 |
R_WAS_MODIFIED = (1 << 0), |
| 41 |
R_INVISIBLE = (1 << 1), |
| 42 |
R_DELETED = (1 << 2), |
| 43 |
R_HAS_BASEFORM = (1 << 3), |
| 44 |
}; |
| 45 |
|
| 46 |
struct Reading { |
| 47 |
uint32_t reading_length; // sum of all data here plus data from all tags |
| 48 |
uint32_t flags; |
| 49 |
Text *baseform; // optional, depends on (flags & R_HAS_BASEFORM) |
| 50 |
|
| 51 |
uint32_t num_tags; |
| 52 |
Text *tags; |
| 53 |
}; |
| 54 |
|
| 55 |
enum COHORT_FLAGS { |
| 56 |
C_HAS_TEXT = (1 << 0), |
| 57 |
C_HAS_PARENT = (1 << 1), |
| 58 |
}; |
| 59 |
|
| 60 |
struct Cohort { |
| 61 |
uint32_t cohort_length; // sum of all data here plus data from all readings |
| 62 |
const uint32_t number; // which cohort is this |
| 63 |
uint32_t flags; |
| 64 |
uint32_t parent; // optional, depends on (flags & C_HAS_PARENT) |
| 65 |
Text wordform; |
| 66 |
|
| 67 |
const uint32_t num_readings; |
| 68 |
Reading *readings; |
| 69 |
|
| 70 |
Text *text; // optional, depends on (flags & C_HAS_TEXT) |
| 71 |
}; |
| 72 |
|
| 73 |
struct Window { |
| 74 |
uint32_t window_length; // sum of all data here plus data from all cohorts |
| 75 |
const uint32_t number; // which window is this |
| 76 |
|
| 77 |
const uint32_t num_cohorts; |
| 78 |
Cohort *cohorts; |
| 79 |
}; |
| 80 |
</screen> |
| 81 |
</section> |
| 82 |
|
| 83 |
<section id="external-dataflow"> |
| 84 |
<title>Protocol Flow</title> |
| 85 |
|
| 86 |
<orderedlist numeration="arabic"> |
| 87 |
<listitem> |
| 88 |
<para> |
| 89 |
Initial packet is simply the protocol_version. This is used to detect when an external may be out of date. |
| 90 |
If an external cannot reliably handle a protocol, I recommend that it terminates to avoid subtle bugs. |
| 91 |
Protocol version is only sent once and no response is allowed. |
| 92 |
</para> |
| 93 |
</listitem> |
| 94 |
<listitem> |
| 95 |
<para> |
| 96 |
Every time an EXTERNAL rule hits, a Window is sent. If you make no changes to the window, send a null_response. |
| 97 |
If you do change the window, you must compose a whole Window as response. |
| 98 |
If you change anything in a Reading, you must set the R_WAS_MODIFIED flag on the Reading. |
| 99 |
If you change a Cohort's wordform, that automatically sets the R_WAS_MODIFIED flags on all Readings. |
| 100 |
You must send some response for every window. |
| 101 |
</para> |
| 102 |
</listitem> |
| 103 |
<listitem> |
| 104 |
<para> |
| 105 |
When CG-3 is shutting down, the final thing it sends to the external is a null_reponse. |
| 106 |
Use this to clean up if necessary. Any data output after the null_response is ignored. |
| 107 |
</para> |
| 108 |
</listitem> |
| 109 |
</orderedlist> |
| 110 |
</section> |
| 111 |
|
| 112 |
</chapter> |