Debian Codegraph @

doc/ace-javascript-target.md

raw
1 # Integrating ANTLR JavaScript parsers with ACE editor
2
3 Having the ability to parse code other than JavaScript is great, but nowadays users expect to be able to edit code with nice edit features such as keyword highlighting, indentation and brace matching, and advanced ones such as syntax checking.
4
5 I have been through the process of integrating an ANTLR parser with ACE, the dominant code editor for web based code editing. Information about ACE can be found on their web site.
6
7 This page describes my experience, and humbly aims to help you get started. It is not however a reference guide, and no support is provided.
8
9 ## Architecture
10
11 The ACE editor is organized as follows
12
13 1. The editor itself is a <div> which once initialized comprises a number of elements. This UI element is responsible for the display, and the generation of edit events.
14 1. The editor relies on a Session, which manages events and configuration.
15 1. The code itself is stored in a Document. Any insertion or deletion of text is reflected in the Document.
16 1. Keyword highlighting, indentation and brace matching are delegated to a mode. There is no direct equivalent of an ACE mode in ANTLR. While keywords are the equivalent of ANTLR lexer tokens, indentation and brace matching are edit tasks, not parsing ones. A given ACE editor can only have one mode, which corresponds to the language being edited. There is no need for ANTLR integration to support keyword highlighting, indentation and brace matching.
17 1. Syntax checking is delegated to a worker. This is where ANTLR integration is needed. If syntax checking is enabled, ACE asks the mode to create a worker. In JavaScript, workers run in complete isolation i.e. they don't share code or variables with other workers, or with the HTML page itself.
18 1. The below diagram describes how the whole system works. In green are the components *you* need to provide. You'll notice that there is no need to load ANTLR in the HTML page itself. You'll also notice that ACE maintains a document in each thread. This is done through low level events sent by the ACE session to the worker which describe the delta. Once applied to the worker document, a high level event is triggered, which is easy to handle since at this point the worker document is a perfect copy of the UI document.
19
20 <img src=images/ACE-Architecture.001.png>
21
22 ## Step-by-step guide
23
24 The first thing to do is to create an editor in your html page. This is thoroughly described in the ACE documentation, so we'll just sum it up here:
25
26 ```xml
27 <script src="../js/ace/ace.js" type="text/javascript" charset="utf-8"></script>
28 <script>
29 var editor = ace.edit("editor");
30 </script>
31 ```
32
33 This should give you a working editor. You may want to control its sizing using CSS. I personally load the editor in an iframe and set its style to position: absolute, top: 0, left: 0 etc... but I'm sure you know better than me how to achieve results.
34
35 The second thing to do is to configure the ACE editor to use your mode i.e. language configuration. A good place to start is to inherit from the built-in TextMode. The following is a very simple example, which only caters for comments, literals, and a limited subset of separators and keywords :
36
37 ```javascript
38 ace.define('ace/mode/my-mode',["require","exports","module","ace/lib/oop","ace/mode/text","ace/mode/text_highlight_rules", "ace/worker/worker_client" ], function(require, exports, module) {
39 var oop = require("ace/lib/oop");
40 var TextMode = require("ace/mode/text").Mode;
41 var TextHighlightRules = require("ace/mode/text_highlight_rules").TextHighlightRules;
42
43 var MyHighlightRules = function() {
44 var keywordMapper = this.createKeywordMapper({
45 "keyword.control": "if|then|else",
46 "keyword.operator": "and|or|not",
47 "keyword.other": "class",
48 "storage.type": "int|float|text",
49 "storage.modifier": "private|public",
50 "support.function": "print|sort",
51 "constant.language": "true|false"
52 }, "identifier");
53 this.$rules = {
54 "start": [
55 { token : "comment", regex : "//" },
56 { token : "string", regex : '["](?:(?:\\\\.)|(?:[^"\\\\]))*?["]' },
57 { token : "constant.numeric", regex : "0[xX][0-9a-fA-F]+\\b" },
58 { token : "constant.numeric", regex: "[+-]?\\d+(?:(?:\\.\\d*)?(?:[eE][+-]?\\d+)?)?\\b" },
59 { token : "keyword.operator", regex : "!|%|\\\\|/|\\*|\\-|\\+|~=|==|<>|!=|<=|>=|=|<|>|&&|\\|\\|" },
60 { token : "punctuation.operator", regex : "\\?|\\:|\\,|\\;|\\." },
61 { token : "paren.lparen", regex : "[[({]" },
62 { token : "paren.rparen", regex : "[\\])}]" },
63 { token : "text", regex : "\\s+" },
64 { token: keywordMapper, regex: "[a-zA-Z_$][a-zA-Z0-9_$]*\\b" }
65 ]
66 };
67 };
68 oop.inherits(MyHighlightRules, TextHighlightRules);
69
70 var MyMode = function() {
71 this.HighlightRules = MyHighlightRules;
72 };
73 oop.inherits(MyMode, TextMode);
74
75 (function() {
76
77 this.$id = "ace/mode/my-mode";
78
79 }).call(MyMode.prototype);
80
81 exports.Mode = MyMode;
82 });
83 ```
84
85 Now if you store the above in a file called "my-mode.js", setting the ACE Editor becomes straightforward:
86
87 ```xml
88 <script src="../js/ace/ace.js" type="text/javascript" charset="utf-8"></script>
89 <script src="../js/my-mode.js" type="text/javascript" charset="utf-8"></script>
90 <script>
91 var editor = ace.edit("editor");
92 editor.getSession().setMode("ace/mode/my-mode");
93 </script>
94 ```
95
96 At this point you should have a working editor, able to highlight keywords. You may wonder why you need to set the tokens when you have already done so in your ANTLR lexer grammar. First, ACE expects a classification (control, operator, type...) which does not exist in ANTLR. Second, there is no need for ANTLR to achieve this, since ACE comes with its own lexer.
97
98 Ok, now that we have a working editor comes the time where we need syntax validation. This is where the worker comes in the picture.
99
100 Creating the worker is the responsibility of the mode you provide. So you need to enhance it with something like the following:
101
102 ```javascript
103 var WorkerClient = require("ace/worker/worker_client").WorkerClient;
104 this.createWorker = function(session) {
105 this.$worker = new WorkerClient(["ace"], "ace/worker/my-worker", "MyWorker", "../js/my-worker.js");
106 this.$worker.attachToDocument(session.getDocument());
107
108 this.$worker.on("errors", function(e) {
109 session.setAnnotations(e.data);
110 });
111
112 this.$worker.on("annotate", function(e) {
113 session.setAnnotations(e.data);
114 });
115
116 this.$worker.on("terminate", function() {
117 session.clearAnnotations();
118 });
119
120 return this.$worker;
121
122 };
123 ```
124
125 The above code needs to be placed in the existing worker, after:
126
127 ```javascript
128 this.$id = "ace/mode/my-mode";
129 ```
130
131 Please note that the mode code runs on the UI side, not the worker side. The event handlers here are for events sent by the worker, not to the worker.
132
133 Obviously the above won't work out of the box, because you need to provide the "my-worker.js" file.
134
135 Creating a worker from scratch is not something I've tried. Simply put, your worker needs to handle all messages sent by ACE using the WorkerClient created by the mode. This is not a simple task, and is better delegated to existing ACE code, so we can focus on tasks specific to our language.
136
137 What I did is I started from "mode-json.js", a rather simple worker which comes with ACE, stripped out all JSON validation related stuff out of it, and saved the remaining code in a file name "worker-base.js" which you can find [here](resources/worker-base.js). Once this done, I was able to create a simple worker, as follows:
138
139 ```javascript
140 importScripts("worker-base.js");
141 ace.define('ace/worker/my-worker',["require","exports","module","ace/lib/oop","ace/worker/mirror"], function(require, exports, module) {
142 "use strict";
143
144 var oop = require("ace/lib/oop");
145 var Mirror = require("ace/worker/mirror").Mirror;
146
147 var MyWorker = function(sender) {
148 Mirror.call(this, sender);
149 this.setTimeout(200);
150 this.$dialect = null;
151 };
152
153 oop.inherits(MyWorker, Mirror);
154
155 (function() {
156
157 this.onUpdate = function() {
158 var value = this.doc.getValue();
159 var annotations = validate(value);
160 this.sender.emit("annotate", annotations);
161 };
162
163 }).call(MyWorker.prototype);
164
165 exports.MyWorker = MyWorker;
166 });
167
168 var validate = function(input) {
169 return [ { row: 0, column: 0, text: "MyMode says Hello!", type: "error" } ];
170 };
171 ```
172
173 At this point, you should have an editor which displays an error icon next to the first line. When you hover over the error icon, it should display: MyMode says Hello!. Is that not a friendly worker? Yum.
174
175 What remains to be done is have our validate function actually validate the input. Finally ANTLR comes in the picture!
176
177 To start with, let's load ANTLR and your parser, listener etc..
178
179 The preferred approach for loading parser code is to bundle your parser, [as described here](javascript-target.md).
180 You can then load it as part of the importScripts instruction at the start of your worker code.
181
182 Another approach is to load it using 'require'. Easy, since you could write:
183
184 ```js
185 var antlr4 = require('antlr4/index');
186 ```
187
188 This may work, but it's actually unreliable. The reason is that the 'require' function that comes with ACE uses a different syntax than the 'require' function used by ANTLR, which follows the NodeJS 'require' convention.
189 So we need to bring in a NodeJS compatible 'require' function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find in li/require.js.
190 But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows:
191
192 ```js
193 // load nodejs compatible require
194 var ace_require = require;
195 require = undefined;
196 var Honey = { 'requirePath': ['..'] }; // walk up to js folder, see Honey docs
197 importScripts("../lib/require.js");
198 var antlr4_require = require;
199 require = ace_require;
200 ```
201 Now it's safe to load antlr and the parsers generated for your language.
202 Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows:
203 ```js
204 // load antlr4 and myLanguage
205 var antlr4, mylanguage;
206 try {
207 require = antlr4_require;
208 antlr4 = require('antlr4/index');
209 mylanguage = require('mylanguage/index');
210 } finally {
211 require = ace_require;
212 }
213 ```
214 Please note the try-finally construct. ANTLR uses 'require' synchronously so it's perfectly safe to ignore the ACE 'require' while running ANTLR code. ACE itself does not guarantee synchronous execution, so you are much safer always switching 'require' back to 'ace_require'.
215
216 Now detecting deep syntax errors in your code is a task for your ANTLR listener or visitor or whatever piece of code you've delegated this to. We're not going to describe this here, since it would require some knowledge of your language. However, detecting grammar syntax errors is something ANTLR does beautifully (isn't that why you went for ANTLR in the first place?). So what we will illustrate here is how to report grammar syntax errors. I have no doubt that from there, you will be able to extend the validator to suit your specific needs.
217 Whenever ANTLR encounters an unexpected token, it fires an error. By default, the error is routed to an error listener which simply writes to the console.
218 What we need to do is replace this listener by our own listener, se we can route errors to the ACE editor. First, let's create such a listener:
219 ```js
220 // class for gathering errors and posting them to ACE editor
221 var AnnotatingErrorListener = function(annotations) {
222 antlr4.error.ErrorListener.call(this);
223 this.annotations = annotations;
224 return this;
225 };
226
227 AnnotatingErrorListener.prototype = Object.create(antlr4.error.ErrorListener.prototype);
228 AnnotatingErrorListener.prototype.constructor = AnnotatingErrorListener;
229
230 AnnotatingErrorListener.prototype.syntaxError = function(recognizer, offendingSymbol, line, column, msg, e) {
231 this.annotations.push({
232 row: line - 1,
233 column: column,
234 text: msg,
235 type: "error"
236 });
237 };
238 ```
239 With this, all that remains to be done is plug the listener in when we parse the code. Here is how I do it:
240 ```js
241 var validate = function(input) {
242 var stream = CharStreams.fromString(input);
243 var lexer = new mylanguage.MyLexer(stream);
244 var tokens = new antlr4.CommonTokenStream(lexer);
245 var parser = new mylanguage.MyParser(tokens);
246 var annotations = [];
247 var listener = new AnnotatingErrorListener(annotations)
248 parser.removeErrorListeners();
249 parser.addErrorListener(listener);
250 parser.parseMyRule();
251 return annotations;
252 };
253 ```
254 You know what? That's it! You now have an ACE editor that does syntax validation using ANTLR! I hope you find this useful, and simple enough to get started.
255 Now wait, hey! How do you debug this? Well, as usual, using Chrome, since no other browser is able to debug worker code. What a shame...