Debian Codegraph @

CHANGELOG-3.0.md

raw
1 # Elastic 3.0
2
3 Elasticsearch 2.0 comes with some [breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking-changes-2.0.html). You will probably need to upgrade your application and/or rewrite part of it due to those changes.
4
5 We use that window of opportunity to also update Elastic (the Go client) from version 2.0 to 3.0. This will introduce both changes due to the Elasticsearch 2.0 update as well as changes that make Elastic cleaner by removing some old cruft.
6
7 So, to summarize:
8
9 1. Elastic 2.0 is compatible with Elasticsearch 1.7+ and is still actively maintained.
10 2. Elastic 3.0 is compatible with Elasticsearch 2.0+ and will soon become the new master branch.
11
12 The rest of the document is a list of all changes in Elastic 3.0.
13
14 ## Pointer types
15
16 All types have changed to be pointer types, not value types. This not only is cleaner but also simplifies the API as illustrated by the following example:
17
18 Example for Elastic 2.0 (old):
19
20 ```go
21 q := elastic.NewMatchAllQuery()
22 res, err := elastic.Search("one").Query(&q).Do() // notice the & here
23 ```
24
25 Example for Elastic 3.0 (new):
26
27 ```go
28 q := elastic.NewMatchAllQuery()
29 res, err := elastic.Search("one").Query(q).Do() // no more &
30 // ... which can be simplified as:
31 res, err := elastic.Search("one").Query(elastic.NewMatchAllQuery()).Do()
32 ```
33
34 It also helps to prevent [subtle issues](https://github.com/olivere/elastic/issues/115#issuecomment-130753046).
35
36 ## Query/filter merge
37
38 One of the biggest changes in Elasticsearch 2.0 is the [merge of queries and filters](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_queries_and_filters_merged). In Elasticsearch 1.x, you had a whole range of queries and filters that were basically identical (e.g. `term_query` and `term_filter`).
39
40 The practical aspect of the merge is that you can now basically use queries where once you had to use filters instead. For Elastic 3.0 this means: We could remove a whole bunch of files. Yay!
41
42 Notice that some methods still come by "filter", e.g. `PostFilter`. However, they accept a `Query` now when they used to accept a `Filter` before.
43
44 Example for Elastic 2.0 (old):
45
46 ```go
47 q := elastic.NewMatchAllQuery()
48 f := elastic.NewTermFilter("tag", "important")
49 res, err := elastic.Search().Index("one").Query(&q).PostFilter(f)
50 ```
51
52 Example for Elastic 3.0 (new):
53
54 ```go
55 q := elastic.NewMatchAllQuery()
56 f := elastic.NewTermQuery("tag", "important") // it's a query now!
57 res, err := elastic.Search().Index("one").Query(q).PostFilter(f)
58 ```
59
60 ## Facets are removed
61
62 [Facets have been removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_removed_features.html#_facets_have_been_removed) in Elasticsearch 2.0. You need to use aggregations now.
63
64 ## Errors
65
66 Elasticsearch 2.0 returns more information about an error in the HTTP response body. Elastic 3.0 now reads this information and makes it accessible by the consumer.
67
68 Errors and all its details are now returned in [`Error`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59).
69
70 ### HTTP Status 404 (Not Found)
71
72 When Elasticsearch does not find an entity or an index, it generally returns HTTP status code 404. In Elastic 2.0 this was a valid result and didn't raise an error from the `Do` functions. This has now changed in Elastic 3.0.
73
74 Starting with Elastic 3.0, there are only two types of responses considered successful. First, responses with HTTP status codes [200..299]. Second, HEAD requests which return HTTP status 404. The latter is used by Elasticsearch to e.g. check for existence of indices or documents. All other responses will return an error.
75
76 To check for HTTP Status 404 (with non-HEAD requests), e.g. when trying to get or delete a missing document, you can use the [`IsNotFound`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L84) helper (see below).
77
78 The following example illustrates how to check for a missing document in Elastic 2.0 and what has changed in 3.0.
79
80 Example for Elastic 2.0 (old):
81
82 ```go
83 res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
84 if err != nil {
85 // Something else went wrong (but 404 is NOT an error in Elastic 2.0)
86 }
87 if !res.Found {
88 // Document has not been found
89 }
90 ```
91
92 Example for Elastic 3.0 (new):
93
94 ```go
95 res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
96 if err != nil {
97 if elastic.IsNotFound(err) {
98 // Document has not been found
99 } else {
100 // Something else went wrong
101 }
102 }
103 ```
104
105 ### HTTP Status 408 (Timeouts)
106
107 Elasticsearch now responds with HTTP status code 408 (Timeout) when a request fails due to a timeout. E.g. if you specify a timeout with the Cluster Health API, the HTTP response status will be 408 if the timeout is raised. See [here](https://github.com/elastic/elasticsearch/commit/fe3179d9cccb569784434b2135ca9ae13d5158d3) for the specific commit to the Cluster Health API.
108
109 To check for HTTP Status 408, we introduced the [`IsTimeout`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L101) helper.
110
111 Example for Elastic 2.0 (old):
112
113 ```go
114 health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
115 if err != nil {
116 // ...
117 }
118 if health.TimedOut {
119 // We have a timeout
120 }
121 ```
122
123 Example for Elastic 3.0 (new):
124
125 ```go
126 health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
127 if elastic.IsTimeout(err) {
128 // We have a timeout
129 }
130 ```
131
132 ### Bulk Errors
133
134 The error response of a bulk operation used to be a simple string in Elasticsearch 1.x.
135 In Elasticsearch 2.0, it returns a structured JSON object with a lot more details about the error.
136 These errors are now captured in an object of type [`ErrorDetails`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59) which is used in [`BulkResponseItem`](https://github.com/olivere/elastic/blob/release-branch.v3/bulk.go#L206).
137
138 ### Removed specific Elastic errors
139
140 The specific error types `ErrMissingIndex`, `ErrMissingType`, and `ErrMissingId` have been removed. They were only used by `DeleteService` and are replaced by a generic error message.
141
142 ## Numeric types
143
144 Elastic 3.0 has settled to use `float64` everywhere. It used to be a mix of `float32` and `float64` in Elastic 2.0. E.g. all boostable queries in Elastic 3.0 now have a boost type of `float64` where it used to be `float32`.
145
146 ## Pluralization
147
148 Some services accept zero, one or more indices or types to operate on.
149 E.g. in the `SearchService` accepts a list of zero, one, or more indices to
150 search and therefor had a func called `Index(index string)` and a func
151 called `Indices(indices ...string)`.
152
153 Elastic 3.0 now only uses the singular form that, when applicable, accepts a
154 variadic type. E.g. in the case of the `SearchService`, you now only have
155 one func with the following signature: `Index(indices ...string)`.
156
157 Notice this is only limited to `Index(...)` and `Type(...)`. There are other
158 services with variadic functions. These have not been changed.
159
160 ## Multiple calls to variadic functions
161
162 Some services with variadic functions have cleared the underlying slice when
163 called while other services just add to the existing slice. This has now been
164 normalized to always add to the underlying slice.
165
166 Example for Elastic 2.0 (old):
167
168 ```go
169 // Would only cleared scroll id "two"
170 // because ScrollId cleared the values when called multiple times
171 client.ClearScroll().ScrollId("one").ScrollId("two").Do()
172 ```
173
174 Example for Elastic 3.0 (new):
175
176 ```go
177 // Now (correctly) clears both scroll id "one" and "two"
178 // because ScrollId no longer clears the values when called multiple times
179 client.ClearScroll().ScrollId("one").ScrollId("two").Do()
180 ```
181
182 ## Ping service requires URL
183
184 The `Ping` service raised some issues because it is different from all
185 other services. If not explicitly given a URL, it always pings `127.0.0.1:9200`.
186
187 Users expected to ping the cluster, but that is not possible as the cluster
188 can be a set of many nodes: So which node do we ping then?
189
190 To make it more clear, the `Ping` function on the client now requires users
191 to explicitly set the URL of the node to ping.
192
193 ## Meta fields
194
195 Many of the meta fields e.g. `_parent` or `_routing` are now
196 [part of the top-level of a document](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_mapping_changes.html#migration-meta-fields)
197 and are no longer returned as parts of the `fields` object. We had to change
198 larger parts of e.g. the `Reindexer` to get it to work seamlessly with Elasticsearch 2.0.
199
200 Notice that all stored meta-fields are now [returned by default](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_crud_and_routing_changes.html#_all_stored_meta_fields_returned_by_default).
201
202 ## HasParentQuery / HasChildQuery
203
204 `NewHasParentQuery` and `NewHasChildQuery` must now include both parent/child type and query. It is now in line with the Java API.
205
206 Example for Elastic 2.0 (old):
207
208 ```go
209 allQ := elastic.NewMatchAllQuery()
210 q := elastic.NewHasChildFilter("tweet").Query(&allQ)
211 ```
212
213 Example for Elastic 3.0 (new):
214
215 ```go
216 q := elastic.NewHasChildQuery("tweet", elastic.NewMatchAllQuery())
217 ```
218
219 ## SetBasicAuth client option
220
221 You can now tell Elastic to pass HTTP Basic Auth credentials with each request. In previous versions of Elastic you had to set up your own `http.Transport` to do this. This should make it more convenient to use Elastic in combination with [Shield](https://www.elastic.co/products/shield) in its [basic setup](https://www.elastic.co/guide/en/shield/current/enable-basic-auth.html).
222
223 Example:
224
225 ```go
226 client, err := elastic.NewClient(elastic.SetBasicAuth("user", "secret"))
227 if err != nil {
228 log.Fatal(err)
229 }
230 ```
231
232 ## Delete-by-Query API
233
234 The Delete-by-Query API is [a plugin now](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_removed_features.html#_delete_by_query_is_now_a_plugin). It is no longer core part of Elasticsearch. You can [install it as a plugin as described here](https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugins-delete-by-query.html).
235
236 Elastic 3.0 still contains the `DeleteByQueryService`, but you need to install the plugin first. If you don't install it and use `DeleteByQueryService` you will most probably get a 404.
237
238 An older version of this document stated the following:
239
240 > Elastic 3.0 still contains the `DeleteByQueryService` but it will fail with `ErrPluginNotFound` when the plugin is not installed.
241 >
242 > Example for Elastic 3.0 (new):
243 >
244 > ```go
245 > _, err := client.DeleteByQuery().Query(elastic.NewTermQuery("client", "1")).Do()
246 > if err == elastic.ErrPluginNotFound {
247 > // Delete By Query API is not available
248 > }
249 > ```
250
251 I have decided that this is not a good way to handle the case of a missing plugin. The main reason is that with this logic, you'd always have to check if the plugin is missing in case of an error. This is not only slow, but it also puts logic into a service where it should really be just opaque and return the response of Elasticsearch.
252
253 If you rely on certain plugins to be installed, you should check on startup. That's where the following two helpers come into play.
254
255 ## HasPlugin and SetRequiredPlugins
256
257 Some of the core functionality of Elasticsearch has now been moved into plugins. E.g. the Delete-by-Query API is [a plugin now](https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugins-delete-by-query.html).
258
259 You need to make sure to add these plugins to your Elasticsearch installation to still be able to use the `DeleteByQueryService`. You can test this now with the `HasPlugin(name string)` helper in the client.
260
261 Example for Elastic 3.0 (new):
262
263 ```go
264 err, found := client.HasPlugin("delete-by-query")
265 if err == nil && found {
266 // ... Delete By Query API is available
267 }
268 ```
269
270 To simplify this process, there is now a `SetRequiredPlugins` helper that can be passed as an option func when creating a new client. If the plugin is not installed, the client wouldn't be created in the first place.
271
272 ```go
273 // Will raise an error if the "delete-by-query" plugin is NOT installed
274 client, err := elastic.NewClient(elastic.SetRequiredPlugins("delete-by-query"))
275 if err != nil {
276 log.Fatal(err)
277 }
278 ```
279
280 Notice that there also is a way to define [mandatory plugins](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#_mandatory_plugins) in the Elasticsearch configuration file.
281
282 ## Common Query has been renamed to Common Terms Query
283
284 The `CommonQuery` has been renamed to `CommonTermsQuery` to be in line with the [Java API](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_java_api_changes.html#_query_filter_refactoring).
285
286 ## Remove `MoreLikeThis` and `MoreLikeThisField`
287
288 The More Like This API and the More Like This Field query [have been removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_more_like_this) and replaced with the `MoreLikeThisQuery`.
289
290 ## Remove Filtered Query
291
292 With the merge of queries and filters, the [filtered query became deprecated](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_filtered_literal_query_and_literal_query_literal_filter_deprecated). While it is only deprecated and therefore still available in Elasticsearch 2.0, we have decided to remove it from Elastic 3.0. Why? Because we think that when you're already forced to rewrite many of your application code, it might be a good chance to get rid of things that are deprecated as well. So you might simply change your filtered query with a boolean query as [described here](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_filtered_literal_query_and_literal_query_literal_filter_deprecated).
293
294 ## Remove FuzzyLikeThis and FuzzyLikeThisField
295
296 Both have been removed from Elasticsearch 2.0 as well.
297
298 ## Remove LimitFilter
299
300 The `limit` filter is [deprecated in Elasticsearch 2.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_limit_literal_filter_deprecated) and becomes a no-op. Now is a good chance to remove it from your application as well. Use the `terminate_after` parameter in your search [as described here](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/search-request-body.html) to achieve similar effects.
301
302 ## Remove `_cache` and `_cache_key` from filters
303
304 Both have been [removed from Elasticsearch 2.0 as well](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_filter_auto_caching).
305
306 ## Partial fields are gone
307
308 Partial fields are [removed in Elasticsearch 2.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_search_changes.html#_partial_fields) in favor of [source filtering](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/search-request-source-filtering.html).
309
310 ## Scripting
311
312 A [`Script`](https://github.com/olivere/elastic/blob/release-branch.v3/script.go) type has been added to Elastic 3.0. In Elastic 2.0, there were various places (e.g. aggregations) where you could just add the script as a string, specify the scripting language, add parameters etc. With Elastic 3.0, you should now always use the `Script` type.
313
314 Example for Elastic 2.0 (old):
315
316 ```go
317 update, err := client.Update().Index("twitter").Type("tweet").Id("1").
318 Script("ctx._source.retweets += num").
319 ScriptParams(map[string]interface{}{"num": 1}).
320 Upsert(map[string]interface{}{"retweets": 0}).
321 Do()
322 ```
323
324 Example for Elastic 3.0 (new):
325
326 ```go
327 update, err := client.Update().Index("twitter").Type("tweet").Id("1").
328 Script(elastic.NewScript("ctx._source.retweets += num").Param("num", 1)).
329 Upsert(map[string]interface{}{"retweets": 0}).
330 Do()
331 ```
332
333 ## Cluster State
334
335 The combination of `Metric(string)` and `Metrics(...string)` has been replaced by a single func with the signature `Metric(...string)`.
336
337 ## Unexported structs in response
338
339 Services generally return a typed response from a `Do` func. Those structs are exported so that they can be passed around in your own application. In Elastic 3.0 however, we changed that (most) sub-structs are now unexported, meaning: You can only pass around the whole response, not sub-structures of it. This makes it easier for restructuring responses according to the Elasticsearch API. See [`ClusterStateResponse`](https://github.com/olivere/elastic/blob/release-branch.v3/cluster_state.go#L182) as an example.
340
341 ## Add offset to Histogram aggregation
342
343 Histogram aggregations now have an [offset](https://github.com/elastic/elasticsearch/pull/9505) option.
344
345 ## Services
346
347 ### REST API specification
348
349 As you might know, Elasticsearch comes with a REST API specification. The specification describes the endpoints in a JSON structure.
350
351 Most services in Elastic predated the REST API specification. We are in the process of bringing all these services in line with the specification. Services can be generated by `go generate` (not 100% automatic though). This is an ongoing process.
352
353 This probably doesn't mean a lot to you. However, you can now be more confident that Elastic supports all features that the REST API specification describes.
354
355 At the same time, the file names of the services are renamed to match the REST API specification naming.
356
357 ### REST API Test Suite
358
359 The REST API specification of Elasticsearch comes along with a test suite that official clients typically use to test for conformance. Up until now, Elastic didn't run this test suite. However, we are in the process of setting up infrastructure and tests to match this suite as well.
360
361 This process in not completed though.
362
363