| 1 |
# Unicode Text Segmentation for Go |
| 2 |
|
| 3 |
[](https://godoc.org/github.com/rivo/uniseg) |
| 4 |
[](https://goreportcard.com/report/github.com/rivo/uniseg) |
| 5 |
|
| 6 |
This Go package implements Unicode Text Segmentation according to [Unicode Standard Annex #29](http://unicode.org/reports/tr29/) (Unicode version 12.0.0). |
| 7 |
|
| 8 |
At this point, only the determination of grapheme cluster boundaries is implemented. |
| 9 |
|
| 10 |
## Background |
| 11 |
|
| 12 |
In Go, [strings are read-only slices of bytes](https://blog.golang.org/strings). They can be turned into Unicode code points using the `for` loop or by casting: `[]rune(str)`. However, multiple code points may be combined into one user-perceived character or what the Unicode specification calls "grapheme cluster". Here are some examples: |
| 13 |
|
| 14 |
|String|Bytes (UTF-8)|Code points (runes)|Grapheme clusters| |
| 15 |
|-|-|-|-| |
| 16 |
|Käse|6 bytes: `4b 61 cc 88 73 65`|5 code points: `4b 61 308 73 65`|4 clusters: `[4b],[61 308],[73],[65]`| |
| 17 |
|🏳️🌈|14 bytes: `f0 9f 8f b3 ef b8 8f e2 80 8d f0 9f 8c 88`|4 code points: `1f3f3 fe0f 200d 1f308`|1 cluster: `[1f3f3 fe0f 200d 1f308]`| |
| 18 |
|🇩🇪|8 bytes: `f0 9f 87 a9 f0 9f 87 aa`|2 code points: `1f1e9 1f1ea`|1 cluster: `[1f1e9 1f1ea]`| |
| 19 |
|
| 20 |
This package provides a tool to iterate over these grapheme clusters. This may be used to determine the number of user-perceived characters, to split strings in their intended places, or to extract individual characters which form a unit. |
| 21 |
|
| 22 |
## Installation |
| 23 |
|
| 24 |
```bash |
| 25 |
go get github.com/rivo/uniseg |
| 26 |
``` |
| 27 |
|
| 28 |
## Basic Example |
| 29 |
|
| 30 |
```go |
| 31 |
package uniseg |
| 32 |
|
| 33 |
import ( |
| 34 |
"fmt" |
| 35 |
|
| 36 |
"github.com/rivo/uniseg" |
| 37 |
) |
| 38 |
|
| 39 |
func main() { |
| 40 |
gr := uniseg.NewGraphemes("👍🏼!") |
| 41 |
for gr.Next() { |
| 42 |
fmt.Printf("%x ", gr.Runes()) |
| 43 |
} |
| 44 |
// Output: [1f44d 1f3fc] [21] |
| 45 |
} |
| 46 |
``` |
| 47 |
|
| 48 |
## Documentation |
| 49 |
|
| 50 |
Refer to https://godoc.org/github.com/rivo/uniseg for the package's documentation. |
| 51 |
|
| 52 |
## Dependencies |
| 53 |
|
| 54 |
This package does not depend on any packages outside the standard library. |
| 55 |
|
| 56 |
## Your Feedback |
| 57 |
|
| 58 |
Add your issue here on GitHub. Feel free to get in touch if you have any questions. |
| 59 |
|
| 60 |
## Version |
| 61 |
|
| 62 |
Version tags will be introduced once Golang modules are official. Consider this version 0.1. |