Debian Codegraph @

pipeline/doc.go

raw
1 // Copyright 2017 Microsoft Corporation. All rights reserved.
2 // Use of this source code is governed by an MIT
3 // license that can be found in the LICENSE file.
4
5 /*
6 Package pipeline implements an HTTP request/response middleware pipeline whose
7 policy objects mutate an HTTP request's URL, query parameters, and/or headers before
8 the request is sent over the wire.
9
10 Not all policy objects mutate an HTTP request; some policy objects simply impact the
11 flow of requests/responses by performing operations such as logging, retry policies,
12 timeouts, failure injection, and deserialization of response payloads.
13
14 Implementing the Policy Interface
15
16 To implement a policy, define a struct that implements the pipeline.Policy interface's Do method. Your Do
17 method is called when an HTTP request wants to be sent over the network. Your Do method can perform any
18 operation(s) it desires. For example, it can log the outgoing request, mutate the URL, headers, and/or query
19 parameters, inject a failure, etc. Your Do method must then forward the HTTP request to next Policy object
20 in a linked-list ensuring that the remaining Policy objects perform their work. Ultimately, the last Policy
21 object sends the HTTP request over the network (by calling the HTTPSender's Do method).
22
23 When an HTTP response comes back, each Policy object in the linked-list gets a chance to process the response
24 (in reverse order). The Policy object can log the response, retry the operation if due to a transient failure
25 or timeout, deserialize the response body, etc. Ultimately, the last Policy object returns the HTTP response
26 to the code that initiated the original HTTP request.
27
28 Here is a template for how to define a pipeline.Policy object:
29
30 type myPolicy struct {
31 node PolicyNode
32 // TODO: Add configuration/setting fields here (if desired)...
33 }
34
35 func (p *myPolicy) Do(ctx context.Context, request pipeline.Request) (pipeline.Response, error) {
36 // TODO: Mutate/process the HTTP request here...
37 response, err := p.node.Do(ctx, request) // Forward HTTP request to next Policy & get HTTP response
38 // TODO: Mutate/process the HTTP response here...
39 return response, err // Return response/error to previous Policy
40 }
41
42 Implementing the Factory Interface
43
44 Each Policy struct definition requires a factory struct definition that implements the pipeline.Factory interface's New
45 method. The New method is called when application code wants to initiate a new HTTP request. Factory's New method is
46 passed a pipeline.PolicyNode object which contains a reference to the owning pipeline.Pipeline object (discussed later) and
47 a reference to the next Policy object in the linked list. The New method should create its corresponding Policy object
48 passing it the PolicyNode and any other configuration/settings fields appropriate for the specific Policy object.
49
50 Here is a template for how to define a pipeline.Policy object:
51
52 // NOTE: Once created & initialized, Factory objects should be goroutine-safe (ex: immutable);
53 // this allows reuse (efficient use of memory) and makes these objects usable by multiple goroutines concurrently.
54 type myPolicyFactory struct {
55 // TODO: Add any configuration/setting fields if desired...
56 }
57
58 func (f *myPolicyFactory) New(node pipeline.PolicyNode) Policy {
59 return &myPolicy{node: node} // TODO: Also initialize any configuration/setting fields here (if desired)...
60 }
61
62 Using your Factory and Policy objects via a Pipeline
63
64 To use the Factory and Policy objects, an application constructs a slice of Factory objects and passes
65 this slice to the pipeline.NewPipeline function.
66
67 func NewPipeline(factories []pipeline.Factory, sender pipeline.HTTPSender) Pipeline
68
69 This function also requires an object implementing the HTTPSender interface. For simple scenarios,
70 passing nil for HTTPSender causes a standard Go http.Client object to be created and used to actually
71 send the HTTP response over the network. For more advanced scenarios, you can pass your own HTTPSender
72 object in. This allows sharing of http.Client objects or the use of custom-configured http.Client objects
73 or other objects that can simulate the network requests for testing purposes.
74
75 Now that you have a pipeline.Pipeline object, you can create a pipeline.Request object (which is a simple
76 wrapper around Go's standard http.Request object) and pass it to Pipeline's Do method along with passing a
77 context.Context for cancelling the HTTP request (if desired).
78
79 type Pipeline interface {
80 Do(ctx context.Context, methodFactory pipeline.Factory, request pipeline.Request) (pipeline.Response, error)
81 }
82
83 Do iterates over the slice of Factory objects and tells each one to create its corresponding
84 Policy object. After the linked-list of Policy objects have been created, Do calls the first
85 Policy object passing it the Context & HTTP request parameters. These parameters now flow through
86 all the Policy objects giving each object a chance to look at and/or mutate the HTTP request.
87 The last Policy object sends the message over the network.
88
89 When the network operation completes, the HTTP response and error return values pass
90 back through the same Policy objects in reverse order. Most Policy objects ignore the
91 response/error but some log the result, retry the operation (depending on the exact
92 reason the operation failed), or deserialize the response's body. Your own Policy
93 objects can do whatever they like when processing outgoing requests or incoming responses.
94
95 Note that after an I/O request runs to completion, the Policy objects for that request
96 are garbage collected. However, Pipeline object (like Factory objects) are goroutine-safe allowing
97 them to be created once and reused over many I/O operations. This allows for efficient use of
98 memory and also makes them safely usable by multiple goroutines concurrently.
99
100 Inserting a Method-Specific Factory into the Linked-List of Policy Objects
101
102 While Pipeline and Factory objects can be reused over many different operations, it is
103 common to have special behavior for a specific operation/method. For example, a method
104 may need to deserialize the response's body to an instance of a specific data type.
105 To accommodate this, the Pipeline's Do method takes an additional method-specific
106 Factory object. The Do method tells this Factory to create a Policy object and
107 injects this method-specific Policy object into the linked-list of Policy objects.
108
109 When creating a Pipeline object, the slice of Factory objects passed must have 1
110 (and only 1) entry marking where the method-specific Factory should be injected.
111 The Factory marker is obtained by calling the pipeline.MethodFactoryMarker() function:
112
113 func MethodFactoryMarker() pipeline.Factory
114
115 Creating an HTTP Request Object
116
117 The HTTP request object passed to Pipeline's Do method is not Go's http.Request struct.
118 Instead, it is a pipeline.Request struct which is a simple wrapper around Go's standard
119 http.Request. You create a pipeline.Request object by calling the pipeline.NewRequest function:
120
121 func NewRequest(method string, url url.URL, options pipeline.RequestOptions) (request pipeline.Request, err error)
122
123 To this function, you must pass a pipeline.RequestOptions that looks like this:
124
125 type RequestOptions struct {
126 // The readable and seekable stream to be sent to the server as the request's body.
127 Body io.ReadSeeker
128
129 // The callback method (if not nil) to be invoked to report progress as the stream is uploaded in the HTTP request.
130 Progress ProgressReceiver
131 }
132
133 The method and struct ensure that the request's body stream is a read/seekable stream.
134 A seekable stream is required so that upon retry, the final Policy object can seek
135 the stream back to the beginning before retrying the network request and re-uploading the
136 body. In addition, you can associate a ProgressReceiver callback function which will be
137 invoked periodically to report progress while bytes are being read from the body stream
138 and sent over the network.
139
140 Processing the HTTP Response
141
142 When an HTTP response comes in from the network, a reference to Go's http.Response struct is
143 embedded in a struct that implements the pipeline.Response interface:
144
145 type Response interface {
146 Response() *http.Response
147 }
148
149 This interface is returned through all the Policy objects. Each Policy object can call the Response
150 interface's Response method to examine (or mutate) the embedded http.Response object.
151
152 A Policy object can internally define another struct (implementing the pipeline.Response interface)
153 that embeds an http.Response and adds additional fields and return this structure to other Policy
154 objects. This allows a Policy object to deserialize the body to some other struct and return the
155 original http.Response and the additional struct back through the Policy chain. Other Policy objects
156 can see the Response but cannot see the additional struct with the deserialized body. After all the
157 Policy objects have returned, the pipeline.Response interface is returned by Pipeline's Do method.
158 The caller of this method can perform a type assertion attempting to get back to the struct type
159 really returned by the Policy object. If the type assertion is successful, the caller now has
160 access to both the http.Response and the deserialized struct object.*/
161 package pipeline