| 1 |
// Copyright 2017 Microsoft Corporation. All rights reserved. |
| 2 |
// Use of this source code is governed by an MIT |
| 3 |
// license that can be found in the LICENSE file. |
| 4 |
|
| 5 |
/* |
| 6 |
Package pipeline implements an HTTP request/response middleware pipeline whose |
| 7 |
policy objects mutate an HTTP request's URL, query parameters, and/or headers before |
| 8 |
the request is sent over the wire. |
| 9 |
|
| 10 |
Not all policy objects mutate an HTTP request; some policy objects simply impact the |
| 11 |
flow of requests/responses by performing operations such as logging, retry policies, |
| 12 |
timeouts, failure injection, and deserialization of response payloads. |
| 13 |
|
| 14 |
Implementing the Policy Interface |
| 15 |
|
| 16 |
To implement a policy, define a struct that implements the pipeline.Policy interface's Do method. Your Do |
| 17 |
method is called when an HTTP request wants to be sent over the network. Your Do method can perform any |
| 18 |
operation(s) it desires. For example, it can log the outgoing request, mutate the URL, headers, and/or query |
| 19 |
parameters, inject a failure, etc. Your Do method must then forward the HTTP request to next Policy object |
| 20 |
in a linked-list ensuring that the remaining Policy objects perform their work. Ultimately, the last Policy |
| 21 |
object sends the HTTP request over the network (by calling the HTTPSender's Do method). |
| 22 |
|
| 23 |
When an HTTP response comes back, each Policy object in the linked-list gets a chance to process the response |
| 24 |
(in reverse order). The Policy object can log the response, retry the operation if due to a transient failure |
| 25 |
or timeout, deserialize the response body, etc. Ultimately, the last Policy object returns the HTTP response |
| 26 |
to the code that initiated the original HTTP request. |
| 27 |
|
| 28 |
Here is a template for how to define a pipeline.Policy object: |
| 29 |
|
| 30 |
type myPolicy struct { |
| 31 |
node PolicyNode |
| 32 |
// TODO: Add configuration/setting fields here (if desired)... |
| 33 |
} |
| 34 |
|
| 35 |
func (p *myPolicy) Do(ctx context.Context, request pipeline.Request) (pipeline.Response, error) { |
| 36 |
// TODO: Mutate/process the HTTP request here... |
| 37 |
response, err := p.node.Do(ctx, request) // Forward HTTP request to next Policy & get HTTP response |
| 38 |
// TODO: Mutate/process the HTTP response here... |
| 39 |
return response, err // Return response/error to previous Policy |
| 40 |
} |
| 41 |
|
| 42 |
Implementing the Factory Interface |
| 43 |
|
| 44 |
Each Policy struct definition requires a factory struct definition that implements the pipeline.Factory interface's New |
| 45 |
method. The New method is called when application code wants to initiate a new HTTP request. Factory's New method is |
| 46 |
passed a pipeline.PolicyNode object which contains a reference to the owning pipeline.Pipeline object (discussed later) and |
| 47 |
a reference to the next Policy object in the linked list. The New method should create its corresponding Policy object |
| 48 |
passing it the PolicyNode and any other configuration/settings fields appropriate for the specific Policy object. |
| 49 |
|
| 50 |
Here is a template for how to define a pipeline.Policy object: |
| 51 |
|
| 52 |
// NOTE: Once created & initialized, Factory objects should be goroutine-safe (ex: immutable); |
| 53 |
// this allows reuse (efficient use of memory) and makes these objects usable by multiple goroutines concurrently. |
| 54 |
type myPolicyFactory struct { |
| 55 |
// TODO: Add any configuration/setting fields if desired... |
| 56 |
} |
| 57 |
|
| 58 |
func (f *myPolicyFactory) New(node pipeline.PolicyNode) Policy { |
| 59 |
return &myPolicy{node: node} // TODO: Also initialize any configuration/setting fields here (if desired)... |
| 60 |
} |
| 61 |
|
| 62 |
Using your Factory and Policy objects via a Pipeline |
| 63 |
|
| 64 |
To use the Factory and Policy objects, an application constructs a slice of Factory objects and passes |
| 65 |
this slice to the pipeline.NewPipeline function. |
| 66 |
|
| 67 |
func NewPipeline(factories []pipeline.Factory, sender pipeline.HTTPSender) Pipeline |
| 68 |
|
| 69 |
This function also requires an object implementing the HTTPSender interface. For simple scenarios, |
| 70 |
passing nil for HTTPSender causes a standard Go http.Client object to be created and used to actually |
| 71 |
send the HTTP response over the network. For more advanced scenarios, you can pass your own HTTPSender |
| 72 |
object in. This allows sharing of http.Client objects or the use of custom-configured http.Client objects |
| 73 |
or other objects that can simulate the network requests for testing purposes. |
| 74 |
|
| 75 |
Now that you have a pipeline.Pipeline object, you can create a pipeline.Request object (which is a simple |
| 76 |
wrapper around Go's standard http.Request object) and pass it to Pipeline's Do method along with passing a |
| 77 |
context.Context for cancelling the HTTP request (if desired). |
| 78 |
|
| 79 |
type Pipeline interface { |
| 80 |
Do(ctx context.Context, methodFactory pipeline.Factory, request pipeline.Request) (pipeline.Response, error) |
| 81 |
} |
| 82 |
|
| 83 |
Do iterates over the slice of Factory objects and tells each one to create its corresponding |
| 84 |
Policy object. After the linked-list of Policy objects have been created, Do calls the first |
| 85 |
Policy object passing it the Context & HTTP request parameters. These parameters now flow through |
| 86 |
all the Policy objects giving each object a chance to look at and/or mutate the HTTP request. |
| 87 |
The last Policy object sends the message over the network. |
| 88 |
|
| 89 |
When the network operation completes, the HTTP response and error return values pass |
| 90 |
back through the same Policy objects in reverse order. Most Policy objects ignore the |
| 91 |
response/error but some log the result, retry the operation (depending on the exact |
| 92 |
reason the operation failed), or deserialize the response's body. Your own Policy |
| 93 |
objects can do whatever they like when processing outgoing requests or incoming responses. |
| 94 |
|
| 95 |
Note that after an I/O request runs to completion, the Policy objects for that request |
| 96 |
are garbage collected. However, Pipeline object (like Factory objects) are goroutine-safe allowing |
| 97 |
them to be created once and reused over many I/O operations. This allows for efficient use of |
| 98 |
memory and also makes them safely usable by multiple goroutines concurrently. |
| 99 |
|
| 100 |
Inserting a Method-Specific Factory into the Linked-List of Policy Objects |
| 101 |
|
| 102 |
While Pipeline and Factory objects can be reused over many different operations, it is |
| 103 |
common to have special behavior for a specific operation/method. For example, a method |
| 104 |
may need to deserialize the response's body to an instance of a specific data type. |
| 105 |
To accommodate this, the Pipeline's Do method takes an additional method-specific |
| 106 |
Factory object. The Do method tells this Factory to create a Policy object and |
| 107 |
injects this method-specific Policy object into the linked-list of Policy objects. |
| 108 |
|
| 109 |
When creating a Pipeline object, the slice of Factory objects passed must have 1 |
| 110 |
(and only 1) entry marking where the method-specific Factory should be injected. |
| 111 |
The Factory marker is obtained by calling the pipeline.MethodFactoryMarker() function: |
| 112 |
|
| 113 |
func MethodFactoryMarker() pipeline.Factory |
| 114 |
|
| 115 |
Creating an HTTP Request Object |
| 116 |
|
| 117 |
The HTTP request object passed to Pipeline's Do method is not Go's http.Request struct. |
| 118 |
Instead, it is a pipeline.Request struct which is a simple wrapper around Go's standard |
| 119 |
http.Request. You create a pipeline.Request object by calling the pipeline.NewRequest function: |
| 120 |
|
| 121 |
func NewRequest(method string, url url.URL, options pipeline.RequestOptions) (request pipeline.Request, err error) |
| 122 |
|
| 123 |
To this function, you must pass a pipeline.RequestOptions that looks like this: |
| 124 |
|
| 125 |
type RequestOptions struct { |
| 126 |
// The readable and seekable stream to be sent to the server as the request's body. |
| 127 |
Body io.ReadSeeker |
| 128 |
|
| 129 |
// The callback method (if not nil) to be invoked to report progress as the stream is uploaded in the HTTP request. |
| 130 |
Progress ProgressReceiver |
| 131 |
} |
| 132 |
|
| 133 |
The method and struct ensure that the request's body stream is a read/seekable stream. |
| 134 |
A seekable stream is required so that upon retry, the final Policy object can seek |
| 135 |
the stream back to the beginning before retrying the network request and re-uploading the |
| 136 |
body. In addition, you can associate a ProgressReceiver callback function which will be |
| 137 |
invoked periodically to report progress while bytes are being read from the body stream |
| 138 |
and sent over the network. |
| 139 |
|
| 140 |
Processing the HTTP Response |
| 141 |
|
| 142 |
When an HTTP response comes in from the network, a reference to Go's http.Response struct is |
| 143 |
embedded in a struct that implements the pipeline.Response interface: |
| 144 |
|
| 145 |
type Response interface { |
| 146 |
Response() *http.Response |
| 147 |
} |
| 148 |
|
| 149 |
This interface is returned through all the Policy objects. Each Policy object can call the Response |
| 150 |
interface's Response method to examine (or mutate) the embedded http.Response object. |
| 151 |
|
| 152 |
A Policy object can internally define another struct (implementing the pipeline.Response interface) |
| 153 |
that embeds an http.Response and adds additional fields and return this structure to other Policy |
| 154 |
objects. This allows a Policy object to deserialize the body to some other struct and return the |
| 155 |
original http.Response and the additional struct back through the Policy chain. Other Policy objects |
| 156 |
can see the Response but cannot see the additional struct with the deserialized body. After all the |
| 157 |
Policy objects have returned, the pipeline.Response interface is returned by Pipeline's Do method. |
| 158 |
The caller of this method can perform a type assertion attempting to get back to the struct type |
| 159 |
really returned by the Policy object. If the type assertion is successful, the caller now has |
| 160 |
access to both the http.Response and the deserialized struct object.*/ |
| 161 |
package pipeline |