| 1 |
Introduction |
| 2 |
============ |
| 3 |
|
| 4 |
ATA over Ethernet is a network protocol that provides simple access to |
| 5 |
block storage on the LAN. |
| 6 |
|
| 7 |
http://support.coraid.com/documents/AoEr11.txt |
| 8 |
|
| 9 |
The EtherDrive (R) HOWTO for 2.6 and 3.x kernels is found at ... |
| 10 |
|
| 11 |
http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html |
| 12 |
|
| 13 |
It has many tips and hints! Please see, especially, recommended |
| 14 |
tunings for virtual memory: |
| 15 |
|
| 16 |
http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO-5.html#ss5.19 |
| 17 |
|
| 18 |
The aoetools are userland programs that are designed to work with this |
| 19 |
driver. The aoetools are on sourceforge. |
| 20 |
|
| 21 |
http://aoetools.sourceforge.net/ |
| 22 |
|
| 23 |
The scripts in this Documentation/admin-guide/aoe directory are intended to |
| 24 |
document the use of the driver and are not necessary if you install |
| 25 |
the aoetools. |
| 26 |
|
| 27 |
|
| 28 |
Creating Device Nodes |
| 29 |
===================== |
| 30 |
|
| 31 |
Users of udev should find the block device nodes created |
| 32 |
automatically, but to create all the necessary device nodes, use the |
| 33 |
udev configuration rules provided in udev.txt (in this directory). |
| 34 |
|
| 35 |
There is a udev-install.sh script that shows how to install these |
| 36 |
rules on your system. |
| 37 |
|
| 38 |
There is also an autoload script that shows how to edit |
| 39 |
/etc/modprobe.d/aoe.conf to ensure that the aoe module is loaded when |
| 40 |
necessary. Preloading the aoe module is preferable to autoloading, |
| 41 |
however, because AoE discovery takes a few seconds. It can be |
| 42 |
confusing when an AoE device is not present the first time the a |
| 43 |
command is run but appears a second later. |
| 44 |
|
| 45 |
Using Device Nodes |
| 46 |
================== |
| 47 |
|
| 48 |
"cat /dev/etherd/err" blocks, waiting for error diagnostic output, |
| 49 |
like any retransmitted packets. |
| 50 |
|
| 51 |
"echo eth2 eth4 > /dev/etherd/interfaces" tells the aoe driver to |
| 52 |
limit ATA over Ethernet traffic to eth2 and eth4. AoE traffic from |
| 53 |
untrusted networks should be ignored as a matter of security. See |
| 54 |
also the aoe_iflist driver option described below. |
| 55 |
|
| 56 |
"echo > /dev/etherd/discover" tells the driver to find out what AoE |
| 57 |
devices are available. |
| 58 |
|
| 59 |
In the future these character devices may disappear and be replaced |
| 60 |
by sysfs counterparts. Using the commands in aoetools insulates |
| 61 |
users from these implementation details. |
| 62 |
|
| 63 |
The block devices are named like this:: |
| 64 |
|
| 65 |
e{shelf}.{slot} |
| 66 |
e{shelf}.{slot}p{part} |
| 67 |
|
| 68 |
... so that "e0.2" is the third blade from the left (slot 2) in the |
| 69 |
first shelf (shelf address zero). That's the whole disk. The first |
| 70 |
partition on that disk would be "e0.2p1". |
| 71 |
|
| 72 |
Using sysfs |
| 73 |
=========== |
| 74 |
|
| 75 |
Each aoe block device in /sys/block has the extra attributes of |
| 76 |
state, mac, and netif. The state attribute is "up" when the device |
| 77 |
is ready for I/O and "down" if detected but unusable. The |
| 78 |
"down,closewait" state shows that the device is still open and |
| 79 |
cannot come up again until it has been closed. |
| 80 |
|
| 81 |
The mac attribute is the ethernet address of the remote AoE device. |
| 82 |
The netif attribute is the network interface on the localhost |
| 83 |
through which we are communicating with the remote AoE device. |
| 84 |
|
| 85 |
There is a script in this directory that formats this information in |
| 86 |
a convenient way. Users with aoetools should use the aoe-stat |
| 87 |
command:: |
| 88 |
|
| 89 |
root@makki root# sh Documentation/admin-guide/aoe/status.sh |
| 90 |
e10.0 eth3 up |
| 91 |
e10.1 eth3 up |
| 92 |
e10.2 eth3 up |
| 93 |
e10.3 eth3 up |
| 94 |
e10.4 eth3 up |
| 95 |
e10.5 eth3 up |
| 96 |
e10.6 eth3 up |
| 97 |
e10.7 eth3 up |
| 98 |
e10.8 eth3 up |
| 99 |
e10.9 eth3 up |
| 100 |
e4.0 eth1 up |
| 101 |
e4.1 eth1 up |
| 102 |
e4.2 eth1 up |
| 103 |
e4.3 eth1 up |
| 104 |
e4.4 eth1 up |
| 105 |
e4.5 eth1 up |
| 106 |
e4.6 eth1 up |
| 107 |
e4.7 eth1 up |
| 108 |
e4.8 eth1 up |
| 109 |
e4.9 eth1 up |
| 110 |
|
| 111 |
Use /sys/module/aoe/parameters/aoe_iflist (or better, the driver |
| 112 |
option discussed below) instead of /dev/etherd/interfaces to limit |
| 113 |
AoE traffic to the network interfaces in the given |
| 114 |
whitespace-separated list. Unlike the old character device, the |
| 115 |
sysfs entry can be read from as well as written to. |
| 116 |
|
| 117 |
It's helpful to trigger discovery after setting the list of allowed |
| 118 |
interfaces. The aoetools package provides an aoe-discover script |
| 119 |
for this purpose. You can also directly use the |
| 120 |
/dev/etherd/discover special file described above. |
| 121 |
|
| 122 |
Driver Options |
| 123 |
============== |
| 124 |
|
| 125 |
There is a boot option for the built-in aoe driver and a |
| 126 |
corresponding module parameter, aoe_iflist. Without this option, |
| 127 |
all network interfaces may be used for ATA over Ethernet. Here is a |
| 128 |
usage example for the module parameter:: |
| 129 |
|
| 130 |
modprobe aoe_iflist="eth1 eth3" |
| 131 |
|
| 132 |
The aoe_deadsecs module parameter determines the maximum number of |
| 133 |
seconds that the driver will wait for an AoE device to provide a |
| 134 |
response to an AoE command. After aoe_deadsecs seconds have |
| 135 |
elapsed, the AoE device will be marked as "down". A value of zero |
| 136 |
is supported for testing purposes and makes the aoe driver keep |
| 137 |
trying AoE commands forever. |
| 138 |
|
| 139 |
The aoe_maxout module parameter has a default of 128. This is the |
| 140 |
maximum number of unresponded packets that will be sent to an AoE |
| 141 |
target at one time. |
| 142 |
|
| 143 |
The aoe_dyndevs module parameter defaults to 1, meaning that the |
| 144 |
driver will assign a block device minor number to a discovered AoE |
| 145 |
target based on the order of its discovery. With dynamic minor |
| 146 |
device numbers in use, a greater range of AoE shelf and slot |
| 147 |
addresses can be supported. Users with udev will never have to |
| 148 |
think about minor numbers. Using aoe_dyndevs=0 allows device nodes |
| 149 |
to be pre-created using a static minor-number scheme with the |
| 150 |
aoe-mkshelf script in the aoetools. |