| 1 |
============ |
| 2 |
Introduction |
| 3 |
============ |
| 4 |
|
| 5 |
The RapidIO standard is a packet-based fabric interconnect standard designed for |
| 6 |
use in embedded systems. Development of the RapidIO standard is directed by the |
| 7 |
RapidIO Trade Association (RTA). The current version of the RapidIO specification |
| 8 |
is publicly available for download from the RTA web-site [1]. |
| 9 |
|
| 10 |
This document describes the basics of the Linux RapidIO subsystem and provides |
| 11 |
information on its major components. |
| 12 |
|
| 13 |
1 Overview |
| 14 |
========== |
| 15 |
|
| 16 |
Because the RapidIO subsystem follows the Linux device model it is integrated |
| 17 |
into the kernel similarly to other buses by defining RapidIO-specific device and |
| 18 |
bus types and registering them within the device model. |
| 19 |
|
| 20 |
The Linux RapidIO subsystem is architecture independent and therefore defines |
| 21 |
architecture-specific interfaces that provide support for common RapidIO |
| 22 |
subsystem operations. |
| 23 |
|
| 24 |
2. Core Components |
| 25 |
================== |
| 26 |
|
| 27 |
A typical RapidIO network is a combination of endpoints and switches. |
| 28 |
Each of these components is represented in the subsystem by an associated data |
| 29 |
structure. The core logical components of the RapidIO subsystem are defined |
| 30 |
in include/linux/rio.h file. |
| 31 |
|
| 32 |
2.1 Master Port |
| 33 |
--------------- |
| 34 |
|
| 35 |
A master port (or mport) is a RapidIO interface controller that is local to the |
| 36 |
processor executing the Linux code. A master port generates and receives RapidIO |
| 37 |
packets (transactions). In the RapidIO subsystem each master port is represented |
| 38 |
by a rio_mport data structure. This structure contains master port specific |
| 39 |
resources such as mailboxes and doorbells. The rio_mport also includes a unique |
| 40 |
host device ID that is valid when a master port is configured as an enumerating |
| 41 |
host. |
| 42 |
|
| 43 |
RapidIO master ports are serviced by subsystem specific mport device drivers |
| 44 |
that provide functionality defined for this subsystem. To provide a hardware |
| 45 |
independent interface for RapidIO subsystem operations, rio_mport structure |
| 46 |
includes rio_ops data structure which contains pointers to hardware specific |
| 47 |
implementations of RapidIO functions. |
| 48 |
|
| 49 |
2.2 Device |
| 50 |
---------- |
| 51 |
|
| 52 |
A RapidIO device is any endpoint (other than mport) or switch in the network. |
| 53 |
All devices are presented in the RapidIO subsystem by corresponding rio_dev data |
| 54 |
structure. Devices form one global device list and per-network device lists |
| 55 |
(depending on number of available mports and networks). |
| 56 |
|
| 57 |
2.3 Switch |
| 58 |
---------- |
| 59 |
|
| 60 |
A RapidIO switch is a special class of device that routes packets between its |
| 61 |
ports towards their final destination. The packet destination port within a |
| 62 |
switch is defined by an internal routing table. A switch is presented in the |
| 63 |
RapidIO subsystem by rio_dev data structure expanded by additional rio_switch |
| 64 |
data structure, which contains switch specific information such as copy of the |
| 65 |
routing table and pointers to switch specific functions. |
| 66 |
|
| 67 |
The RapidIO subsystem defines the format and initialization method for subsystem |
| 68 |
specific switch drivers that are designed to provide hardware-specific |
| 69 |
implementation of common switch management routines. |
| 70 |
|
| 71 |
2.4 Network |
| 72 |
----------- |
| 73 |
|
| 74 |
A RapidIO network is a combination of interconnected endpoint and switch devices. |
| 75 |
Each RapidIO network known to the system is represented by corresponding rio_net |
| 76 |
data structure. This structure includes lists of all devices and local master |
| 77 |
ports that form the same network. It also contains a pointer to the default |
| 78 |
master port that is used to communicate with devices within the network. |
| 79 |
|
| 80 |
2.5 Device Drivers |
| 81 |
------------------ |
| 82 |
|
| 83 |
RapidIO device-specific drivers follow Linux Kernel Driver Model and are |
| 84 |
intended to support specific RapidIO devices attached to the RapidIO network. |
| 85 |
|
| 86 |
2.6 Subsystem Interfaces |
| 87 |
------------------------ |
| 88 |
|
| 89 |
RapidIO interconnect specification defines features that may be used to provide |
| 90 |
one or more common service layers for all participating RapidIO devices. These |
| 91 |
common services may act separately from device-specific drivers or be used by |
| 92 |
device-specific drivers. Example of such service provider is the RIONET driver |
| 93 |
which implements Ethernet-over-RapidIO interface. Because only one driver can be |
| 94 |
registered for a device, all common RapidIO services have to be registered as |
| 95 |
subsystem interfaces. This allows to have multiple common services attached to |
| 96 |
the same device without blocking attachment of a device-specific driver. |
| 97 |
|
| 98 |
3. Subsystem Initialization |
| 99 |
=========================== |
| 100 |
|
| 101 |
In order to initialize the RapidIO subsystem, a platform must initialize and |
| 102 |
register at least one master port within the RapidIO network. To register mport |
| 103 |
within the subsystem controller driver's initialization code calls function |
| 104 |
rio_register_mport() for each available master port. |
| 105 |
|
| 106 |
After all active master ports are registered with a RapidIO subsystem, |
| 107 |
an enumeration and/or discovery routine may be called automatically or |
| 108 |
by user-space command. |
| 109 |
|
| 110 |
RapidIO subsystem can be configured to be built as a statically linked or |
| 111 |
modular component of the kernel (see details below). |
| 112 |
|
| 113 |
4. Enumeration and Discovery |
| 114 |
============================ |
| 115 |
|
| 116 |
4.1 Overview |
| 117 |
------------ |
| 118 |
|
| 119 |
RapidIO subsystem configuration options allow users to build enumeration and |
| 120 |
discovery methods as statically linked components or loadable modules. |
| 121 |
An enumeration/discovery method implementation and available input parameters |
| 122 |
define how any given method can be attached to available RapidIO mports: |
| 123 |
simply to all available mports OR individually to the specified mport device. |
| 124 |
|
| 125 |
Depending on selected enumeration/discovery build configuration, there are |
| 126 |
several methods to initiate an enumeration and/or discovery process: |
| 127 |
|
| 128 |
(a) Statically linked enumeration and discovery process can be started |
| 129 |
automatically during kernel initialization time using corresponding module |
| 130 |
parameters. This was the original method used since introduction of RapidIO |
| 131 |
subsystem. Now this method relies on enumerator module parameter which is |
| 132 |
'rio-scan.scan' for existing basic enumeration/discovery method. |
| 133 |
When automatic start of enumeration/discovery is used a user has to ensure |
| 134 |
that all discovering endpoints are started before the enumerating endpoint |
| 135 |
and are waiting for enumeration to be completed. |
| 136 |
Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering |
| 137 |
endpoint waits for enumeration to be completed. If the specified timeout |
| 138 |
expires the discovery process is terminated without obtaining RapidIO network |
| 139 |
information. NOTE: a timed out discovery process may be restarted later using |
| 140 |
a user-space command as it is described below (if the given endpoint was |
| 141 |
enumerated successfully). |
| 142 |
|
| 143 |
(b) Statically linked enumeration and discovery process can be started by |
| 144 |
a command from user space. This initiation method provides more flexibility |
| 145 |
for a system startup compared to the option (a) above. After all participating |
| 146 |
endpoints have been successfully booted, an enumeration process shall be |
| 147 |
started first by issuing a user-space command, after an enumeration is |
| 148 |
completed a discovery process can be started on all remaining endpoints. |
| 149 |
|
| 150 |
(c) Modular enumeration and discovery process can be started by a command from |
| 151 |
user space. After an enumeration/discovery module is loaded, a network scan |
| 152 |
process can be started by issuing a user-space command. |
| 153 |
Similar to the option (b) above, an enumerator has to be started first. |
| 154 |
|
| 155 |
(d) Modular enumeration and discovery process can be started by a module |
| 156 |
initialization routine. In this case an enumerating module shall be loaded |
| 157 |
first. |
| 158 |
|
| 159 |
When a network scan process is started it calls an enumeration or discovery |
| 160 |
routine depending on the configured role of a master port: host or agent. |
| 161 |
|
| 162 |
Enumeration is performed by a master port if it is configured as a host port by |
| 163 |
assigning a host destination ID greater than or equal to zero. The host |
| 164 |
destination ID can be assigned to a master port using various methods depending |
| 165 |
on RapidIO subsystem build configuration: |
| 166 |
|
| 167 |
(a) For a statically linked RapidIO subsystem core use command line parameter |
| 168 |
"rapidio.hdid=" with a list of destination ID assignments in order of mport |
| 169 |
device registration. For example, in a system with two RapidIO controllers |
| 170 |
the command line parameter "rapidio.hdid=-1,7" will result in assignment of |
| 171 |
the host destination ID=7 to the second RapidIO controller, while the first |
| 172 |
one will be assigned destination ID=-1. |
| 173 |
|
| 174 |
(b) If the RapidIO subsystem core is built as a loadable module, in addition |
| 175 |
to the method shown above, the host destination ID(s) can be specified using |
| 176 |
traditional methods of passing module parameter "hdid=" during its loading: |
| 177 |
|
| 178 |
- from command line: "modprobe rapidio hdid=-1,7", or |
| 179 |
- from modprobe configuration file using configuration command "options", |
| 180 |
like in this example: "options rapidio hdid=-1,7". An example of modprobe |
| 181 |
configuration file is provided in the section below. |
| 182 |
|
| 183 |
NOTES: |
| 184 |
(i) if "hdid=" parameter is omitted all available mport will be assigned |
| 185 |
destination ID = -1; |
| 186 |
|
| 187 |
(ii) the "hdid=" parameter in systems with multiple mports can have |
| 188 |
destination ID assignments omitted from the end of list (default = -1). |
| 189 |
|
| 190 |
If the host device ID for a specific master port is set to -1, the discovery |
| 191 |
process will be performed for it. |
| 192 |
|
| 193 |
The enumeration and discovery routines use RapidIO maintenance transactions |
| 194 |
to access the configuration space of devices. |
| 195 |
|
| 196 |
NOTE: If RapidIO switch-specific device drivers are built as loadable modules |
| 197 |
they must be loaded before enumeration/discovery process starts. |
| 198 |
This requirement is cased by the fact that enumeration/discovery methods invoke |
| 199 |
vendor-specific callbacks on early stages. |
| 200 |
|
| 201 |
4.2 Automatic Start of Enumeration and Discovery |
| 202 |
------------------------------------------------ |
| 203 |
|
| 204 |
Automatic enumeration/discovery start method is applicable only to built-in |
| 205 |
enumeration/discovery RapidIO configuration selection. To enable automatic |
| 206 |
enumeration/discovery start by existing basic enumerator method set use boot |
| 207 |
command line parameter "rio-scan.scan=1". |
| 208 |
|
| 209 |
This configuration requires synchronized start of all RapidIO endpoints that |
| 210 |
form a network which will be enumerated/discovered. Discovering endpoints have |
| 211 |
to be started before an enumeration starts to ensure that all RapidIO |
| 212 |
controllers have been initialized and are ready to be discovered. Configuration |
| 213 |
parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which |
| 214 |
a discovering endpoint will wait for enumeration to be completed. |
| 215 |
|
| 216 |
When automatic enumeration/discovery start is selected, basic method's |
| 217 |
initialization routine calls rio_init_mports() to perform enumeration or |
| 218 |
discovery for all known mport devices. |
| 219 |
|
| 220 |
Depending on RapidIO network size and configuration this automatic |
| 221 |
enumeration/discovery start method may be difficult to use due to the |
| 222 |
requirement for synchronized start of all endpoints. |
| 223 |
|
| 224 |
4.3 User-space Start of Enumeration and Discovery |
| 225 |
------------------------------------------------- |
| 226 |
|
| 227 |
User-space start of enumeration and discovery can be used with built-in and |
| 228 |
modular build configurations. For user-space controlled start RapidIO subsystem |
| 229 |
creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate |
| 230 |
an enumeration or discovery process on specific mport device, a user needs to |
| 231 |
write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a |
| 232 |
sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device |
| 233 |
registration. For example for machine with single RapidIO controller, mport_ID |
| 234 |
for that controller always will be 0. |
| 235 |
|
| 236 |
To initiate RapidIO enumeration/discovery on all available mports a user may |
| 237 |
write '-1' (or RIO_MPORT_ANY) into the scan attribute file. |
| 238 |
|
| 239 |
4.4 Basic Enumeration Method |
| 240 |
---------------------------- |
| 241 |
|
| 242 |
This is an original enumeration/discovery method which is available since |
| 243 |
first release of RapidIO subsystem code. The enumeration process is |
| 244 |
implemented according to the enumeration algorithm outlined in the RapidIO |
| 245 |
Interconnect Specification: Annex I [1]. |
| 246 |
|
| 247 |
This method can be configured as statically linked or loadable module. |
| 248 |
The method's single parameter "scan" allows to trigger the enumeration/discovery |
| 249 |
process from module initialization routine. |
| 250 |
|
| 251 |
This enumeration/discovery method can be started only once and does not support |
| 252 |
unloading if it is built as a module. |
| 253 |
|
| 254 |
The enumeration process traverses the network using a recursive depth-first |
| 255 |
algorithm. When a new device is found, the enumerator takes ownership of that |
| 256 |
device by writing into the Host Device ID Lock CSR. It does this to ensure that |
| 257 |
the enumerator has exclusive right to enumerate the device. If device ownership |
| 258 |
is successfully acquired, the enumerator allocates a new rio_dev structure and |
| 259 |
initializes it according to device capabilities. |
| 260 |
|
| 261 |
If the device is an endpoint, a unique device ID is assigned to it and its value |
| 262 |
is written into the device's Base Device ID CSR. |
| 263 |
|
| 264 |
If the device is a switch, the enumerator allocates an additional rio_switch |
| 265 |
structure to store switch specific information. Then the switch's vendor ID and |
| 266 |
device ID are queried against a table of known RapidIO switches. Each switch |
| 267 |
table entry contains a pointer to a switch-specific initialization routine that |
| 268 |
initializes pointers to the rest of switch specific operations, and performs |
| 269 |
hardware initialization if necessary. A RapidIO switch does not have a unique |
| 270 |
device ID; it relies on hopcount and routing for device ID of an attached |
| 271 |
endpoint if access to its configuration registers is required. If a switch (or |
| 272 |
chain of switches) does not have any endpoint (except enumerator) attached to |
| 273 |
it, a fake device ID will be assigned to configure a route to that switch. |
| 274 |
In the case of a chain of switches without endpoint, one fake device ID is used |
| 275 |
to configure a route through the entire chain and switches are differentiated by |
| 276 |
their hopcount value. |
| 277 |
|
| 278 |
For both endpoints and switches the enumerator writes a unique component tag |
| 279 |
into device's Component Tag CSR. That unique value is used by the error |
| 280 |
management notification mechanism to identify a device that is reporting an |
| 281 |
error management event. |
| 282 |
|
| 283 |
Enumeration beyond a switch is completed by iterating over each active egress |
| 284 |
port of that switch. For each active link, a route to a default device ID |
| 285 |
(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written |
| 286 |
into the routing table. The algorithm recurs by calling itself with hopcount + 1 |
| 287 |
and the default device ID in order to access the device on the active port. |
| 288 |
|
| 289 |
After the host has completed enumeration of the entire network it releases |
| 290 |
devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint |
| 291 |
in the system, it sets the Discovered bit in the Port General Control CSR |
| 292 |
to indicate that enumeration is completed and agents are allowed to execute |
| 293 |
passive discovery of the network. |
| 294 |
|
| 295 |
The discovery process is performed by agents and is similar to the enumeration |
| 296 |
process that is described above. However, the discovery process is performed |
| 297 |
without changes to the existing routing because agents only gather information |
| 298 |
about RapidIO network structure and are building an internal map of discovered |
| 299 |
devices. This way each Linux-based component of the RapidIO subsystem has |
| 300 |
a complete view of the network. The discovery process can be performed |
| 301 |
simultaneously by several agents. After initializing its RapidIO master port |
| 302 |
each agent waits for enumeration completion by the host for the configured wait |
| 303 |
time period. If this wait time period expires before enumeration is completed, |
| 304 |
an agent skips RapidIO discovery and continues with remaining kernel |
| 305 |
initialization. |
| 306 |
|
| 307 |
4.5 Adding New Enumeration/Discovery Method |
| 308 |
------------------------------------------- |
| 309 |
|
| 310 |
RapidIO subsystem code organization allows addition of new enumeration/discovery |
| 311 |
methods as new configuration options without significant impact to the core |
| 312 |
RapidIO code. |
| 313 |
|
| 314 |
A new enumeration/discovery method has to be attached to one or more mport |
| 315 |
devices before an enumeration/discovery process can be started. Normally, |
| 316 |
method's module initialization routine calls rio_register_scan() to attach |
| 317 |
an enumerator to a specified mport device (or devices). The basic enumerator |
| 318 |
implementation demonstrates this process. |
| 319 |
|
| 320 |
4.6 Using Loadable RapidIO Switch Drivers |
| 321 |
----------------------------------------- |
| 322 |
|
| 323 |
In the case when RapidIO switch drivers are built as loadable modules a user |
| 324 |
must ensure that they are loaded before the enumeration/discovery starts. |
| 325 |
This process can be automated by specifying pre- or post- dependencies in the |
| 326 |
RapidIO-specific modprobe configuration file as shown in the example below. |
| 327 |
|
| 328 |
File /etc/modprobe.d/rapidio.conf:: |
| 329 |
|
| 330 |
# Configure RapidIO subsystem modules |
| 331 |
|
| 332 |
# Set enumerator host destination ID (overrides kernel command line option) |
| 333 |
options rapidio hdid=-1,2 |
| 334 |
|
| 335 |
# Load RapidIO switch drivers immediately after rapidio core module was loaded |
| 336 |
softdep rapidio post: idt_gen2 idtcps tsi57x |
| 337 |
|
| 338 |
# OR : |
| 339 |
|
| 340 |
# Load RapidIO switch drivers just before rio-scan enumerator module is loaded |
| 341 |
softdep rio-scan pre: idt_gen2 idtcps tsi57x |
| 342 |
|
| 343 |
-------------------------- |
| 344 |
|
| 345 |
NOTE: |
| 346 |
In the example above, one of "softdep" commands must be removed or |
| 347 |
commented out to keep required module loading sequence. |
| 348 |
|
| 349 |
5. References |
| 350 |
============= |
| 351 |
|
| 352 |
[1] RapidIO Trade Association. RapidIO Interconnect Specifications. |
| 353 |
http://www.rapidio.org. |
| 354 |
|
| 355 |
[2] Rapidio TA. Technology Comparisons. |
| 356 |
http://www.rapidio.org/education/technology_comparisons/ |
| 357 |
|
| 358 |
[3] RapidIO support for Linux. |
| 359 |
https://lwn.net/Articles/139118/ |
| 360 |
|
| 361 |
[4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 |
| 362 |
https://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf |