MYRINET
monitoring on every link. The newest,
Abstract: Myrinet, ANSI/VITA 26-1998,
is a high-speed local area networking
system designed by Myricom to be used as
an inter connect between multiple
machines to form computer clusters.
Myrinet has much less protocol overhead
than standards such as Ethernet, and
therefore provides better throughput, less
interference, and lower latency while
using the host CPU. Although it can be
used as a traditional networking system,
Myrinet is often used directly by programs
that "know" about it, thereby bypassing a
call into the operating system.
Keywords: Myrinet, myricom, Ethrenet
Latency, Throughput, switching, Duplex,
Gigabit Ethernet.
Introduction: Myrinet is a cost-effective,
high-performance, packet-communication
and switching technology that is widely
used to interconnect clusters of
workstations, PCs, servers, blade servers,
or single-board computers. Clusters
provide an economical way of achieving
high performance, high availability.
Myrinet physically consists of two fibre
optic cables, upstream and downstream,
connected to the host computers with a
single connector. Machines are connected
via low-overhead routers and switches, as
opposed to connecting one machine
directly to another. Myrinet includes a
number of fault-tolerance features, mostly
backed by the switches. These include
flow control, error control, and "heartbeat"
"fourth-generation" Myrinet, called Myri-
10G, supports a 10 Gbit/s data rate and is
inter-operable with 10 Gigabit Ethernet on
PHY, the physical layer (cables,
connectors, distances, signaling).
Myrinet Overview:
Myricom Myrinet is the market leader in
high-performance, high-availability,
cluster interconnects. Myricom shipped its
first Myrinet products in August 1994 and
there are now thousands of Myrinet
installations, ranging in size up to more
than 1,000 hosts. These sites include many
of the world's premier cluster-computing
systems.
Characteristics that distinguish Myrinet
from other networks include:
 Full-duplex 2+2 Gigabits/s or
10+10 Gigabits/s data rate links,
switch ports, and interface ports
 Flow control, error control, and
"heartbeat" continuity monitoring
on every link
 Low-latency, cut-through, crossbar
switches, with monitoring for high-
availability applications
 Switch networks that can
interconnect hundreds of hosts
with a single, full-bisection
network component, can scale to
tens of thousands of hosts, & that
can also provide alternative
communication paths between
hosts
 Host interfaces that execute a
control program to interact directly
with host processes ("OS bypass")
 for low-latency communication,
and directly with the network to
send, receive, and buffer packets
Myrinet is an American National Standard
-- ANSI/VITA 26-1998. The link and
routing specifications are public,
published, and open.
Myrinet components :
 Myrinet Switches
 Myrinet software support for most
common hosts and operating
systems.
 Myrinet Interfaces
 Myrinet link fiber or cables
Myrinet Switches: Myrinet switches
provide the high speed cluster message
passing network for passing messages
between computer nodes and for I/O. The
Myrinet switches have a few counters that
can be accessed from an Ethernet
connection to the switch. These counters
can be accessed to monitor the health of
the connections, cables, etc. The following
information refers to the 16-port, the clos-
64 switches, and the Myrinet2000
switches.
Myrinet Packets:
--Packet Format
–--Header ( up
Arbitrary to 24payload
length bytes)
– CRC, error-control
Myrinet Switch Counters
The Myrinet switches maintain counters
for the following types of received packets
between reboots or clears:
• good_high, good_low - A 64-bit
counter of the number of good
packets received in on this port.
• bad_high, bad_low - A 64-bit
counter of the number of non-zero
CRC8 packets received on this
port.
• timeouts - The number of times
that the switch didn't see a tail for
too long coming in on a port.
Timeouts are on the receive side,
so they indicate that a packet went
too long without a tail, or that the
packet was blocked for too long.
• bad_routes - The number of
packets that entered this port
(receive), but had a routing byte
that was off either end (out-
ofrange) of the switch.
• dead_routes - The number of
packets that entered this port
(receive), but had a routing byte
destined for a valid port that was
down, disconnected, or off.
 • bad_packets- Received packets
with non-zero CRC bytes at the
tail.
• missed_beats - A port is not
connected. Connected ports receive
heart beat signals from their
opposites, and when these stop
arriving, the counter goes up. If
you have HA features enabled, the
port will also shut itself down
(portDownOnMissedBeat).
• A missed beat is no beat in 10
usecs, but this is polled for by the
xbar microcontrollers around every
10 msecs and updated in the
monitoring card probably about
every 100 msecs.
• port_state - A mask of the
following bits:
 #define SW_PORT_ON
0x01
 #define SW_PORT_BEAT
0x02
 #define
SW_PORT_TRANSMIT
0x04
 #define
SW_PORT_RECEIVE 0x08
• control is in 2 parts.
o The 0xff bits are an internal
state machine state of the
following values:
case 0: /* port was in reset. initialize
control register */
case 1: /* port is initialized */
case 2: /* port is off, but automatic
shutdown is enabled */
case 3: /* port is on, automatic shutdown
is on */
case 10: /*port was disabled by user. */
o The 0xff00 bits are a bit mask of
the following bits:
#define HW_CTRL_XMIT_OFF 0x80
#define HW_CTRL_RECV_OFF 0x40
#define HW_CTRL_MISS_BEAT_OFF
0x20
#define HW_CTRL_ILL_OFF 0x10
System Requirements for Switch
Monitoring :
Clos-64 or 16 port switch requirements
• The Linux kernel must support the
RARP protocol. Kernel
configuration defaults to "no" so a
new kernel may have to be
configured. This can be a a module
or as part of the kernel but as a
module has to be loaded. It's not
very big.
 RARP support can be
confirmed by looking for a
/proc/net/rarp. If the file exists,
then RARP is supported. If not,
a module may have to loaded
or it is not supported.
RARP support might be configured as a
module.
The command > lsmod can be used to list
modules such as rarp.o that have been
installed.
If there is a file in /lib/modules/[release
number]/rarp.o then that release is
configured as a module.
RARP support can be added with the
command:
> insmod /lib/modules/[release
number]/rarp.o.
IP addresses must be assigned for each
switch. This implies that at the switches
must be connected to an Ethernet hub.
Each switch has its hardware MAC
address printed on the switch. The IP
address must be broadcast from a Linux
host using the rarp command. This
command informs the Myrinet switch its
IP address.
• Setting the IP address in the
switch: /sbin/rarp -s [IP address]
[MAC Address]
• The "ping" command can be used
to verify that the Myrinet switch
has assigned its IP address.
• The Myrinet switch will store that
IP address until it is powered off or
rebooted with the bob_test -reboot
command. If it is powered off or
rebooted, the rarp command will
have to be repeated. The switch
will not recognize a changed IP
address with a second rarp
command. The switch will have to
be powered off or rebooted to
change the previous address.
• Download the corresponding 16-
port or 64-port version of the
Myrinet Switch Query Program,
bob_test, from Myrinet at
• There are several precompiled
versions of the 64-port version.
There is no Alpha version. The
Makefile requires some simple
modifications to make it work. It
does build correctly on the Alpha
platform using gcc.
Myrinet2000, or M3, Switches:
Myrinet 2000 Switches are considerably
different from the Clos-64(m2) and 16-
port switches. First, these switches get
their IP addresses from the dhcpd demon,
not from the RARP command. So, the
switches' IP and MAC addresses must be
in the dhcpd's configuration file,
/etc/dhcpd.conf, which is generated from
the Cluster Tools on Cplant.. Dhcpd must
be restarted when the file is modified, it
does not respond to a SIGHUP or notice
that the file is changed. Once the m3
switch has its address it is accessed by
HTML browsing, SNMP commands, or
one of several small programs supplied by
Myrinet. Note that these programs do not
all run correctly so use with caution.
Since then hardware upgrades and
software improvements have made
Myrinet the network of choice for many
cluster builders and until recently there
was hardly an alternative when a fast, low-
latency network was required.
Like Infiniband, Myrinet uses cut-through
routing for an efficient utilisation of the
network. Also RDMA is used to write
to/read from the remote memory of other
host adaptor cards, called Lanai cards.
These cards interface with the PCI-X bus
of the host they are attached to. The latest
Myrinet implementation only uses fibers
as signal carriers. This gives a high
flexibility in the connection and much
headroom in the speed of signals but the
fiber cables and connectors are rather
delicate which can lead to damage when
cluster nodes have to be serviced.
Myrinet offers ready-made 8--256 port
switches. The 8 and 16 port switches are
full crossbars. In principle all larger
networks are build form these using a Clos
network topology. An example for a 64-
port systems is shown in Figure.

Figure. An 8×16 Clos network using 8 an
16 port crossbar switches to connect 64
processors.
A Clos network is another example of a
logarithmic network with the maximum
bisectional bandwidth of the endpoints.
Note that 4 ports of the 16×16 crossbar
switches are unused but other
configurations need either more switches
or connections or both.
Myricom provides benchmark
measurements of its networks at which for
the newest Myinet2000 switches give
quite good results: about 230 MB/s for a
Ping-pong experiment with latencies
around 7 µs for small messages with the
MPI version that Myricom distributes with
its networks.
Myrinet Interfaces:
Myrinet packets may be of any length, and
thus can encapsulate other types of
packets, including IP packets, without an
adaptation layer. Each packet is identified
by type, so that a Myrinet, like an
Ethernet, may carry packets of many types
or protocols concurrently. Thus, Myrinet
supports several software interfaces.
Myricom supplies two alternative low-
level message-passing systems for
Myrinet: GM, which is used in most
Myrinet clusters today, and a new system
called MX (Myrinet Express). Other
software interfaces such as MPI, Sockets,
DAPL, VI, and PVM are layered
efficiently over GM or MX, and are
available from Myricom and from third
parties. Both GM and MX provide
"Ethernet emulation," in which any
protocols carried over Ethernet, typically
TCP/IP and UDP/IP, can be carried over
Myrinet. The GM and MX systems
provide protected user-level access to the
Myrinet (secure in multi-user,
multiprogramming environments);
reliable, ordered delivery of messages;
network mapping and route computation;
and other features that support robust and
error-free communication. The Software
and Customer Support page provides a
complete outline of Myrinet software, with
links to documentation, and the
Performance Measurements page shows
the performance achieved with the various
software interfaces.
Myrinet Express (MX) Software:
Myrinet Express (MX) is a new message-
passing system from Myricom that
provides lower latency with middleware
layers such as MPI. MX includes Ethernet
emulation and support for all protocols
that run over Ethernet, including TCP/IP
and UDP/IP. MX was developed for the
Myri-10G series of NICs, but MX-2G is
already available on several operating
systems for the PCI-X series of Myrinet
NICs. MX was developed for the Myri-
10G series of NICs and is fully compatible
with MX-2G at the API and application
levels. Myrinet-2000 and Myri-10G
employ the same network architecture and
protocols, and, when used with MX
software support, are fully compatible
with respect to applications and
application-programming interfaces.
Myrinet Components and Software:

Myricom supplies Myrinet components

and software in two series: Myrinet-2000
and Myri-10G. Myrinet-2000 is a superior
alternative to Gigabit Ethernet for clusters, Myrinet control program
whereas Myri-10G offers performance and
cost advantages over 10-Gigabit Ethernet. NIC
Myri-10G uses the same physical layers
(PHYs: cables, connectors, signaling) as
10-Gigabit Ethernet, and is highly
interoperable with 10-Gigabit Ethernet. In
Fabric Management System:
fact, Myri-10G NICs are both 10-Gigabit
Myrinet NICs and 10-Gigabit Ethernet
Myricom's Fabric Management System
NICs.
(FMS) provides centralized diagnostic
monitoring of the Myrinet fabric from a
Myrinet-2000 and Myri-10G employ the
command-line or web interface. FMS
same network architecture and protocols,
consists of one FMS server process to
and, when used with MX software
manage the fabric, and many Fabric
support, are fully compatible with respect
Management Agent (FMA) processes, one
to applications and application-
running on each Myrinet node in the
programming interfaces.
fabric. It is an important diagnostic tool
for verifying the health of the Myrinet
Myricom supplies Myrinet software
hardware.
support for most common hosts and
operating systems. The software is
The Fabric Management System (FMS)
supplied "open source," and other Myrinet
for Myrinet networks may be used with
software is available from third parties.
either the MX or GM low-level firmware.
FMS can be used with GM-1 and
You or an integrator install the NICs and
supersedes Mute, the previous
software in the hosts, and connect the
management software provided by
network with cables and switches. The
Myricom.
software maps the network, and uses
whatever communication paths are
Performance:
available from host to host. No switch
programming or routing-table
Myrinet is a lightweight protocol with
configuration is necessary.
little overhead that allows it to operate
with throughput close to the basic
Software Stack:
signaling speed of the physical layer. On
the latest 2.0 Gbit/s links, Myrinet often
runs at 1.98 Gbit/s of sustained
Host throughput, considerably better than what
Application
Ethernet offers, which varies from 0.6 to
1.9 Gbit/s, depending on load. However,
MPI, Sockets ,etc for supercomputing, the low latency of
Myrinet is even more important than its
throughput performance, since, according
Kernal agent user level API GM
to Amdahl's law, a high-performance
parallel system tends to be bottlenecked by
its slowest sequential process, which is
often the latency of transmission of
messages across the network in all but the
most embarrassingly parallel
supercomputer workloads.
Deployment:
According to Myricom, 141 (28.2%) of
the June 2005 TOP500 supercomputers
used Myrinet technology. In the
November 2005 TOP500, the number of
supercomputers using Myrinet was down
to 101 computers, or 20.2%, in November
2006, 79 (15.8%), and by November 2007,
18 (3.6%), a long way behind Gigabit
Ethernet at 54% and Infiniband at 24.2%.
Technology and Reliability:
Myrinet components are implemented
with the same advanced technology -- full-
custom-VLSI CMOS chips -- as today's
workstations, servers, PCs, and single-
board computers. This use of CMOS
technology is one reason why Myrinet
performance has advanced and will
continue to advance in step with advances
in the hosts, without changes to the
network architecture and software
interfaces.
These CMOS-based Myrinet components
are also extremely reliable. The MTBF of
current-production Myrinet switches and
NICs exceeds 5 million hours per port.
Myrinet exhibits a very low bit-error rate,
and is highly robust with respect to host,
switch, and cable faults. Myrinet can map
itself continuously, and use alternate
routes to circumvent faults (e.g.,
disconnections and powering-down). The
hardware computes and checks a CRC for
each packet on each link. The NICs
provide parity checking both in their
memory and across host IO buses.
Reality:
Myrinet is the clear market leader in high-
performance, high-availability, cluster
interconnect. Myricom shipped its first
Myrinet products in August 1994.
Including the installations supplied by
Myricom's OEM customers and by
Myrinet resellers and integrators, there are
now many thousands of Myrinet
installations, ranging in size to more than
2,000 hosts and more than 4,000
processors. These sites include many of
the world's premier cluster-computing
systems. A total of 141 (28.2%) of the
June-2005 TOP500 supercomputers use
Myrinet technology.
Conclusion: Myrinet is a cost-effective,
high-performance, packet communication
and switching technology that is widely
used to interconnect clusters of
workstations, PCs, or single-board
computers. Conventional networks such as
Ethernet can be used to build clusters, but
do not provide the performance or features
required for high-performance or high-
availability clustering.
References:
– A. Gulati, D. K. Panda, P. Sadayappan,
and P. Wyckoff, NIC-based Rate
Control for Proportional Bandwidth
Allocation in Myrinet Clusters, ICPP ‘01
– S. Senapathi, B. Chandrasekharan, D.
Stredney, H.-W. Shen, and D. K.
Panda, QoS-aware Middleware for
Cluster-based Servers to Support
Interactive and Resource-Adaptive
Applications, HPDC ’03
– D. Buntinas, D. K. Panda, J. Duato, and
P. Sadayappan,
Broadcast/Multicast over Myrinet using
NIC-Assisted Multidestination
Messages, CANPC ‘03
– D. Buntinas, D. K. Panda and P.
Sadayappan, Fast NIC-Based Barrier
over Myrinet/GM, IPDPS ‘01.
– D. Buntinas, D.K. Panda, and W. Gropp,
NIC-Based Atomic Operations
on Myrinet/GM, SAN-1
– D. Buntinas and D. K. Panda, NIC-
Based Reduction in Myrinet Clusters:
Is It Beneficial? SAN-2
– W. Yu, D. Buntinas, and D. K. Panda,
High Performance and Reliable
NIC-Based Multicast over Myrinet/GM-2,
ICPP ’03