It's All In the Queue
BY JOHN TOOHEY
Today's large, dispersed enterprise environments place new demands on application
developers. Programs need to communicate with each other in a fast reliable way, often
sharing bandwidth with many other systems. The communication channels must be secure and
take into account various network protocols. At times, communication channels will not be
available due to network outages or hardware failures.
Traditional methods such as RPC or sockets are not enough to support these growing
demands, and applications that use these methods are often difficult to scale and tend to
be monolithic in nature. Programs are often required to serialize access to resources and
hold state information between calls. The solution to building scalable distributed
applications is in the use of middleware servers, which sit between the client software
and back end servers. Traditional middleware servers are often transaction monitors, and
provide resource pooling and serialized access to database servers.
Message queue servers are a form of middleware, and allow application programs to send
messages to each other similarly to how e-mail systems allow people to exchange messages.
Message queue servers allow applications running at different times to communicate across
heterogeneous networks and systems that may be temporarily offline. Within a messaging
system, applications send messages to queues and read messages from queues. The servers
guarantee that all messages are delivered, whether a message is sent to a queue or a
message is read from a queue. They also provide efficient routing, security,
priority-based messaging and connectivity between different applications, even if they are
from different vendors.
Message queue servers have been around for many years, and have been used extensively
in the airline and financial industries. They were often proprietary systems developed
in-house, or large systems such as the MQ series from IBM. The introduction of the MSMQ
Server (Microsoft Message Queue) from Microsoft has made message queuing available on the
Windows NT platform. The rest of this article will give an overview of MSMQ and show how
message queue technology can be used to solve problems associated with routing faxes over
the Internet or corporate intranets.
FEATURES OF A MESSAGE QUEUE SYSTEM
With store-and-forward message queuing, applications aren't affected by network
fluctuations and do not have to establish sessions. Because MSMQ uses a sessionless model
at the application level, the sender and receiver don't need to support the same protocol.
MSMQ supports Internet Protocol (IP) and Internet Packet eXchange (IPX). In a later
release, asynchronous DCOM will be implemented using MSMQ.
Network Traffic Prioritization
Message prioritization allows urgent traffic to preempt less-important traffic to
guarantee adequate response time for critical applications at the expense of less
Messages can be logged to a disk-based queue to provide guaranteed delivery. MSMQ supports
express and recoverable delivery methods. Choosing between express and recoverable
delivery is a matter of trading performance and resource use for reliability and failure
recovery. In general, express messages use fewer resources and have faster throughput than
recoverable messages. However, express messages cannot be recovered if the computer
storing the memory-mapped message files fails. Recoverable messages use more resources and
have slower throughput than express messages, but can be recovered no matter which
For example, if an MSMQ-based application is sending express messages to a queue
through an MSMQ routing server, and the network link between the MSMQ routing server and
the destination computer fails, the messages will continue to be stored in memory on the
MSMQ routing server. If the MSMQ routing server is shut down before the network link is
restored, the express messages are lost. However, if recoverable messages are used, the
messages are not lost and are delivered after the MSMQ routing server and network link are
brought back online.
The MSMQ transaction flag allows implementation of transaction-based applications. It
ensures messages are delivered in order, are delivered no more than once, and confirms
messages reached or were retrieved from the destination queue. MSMQ supports four types of
- Internal, message-based transactions
- External transactions that involve a multiple resource manager using MSDTC
- MTS-based transactions
- XA-compliant transactions
Queue information resides in a dynamic/replicated database so administrators can change
queue properties without affecting messaging applications. Using MSMQ Explorer,
administrators can make these changes from any computer running that application.
MSMQ supports smart routing, based on the physical topology of the network, session
concentration, which allows efficient usage of slow links, and transport connectivity.
MSMQ routes and delivers messages based first on queue priority and then on message
MSMQ supports privacy and security through access control, auditing, encryption, and
authentication. Access control is implemented using Windows NT security and digital
signatures, and auditing is implemented through the Windows NT event logging service.
Encryption and authentication (using digital signatures) are supported using public and
private keys. User certificates can be registered internally with MSMQ or taken from the
IE certificate database.
MSMQ ENTERPRISE COMPONENTS
The MSMQ model is based on a collection of one or more sites known as an enterprise, which
represents a business or organization. Each site is usually a separate physical location
such as a building, but could also be a department within a company. Sites are connected
to each other through site-links, and the administrator assigns a cost to each link which
is used by MSMQ when routing messages between sites. Finally, there are individual client
machines, each of which has its own queue manager, implemented as a service. The primary
enterprise controller can also be a primary site controller, and any controller can be a
client. The system is then broken down into future message queues and messages.
The Primary Enterprise Controller
Each MSMQ Enterprise has a Message Queue Information Store (MQIS) database, which runs on
top of SQL Server. This store is a distributed database and all servers receive a copy,
while MSMQ ensures that all changes are replicated throughout the enterprise. The server
that controls the MQIS is called the primary enterprise controller, and MSMQ only uses SQL
Server for storing configuration information and user certificates. Individual messages
and queues are stored on disk files managed by MSMQ, and express mode and reliable mode
messages are held in RAM or in a memory-mapped file on disk, respectively. The dependency
on SQL Server will disappear in a future release of the product, probably when NT 5.0 is
The Primary Site Controller
Each site has its own MQIS, called a Primary Site Controller (PSC), which maintains a copy
of the enterprise MQIS database.
The Backup Site Controller
A site can have zero or more backup site controllers, which assist in load balancing
routing servers and providing backup in case of a failure on the PSC. The BSC maintains a
read-only copy of the MQIS.
Routing servers are responsible for moving messages between queues in the enterprise.
PECs, PSCs and BSCs all function as routing servers, but additional routing servers can be
installed if needed. They allow dynamic routing, and allow computers using different
network protocols to communicate.
A connector server is used to link your enterprise with a non-MSMQ system. Servers are
available for the IBM MQ series of message queue servers.
Dependant And Independent Clients
An independent client is one which is running the MSMQ service and therefore can
communicate asynchronously with other queues. Sales representatives with MSMQ service
installed on their laptops could process orders during the day using an MSMQ-enabled order
entry application. In the evening they could dial in and connect their computers to the
corporate network. MSMQ would then automatically send messages to the outbound queues and
retrieve any waiting messages. Dependant clients, on the other hand, require a persistent
connection to an MSMQ server at all times.
Message queues are repositories for messages. Each queue is located on only one machine
but can be accessed from any other machine in the enterprise providing that the remote
machines have the correct security privileges. Queues can be assigned quotas based on
size, and when a queue reaches its quota, messages can no longer be sent to it. MSMQ
supports many types of queues, including:
- Public: A queue registered in MQIS that can be located by any MSMQ
application. Public queues are persistent and their registration information can be backed
up on the MSMQ enterprise.
- Private: A queue registered on the local computer (not in MQIS) that
typically cannot be located by other applications. Private queues have the advantage of no
MQIS overhead (faster to create, no latency, and no replication), and they can be created
and deleted when MQIS is not working.
- Journal: Journal queues are created by MSMQ each time a queue is
created. A copy of all messages removed from a queue is placed in the journal queue, and
these messages create an audit trail of all messages retrieved from a queue.
- Dead Letter and Transactional Dead Letter: Dead letter queues hold
messages that could not be delivered. In a non-transactional queue the messages are stored
on the last server that attempted to deliver the messages. In a transactional queue
messages are always stored on the source server.
- System: MSMQ creates six private system queues when it is installed,
for internal use.
- Report: Report queues are used to trace messages and to hold diagnostic
messages from MSMQ. They are created by the administrator, and referenced by applications
when sending messages.
- Administration: These queues are created by applications and are used
to store positive or negative acknowledgment messages, and are also used to track message
Messages are defined by the application that is sending them, and are composed of
properties and contents. Microsoft provides a rich set of properties that can be assigned
to a message, including if the message requires a response, if it should be encrypted, and
how long the system should attempt to successfully deliver the message. Full details of
these properties can be found in the MSMQ documentation. The body or contents of the
message can be anything that the application wants to send. The only limitation is that a
message cannot be larger than 4MB.
John Toohey is director of engineering for Object-Fax, Castelle Inc. Castelle
provides high-performance, cost-effective products to allow LAN managers to optimize
network functions over the Internet and Intranets. Visit the Castelle Web site at www.castelle.com.