January 1999

It's All In the Queue

BY JOHN TOOHEY

Today's large, dispersed enterprise environments place new demands on application developers. Programs need to communicate with each other in a fast reliable way, often sharing bandwidth with many other systems. The communication channels must be secure and take into account various network protocols. At times, communication channels will not be available due to network outages or hardware failures.

Traditional methods such as RPC or sockets are not enough to support these growing demands, and applications that use these methods are often difficult to scale and tend to be monolithic in nature. Programs are often required to serialize access to resources and hold state information between calls. The solution to building scalable distributed applications is in the use of middleware servers, which sit between the client software and back end servers. Traditional middleware servers are often transaction monitors, and provide resource pooling and serialized access to database servers.

Message queue servers are a form of middleware, and allow application programs to send messages to each other similarly to how e-mail systems allow people to exchange messages. Message queue servers allow applications running at different times to communicate across heterogeneous networks and systems that may be temporarily offline. Within a messaging system, applications send messages to queues and read messages from queues. The servers guarantee that all messages are delivered, whether a message is sent to a queue or a message is read from a queue. They also provide efficient routing, security, priority-based messaging and connectivity between different applications, even if they are from different vendors.

Message queue servers have been around for many years, and have been used extensively in the airline and financial industries. They were often proprietary systems developed in-house, or large systems such as the MQ series from IBM. The introduction of the MSMQ Server (Microsoft Message Queue) from Microsoft has made message queuing available on the Windows NT platform. The rest of this article will give an overview of MSMQ and show how message queue technology can be used to solve problems associated with routing faxes over the Internet or corporate intranets.

FEATURES OF A MESSAGE QUEUE SYSTEM
Asynchronous Messaging
With store-and-forward message queuing, applications aren't affected by network fluctuations and do not have to establish sessions. Because MSMQ uses a sessionless model at the application level, the sender and receiver don't need to support the same protocol. MSMQ supports Internet Protocol (IP) and Internet Packet eXchange (IPX). In a later release, asynchronous DCOM will be implemented using MSMQ.

Network Traffic Prioritization
Message prioritization allows urgent traffic to preempt less-important traffic to guarantee adequate response time for critical applications at the expense of less important applications.

Guaranteed Delivery
Messages can be logged to a disk-based queue to provide guaranteed delivery. MSMQ supports express and recoverable delivery methods. Choosing between express and recoverable delivery is a matter of trading performance and resource use for reliability and failure recovery. In general, express messages use fewer resources and have faster throughput than recoverable messages. However, express messages cannot be recovered if the computer storing the memory-mapped message files fails. Recoverable messages use more resources and have slower throughput than express messages, but can be recovered no matter which computer fails.

For example, if an MSMQ-based application is sending express messages to a queue through an MSMQ routing server, and the network link between the MSMQ routing server and the destination computer fails, the messages will continue to be stored in memory on the MSMQ routing server. If the MSMQ routing server is shut down before the network link is restored, the express messages are lost. However, if recoverable messages are used, the messages are not lost and are delivered after the MSMQ routing server and network link are brought back online.

Transactions
The MSMQ transaction flag allows implementation of transaction-based applications. It ensures messages are delivered in order, are delivered no more than once, and confirms messages reached or were retrieved from the destination queue. MSMQ supports four types of transactions:

1. Internal, message-based transactions
2. External transactions that involve a multiple resource manager using MSDTC
3. MTS-based transactions
4. XA-compliant transactions

Dynamic Queues
Queue information resides in a dynamic/replicated database so administrators can change queue properties without affecting messaging applications. Using MSMQ Explorer, administrators can make these changes from any computer running that application.

Routing
MSMQ supports smart routing, based on the physical topology of the network, session concentration, which allows efficient usage of slow links, and transport connectivity. MSMQ routes and delivers messages based first on queue priority and then on message priority.

Security
MSMQ supports privacy and security through access control, auditing, encryption, and authentication. Access control is implemented using Windows NT security and digital signatures, and auditing is implemented through the Windows NT event logging service. Encryption and authentication (using digital signatures) are supported using public and private keys. User certificates can be registered internally with MSMQ or taken from the IE certificate database.

MSMQ ENTERPRISE COMPONENTS
The MSMQ model is based on a collection of one or more sites known as an enterprise, which represents a business or organization. Each site is usually a separate physical location such as a building, but could also be a department within a company. Sites are connected to each other through site-links, and the administrator assigns a cost to each link which is used by MSMQ when routing messages between sites. Finally, there are individual client machines, each of which has its own queue manager, implemented as a service. The primary enterprise controller can also be a primary site controller, and any controller can be a client. The system is then broken down into future message queues and messages.

The Primary Enterprise Controller
Each MSMQ Enterprise has a Message Queue Information Store (MQIS) database, which runs on top of SQL Server. This store is a distributed database and all servers receive a copy, while MSMQ ensures that all changes are replicated throughout the enterprise. The server that controls the MQIS is called the primary enterprise controller, and MSMQ only uses SQL Server for storing configuration information and user certificates. Individual messages and queues are stored on disk files managed by MSMQ, and express mode and reliable mode messages are held in RAM or in a memory-mapped file on disk, respectively. The dependency on SQL Server will disappear in a future release of the product, probably when NT 5.0 is released.

The Primary Site Controller
Each site has its own MQIS, called a Primary Site Controller (PSC), which maintains a copy of the enterprise MQIS database.

The Backup Site Controller
A site can have zero or more backup site controllers, which assist in load balancing routing servers and providing backup in case of a failure on the PSC. The BSC maintains a read-only copy of the MQIS.

Routing Servers
Routing servers are responsible for moving messages between queues in the enterprise. PECs, PSCs and BSCs all function as routing servers, but additional routing servers can be installed if needed. They allow dynamic routing, and allow computers using different network protocols to communicate.

Connector Server
A connector server is used to link your enterprise with a non-MSMQ system. Servers are available for the IBM MQ series of message queue servers.

Dependant And Independent Clients
An independent client is one which is running the MSMQ service and therefore can communicate asynchronously with other queues. Sales representatives with MSMQ service installed on their laptops could process orders during the day using an MSMQ-enabled order entry application. In the evening they could dial in and connect their computers to the corporate network. MSMQ would then automatically send messages to the outbound queues and retrieve any waiting messages. Dependant clients, on the other hand, require a persistent connection to an MSMQ server at all times.

Message Queue
Message queues are repositories for messages. Each queue is located on only one machine but can be accessed from any other machine in the enterprise providing that the remote machines have the correct security privileges. Queues can be assigned quotas based on size, and when a queue reaches its quota, messages can no longer be sent to it. MSMQ supports many types of queues, including:

• Public: A queue registered in MQIS that can be located by any MSMQ application. Public queues are persistent and their registration information can be backed up on the MSMQ enterprise.
• Private: A queue registered on the local computer (not in MQIS) that typically cannot be located by other applications. Private queues have the advantage of no MQIS overhead (faster to create, no latency, and no replication), and they can be created and deleted when MQIS is not working.
• Journal: Journal queues are created by MSMQ each time a queue is created. A copy of all messages removed from a queue is placed in the journal queue, and these messages create an audit trail of all messages retrieved from a queue.
• Dead Letter and Transactional Dead Letter: Dead letter queues hold messages that could not be delivered. In a non-transactional queue the messages are stored on the last server that attempted to deliver the messages. In a transactional queue messages are always stored on the source server.
• System: MSMQ creates six private system queues when it is installed, for internal use.
• Report: Report queues are used to trace messages and to hold diagnostic messages from MSMQ. They are created by the administrator, and referenced by applications when sending messages.
• Administration: These queues are created by applications and are used to store positive or negative acknowledgment messages, and are also used to track message delivery.

Messages
Messages are defined by the application that is sending them, and are composed of properties and contents. Microsoft provides a rich set of properties that can be assigned to a message, including if the message requires a response, if it should be encrypted, and how long the system should attempt to successfully deliver the message. Full details of these properties can be found in the MSMQ documentation. The body or contents of the message can be anything that the application wants to send. The only limitation is that a message cannot be larger than 4MB.

John Toohey is director of engineering for Object-Fax, Castelle Inc. Castelle provides high-performance, cost-effective products to allow LAN managers to optimize network functions over the Internet and Intranets. Visit the Castelle Web site at www.castelle.com.