Pulses

Receiving a pulse is very simple: a tiny, well-defined message is presented to the MsgReceive(), as if a thread had sent a normal message. The only difference is that you can't MsgReply() to this message—after all, the whole idea of a pulse is that it's asynchronous. Here, we take a look at another function, MsgReceivePulse(), that's useful for dealing with pulses. The only funny thing about a pulse is that the receive ID that comes back from the MsgReceive() function is zero. That's your indication that this is a pulse, rather than a regular message from a client. You often see code in servers that looks like this:

#include <sys/neutrino.h>

    rcvid = MsgReceive (chid, …);
    if (rcvid == 0) {   // it's a pulse
        // determine the type of pulse

        // handle it
    } else {            // it's a regular message
        // determine the type of message

        // handle it
    }

Okay, so you receive this message with a receive ID of zero. What does it actually look like? From the <sys/neutrino.h> header file, here's the definition of the _pulse structure:

struct _pulse {
    uint16_t        type;
    uint16_t        subtype;
    int8_t          code;
    uint8_t         zero [3];
    union sigval    value;
    int32_t         scoid;
};

Both the type and subtype members are zero (a further indication that this is a pulse). The code and value members are set to whatever the sender of the pulse determined. Generally, the code is an indication of why the pulse was sent; the value is a 32-bit data value associated with the pulse. Those two fields are where the 40 bits of content comes from; the other fields aren't user adjustable.

The kernel reserves negative values of code, leaving 127 values for programmers to use as they see fit.

The value member is actually a union:

union sigval {
    int     sival_int;
    void    *sival_ptr;
};

Therefore (expanding on the server example above), you often see code like:

#include <sys/neutrino.h>

    rcvid = MsgReceive (chid, …

    if (rcvid == 0) {   // it's a pulse

        // determine the type of pulse
        switch (msg.pulse.code) {

        case    MY_PULSE_TIMER:
            // One of your timers went off, do something 
            // about it...

            break;

        case    MY_PULSE_HWINT:
            // A hardware interrupt service routine sent 
            // you a pulse.  There's a value in the "value" 
            // member that you need to examine:

            val = msg.pulse.value.sival_int;

            // Do something about it...

            break;

        case    _PULSE_CODE_UNBLOCK:
            // A pulse from the kernel, indicating a client 
            // unblock was received, do something about it...

            break;

        // and so on.

    } else {            // it's a regular message

        // determine the type of message
        // handle it

    }

This code assumes, of course, that you've set up your msg structure to contain a struct _pulse pulse; member, and that the manifest constants MY_PULSE_TIMER and MY_PULSE_HWINT are defined. The pulse code _PULSE_CODE_UNBLOCK is one of those negative-numbered kernel pulses mentioned above. You can find a complete list of them in <sys/neutrino.h> along with a brief description of the value field.

The MsgReceive() and MsgReceivev() functions receives either a regular message or a pulse. There may be situations where you want to receive only pulses. The best example of this is in a server where you've received a request from a client to do something, but can't complete the request just yet (perhaps you have to do a long hardware operation). In such a design, you generally set up the hardware (or a timer, or whatever) to send you a pulse whenever a significant event occurs.

If you write your server using the classic wait in an infinite loop for messages design, you might run into a situation where one client sends you a request, and then, while you're waiting for the pulse to come in (to signal completion of the request), another client sends you another request. Generally, this is exactly what you want—after all, you want to be able to service multiple clients at the same time. However, there might be good reasons why this is not acceptable—servicing a client might be so resource-intensive that you want to limit the number of clients.

In that case, you now need to be able to selectively receive only a pulse, and not a regular message. This is where MsgReceivePulse() comes into play:

#include <sys/neutrino.h>

int MsgReceivePulse (int chid,
                     void *rmsg,
                     int rbytes,
                     struct _msg_info *info);

As you can see, you use the same parameters as MsgReceive(); the channel ID, the buffer (and its size), as well as the info parameter. (We discussed the info parameter above, in Who sent the message? .) Note that the info parameter is not used in the case of a pulse; you might ask why it's present in the parameter list. Simple answer: it was easier to do it that way in the implementation. Just pass a NULL!

The MsgReceivePulse() function receives nothing but pulses. So, if you had a channel with a number of threads blocked on it via MsgReceivePulse(), (and no threads blocked on it via MsgReceive()), and a client attempted to send your server a message, the client would remain SEND-blocked until a thread issued the MsgReceive() call. Pulses would be transferred via the MsgReceivePulse() functions in the meantime.

The only thing you can guarantee if you mix both MsgReceivePulse() and MsgReceive() is that the MsgReceivePulse() gets pulses only. The MsgReceive() can get pulses or messages! This is because, generally, the use of the MsgReceivePulse() function is reserved for the cases where you want to exclude regular message delivery to the server.

This does introduce a bit of confusion. Since the MsgReceive() function can receive both a message and a pulse, but the MsgReceivePulse() function can receive only a pulse, how do you deal with a server that makes use of both functions? Generally, the answer here is that you have a pool of threads that are performing MsgReceive(). This pool of threads (one or more threads; the number depends on how many clients you're prepared to service concurrently) is responsible for handling client calls (requests for service). Since you're trying to control the number of service-providing threads, and since some of these threads may need to block, waiting for a pulse to arrive (for example, from some hardware or from another thread), you typically block the service-providing thread using MsgReceivePulse(). This ensures that a client request won't sneak in while you're waiting for the pulse (since MsgReceivePulse() receives only a pulse).

There are cases when you need to break the natural flow of sends. Such a case might occur if you had a client that sent a message to the server, the result might not be available for a while, and the client didn't want to block. Of course, you can also partly solve this with threads, by having the client use up a thread on the blocking server call, but this may not scale well for larger systems (where you are using up lots of threads to wait for many different servers). Let's say you didn't want to use a thread, but instead wanted the server to reply immediately to the client, I'll get around to your request shortly. At this point, since the server replied, the client is now free to continue processing. Once the server has completed whatever task the client gave it, the server now needs some way to tell the client, hey, wake up, I'm done. Obviously, as we saw in the send-hierarchy discussion above, you can't have the server send a message to the client, because this might cause deadlock if the client sent a message to the server at that exact same instant. So, how does the server send a message to a client without violating the send hierarchy?

It's actually a multi-step operation. Here's how it works:

1. The client creates a struct sigevent structure, and fills it in.
2. The client sends a message to the server, effectively stating, perform this specific task for me, reply right away, and by the way, here's a struct sigevent that you should use to notify me when the work is completed.
3. The server receives the message (which includes the struct sigevent), stores the struct sigevent and the receive ID away, and replies immediately to the client.
4. The client is now running, as is the server.
5. When the server completes the work, the server uses MsgDeliverEvent() to inform the client that the work is now complete.

We'll take a look in detail at the struct sigevent in Clocks, timers, and getting a kick every so often. For now, just think of the struct sigevent as a black box that somehow contains the event that the server uses to notify the client.

Since the server stored the struct sigevent and the receive ID from the client, the server can now call MsgDeliverEvent() to deliver the event, as selected by the client, to the client:

int
MsgDeliverEvent (int rcvid,
                 const struct sigevent *event);

Notice that the MsgDeliverEvent() function takes two parameters, the receive ID (in rcvid) and the event to deliver in event. The server does not modify or examine the event in any way! This point is important, because it allows the server to deliver whatever kind of event the client chose, without any specific processing on the server's part. (The server can, however, verify that the event is valid by using the MsgVerifyEvent() function.)

The rcvid is a receive ID that the server got from the client. Note that this is indeed a special case. Generally, after the server has replied to a client, the receive ID ceases to have any meaning (the reasoning being that the client is unblocked, and the server can't unblock it again, or read or write data from/to the client, and so on). But in this case, the receive ID contains just enough information for the kernel to be able to decide which client the event should be delivered to. When the server calls the MsgDeliverEvent() function, the server doesn't block — this is a non-blocking call for the server. The client has the event delivered to it (by the kernel), and may then perform whatever actions are appropriate.

When we introduced the server, we mentioned that the ChannelCreate() function takes a flags parameter and that we'd just leave it as zero. Now it's time to explain the flags. Here are a few of the possible flags values:

_NTO_CHF_FIXED_PRIORITY

The receiving thread does not change priority based on the priority of the sender. (We talk more about priority issues in Priority inheritance). Ordinarily (that is, if you don't specify this flag), the receiving thread's priority is changed to that of the sender.

_NTO_CHF_UNBLOCK

The kernel delivers a pulse whenever a client thread attempts to unblock. The server must reply to the client to allow the client to unblock.

_NTO_CHF_THREAD_DEATH

The kernel delivers a pulse whenever a thread blocked on this channel dies. This is useful for servers that want to maintain a fixed pool of threads available to service requests at all times.

_NTO_CHF_DISCONNECT

The kernel delivers a pulse whenever all connections from a single client have been disconnected from the server.

_NTO_CHF_SENDER_LEN

The kernel delivers the client's message size as part of the information given to the server (the srcmsglen member of the struct _msg_info structure).

_NTO_CHF_REPLY_LEN

The kernel delivers the client's reply message buffer size as part of the information given to the server (the dstmsglen member of the struct _msg_info structure).

_NTO_CHF_COID_DISCONNECT

The kernel delivers a pulse whenever any connection owned by this process is terminated due to the channel on the other end going away.

Let's look at the _NTO_CHF_UNBLOCK flag; it has a few interesting wrinkles for both the client and the server. Normally (that is, where the server does not specify the _NTO_CHF_UNBLOCK flag) when a client wants to unblock from a MsgSend() (and related MsgSendv(), MsgSendvs(), and so on family of functions), the client unblocks. The client could want to unblock due to receiving a signal or a kernel timeout (see the TimerTimeout() function in the BlackBerry 10 OS C Library Reference and Clocks, timers, and getting a kick every so often). The unfortunate aspect to this is that the server has no idea that the client has unblocked and is no longer waiting for a reply. Note that it isn't possible to write a reliable server with this flag off, except in very special situations which require cooperation between the server and all its clients.

Let's assume that you have a server with multiple threads, all blocked on the server's MsgReceive() function. The client sends a message to the server, and one of the server's threads receives it. At this point, the client is blocked, and a thread in the server is actively processing the request. Now, before the server thread has a chance to reply to the client, the client unblocks from the MsgSend() (let's assume it was because of a signal).

Remember, a server thread is still processing the request on behalf of the client. But since the client is now unblocked (the client's MsgSend() would have returned with EINTR), the client is free to send another request to the server. Thanks to the architecture of BlackBerry 10 OS servers, another thread would receive another message from the client, with the exact same receive ID. The server has no way to tell these two requests apart. When the first thread completes and replies to the client, it's really replying to the second message that the client sent, not the first message (as the thread actually believes that it's doing). So, the server's first thread replies to the client's second message.

This is bad enough; but let's take this one step further. Now the server's second thread completes the request and tries to reply to the client. But since the server's first thread already replied to the client, the client is now unblocked and the server's second thread gets an error from its reply.

This problem is limited to multithreaded servers, because in a single-threaded server, the server thread would still be busy working on the client's first request. This means that even though the client is now unblocked and sends again to the server, the client would now go into the SEND-blocked state (instead of the REPLY-blocked state), allowing the server to finish the processing, reply to the client (which would result in an error, because the client isn't REPLY-blocked any more), and then the server would receive the second message from the client. The real problem here is that the server is performing useless processing on behalf of the client (the client's first request). The processing is useless because the client is no longer waiting for the results of that work.

The solution (in the multithreaded server case) is to have the server specify the _NTO_CHF_UNBLOCK flag to its ChannelCreate() call. This says to the kernel, tell me when a client tries to unblock from me (by sending me a pulse), but don't let the client unblock! I'll unblock the client myself.

The key thing to keep in mind is that this server flag changes the behavior of the client by not allowing the client to unblock until the server says it's okay to do so.

In a single-threaded server, the following happens:

This didn't help the client unblock when it should have, but it did ensure that the server didn't get confused. In this kind of example, the server would most likely ignore the pulse that it got from the kernel. This is okay to do—the assumption being made here is that it's safe to let the client block until the server is ready with the data.

If you want the server to act on the pulse that the kernel sent, there are two ways to do this:

• Create another thread in the server that listens for messages (specifically, listening for the pulse from the kernel). This second thread would be responsible for canceling the operation that's under way in the first thread. One of the two threads would reply to the client.
• Don't do the client's work in the thread itself, but rather queue up the work. This is typically done in applications where the server is going to store the client's work on a queue and the server is event driven. Usually, one of the messages arriving at the server indicates that the client's work is now complete, and that the server should reply. In this case, when the kernel pulse arrives, the server cancels the work being performed on behalf of the client and replies.

Which method you choose depends on the type of work the server does. In the first case, the server is actively performing the work on behalf of the client, so you really don't have a choice—you have to have a second thread that listens for unblock-pulses from the kernel (or you can poll periodically within the thread to see if a pulse has arrived, but polling is generally discouraged).

In the second case, the server has something else doing the work—perhaps a piece of hardware has been commanded to go and collect data. In that case, the server's thread is blocked on the MsgReceive() function anyway, waiting for an indication from the hardware that the command has completed.

In either case, the server must reply to the client, otherwise the client remains blocked.

Even if you use the _NTO_CHF_UNBLOCK flag as described above, there's still one more synchronization problem to deal with. Suppose that you have multiple server threads blocked on the MsgReceive() function, waiting for messages or pulses, and the client sends you a message. One thread goes off and begins the client's work. While that's happening, the client wants to unblock, so the kernel generates the unblock pulse. Another thread in the server receives this pulse. At this point, there's a race condition—the first thread could be just about ready to reply to the client. If the second thread (that got the pulse) does the reply, then there's a chance that the client would unblock and send another message to the server, with the server's first thread now getting a chance to run and replying to the client's second request with the first request's data:

Or, if the thread that got the pulse is just about to reply to the client, and the first thread does the reply, then you have the same situation—the first thread unblocks the client, who sends another request, and the second thread (that got the pulse) now unblocks the client's second request.

The situation is that you have two parallel flows of execution (one caused by the message, and one caused by the pulse). Ordinarily, we'd immediately recognize this as a situation that requires a mutex. Unfortunately, this causes a problem—the mutex would have to be acquired immediately after the MsgReceive() and released before the MsgReply(). While this works, it defeats the whole purpose of the unblock pulse! (The server would either get the message and ignore the unblock pulse until after it had replied to the client, or the server would get the unblock pulse and cancel the client's second operation.)

A solution that looks promising (but is ultimately doomed to failure) would be to have a fine-grained mutex. What we mean by that is a mutex that gets locked and unlocked only around small portions of the control flow (the way that you're supposed to use a mutex, instead of blocking the entire processing section, as proposed above). You set up a, have we replied yet, flag in the server, and this flag is cleared when you receive a message and set when you reply to a message. Just before you reply to the message, you check the flag. If the flag indicates that the message has already been replied to, you skip the reply. The mutex is locked and unlocked around the checking and setting of the flag.

Unfortunately, this won't work because we're not always dealing with two parallel flows of execution—the client won't always get hit with a signal during processing (causing an unblock pulse). Here's the scenario where it breaks:

• The client sends a message to the server; the client is now blocked, the server is now running.
• Since the server received a request from the client, the flag is reset to 0, indicating that we still need to reply to the client.
• The server replies normally to the client (because the flag was set to 0) and sets the flag to 1 indicating that, if an unblock-pulse arrives, it should be ignored.
• (Problems begin here.) The client sends a second message to the server, and almost immediately after sending it gets hit with a signal; the kernel sends an unblock-pulse to the server.
• The server thread that receives the message was about to acquire the mutex in order to check the flag, but didn't quite get there (it got preempted).
• Another server thread now gets the pulse and, because the flag is still set to a 1 from the last time, ignores the pulse.
• Now the server's first thread gets the mutex and clears the flag.
• At this point, the unblock event has been lost.

If you refine the flag to indicate more states (such as pulse received, pulse replied to, message received, message replied to), you still run into a synchronization race condition because there's no way for you to create an atomic binding between the flag and the receive and reply function calls. (Fundamentally, that's where the problem lies—the small timing windows after a MsgReceive() and before the flag is adjusted, and after the flag is adjusted just before the MsgReply().) The only way to get around this is to have the kernel keep track of the flag for you.

Luckily, the kernel keeps track of the flag for you as a single bit in the message info structure (the struct _msg_info that you pass as the last parameter to MsgReceive(), or that you can fetch later, given the receive ID, by calling MsgInfo()). This flag is called _NTO_MI_UNBLOCK_REQ and is set if the client wants to unblock (for example, after receiving a signal).

This means that in a multithreaded server, you typically have a worker thread that's performing the client's work, and another thread that's going to receive the unblock message (or some other message; we just focus on the unblock message for now). When you get the unblock message from the client, you set a flag to yourself, letting your program know that the thread wants to unblock.

There are two cases to consider:

• The worker thread is blocked; or
• The worker thread is running.

If the worker thread is blocked, you need to have the thread that got the unblock message awaken it. It might be blocked if it's waiting for a resource, for example. When the worker thread wakes up, it should examine the _NTO_MI_UNBLOCK_REQ flag, and, if set, reply with an abort status. If the flag isn't set, then the thread can do whatever normal processing it does when it wakes up.

Alternatively, if the worker thread is running, it should periodically check the flag to self that the unblock thread may have set, and if the flag is set, it should reply to the client with an abort status. Note that this is just an optimization: in the unoptimized case, the worker thread constantly calls MsgInfo on the receive ID and checks the _NTO_MI_UNBLOCK_REQ bit itself.