Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
BroadcastCommunity.qnx.com will be offline from May 31 6:00pm until June 2 12:00AM for upcoming system upgrades. For more information please go to https://community.qnx.com/sf/discussion/do/listPosts/projects.bazaar/discussion.bazaar.topc28418
Forum Topic - Thread pool outside resource manager.: (15 Items)
   
Thread pool outside resource manager.  
I've made a post already about thread pools, but as it involved resource manager, and there were some inconsistencies 
about C++ usage, I decided to make a new one.

I am developing a controller that is based in a pool of threads, that keep listening to Qnet (thus messages from 
different parts of my system) and then taking actions accordingly. 
Again, this is not a resource manager, it creates the dispatch and channel through name_attach().

The lifecycle of a single thread works as expected (context_alloc -> block_fun -> handler_fun -> block_fun -> ...), but 
even with the thread_pool being set and initialised as below, new threads are not being created as it should (having a 
number of threads in blocked state between lo water and hi water).

int Comm::InitComm()
{
    //  Set up the the server path and attach to channel
    name_attach_t *name_att;
    if((name_att = name_attach(NULL, CmsData::Instance()->cms_self_.hostname.c_str(), NAME_FLAG_ATTACH_GLOBAL)) == NULL)

    {
        //  Throw error
        return -1;
    }
    g_comm_chid = name_att->chid;

    //  Set up the thread parameters for the threads that will populate the pool
    pthread_attr_t t_attr;
    pthread_attr_init (&t_attr);

    //  Set up the thread pool parameters
    thread_pool_attr_t tp_attr;
    thread_pool_t    *tpp;
    memset(&tp_attr, 0, sizeof (tp_attr));
    tp_attr.handle = name_att->dpp;
    tp_attr.block_func = WaitForMessage;
    tp_attr.unblock_func = UnblockFunc;
    tp_attr.context_alloc = ContextAlloc;
    tp_attr.handler_func = ParseMessage;
    tp_attr.context_free = ContextFree;
    tp_attr.lo_water = 3;
    tp_attr.hi_water = 6;
    tp_attr.increment = 2;
    tp_attr.maximum = 10;
    tp_attr.attr = &t_attr;

    //  Create a thread pool that returns
    tpp = thread_pool_create (&tp_attr, 0);
    if (tpp == NULL)
    {
        //  Could not create thread pool
        name_detach(name_att, 0);
        return -1;
    }

    if (thread_pool_start(tpp) == -1)
    {
        //  Could not start thread pool
        name_detach(name_att, 0);
        return -1;
    }

    //  If it gets here, thread pool is up and running
    return 1;
}

THREAD_POOL_PARAM_T * Comm::ContextAlloc(THREAD_POOL_HANDLE_T *handle)
{
    resmgr_context_t *ctp = resmgr_context_alloc(handle);
    CommServerStruct *comm_server = new CommServerStruct;
    SETIOV(ctp->iov, comm_server, sizeof(*comm_server));
    return ctp;
}

//  Function to block waiting for a message from client
THREAD_POOL_PARAM_T * Comm::WaitForMessage(THREAD_POOL_PARAM_T *ctp)
{
    //  Cast pointer
    CommServerStruct *comm_server = (CommServerStruct *)ctp->iov->iov_base;
    //  Blocks waiting for message
    comm_server->comm_rcvid = MsgReceive(g_comm_chid, comm_server->msg, 8, NULL);
    //ctp->rcvid = MsgReceive(g_comm_chid, comm_server.msg, 8, NULL);
    char rmsg[8];
    //  Reply to sender to unblock it immediately
    MsgReply(comm_server->comm_rcvid, EOK, rmsg, 8);

    SETIOV(ctp->iov, comm_server, sizeof(*comm_server));
    return ctp;
}

//  Message parsing function when unblocks with received message
int Comm::ParseMessage(THREAD_POOL_PARAM_T *ctp)
{
    //  Cast pointer
    CommServerStruct *comm_server = (CommServerStruct *)ctp->iov->iov_base;
	//	If the message is coming from the UI
	if (comm_server->msg[0] == SCREEN)
	{
...
...
}


Is anyone working with thread pools outside the context of a resource manager?
Can anyone give me some directions on why the threads are not being created and destroyed as expected?
Re: Thread pool outside resource manager.  
Hi,

I believe your problem lies (at least) with the MsgReceive() call. The size of the receive buffer must be at least large
 enough to accomodate one  struct _pulse , which (IIRC) is currently 16 bytes. It is usually recommended to define a 
union type across all expected message type (including pulses!), like this:
   union msg_u {
      uint16_t                   type;
      struct _pulse          pulse;
      struct my_msg_1  msg_1;
      struct my_msg_2  msg_2;
      ...
      struct my_msg_n  msg_n;
   };

...then have a receive buffer of this type and use its sizeof() in the receive:
   union msg_u   msg;
   ...
   rcvid = MsgReceive(chid, &msg, sizeof msg, NULL);

As always, checking the return values of functions like MsgReceive() is strongly recommended...  ;-)

Hope this helps,
Thomas
Re: Thread pool outside resource manager.  
Hi Thomas, 

First of all, thank you for your reply!

Well, it was not that actually, even because since I am the only one talking to the created channel (at least 
supposedly), I have very good control of the kind of messages that will be transmitted/received.

But I did solve the problem in a different manner. I cannot actually explain why very well, but it seems that if QNX 
returns from the function that starts the thread pool, the thread pool does not work well.

So from the code above, the only change was creating the thread pool as:

tpp = thread_pool_create (&tp_attr, POOL_FLAG_USE_SELF);

But in order to not block the whole program, the main program that used to just call InitComm(), now creates a new 
thread to do so.

It works beautifully now! Thanks anyway…

Best regards

John
Re: Thread pool outside resource manager.  
Hi John,

I'm glad things now seem to work for you. Still, you should be aware of two things:
First, since you leave the channel creation to name_attach(), you are not as much in control of received messages as you
 believe. name_attach() will set channel flags NTO_CHF_DISCONNECT, NTO_CHF_COID_DISCONNECT, and NTO_CHF_UNBLOCK. Thus, 
you should be prepared to handle the associated pulses PULSE_CODE_DISCONNECT (client has disconnected), 
PULSE_CODE_COID_DISCONNECT (a server has disappeared), and PULSE_CODE_UNBLOCK (a REPLY-blocked client wants to unblock 
due to signal, timeout, or cancellation).
And second, even if you never received any pulse at all, the MsgReceive() call will still fail if the receive buffer is 
not sufficiently large for a pulse. Please do me (and much more yourself) a favor and check the return code from 
MsgReceive() for potential errors.

Best regards and good luck,
Thomas
Re: Thread pool outside resource manager.  
Hi Thomas,

I will look into it!
Thank you very much.

Best regards

John
Re: Thread pool outside resource manager.  
Hi,

seems to me, you have the same problem, I had a week ago.

The pool create function copies your thread_pool_attr_t  structure.
But it does NOT copy your thread_attr_t structure!!
So, after you return from your InitComm method, your thread_attr_t structure belongs to free memory.
=> With a good chance the pthread_create funtion inside pool-handling gets EINVAL and no additional thread is created.

-Michael
Re: Thread pool outside resource manager.  
Bullseye Michael!

Thank you very much! That does actually makes total sense.
Just made the thread_attr_t a class member and it went perfectly!

Best regards

John
Re: Thread pool outside resource manager.  
The documentation for thread_pool_create() has a note about the shallow copy, but maybe it isn't prominent enough. I'll 
move it up in the description.
Re: Thread pool outside resource manager.  
Yes it says that - but "shallow copy" was  not a meaningful term to me as a C-programmer.   Apparently, it is an object-
oriented term according to wikipedia.
Re: Thread pool outside resource manager.  
But the note doesn't stop at "shallow copy". Here's the full note:

The thread_pool_create() function does a shallow copy of the thread_pool_attr_t structure; it doesn't make copies of 
anything that the structure points to. This means that items that the structure points to (e.g., the tid_name string and
 the pthread_attr_t structure) must exist for the lifetime of the thread pool itself. For example, this is valid:

pool_attr.tid_name = "fsys_resmgr";

but using a local or automatic variable like this isn't:

{
   char  name[32];
   snprintf(name, sizeof(name), "cam %d:%d", cam.path, cam.target);
   pool_attr.tid_name = name;
   ...
   return;
}

Does it need to say anything else?
Re: Thread pool outside resource manager.  
You are right.   The text is very clear - my fault for not paying more attention to it!
Re: Thread pool outside resource manager.  
Thanks guys!

But which document is this one? It doesn't seem to be the ones that I am following (which are mainly 
Get_Programming_with_the_QNX_Neutrino_RTOS and QNX_Neutrino_RTOS_C_Library_Reference)
Re: Thread pool outside resource manager.  
http://www.qnx.com/developers/docs

Then pick topmost "Go" button.  Use search box in upper left to find thread_pool_create() entry.
Re: Thread pool outside resource manager.  
Re: Thread pool outside resource manager.  
Thanks again!