Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Adaptive Partitioning: Processor/Partition/Thread affinity: (8 Items)
   
Adaptive Partitioning: Processor/Partition/Thread affinity  
Hello

I'm currently trying to implement a system on a quad-core machine with the following concept:

* Processing Partition, budget 75%: contains 3 worker threads that run on processors 2, 3 and 4 with hard affinity (no 
thread migrations).

* System Partition, budget 25%: runs on processor 1 _only_ and runs all other threads on the machine.


My worker threads (fixed on cpus 2, 3 and 4) are being interrupted from time to time with "kernel nops" (That's what it 
says in the kernel trace). I want to avoid this by setting the "default" runmask for all threads (new and existing) to 
0x1 (only processor 1).
I figured using partitioned scheduling would be an elegant solution, but after reading the documentation several times, 
I haven't been able to achieve what I need.

Does anybody have suggestions or solutions?

Thanks in advance,
~Lorenz
Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
How are you getting the system partition to run on processor 1 only?

Lorenz Bucher wrote:
> Hello
> 
> I'm currently trying to implement a system on a quad-core machine with the following concept:
> 
> * Processing Partition, budget 75%: contains 3 worker threads that run on processors 2, 3 and 4 with hard affinity (no
 thread migrations).
> 
> * System Partition, budget 25%: runs on processor 1 _only_ and runs all other threads on the machine.
> 
> 
> My worker threads (fixed on cpus 2, 3 and 4) are being interrupted from time to time with "kernel nops" (That's what 
it says in the kernel trace). I want to avoid this by setting the "default" runmask for all threads (new and existing) 
to 0x1 (only processor 1).
> I figured using partitioned scheduling would be an elegant solution, but after reading the documentation several times
, I haven't been able to achieve what I need.
> 
> Does anybody have suggestions or solutions?
> 
> Thanks in advance,
> ~Lorenz
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post18185
> 

Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
> How are you getting the system partition to run on processor 1 only?

I'm not... It was rather a question whether I could :-)
With the system partition, it's a certainly a special case, because I assume the kernel process (that's in the system 
partition) cannot have its affinity adjusted. Am I correct with this assumption?

Generally speaking, is there a way to directly set a partition's CPU affinity? (i.e. if I join a partition with a 
certain thread, it will automatically inherit the affinity of the partition).

And about process affinity: In Momentics, I can open the Target Navigator, select all the processes (except the kernel, 
procnto-smp-instr) on the system and set the CPU Affinity as well as the Inherited CPU Affinity to CPU0 only.
This is what I want to do .. but automatically.

I guess it could be done via shell scripting (taking the output of ps -A or something..), but I'm sure there's a better 
solution.

Sorry if I'm missing some obvious points.. 
<Many things I must learn, for a QNXpert to become>

~Lorenz
Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
there is no chance to get a relation between a partition and a core.
APS is looking at all available cores as 100%

/hp
Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
> there is no chance to get a relation between a partition and a core.
> APS is looking at all available cores as 100%
> 
> /hp

Ok, thanks for clearing that up. I somehow kept wanting to combine partitions and cpu affinity, while they're really two
 separate mechanisms.

After more reading and thinking, I've come to the conclusino that what I really want/need is just to set CPU affinity on
 all processes (at least all I can) to CPU0, so that my worker threads can execute without interruptions on the other 3 
cores.

Dankeschön,
~Lorenz
Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
Lorenz Bucher wrote:
>> How are you getting the system partition to run on processor 1 only?
> 
> I'm not... It was rather a question whether I could :-)
> With the system partition, it's a certainly a special case, because I assume the kernel process (that's in the system 
partition) cannot have its affinity adjusted. Am I correct with this assumption?

That's why I asked. I'm not aware of any easy way of setting affinity on proc theads. There is a hard way: use a series 

of DCMD_DEBUG_THREAD devctls on PROC to list out all of the thread IDs (see the source code of pidin for an example) and
 
then call ThreadCtl() on each one.

I'm not at all sure that it's worth it.


> Generally speaking, is there a way to directly set a partition's CPU affinity?

No. Partitions do not have cpu affinity. No do processes. Only individual threads have cpu affinity.

(Btw, we tend to call it "runmasked" rather than "cpu affinity" because the word "affinity" connotes to some that a 
"thread will spend most of its time on that cpu". Whereas, when a runmask is set, a thread is never permitted to run on 

a cpu incompatible with its runmask.)


However, there may be a way to do what you want without cpu affinity, using aps only. If you want to give a bunch of 
threads 75% total cpu and another bunch 25% of total cpu, APS can do that with two partition. The only requirement is 
that on an <n> cpu system, each partition have at least <n> threads (other wise one partition may not use up its entire 

budget).

-ad


Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
Thank you for your reply, Attilla.
I was indeed confused by partitions and runmasks. Now I see clearly that they're actually two completely separate things
.

> That's why I asked. I'm not aware of any easy way of setting affinity on proc 
> theads. There is a hard way: use a series 
> of DCMD_DEBUG_THREAD devctls on PROC to list out all of the thread IDs (see 
> the source code of pidin for an example) and 
> then call ThreadCtl() on each one.

Ok, that seems at least a little more elegant than the grep-ing and sed-ing I had planned ;-)

> However, there may be a way to do what you want without cpu affinity, using 
> aps only. If you want to give a bunch of 
> threads 75% total cpu and another bunch 25% of total cpu, APS can do that with
>  two partition. 

I tried removing the affinity , but then the worker threads in the partition with 75% would migrate between all 4 cpus 
all the time.. which I want to avoid for efficiency reasons.

So right now I have exactly that: System partition with 25%, Worker partition with 75% CPU. 

> The only requirement is 
> that on an <n> cpu system, each partition have at least <n> threads (other wise one 
> partition may not use up its entire 
> budget).

Yep .. that's something to consider. But as my worker partition has 3 threads that each run exclusively on their 
respective cpu and are always running (busy loop), I should always be able to reach the full budget (75% = 3 of 4 cpus).
  That's at least my design concept.

To sum up my goal, this is what I want:

CPU1 = runs *only* worker thread 1 (never interrupted!)
CPU2 = runs *only* worker thread 2 (never interrupted!)
CPU3 = runs *only* worker thread 3 (never interrupted!)
CPU0 = handles ALL other threads.

Can I really achieve this through APS without manually setting each thread's runmask?


I'm quite close of achieving this, but my worker threads get interrupted, even though they have max priority. 
However, I've noticed that the threads that interrupt my threads are mostly (all?) interrupt-driven stuff (device 
drivers), which would explain why they're allowed to preempt my threads.

So I might get away with just runmasking each device driver to CPU0, but as Attilla said before, this can get ugly.
Any other suggestions?
I mean, it's not like I can/want to mask all other interrupt handlers indefinitely long..

--
Another question on the side: What are these KER NOP enter/exit I see about every 1ms? Even when a thread is not 
rescheduled, the trace shows these events. Is that the "system tick"?

Thanks,
~Lorenz


p.s. And thanks for pointing out to me the difference between "runmasked" and "affinity" - that definitely helps 
minimize the confusion.
Re: Adaptive Partitioning: Processor/Partition/Thread affinity  
I have to add that I've already tried the following to set the runmask of some existing processes:

According to:
http://www.qnx.com/developers/docs/6.3.2/momentics/release_notes/rel_6.3.2.html#Changes_core_utilities

I should be able to set the runmask of a running process using slay. E.g. I execute "slay -C 0 qconn"  to limit qconn to
 run on CPU0. When I do this and then check it using "pidin -p qconn sched", I see that qconn is running 4 threads, but 
only the first is indeed running on CPU0, the other 3 are running on CPU1 and CPU3.

--

Also, when I set the affinity in Momentics (as I described in my previous post), and then open the dialog again, the 
processes have all cpus checked... is this some kind of "write only" dialog, i.e. I cannot look a the current status of 
the runmask with this?

~Lorenz