Project Home
Project Home
Source Code
Source Code
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - accelerate the compilation via multi-core machines: (6 Items)
   
accelerate the compilation via multi-core machines  
Hi,

Someone would like to accelerate the compilation of programms and is now looking for faster machines.

In this context following came up:

Does QNX compiler on Windows take advantage of dual and quad core ?
Does it also take advantage of multiple dual and quad core processors?

THX,
Jeevan
 
Re: accelerate the compilation via multi-core machines  
> Hi,
> 
> Someone would like to accelerate the compilation of programms and is now 
> looking for faster machines.
> 
> In this context following came up:
> 
> Does QNX compiler on Windows take advantage of dual and quad core ?
> Does it also take advantage of multiple dual and quad core processors?

The compiler, assembler and linker are all single threaded applications so the answer is no. However someone with a 
multicore/multiprocessor system could use make to parallelize the build (make -j) which invokes the compiler  several 
times simultaneously.

http://www.gnu.org/software/make/manual/make.html#Parallel

Regards,

Ryan Mansfield



Re: accelerate the compilation via multi-core machines  
On Tue, 2008-04-15 at 09:07 -0400, Ryan Mansfield wrote:

> The compiler, assembler and linker are all single threaded
> applications so the answer is no. However someone with a
> multicore/multiprocessor system could use make to parallelize the
> build (make -j) which invokes the compiler  several times
> simultaneously.
> 
> http://www.gnu.org/software/make/manual/make.html#Parallel

Though a couple of things should be clarified:

1) If you work with the qnx makefiles, you need to "make JLEVEL=4 all"
instead of "make -j 4 all".

2) qnx makefiles have some bugs with regards to JLEVEL (the next release
will have the necessary fixes)
Re: accelerate the compilation via multi-core machines  
Once upon a time I did manage to get distcc building the C library and it
made a
significant difference in time.  It was a bit of a pain since I had to
create a custom
compiler configuration to front end for qcc but it did work.

If someone is looking for a good summer coop project, resurecting distcc for
qcc
for QNX builds would be great for the community.

Thomas

On Tue, Apr 15, 2008 at 10:21 AM, Stephane Boucher <sboucher@qnx.com> wrote:

> On Tue, 2008-04-15 at 09:07 -0400, Ryan Mansfield wrote:
>
> > The compiler, assembler and linker are all single threaded
> > applications so the answer is no. However someone with a
> > multicore/multiprocessor system could use make to parallelize the
> > build (make -j) which invokes the compiler  several times
> > simultaneously.
> >
> > http://www.gnu.org/software/make/manual/make.html#Parallel
>
> Though a couple of things should be clarified:
>
> 1) If you work with the qnx makefiles, you need to "make JLEVEL=4 all"
> instead of "make -j 4 all".
>
> 2) qnx makefiles have some bugs with regards to JLEVEL (the next release
> will have the necessary fixes)
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post6800
>
>
Re: accelerate the compilation via multi-core machines  
> Once upon a time I did manage to get distcc building the C library and it
> made a
> significant difference in time.  It was a bit of a pain since I had to
> create a custom
> compiler configuration to front end for qcc but it did work.
> 
> If someone is looking for a good summer coop project, resurecting distcc for
> qcc
> for QNX builds would be great for the community.
> 
> Thomas
> 
> On Tue, Apr 15, 2008 at 10:21 AM, Stephane Boucher <sboucher@qnx.com> wrote:
> 
> > On Tue, 2008-04-15 at 09:07 -0400, Ryan Mansfield wrote:
> >
> > > The compiler, assembler and linker are all single threaded
> > > applications so the answer is no. However someone with a
> > > multicore/multiprocessor system could use make to parallelize the
> > > build (make -j) which invokes the compiler  several times
> > > simultaneously.
> > >
> > > http://www.gnu.org/software/make/manual/make.html#Parallel
> >
> > Though a couple of things should be clarified:
> >
> > 1) If you work with the qnx makefiles, you need to "make JLEVEL=4 all"
> > instead of "make -j 4 all".
> >
> > 2) qnx makefiles have some bugs with regards to JLEVEL (the next release
> > will have the necessary fixes)
> >
> >
> > _______________________________________________
> > General
> > http://community.qnx.com/sf/go/post6800
> >
> >

Thanks all. Hi Thomas,

can perhaps You confrim that this is working (and if yes, will keep working)?:

http://lists.samba.org/archive/distcc/2006q3/003431.html

There we can read this:
--------------------------------------------------------------------------------
We have distcc working with QNX 6.3.0 - the QNX cross-compiler 'qcc'
is more-or-less identical to gcc.

The main complication with QNX is its recursive makefile structure:

In order to use distcc you must edit c:/QNX630/target/qnx6/usr/include/qconfig.mk in
which 'make' variables are set up to invoke the various target-dependent compiler
derivatives. For example:

CC_nto_x86_gcc_qcc = $(QNX_HOST)/usr/bin/qcc -V$(gcc_ver_string)gcc_ntox86 -c
AS_nto_x86_gcc_qcc = $(QNX_HOST)/usr/bin/qcc -V$(gcc_ver_string)gcc_ntox86 -c
LR_nto_x86_gcc_qcc = $(QNX_HOST)/usr/bin/qcc -V$(gcc_ver_string)gcc_ntox86 -r
LD_nto_x86_gcc_qcc = $(QNX_HOST)/usr/bin/qcc -V$(gcc_ver_string)gcc_ntox86
AR_nto_x86_gcc_qcc = $(AR_nto_x86_gcc)
UM_nto_x86_gcc_qcc = $(UM_nto_x86_gcc)
LDBOOTSTRAP_nto_x86_gcc_qcc = $(LDBOOTSTRAP_nto_x86_gcc)
LDBOOTSTRAPPOST_nto_x86_gcc_qcc = $(LDBOOTSTRAPPOST_nto_x86_gcc)

must become:

CC_nto_x86_gcc_qcc = distcc qcc -V$(gcc_ver_string)gcc_ntox86 -c
AS_nto_x86_gcc_qcc = distcc qcc -V$(gcc_ver_string)gcc_ntox86 -c
LR_nto_x86_gcc_qcc = distcc qcc -V$(gcc_ver_string)gcc_ntox86 -r
LD_nto_x86_gcc_qcc = distcc qcc -V$(gcc_ver_string)gcc_ntox86
AR_nto_x86_gcc_qcc = $(AR_nto_x86_gcc)
UM_nto_x86_gcc_qcc = $(UM_nto_x86_gcc)
LDBOOTSTRAP_nto_x86_gcc_qcc = $(LDBOOTSTRAP_nto_x86_gcc)
LDBOOTSTRAPPOST_nto_x86_gcc_qcc = $(LDBOOTSTRAPPOST_nto_x86_gcc)

Then ensure that the PATH environment variable has c:/QNX630/host/win32/x86/usr/bin
early in its list. (N.B. QNX_HOST is set to c:/QNX630/host/win32/x86 on our
QNX distribution)

This works both from Momentics and a bash shell

In order to gain control of the 'make' -j option which QNX normally defaults to 1,
try 'make JLEVEL=2 CPULIST=<your target> ....' or add an equivalent build target to
Momentics.

Choosing '-j N' or JLEVEL=N where N is greater than the number of volunteer
hosts, we have found that distcc will occasionally cough out error message:

'cc: mkdir: cannot create temporary working directory'
'distcc[2024] ERROR: compile c:/tom/dick/harry.cpp failed'

Other than that, it seems to work as advertised. One is impressed.

Steven Whittard-Swift,...
Re: accelerate the compilation via multi-core machines  
> > Hi,
> > 
> > Someone would like to accelerate the compilation of programms and is now 
> > looking for faster machines.
> > 
> > In this context following came up:
> > 
> > Does QNX compiler on Windows take advantage of dual and quad core ?
> > Does it also take advantage of multiple dual and quad core processors?
> 
> The compiler, assembler and linker are all single threaded applications so the
>  answer is no. However someone with a multicore/multiprocessor system could 
> use make to parallelize the build (make -j) which invokes the compiler  
> several times simultaneously.
> 
> http://www.gnu.org/software/make/manual/make.html#Parallel
> 
> Regards,
> 
> Ryan Mansfield
> 
> 
> 

I've been using -j option to make for over a year now, with our own makefile. We got 4 core machines and it's compiles 
almosts 4 times faster.   Can't wait to try it on an 8 core machine ( 2 quad ).

Also setting devb-eide to commit=none and noatime does help, at the cost of robusness and access time update seems to 
help speed the compilation.  I also increase cache size but I have not speed time to find the sweet spot.