TeamForge : Post

Forum Topic - ntox86-ld Hanging and consuming 2MB/sec of RAM: (5 Items)

View: as

zane mccaig

07/11/2014 4:14 PM

post111039

ntox86-ld Hanging and consuming 2MB/sec of RAM

Hi everyone,

I am continuing my work on porting LLVM to QNX650 and I am running into a problem that appears to be a hanging within a 
g++ call. The busiest process during the hanging is ntox86-ld.

The g++ call is generated from a Makefile and here is the relevant portion of the output when I run make -n  on the 
problematic directory.

make[2]: Entering directory `/home/llvm/tools/lto'
echo llvm[2]: Linking Debug+Asserts "Shared Library" \
          libLTO.so
g++  -Wl,-R -Wl,'$ORIGIN' -Wl,--gc-sections  -L/home/llvm/Debug+Asserts/lib -L/home/llvm/Debug+Asserts/lib    -shared -o
 /home/llvm/Debug+Asserts/lib/libLTO.so /home/llvm/tools/lto/Debug+Asserts/LTODisassembler.o /home/llvm/tools/lto/Debug+
Asserts/lto.o \
           -lLLVMMCDisassembler -lLLVMLTO -lLLVMObjCARCOpts -lLLVMLinker -lLLVMipo -lLLVMVectorize -lLLVMBitWriter -
lLLVMR600CodeGen -lLLVMR600Desc -lLLVMR600Info -lLLVMR600AsmPrinter -lLLVMSystemZDisassembler -lLLVMSystemZCodeGen -
lLLVMSystemZAsmParser -lLLVMSystemZDesc
 -lLLVMSystemZInfo -lLLVMSystemZAsmPrinter -lLLVMHexagonCodeGen -lLLVMHexagonAsmPrinter -lLLVMHexagonDesc -
lLLVMHexagonInfo -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMCppBackendCodeGen -
lLLVMCppBackendInfo -lLLVMMSP430CodeGen -lLLVMMSP
430Desc -lLLVMMSP430Info -lLLVMMSP430AsmPrinter -lLLVMXCoreDisassembler -lLLVMXCoreCodeGen -lLLVMXCoreDesc -
lLLVMXCoreInfo -lLLVMXCoreAsmPrinter -lLLVMMipsDisassembler -lLLVMMipsCodeGen -lLLVMMipsAsmParser -lLLVMMipsDesc -
lLLVMMipsInfo -lLLVMMipsAsmPrinter -lLLVMAArch64D
isassembler -lLLVMAArch64CodeGen -lLLVMAArch64AsmParser -lLLVMAArch64Desc -lLLVMAArch64Info -lLLVMAArch64AsmPrinter -
lLLVMAArch64Utils -lLLVMARMDisassembler -lLLVMARMCodeGen -lLLVMARMAsmParser -lLLVMARMDesc -lLLVMARMInfo -
lLLVMARMAsmPrinter -lLLVMPowerPCDisassembler -lLL
VMPowerPCCodeGen -lLLVMPowerPCAsmParser -lLLVMPowerPCDesc -lLLVMPowerPCInfo -lLLVMPowerPCAsmPrinter -
lLLVMSparcDisassembler -lLLVMSparcCodeGen -lLLVMSparcAsmParser -lLLVMSparcDesc -lLLVMSparcInfo -lLLVMSparcAsmPrinter -
lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86Cod
eGen -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMipa
 -lLLVMAnalysis -lLLVMTarget -lLLVMX86Desc -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMCore -lLLVMX86Info -
lLLVMX86AsmPrinter -lLLVMMC -lLLVMX86U
tils -lLLVMSupport -Wl,--version-script,/home/llvm/tools/lto/Debug+Asserts/lto.exports.map -lz -lncurses -lm
make[2]: Leaving directory `/home/llvm/tools/lto'
make[2]: Entering directory `/home/llvm/tools/llvm-lto'
echo llvm[2]: Linking Debug+Asserts executable llvm-lto
g++  -Wl,--gc-sections -Wl,-R -Wl,'$ORIGIN/../lib' -L/home/llvm/Debug+Asserts/lib -L/home/llvm/Debug+Asserts/lib    -o /
home/llvm/Debug+Asserts/bin/llvm-lto  /home/llvm/tools/llvm-lto/Debug+Asserts/llvm-lto.o  \
        -lLLVMR600CodeGen -lLLVMR600Desc -lLLVMR600Info -lLLVMR600AsmPrinter -lLLVMSystemZDisassembler -
lLLVMSystemZCodeGen -lLLVMSystemZAsmParser -lLLVMSystemZDesc -lLLVMSystemZInfo -lLLVMSystemZAsmPrinter -
lLLVMHexagonCodeGen -lLLVMHexagonAsmPrinter -lLLVMHexagonDesc -
lLLVMHexagonInfo -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMCppBackendCodeGen -
lLLVMCppBackendInfo -lLLVMMSP430CodeGen -lLLVMMSP430Desc -lLLVMMSP430Info -lLLVMMSP430AsmPrinter -lLLVMXCoreDisassembler
 -lLLVMXCoreCodeGen -lLLVMXCoreDesc
-lLLVMXCoreInfo -lLLVMXCoreAsmPrinter -lLLVMMipsDisassembler -lLLVMMipsCodeGen -lLLVMMipsAsmParser -lLLVMMipsDesc -
lLLVMMipsInfo -lLLVMMipsAsmPrinter -lLLVMAArch64Disassembler -lLLVMAArch64CodeGen -lLLVMAArch64AsmParser -
lLLVMAArch64Desc -lLLVMAArch64Info -lLLVMAArch64As
mPrinter -lLLVMAArch64Utils -lLLVMARMDisassembler -lLLVMARMCodeGen -lLLVMARMAsmParser -lLLVMARMDesc -lLLVMARMInfo -
lLLVMARMAsmPrinter -lLLVMPowerPCDisassembler -lLLVMPowerPCCodeGen -lLLVMPowerPCAsmParser -lLLVMPowerPCDesc -
lLLVMPowerPCInfo -lLLVMPowerPCAsmPrinter -lLLVMS
parcDisassembler -lLLVMSparcCodeGen...

View Full Message

bogdan celer

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

bogdan celer

07/15/2014 6:06 AM

post111058

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

Zane 
I see that you build Debug type of LLVM.
Library is very very big ! I thing that you should build Release typ of LLVM.
You use cmake or make to build LLVM ?  
There is my version of LLVM 3.4 for QNX650: LLVM_Clang.3.4-i386-QNX6.5.7z 
You can show your port ?

Attachment:

LLVM_Clang.3.4-i386-QNX6.5.7z 198.52 KB

Armin Steinhoff

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

Armin Steinhoff

07/15/2014 8:36 AM

post111060

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

zane mccaig wrote:
> Hi everyone,
>
> [ clip]
>
> At this point it begins to consume 2-3MB/sec with ntox86-ld running at 14-15% of the cpu with two devb-eide being the 
second busiest processes running at about 5% cpu usage each.
>
> With 512Mb of RAM this would continue until all memory was consumed and then it would crash. I increased the RAM to 
4Gb to see if that would help. It has not taken all the ram yet but it is well on its way.
>
> The version of ntox86-ld is 2.24
> Binutils is 2.24 (From here)
> GCC is 4.8.1 (from Ajamweb.pl)
> QNX is 6.5.0 2010/07/09
>
> Does this consumption seem normal given the number of libraries being linked or does this seem out of the ordinary? 

That's normal ... also when your system has 4GB memory e.g.

> Does anyone know about any compiler options I can add on that will give verbosity from the loaders perspective?
>
> Any other insight is more than welcome!
Here some other statements about the timely behavior of the build process:

   Csaba Raduly wrote:
> Hi all,
>
> On Thu, Jun 12, 2014 at 2:37 PM, Armin Steinhoff  wrote:
>> Dan Liew wrote:
>>
>> If you want things to go faster
>>
>> * Use the -jN flag with make to build in parallel where N is the maximum
>> number of jobs run in parallel. E.g.
>>
>> $ make -j4
>>
>>
>> This could be critical if you have 4 or less GigaBytes of memory. If you run
>> "make -j2" it makes sure that no swapping happens  ... mostly :)
>>
> I use the following hack to ensure that at most one linker process runs:
>
> ~/workspace/LLVM/llvm $ svn diff Makefile.rules
> Index: Makefile.rules
> ===================================================================
> --- Makefile.rules    (revision 212018)
> +++ Makefile.rules    (working copy)
> @@ -767,7 +767,7 @@
>  Preprocess.CXX= $(Compile.Wrapper) \
>                $(CXX) $(CPP.Flags) $(TargetCommonOpts) $(CPPFLAGS) \
>                  $(CompileCommonOpts) $(CXX.Flags) -E
> -Link          = $(Compile.Wrapper) \
> +Link          = $(Compile.Wrapper) ${FLOCK_COMMAND} \
>                $(CXX) $(CXXFLAGS) $(LD.Flags) $(LDFLAGS) \
>                  $(TargetCommonOpts) $(Strip)
>
> and the following command line:
>
> $ make -j 8 FLOCK_COMMAND='flock /tmp/linker.lock'
>
> This is more or less required on my 8-processor/8GB RAM machine when
> building the debug version :(
>

Cheers

--Armin

>
> Thanks Everyone!
>
> Regards,
>
>
> Zane
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post111039
> To cancel your subscription to this discussion, please e-mail general-toolchain-unsubscribe@community.qnx.com
>

zane mccaig

07/16/2014 10:14 AM

post111068

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

hi guys,

Thank you both for your insight. I ended up making it through that particular linking operation that i originally posted
 about. It just took a while to do it. Then I had the same problem happen when it tried to link clang. 

So I restarted the build in release mode with make-j2 instead of j4 and it seems to be a lot more conservative with the 
RAM so I am hoping it will be fine now. Thus far I have only had the issue occur when a single linker is running so I 
did not need to put in Armin's changes in the makefile but if I end up having an issue I will put them in. 

Bogdan I have attached the diff files from llvm and clang. They are a little outdated from what I am currently trying to
 build but I cannot easily get a diff from within QNX because the svn version is too old for the LLVM repository. For 
the most part the changes are the same aside from some paths being different. (ie I have gcc 4.8.1 and then QNX's gcc 4.
4.2, where you had gcc 4.7.3 and 4.6.3).

I have however pushed almost everything from the changes you posted above into Clang so the clang.diff I am posting here
 is not as good of an indicator of what I am trying to build as your own changes would be. That said I have not pushed 
anything related to pthreads yet as I have not run into an issue with it. 
What kind of problems did you run into that you needed to make all of the pthread changes?

Also I am building using Make not cmake.

Thanks & regards,

Zane

Attachment:

llvm3.5_QNX6.5.0.tar.bz2 4.85 KB

Armin Steinhoff

07/16/2014 10:30 AM

post111069

Re: ntox86-ld Hanging and consuming 2MB/sec of RAM

Hi Zane,

don't give up ! The linker can run a very, very long time!

Cheers

--Armin

zane mccaig wrote:
> hi guys,
>
> Thank you both for your insight. I ended up making it through that particular linking operation that i originally 
posted about. It just took a while to do it. Then I had the same problem happen when it tried to link clang. 
>
> So I restarted the build in release mode with make-j2 instead of j4 and it seems to be a lot more conservative with 
the RAM so I am hoping it will be fine now. Thus far I have only had the issue occur when a single linker is running so 
I did not need to put in Armin's changes in the makefile but if I end up having an issue I will put them in. 
>
> Bogdan I have attached the diff files from llvm and clang. They are a little outdated from what I am currently trying 
to build but I cannot easily get a diff from within QNX because the svn version is too old for the LLVM repository. For 
the most part the changes are the same aside from some paths being different. (ie I have gcc 4.8.1 and then QNX's gcc 4.
4.2, where you had gcc 4.7.3 and 4.6.3).
>
> I have however pushed almost everything from the changes you posted above into Clang so the clang.diff I am posting 
here is not as good of an indicator of what I am trying to build as your own changes would be. That said I have not 
pushed anything related to pthreads yet as I have not run into an issue with it. 
> What kind of problems did you run into that you needed to make all of the pthread changes?
>
> Also I am building using Make not cmake.
>
> Thanks & regards,
>
> Zane
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post111068
> To cancel your subscription to this discussion, please e-mail general-toolchain-unsubscribe@community.qnx.com

Return