Filesystems/Discussion/General/Not able to read a binary file of 2.64GB through fread() on QNX/List of Posts
 
Forum Topic - Not able to read a binary file of 2.64GB through fread() on QNX: Page 1 of 2 (29 Items)
   
Update
Shivendra Tiwari
08/16/2008 7:18 AM
post11897
Not able to read a binary file of 2.64GB through fread() on QNX   
Hello,

I have been trying to read the data from a large sized binary file (2.64GB), but fread() returns the size of the read 
data as 0. It works fine for all the small size files. I checked "errno" variable which has the value 0 (no errors).

Here is the code:
    char buf[4096];
    int numBytes =0;
    int bufSize=0;
    FILE* kiwiFile;

    kiwiFile = fopen( "/SharedSpace/01KIWI_20060929/ALLDATA.KWI", "r" );
    bufSize = sizeof(buf);
    memset(buf,0,bufSize);
    numBytes = fread(buf,1,bufSize,kiwiFile);



Can someone please suggest me the solutions/reason.

Thanks,
Shivendra
  Post Deleted
Shivendra Tiwari
08/18/2008 3:04 AM
post11919
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
I tried various options to read the data from the file through fopen(), fopen64(), open(), open64() and readblock() APIs
 but still no luck. 

Any clue would be great help.

-Shivendra
  Post Deleted
David Sarrazin
08/18/2008 9:14 AM
post11939
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
Shivendra,

Which filesystem is this 2.64gb file located on?  Some of them don't  
support files larger than 2gb.

David

On 16-Aug-08, at 7:18 AM, "Shivendra Tiwari" <community- 
noreply@qnx.com> wrote:

> Hello,
>
> I have been trying to read the data from a large sized binary file  
> (2.64GB), but fread() returns the size of the read data as 0. It  
> works fine for all the small size files. I checked "errno" variable  
> which has the value 0 (no errors).
>
> Here is the code:
>    char buf[4096];
>    int numBytes =0;
>    int bufSize=0;
>    FILE* kiwiFile;
>
>    kiwiFile = fopen( "/SharedSpace/01KIWI_20060929/ALLDATA.KWI",  
> "r" );
>    bufSize = sizeof(buf);
>    memset(buf,0,bufSize);
>    numBytes = fread(buf,1,bufSize,kiwiFile);
>
>
>
> Can someone please suggest me the solutions/reason.
>
> Thanks,
> Shivendra
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post11897
>
Shivendra Tiwari
08/18/2008 9:26 AM
post11941
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
Hi David,

The File format is NTFS (on Windows XP); however I am using VMWare for installing the guest OS (i.e. QNX) on the same 
machine. 

Thanks,
Shivendra.
Shivendra Tiwari
08/18/2008 9:29 AM
post11942
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
I could also see that there are some certain limitation on QNX platform:

http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/user_guide/limits.html 

Could you please let me know if I can change the system configurations/preferences in order to make the file size limit 
to 3GB or something?

-Shivendra
  Post Deleted
Mario Charest
08/19/2008 2:43 PM
post12045
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
> I could also see that there are some certain limitation on QNX platform:
> 
> http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/user_guide/limits.html 
> 
> Could you please let me know if I can change the system configurations/
> preferences in order to make the file size limit to 3GB or something?

You can't.

Maybe the new QNX6FS can support files bigger then 2G?  Even if it does it would be complex to switch a QNX4FS partition
 to a QNX6FS partition.

You might be able to test it by setting up a second virtual HD and use QNX6FS on it.

> 
> -Shivendra
David Sarrazin
08/19/2008 2:48 PM
post12048
RE: Not able to read a binary file of 2.64GB through fread() on QNX   
 

> -----Original Message-----
> From: Mario Charest [mailto:community-noreply@qnx.com] 
> >
> > Could you please let me know if I can change the system 
> > configurations/ preferences in order to make the file size 
> limit to 3GB or something?
> 
> You can't.
> 
> Maybe the new QNX6FS can support files bigger then 2G?  Even 
> if it does it would be complex to switch a QNX4FS partition 
> to a QNX6FS partition.
> 
> You might be able to test it by setting up a second virtual 
> HD and use QNX6FS on it.


The new QNX6FS does support > 2GB file sizes.  The "32-bit" flag is not
set if you look at the statvfs() data, and I've created 2.75GB files
myself on qnx6fs.
  Post Deleted
Shivendra Tiwari
08/22/2008 7:48 AM
post12240
Re: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
Hi David,

Thanks for your reply.

I would like to know few details of the environment you are using in order to access the large files (>2GB):

1. What is the hardware configuration used? Whether the hardware is 64bit?
2. Do you think there is any difference using QNX on VMWare vs QNX installed on a machine directly?
3. How to get the QNX6FS? By default QNX supports QNX4FS.


Here is what I have been trying:
1. Installed QNX on a  machine (HP DX 2700 Core2 Duo).
2. It doesnt ask for the FS we want, it just asks for partition and by default uses QNX4.

In another try:
1. Installed VMWare on my Windows XP machine and installed QNX on VMWare.
2. Used _LARGE_FILE_BITS=64 while compiling the source.


Please let me know what else I need in order to get the large file working.


thanks a lot.

-Shivendra
David Sarrazin
08/22/2008 1:28 PM
post12273
RE: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
 

> -----Original Message-----
> From: Shivendra Tiwari [mailto:community-noreply@qnx.com] 
> Sent: August 22, 2008 7:49 AM
> To: general-filesystems
> Subject: Re: RE: Not able to read a binary file of 2.64GB 
> through fread() on QNX
> 
> Hi David,
> 
> Thanks for your reply.
> 
> I would like to know few details of the environment you are 
> using in order to access the large files (>2GB):
> 
> 1. What is the hardware configuration used? Whether the 
> hardware is 64bit?

I was testing QNX 6.4.0 on VMware, hosted on Windows.

> 2. Do you think there is any difference using QNX on VMWare 
> vs QNX installed on a machine directly?

No, there should be no difference.

> 3. How to get the QNX6FS? By default QNX supports QNX4FS.

The QNX6fs ships as the new default filesystem in the upcoming 6.4
release.  The QNX4 filesystem does not support files larger than 2GB.
The qnx6fs is on the Foundry27 Filesystems project, in the directory
services/blk/fs/qnx6.  It will be necessary to use the new io-blk along
with the updated filesystems.   You will also need to create a new
partition for the qnx6fs, or reformat your HDD.
> 
> 
> Here is what I have been trying:
> 1. Installed QNX on a  machine (HP DX 2700 Core2 Duo).
> 2. It doesnt ask for the FS we want, it just asks for 
> partition and by default uses QNX4.

OS versions prior to 6.4 only offered QNX4 as a choice.

> 
> In another try:
> 1. Installed VMWare on my Windows XP machine and installed 
> QNX on VMWare.
> 2. Used _LARGE_FILE_BITS=64 while compiling the source.

The _LARGE_FILE_BITS won't help for the filesystem.  Fs-qnx4 simply
cannot handle files larger than 2GB.

> 
> 
> Please let me know what else I need in order to get the large 
> file working.

There are programs which will allow to use a large file on filesystems
which don't support it.  They break the large file into multiple 2GB
files.  You can also try the ext2 filesystem if you require file sizes
larger than 2GB.

> 
> 
> thanks a lot.
> 
> -Shivendra
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post12240
> 
>
Shivendra Tiwari
08/22/2008 2:44 PM
post12280
Re: RE: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
Hi David,

Thanks a lot for very useful answers.

It would be really great if you could share few more information:

1. I can see the binaries for QNX6.3.2 but not for QNX6.4 (http://qnx.com/products/getmomentics/). Could you please let 
me know from where I can download QNX6.4 so that I can install it on VMWare? Or could you please share the binaries?

2. I have downloaded the complete "trunc" source code for file systems; however it gives error while compilig, as my QNX
 on VMWare is x86. Do you think I should Install full version first in order to compile the code easily?


Thanks a lot.

-Shivendra
  Post Deleted
Kerry Johnson
10/17/2008 9:19 AM
post15141
Re: RE: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
Hi,

I am glad that the qnx6 file system works for you.  One thing to note, if you intend to user the qnx6 file system for 
commercial purposes (as part of your product) there are licensing requirements for using this file system.  For more 
information, please contact your QNX sales rep.  (If you are using QNX products as a hobbiest, then this doesn't apply.)


From a product perspective, the qnx6 file system is part of the new QNX 6.4 release and this is the official QNX 
supported configuration.

Using the qnx6 filesystem that you build from Foundry and deploy in a 6.3.2 environment is possible, but is not covered 
by QNX standard support.  Community (Foundry27) support and other commercial support arrangements are availale if needed
.

Kerry
Thomas Fletcher
10/17/2008 8:15 PM
post15186
Re: Not able to read a binary file of 2.64GB through fread() on QNX   
Kerry Johnson wrote:
>  Hi,
>
>  I am glad that the qnx6 file system works for you.  One thing to
>  note, if you intend to user the qnx6 file system for commercial
>  purposes (as part of your product) there are licensing requirements
>  for using this file system.  For more information, please contact
>  your QNX sales rep.  (If you are using QNX products as a hobbiest,
>  then this doesn't apply.)
>
>  From a product perspective, the qnx6 file system is part of the new
>  QNX 6.4 release and this is the official QNX supported configuration.
>
>
>  Using the qnx6 filesystem that you build from Foundry and deploy in a
>  6.3.2 environment is possible, but is not covered by QNX standard
>  support.  Community (Foundry27) support and other commercial support
>  arrangements are availale if needed.

... and if you want to read a large amount of data in one shot, you will
want to look at fopen64() vs the normal fopen().  Depending on how you
compile your source (and it may even be the default in 6.4, I haven't 
checked)
you will get fopen() automatically mapped to fopen64()).

Thomas

Thomas
  Post Deleted
  Post Deleted
Shivendra Tiwari
08/22/2008 2:55 PM
post12285
RE: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
 I see,  there is a pre-release version of QNX6.4. 
 
https://www.qnx.com/account/authenticate.html?id=18301&redirect=%2Fdownload%2Fgroup.html%3Fprogramid%3D18301
 
Thanks,
Shivendra
  Post Deleted
  Post Deleted
David Sarrazin
08/19/2008 2:50 PM
post12049
RE: Not able to read a binary file of 2.64GB through fread() on QNX   
> -----Original Message-----
> From: Shivendra Tiwari [mailto:community-noreply@qnx.com] 
> 
> I could also see that there are some certain limitation on 
> QNX platform:
> 
> http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/user_guid
> e/limits.html 
> 
> Could you please let me know if I can change the system 
> configurations/preferences in order to make the file size 
> limit to 3GB or something?
> 
Shivendra,

Which filesystem are you running on the QNX machine?  The easiest way to
determine this is to open the command-line, and run "df -g /".

David
  Post Deleted
Shivendra Tiwari
08/24/2008 1:02 AM
post12317
Re: RE: Not able to read a binary file of 2.64GB through fread() on QNX   
Hi David,

I installed fresh QNX6.4 which has QNX6FS and hence it works with large files.

Now our requirement is installing QNX6FS on QNX6.3.2 OS (default is QNX4FS) so that with the exsiting setup and 
configuration we can support large files. I can see the Filesystem source code (http://community.qnx.com/sf/wiki/do/
viewPage/projects.filesystems/wiki/SourceGuide
) which can be compiled and installed.

Do you think I can plugin QNX6FS on QNX6.3.2 and it will support large files. Though I think it should work but still I 
want to confirm with you.

Thanks a lot.

-Shivendra
12Next

Return