1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

fwrite fread for large files

Discussion in 'C' started by ccoolgoose, Jun 2, 2009.

  1. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Hi,

    I am trying to write a 1 GB file in blocks of 100KB and read them back in 100KB blocks.
    First, I would like to write the whole 1 GB in consecutive blocks of 100kb and then read them back in 100kb consecutive blocks.
    How do I get the pointer set to the correct value to read the next block? Could anyone please help?

    Thank you!
     
  2. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    Assuming your file pointer is pFile; you can use :

    Code:
    for(int block = 0; block < MAX_BLOCKS; ++block)
    {
          // We have to move cursor by block*100*1024 bytes.
          fseek(pFile, block*102400, SEEK_SET);
          .
          .
    }
    
     
    Last edited: Jun 2, 2009
  3. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    But, does SEEK_SET set the pointer to the beginning of the file. Will I get a 1GB in such a case.

    I was thinking of something like this:
    long int size;
    file *f;
    int *buff[102400];
    if(size<1024000000)
    {
    fwrite(buff, 1, 102400, file);
    size+=102400;
    }
    Please suggest. Thanks!
     
  4. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    Yeah you are right. The value of block * 102400 will be huge.
    [ Please ALWAYS post code inside code-blocks]

    So, you can use this :

    Code:
    fseek(pFile, 0, SEEK_SET);
    for(int block = 0; block < MAX_BLOCKS; ++block)
          fwrite(buff, 1, 102400, file);
    
    I think the write cursor will automatically move till the end of the written block.
    So your pointer is auto-adjusted.
     
    Last edited: Jun 2, 2009
  5. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    so, what value is max_blocks set to? Thanks a ton!
     
  6. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    also, is fseek really necessary..since we already have FILE *fp=NULL
     
  7. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    MAX_BLOCKS = Number of 100 kB blocks inside 1 GB = 1 GB / 100 kB = ~10485 :smile:

    Just for making sure. Not really necessary !
     
  8. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Awesome!

    So, do you know if there is anyway for me to write these 10485 blocks not consecutively....i mean not one by one..just randomly..but need to make sure that i write all the 10485 blocks

    I think may be for loop doesn't work because"for" loop does each block one by one. What do u suggest?
     
  9. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    Sorry, I was busy creating another thread. So, I didn't see your post.
    Hmmm.. I'll think for a while and reply.

    OK. Randomly in the sense ?!?
    After you write a block, your pointer stays there till you manually move it by reading writing or seeking.
    So, you can write a block. Take a break :). Write 100 more. Take another break. Then write 10000 more. And then after final break you may write the rest 384 blocks !

    I don't exactly understand what do you mean by RANDOMLY ?!
     
    Last edited: Jun 2, 2009
  10. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    If you mean, you want to write to random positions of the file, then I think you can make a func like this :

    Code:
    void SeekTo_NthBlock(FILE *pFile, int n, int blocksize)
    {
          // Note that this func treats the first block as Block 0, second as Block 1 and so on ..
    
          fseek(pFile, 0, SEEK_SET);
          for(int i = 0; i < n; ++i)
                fseek(pFile, blocksize, SEEK_CUR);
    }
    
    After defining the above func, you can use it like this :
    Code:
    // Write to 100th block
    SeekTo_NthBlock(fp, 99, 102400);
    fwrite(buf, 1, 102400, fp);
    
    // Write to 10001st block
    SeekTo_NthBlock(fp, 10000, 102400);
    fwrite(buf, 1, 102400, fp);
    
    Is that what you wanted ?
     
  11. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Not a problem..I mean...not consecutive blocks...but want to write/read the whole file. not 1,2,3...something like...1,50,2000, etc....till 1 GB
     
  12. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    Did you see my last post ?
    You can use the func SeekTo_NthBlock to write to any block at any time.

    Code:
    // Writing randomly
    short IsWritten[10485];
    memset(IsWritten, 0, 10485*sizeof(short));
    int TotalWritten = 0;
    int WriteToBlock;
    
    while (TotalWritten < 10485 )
    {
          WriteToBlock = rand() % 10485;
          if ( IsWritten[WriteToBlock] )      continue;
          SeekTo_NthBlock(fp, WriteToBlock, 102400);
          fwrite(buff, 1, 102400, fp);
          IsWritten[WriteToBlock] = 1;
          ++TotalWritten;
    }
    
    Is it okay now ?
     
  13. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Random ...i meant not sequential...but i wanted to write all 10485 blocks. how do i make sure i have written all the blocks
     
  14. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    I guess you are able to read the post just above yours. :mad:
    What does while(TotalWritten < 10485) mean ??
     
  15. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    I was trying to post...but I don't know it doesn't allow me :(
     
  16. ccoolgoose

    ccoolgoose New Member

    Joined:
    Jun 2, 2009
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Sorry...somehow I couldn't see your posts. Thank you very much!
     
  17. SaswatPadhi

    SaswatPadhi ~ Б0ЯИ Τ0 С0δЭ ~

    Joined:
    May 5, 2009
    Messages:
    1,343
    Likes Received:
    55
    Trophy Points:
    0
    Occupation:
    STUDENT !
    Location:
    Orissa, INDIA
    Home Page:
    You are welcome ! :biggrin:
     

Share This Page