Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / Languages / C# / August 2007

Tip: Looking for answers? Try searching our database.

GZipStream Decompression Failure

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
BDRichardson - 30 Aug 2007 11:44 GMT
Hi,

I have been using the new GZipStream classes, and have been experiencing
problems when attemping to decompress files, which from experience, seem to
be failing when the original file size exceeds something like 64 MB.

For example, when I attempt to decompress a text file of size 1.18 MB to its
original size of 106 MB, I receive the following error message:

System.IO.IOException:  Insufficient system resources exist to complete the
requested service.
Source:  mscorlib
StackTrace:  "   at System.IO.__Error.WinIOError(Int32 errorCode, String
maybeFullPath)\r\n   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32
offset, Int32 count)\r\n   at System.IO.FileStream.Write(Byte[] array, Int32
offset, Int32 count)\r\n   at %MethodName% in %SourceFilePath%:line 96"
TargetSite:  {Void WinIOError(Int32, System.String)}

However, it works perfectly fine for files of an original size which is,
approximately, less than 64 MB in size.

Can anyone kindly provide any assistance as to whether there is a solution,
or whether there is just a flaw in the class?

Thanks
Jon Skeet [C# MVP] - 30 Aug 2007 11:56 GMT
On Aug 30, 11:44 am, BDRichardson
<BDRichard...@discussions.microsoft.com> wrote:
> I have been using the new GZipStream classes, and have been experiencing
> problems when attemping to decompress files, which from experience, seem to
> be failing when the original file size exceeds something like 64 MB.

<snip>

> Can anyone kindly provide any assistance as to whether there is a solution,
> or whether there is just a flaw in the class?

Well, you haven't shown any of the code you're using to decompress.

Could you produce a short but complete program which demonstrates the
problem?
See http://pobox.com/~skeet/csharp/complete.html for what I mean by
that.

Jon
BDRichardson - 30 Aug 2007 12:08 GMT
The Method:

public static void DecompressFile( FileInfo FileToDecompress, String
DestinationDirectory, Boolean DeleteOriginalFile)
{
    FileStream fsSource = null;
    FileStream fsDestination = null;
    GZipStream compressedStream = null;

    try
    {
        Byte[] buffer;

        fsSource = new FileStream( FileToDecompress.FullName, FileMode.Open,
FileAccess.Read, FileShare.Read);

        // The original file size may be obtained from the footer of the
compressed file
        buffer = new Byte[ 4];
        fsSource.Position = Convert.ToInt32( fsSource.Length) - 4;
        fsSource.Read( buffer, 0, 4);
        Int32 _OriginalFileSize = BitConverter.ToInt32( buffer, 0);

        // Read the decompressed file contents
        buffer = new Byte[ _OriginalFileSize];
        fsSource.Position = 0;

        compressedStream = new GZipStream( fsSource, CompressionMode.Decompress,
true);
        compressedStream.Read( buffer, 0, _OriginalFileSize);
        compressedStream.Flush();
        compressedStream.Close();

        // Write the decompressed data to a new file
        String _OriginalFileName = FileToDecompress.Name.Substring( 0,
FileToDecompress.Name.Length - 5);
        fsDestination = new FileStream( DestinationDirectory +
Path.DirectorySeparatorChar + _OriginalFileName, FileMode.Create,
FileAccess.Write, FileShare.Write);
        fsDestination.Write( buffer, 0, _OriginalFileSize);
        fsDestination.Close();
        fsSource.Close();

        System.Diagnostics.Debug.WriteLine( String.Format( "The file {0} was
decompressed as {1}.", FileToDecompress.Name, _OriginalFileName));

        if( DeleteOriginalFile)
        {
            FileToDecompress.Delete();
            System.Diagnostics.Debug.WriteLine( String.Format( "The following file
was deleted: {0}", FileToDecompress.Name));
        }
    }
    catch( System.Exception ex)
    {
        System.Diagnostics.Debug.WriteLine( String.Format( "The following
exception occurred within the application:\n{0}", ex.Message));
    }
    finally
    {
        if( fsSource != null) fsSource.Close();
        if( compressedStream != null)
        {
            compressedStream.Flush();
            compressedStream.Close();
        }

        if( fsDestination != null) fsDestination.Close();
    }
}
Jon Skeet [C# MVP] - 30 Aug 2007 12:14 GMT
On Aug 30, 12:08 pm, BDRichardson
<BDRichard...@discussions.microsoft.com> wrote:
> The Method:

Please see http://pobox.com/~skeet/csharp/incomplete.html

However, it looks to me like the problem is that you're trying to read
the whole thing in one go. Aside from anything else, this is
horrendously inefficient in terms of memory. It's much better to use a
buffer, read into that, write from it, then repeat until you're done.

I've got a method in my MiscUtil library for copying a whole stream.
See
http://pobox.com/~skeet/csharp/miscutil

It won't *quite* work out of the box in this case because you've
appended the original size to the end of the compressed stream. Is
this absolutely required? If you really need the information, could
you not put it at the *start* of the stream rather than the end? (That
way it can be skipped over very easily, rather than giving a
compressed stream which has invalid data at the end.)

Jon
Marc Gravell - 30 Aug 2007 12:22 GMT
Yowser! Buffer bomb!

You shouldn't be allocating a buffer for the entire file (which could
be huge), but rather just create a small buffer and loop over the
data - something like (untested):

static void Main() {
       using(Stream inFile = File.OpenRead("in.gzip"))
       using(GZipStream zip = new GZipStream(inFile,
CompressionMode.Decompress))
       using (Stream outFile = File.OpenWrite("out.whatever")) {
           CopyStream(zip, outFile);
           outFile.Close();
       }

   }
   static long CopyStream(Stream source, Stream destination) {
       if (source == null) throw new ArgumentNullException("source");
       if (!source.CanRead) throw new ArgumentException("Cannot read
from source");
       if (destination == null) throw new
ArgumentNullException("destination");
       if (!destination.CanWrite) throw new ArgumentException("Cannot
write to destination");
       const int BUFFER_SIZE = 4096;
       byte[] buffer = new byte[BUFFER_SIZE];
       int bytesRead;
       long totalBytesRead = 0;
       while ((bytesRead = source.Read(buffer, 0, BUFFER_SIZE)) > 0)
{
           destination.Write(buffer, 0, bytesRead);
           totalBytesRead += bytesRead;
       }
       destination.Flush();
       return totalBytesRead;
   }

Marc
BDRichardson - 30 Aug 2007 17:14 GMT
KABOOOOM!

OK, shoot me now!

I'm sure its not that obvious that I'm new to Streams... yeah right!  
Although I didn't & don't totally understand streams, I do recall when I
first started carving the code, I suspected that it might be dumping the
whole buffer at once.  Obvious really when you take a careful look at the
code!

I've altered it slightly so that it now only reads and writes a small buffer
at a time, and it works a treat.

Many thanks guys, you've given me the slap I needed!
Marc Gravell - 30 Aug 2007 20:48 GMT
> [being new to streams]
Please rest assured that I also learnt this the hard way. No doubt Jon
or one of the other stalwarts set me on the right path - sorry if it
sounded condascending, it wasn't my intent - just to stress that a
radical change of direction was needed ;-p

> Many thanks guys, you've given me the slap I needed!
No problem

Marc

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.