Re: .compress()

John Roberts wrote:
> ..compress() looks like it would be useful to make a backup/zip file with -

> seeing my current SB routine fails with Vista. Compress and append files 
> then write a header with offsets. Anything that isn't obvious that I should

> be aware of when using .compress()?

I would suggest creating a format that uses a clear chunk size with a
header that describes the chunk size. The .compress1() function is not
ultra fast during compression, but the .decompress1() is extremely fast
at decompression. It may be worth experimenting with to find the best
chunk size. Also, .compress1() is more effective with text files than
other file types, though it should still give reasonably good
performance (size/speed-wise). I would probably design it with a header
that contains the list of files (probably itself compressed), plus
metadata about the files (date/time stamp, maybe attributes). I would
make sure the decompress routine could restore the date/time stamp and
the attributes.

.compress() looks like it would be useful to make a backup/zip file with - 
seeing my current SB routine fails with Vista. Compress and append files 
then write a header with offsets. Anything that isn't obvious that I should 
be aware of when using .compress()?
John 

I have this same complaint about xp and all the rest: to work on a separate
drive, so that when Windows crashes and has to be totally re-installed (which is
now a built-in function on most of the computers I've bought lately), I don't
lose everything to the last backup, I have to build and maintain a separate
"desktop" folder for the icons and a separate "programs"
directory for the programs that don't force me into "Program Files".
Then, after the re-install, I have to fix the Registry again so that the
already-installed programs work.

Recently I did a "restore" to a previous date, and it re-set every
.ini file on all my drives to the previous URL's, email addresses, etc., etc.,
including Opera's mail account settings. Only then did I find I had to switch
off the drives I did not want messed up. All the programs I had installed in the
interval were REMOVED. That is, whatever files were newer than the "restore
date" were just gone.

I am so sick of Windows trying to obscure the system from its owner (me, not
Bill), I can't wait for the Linux stuff to work! I'm setting up yet another box
with wine, this time I hope it is going to accommodate my Windows/Superbase
stuff.

Vista? Me? NEVER!!!

> pant, pant<

Sorry, I had to get this off my chest...





On Mon, 17 Sep 2007 08:42:35 -0500, John Roberts <remjohnr@iinet.net.au>
wrote:

>
> "Neil Robinson" <neil@simpol.com> wrote in message
> news:fcdo92$nqj$1@ipx22096.ipxserver.de...
>> John Roberts wrote:
>>> ..compress() looks like it would be useful to make a backup/zip
file
>>> with -
>>> seeing my current SB routine fails with Vista. Compress and append
files
>>> then write a header with offsets. Anything that isn't obvious that
I
>>> should
>>> be aware of when using .compress()?
>>
>> I would suggest creating a format that uses a clear chunk size with a
>> header that describes the chunk size. The .compress1() function is not
>> ultra fast during compression, but the .decompress1() is extremely
fast
>> at decompression. It may be worth experimenting with to find the best
>> chunk size. Also, .compress1() is more effective with text files than
>> other file types, though it should still give reasonably good
>> performance (size/speed-wise). I would probably design it with a
header
>> that contains the list of files (probably itself compressed), plus
>> metadata about the files (date/time stamp, maybe attributes). I would
>> make sure the decompress routine could restore the date/time stamp and
>> the attributes.
>>
>> Ciao, Neil
>
> I don't need this to work as I have a work around but I have done it out
of
> interest.
>
> .compress1() works well with small files without breaking things up into
> chunks. I can compress and output many files into one output file and then
> decompress/retrieve them. I think there is a limit to how large a string
can
> be held in memory or added to another without causing an unexpected
program
> end. Using a temporary file on disk should sort that out.
>
>  .jpg and .pdf files come out larger than the original by no small amount
> (~200kb on top of a 550kb file) but that is to be expected given that they
> are already compressed. It is up to the programmer to determine which file
> types are problematic and what to do with these. Probably best to include
> them in the output file (if writing a single output file) as original
> uncompressed and not to be decompressed.
>
> Without breaking files up into chunks, large files are lengthy in time to
> compress. So much so as to be unusable for regular backup usage if there
are
> many. I haven't yet tried breaking them into chunks to see if that helps.
>
> When breaking files up into chunks, it will be easy to get a standard read
> size. However, each post compression chunk will be of varying size
depending
> on the content. This means storing not one but every individual chunk size
> in the header for later retrieval. For a large file this will mean many
> entries and a large header full of numbers that might not compress well.
> Only one way to find out I guess...
>
> What an unpleasant piece of work Vista is. Now we have "program
data" as
> well as "program files", the latter difficult to write to without
resetting
> permissions. Also "...\virtual store\program files\myprogram files or
> whatever". One minute you are looking at the real dir\files and the
next
> minute explorer has hijacked your view somewhere else.
> Cheers, John
>
>
>

"Neil Robinson" <neil@simpol.com> wrote in message 
news:fcdo92$nqj$1@ipx22096.ipxserver.de...
> John Roberts wrote:
>> ..compress() looks like it would be useful to make a backup/zip file 
>> with -
>> seeing my current SB routine fails with Vista. Compress and append
files
>> then write a header with offsets. Anything that isn't obvious that I 
>> should
>> be aware of when using .compress()?
>
> I would suggest creating a format that uses a clear chunk size with a
> header that describes the chunk size. The .compress1() function is not
> ultra fast during compression, but the .decompress1() is extremely fast
> at decompression. It may be worth experimenting with to find the best
> chunk size. Also, .compress1() is more effective with text files than
> other file types, though it should still give reasonably good
> performance (size/speed-wise). I would probably design it with a header
> that contains the list of files (probably itself compressed), plus
> metadata about the files (date/time stamp, maybe attributes). I would
> make sure the decompress routine could restore the date/time stamp and
> the attributes.
>
> Ciao, Neil

I don't need this to work as I have a work around but I have done it out of 
interest.

.compress1() works well with small files without breaking things up into 
chunks. I can compress and output many files into one output file and then 
decompress/retrieve them. I think there is a limit to how large a string can 
be held in memory or added to another without causing an unexpected program 
end. Using a temporary file on disk should sort that out.

 .jpg and .pdf files come out larger than the original by no small amount 
(~200kb on top of a 550kb file) but that is to be expected given that they 
are already compressed. It is up to the programmer to determine which file 
types are problematic and what to do with these. Probably best to include 
them in the output file (if writing a single output file) as original 
uncompressed and not to be decompressed.

Without breaking files up into chunks, large files are lengthy in time to 
compress. So much so as to be unusable for regular backup usage if there are 
many. I haven't yet tried breaking them into chunks to see if that helps.

When breaking files up into chunks, it will be easy to get a standard read 
size. However, each post compression chunk will be of varying size depending 
on the content. This means storing not one but every individual chunk size 
in the header for later retrieval. For a large file this will mean many 
entries and a large header full of numbers that might not compress well. 
Only one way to find out I guess...

What an unpleasant piece of work Vista is. Now we have "program data"
as 
well as "program files", the latter difficult to write to without
resetting 
permissions. Also "...\virtual store\program files\myprogram files or 
whatever". One minute you are looking at the real dir\files and the next 
minute explorer has hijacked your view somewhere else.
Cheers, John 

"John Roberts" <remjohnr@iinet.net.au> wrote in message 
news:fcm08p$i0o$1@ipx22096.ipxserver.de...
> I think there is a limit to how large a string can be held in memory or 
> added to another without causing an unexpected program end.

Crashes with the ide were due to the ide screen output buffer not being 
large enough to accomodate the list of compressed file names. 863 file 
path/names for an SB app datafile and Word doc backup were too much for it. 
ie not a string var overflow in simpol.

> Without breaking files up into chunks, large files are lengthy in time to 
> compress. So much so as to be unusable for regular backup usage if there 
> are many. I haven't yet tried breaking them into chunks to see if that 
> helps.

I have implemented this and it makes a very real difference in speed. The 
above backup dropped in time from about 12 mins to 3 using a 30000 char 
chunk. I have yet to test different chunk sizes for speed but will report on 
this when done. 863 files of approx 50 MB compressed to a 15 MB file. Winzip 
8 (last of the free unlimited trial versions) compresses the same file set 
to 8.7 MB in ~20 secs or less. 7-zip open source is much slower than winzip 
but is free. Both simpol fsfileinputstream and 7-zip will not open a file to 
read that is in use by SB whereas winzip will but reports potential 
corruption of that file in the zip file.
There is potential for good use here but it needs further testing.
John