[Discuss] large log file transfer

[jack at coats.org (Jack Coats)]

Why split it?  It depends on how big big is.  Even compressed by gzip -9 if
the file is 'too large', splitting it in half (or more parts) to get the
size of each compressed portion below the threshold of data transfer 'pain'
can make it possible to transfer large files over 'limited' links or with
'limited size media'.  I first ran into split back when using 128K floppy
diskettes to move files.  The concept is still the same, just the
definition of 'large' has changed.

On Tue, Jun 23, 2015 at 10:17 AM, Mike Small <smallm at panix.com> wrote:

> John Malloy <jomalloy at gmail.com> writes:
>
> > I have a log file (325MB)  that I need to transfer from a restricted
> > network, that I cannot plug a USB into.
> >
> > Is there an easy way to "split up" the log file into smaller chunks and
> Zip
> > it to get it over the net?
>
> So Jerry and Jack have pointed you to split, but why do you want to
> split it into chunks?
>
> I must admit I've not dealt with any networks lately that I couldn't
> reliably download a 300 MB file on, but is it not still possible to
> resume interrupted transfers?  Are you using vanilla ftp, http or scp?
>
> If you're using scp, I'd be inclined not to zip it but to use ssh's
> compression so when it finally does arrive I wouldn't have to unzip and
> then again find something else to do while that happens. But I guess scp
> doesn't support resume (hmmm, maybe ssh dd and adjust the offset based
> on the file size after the failure? -- perhaps that's more trouble than
> splitting).
>
> By "restricted network" do you mean that rsync is not an option?
>
> --
> Mike Small
> smallm at panix.com
> _______________________________________________
> Discuss mailing list
> Discuss at blu.org
> http://lists.blu.org/mailman/listinfo/discuss
>



-- 
><> ... Jack

The Four Boxes of Liberty - "There are four boxes to be used in the defense
of liberty: soap, ballot, jury and ammo. Please use in that order."
"Whatever you do, work at it with all your heart"... Colossians 3:23
"Anyone who has never made a mistake, has never tried anything new." -
Albert Einstein
"You don't manage people; you manage things. You lead people." - Admiral
Grace Hopper, USN
"Tell me and I forget. Teach me and I remember. Involve me and I learn." -
Ben Franklin