[Mondo-devel] Backup speed

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Mondo-devel] Backup speed

David C. Partridge
I'm running a full backup now and the thing that's most noticeable is that
the tape streamer isn't streaming even with the block size of 64K.

This could be just that the processor (Intel Atom D525) is relatively slow,
but I don't think that's the problem as CPU% is fairly  low.

I think this may be because the afioballs are relatively small files.  Is
there a (relatively easy) way to change this.

I think that the large file part of the backup does manage to stream OK
(from memory of my earlier testing).

Cheers
Dave


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] Backup speed

Bruno Cornec-4
Hello David,

David C. Partridge said on Thu, Dec 22, 2016 at 12:46:02PM +0000:
>I'm running a full backup now and the thing that's most noticeable is that
>the tape streamer isn't streaming even with the block size of 64K.

Maybe the buffer process in the middle is not giving enough to eat to your tape drive.

>I think this may be because the afioballs are relatively small files.  Is
>there a (relatively easy) way to change this.

Not at the moment.
In the code the variable bkpinfo->optimal_set_size is the max value for an afio tar ball (by reading the code, I've not xritten that part).

And it's a fixed value:
bkpinfo->optimal_set_size = (IS_THIS_A_STREAMING_BACKUP(bkpinfo->backup_media_type) ? 16 : 16) * 1024L;
So 16 MB IIUC.

So whether we use a tape or not, it's the same atm. Maybe we need to change that no ?
I could add a flag to be able to change it e.g, and we could also keep it fixed as of now, but with a different default for tapes.

>I think that the large file part of the backup does manage to stream OK
>(from memory of my earlier testing).

There the -M parameter to change the size of what a Big file is, so you could try lowering that value (default 64MB).
Maybe creating more big files could help in your case, even if they are smaller.

Bruno.
--
Open Source Profession, WW Linux Community Lead     http://www.hpintelco.net
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] Backup speed

David C. Partridge
Bruno.

Let me tinker with that bit of the code trying a different
"optimal_set_size" for streaming backups (does any device other than tape do
streaming?).

Certainly the larger afioballs got a lot closer to getting the tape
streaming properly than the smaller ones.

I'll let you know the results.

> Maybe the buffer process in the middle is not giving enough to eat to your
tape drive.

Hmmm how on earth could I check that!??

Dave


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] Backup speed

Bruno Cornec-4
David C. Partridge said on Fri, Dec 23, 2016 at 01:59:34PM +0000:
>Let me tinker with that bit of the code trying a different
>"optimal_set_size" for streaming backups (does any device other than tape do
>streaming?).

 I thnik that some DVD/CD can support a straming approach as well to be written.
 Honestly I don't really care, as I think most people use DVD/CD with xorriso/genisoimage/mkisofs or USB with copy, or tapes with streaming through buffer.
 So go for your tests, if someone has something to say here, it should be now !

>Certainly the larger afioballs got a lot closer to getting the tape
>streaming properly than the smaller ones.
>I'll let you know the results.

Ok thanks a lot !

>> Maybe the buffer process in the middle is not giving enough to eat to your
>tape drive.
>
>Hmmm how on earth could I check that!??

You could try to increase the buffer used by buffer maybe by using -b 32 MB e.g. to see the impact. and varying that number.
It will have a direct impact on the buffer command behaviour normally.

Just an idea, feel free to experiment and come back with questions/points.

bruno.
--
Open Source Profession, WW Linux Community Lead     http://www.hpintelco.net
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] Backup speed

David C. Partridge
Well I tried 64MB instead of 16MB for optimal_set_size but it made little
difference.   I'll put my thinking cap on and ponder this.

Dave


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel