Not thinking it through
I was investigating the failure of a backup job , written and produced by another party, and just thought I'd share this.
The job backs up more than one database in a cursor loop, if the job fails it retries three times. So far this sounds a good solution?
Well not really, in this case of failure the database sizes were diverse, from a few hundred mb to tens of gb, the backup job failed beacuse there was insufficient disk space to backup the largest database.
What happens is that you then get multiple backups of the smaller databases until the four iterations complete or the smaller backups have filled the available disk space - in worst case the process would repeat until all available disk space was filled, as the backup drive also stored hourly transaction log backups for the databases the potential to bring the whole server to a stop was quite high.
So the point I'm making is that retry on fail might not actually be the best solution, it isn't in this case.