Comment 1 for bug 1500442

In the initial failure log we see that the rsync has failed prematurely and repeatedly. As a result, the charm is attempting to clean up the local metadata directory which had been used. When attempting to delete the entire destination metadata tree there is a permission failure which raises an exception. I have not recreated the exact failure as I am not recreating the internal errors that saw rsync exit early, however my tests have created a failure which matches this behavior.

I ran with a locally modified charm that would skip the rsync and fail the mirror_base() function. I that case cleanup failed as we see from the initial environment. Looking at the environment (and code) we see that the destination directory has permissions 0555 when it is create and after a successful rsync they are 0775. I appears that the early exit from rsync leaves the parent directory without write permissions this is causing the clean_metadata_dir function to fail. The fix is to create the directory with user and group write permissions (0o775 rather than 0o555 as we have done).

Summary:
Initial metadata directory permissions blocked clean on failure (LP: #1500442)

  If a unit has an rsync problem, the clean_metadata_dir() call can fail
  with a permission denied error for the metadata directory (or one of its
  entries).

  Successful 'rsync' execution has a side-effect of changing the metadata
  directory permissions from 0555 to 0775. Without rsync success the
  directory can not be removed with shutils.rmtree() due to the missing
  write permissions on the parent. The initial permissions during directory
  creation have been changed to 0775 so that cleanup can proceed even if
  rsync has failed.