Skip to content

CopyDir issues four stat calls and two utimensat calls per file #1048

Description

@mzihlmann

CopyDir copies a directory tree by walking it, then copying each entry. Per regular file it issues four stat calls and two utimensat calls where one of each would do.

CopyDir already has the entry's os.Lstat result in hand. CopyFile then calls os.Stat on the same source again. CreateFile calls FilepathExists on a destination it is about to os.Create. CopyTimestamps calls os.Lstat on the source a third time.

Timestamps are applied twice to every regular file. CopyFile calls CopyTimestamps, and CopyDir records the same entry in updates and replays CopyTimestamps over all of them after the loop. The replay is needed for directories, whose mtime moves when children are written into them. Regular files do not need it.

Measured on a copy of this repo's vendor/ tree, 4861 files and 1217 directories, 55.8 MB, no hardlinks and no symlinks, against otiai10/copy doing the same work on the same tree:

                     kaniko   otiai10    delta
newfstatat            35252     14590   +20662
utimensat             10939      6078    +4861
lgetxattr              4861         0    +4861
fchmodat               4861      6078    -1217
fcntl                 39398     44266    -4868
openat, close, copy_file_range, mkdirat, fchownat, getdents64   identical

Bulk data movement is identical, 9718 copy_file_range on both sides. The whole difference is metadata. Wall clock on this host is 0.603s against 0.440s, mean of six runs with the order alternated, so 37% slower on a tree where kaniko's extra work buys nothing.

The lgetxattr column is CopyCapabilities, which is a feature otiai10/copy does not have. The other rows are not.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceBuild speed or resource usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions