Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy operations are (relatively) slow #1685

Open
valoq opened this issue Apr 9, 2024 · 1 comment
Open

Copy operations are (relatively) slow #1685

valoq opened this issue Apr 9, 2024 · 1 comment

Comments

@valoq
Copy link
Contributor

valoq commented Apr 9, 2024

It takes my setup about 1 second to copy a 1GB file using "cp"
But the same operation takes about 5 seconds in lf.

After digging through the code I am still unsure what causes this issue.

@MahouShoujoMivutilde
Copy link

Lf copies a file in relatively small chunks of fixed size (4KB)

lf/copy.go

Lines 51 to 86 in ce12116

buf := make([]byte, 4096)
r, err := os.Open(src)
if err != nil {
return err
}
defer r.Close()
w, err := os.OpenFile(dst, os.O_RDWR|os.O_CREATE|os.O_TRUNC, dst_mode)
if err != nil {
return err
}
for {
n, err := r.Read(buf)
if err != nil && err != io.EOF {
w.Close()
os.Remove(dst)
return err
}
if n == 0 {
break
}
if _, err := w.Write(buf[:n]); err != nil {
return err
}
nums <- int64(n)
}
if err := w.Close(); err != nil {
os.Remove(dst)
return err
}

which could be suboptimal.

In an ideal world it would use io.Copy() (which could benefit from copy_file_range() to speed things up, allowing e.g. CoW fs reflinks, samba server side file copy, etc) instead of a custom implementation (which exists for a reason ⬇️).

However, it's also nice to have a % progress of an ongoing copy, which io.Copy() doesn't provide an easy way* to implement while keeping full scope of its benefits.

Increasing it to 4MB decreases the copy duration by about 30%.

* - at least I haven't found any in 2021.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants