Plan Preview frequently failed

[ffjlabo]

Plan Preview frequently failed like this↓

[ffjlabo]

It may happen when copying a repo cloned locally with cp while working git gc in the background.
git gc: https://git-scm.com/docs/git-gc

When some git subcommands (merge, rebase, commit...) are executed, git gc is also executed automatically in the background to optimize object files. https://www.atlassian.com/git/tutorials/git-gc
So gc.pid and gc.log.lock files are created on the process. (shown on the log above )
These files are deleted after finishing git gc

Piped prepare the local repo for executing Plan Preview like below

• fetches and merges the branch created for the PR

  • pipecd/pkg/app/piped/planpreview/builder.go

    Line 142 in 8e302a8

    repo, err := b.cloneHeadCommit(ctx, cmd.HeadBranch, cmd.HeadCommit)

  • pipecd/pkg/app/piped/planpreview/builder.go

    Lines 288 to 296 in 8e302a8

    	repo, err := b.gitClient.Clone(ctx, b.repoCfg.RepoID, remote, baseBranch, dir)
    	if err != nil {
    	return nil, fmt.Errorf("failed to clone git repository %s at branch %s", b.repoCfg.RepoID, baseBranch)
    	}
    	mergeCommitMessage := fmt.Sprintf("Plan-preview: merged %s commit from %s branch into %s base branch", headCommit, headBranch, baseBranch)
    	if err := repo.MergeRemoteBranch(ctx, headBranch, headCommit, mergeCommitMessage); err != nil {
    	return nil, fmt.Errorf("detected conflicts between commit %s at %s branch and the base branch %s (%w)", headCommit, headBranch, baseBranch, err)
    	}

• then copies the repo with cp locally

  • pipecd/pkg/app/piped/deploysource/deploysource.go

    Lines 145 to 149 in 8e302a8

    	// Clone the specified revision of the repository.
    	if err := p.cloner.Clone(ctx, repoDir); err != nil {
    	fmt.Fprintf(lw, "Unable to clone the %s commit (%v)\n", p.revisionName, err)
    	return nil, err
    	}

  • pipecd/pkg/app/piped/deploysource/sourcecloner.go

    Lines 91 to 94 in 8e302a8

    	func (d *localSourceCloner) Clone(ctx context.Context, dest string) error {
    	_, err := d.repo.Copy(dest)
    	return err
    	}

  • pipecd/pkg/git/repo.go

    Lines 81 to 97 in 8e302a8

    	// Copy does copying the repository to the given destination.
    	// NOTE: the given “dest” must be a path that doesn’t exist yet.
    	// If you don't, it copies the repo root itself to the given dest as a subdirectory.
    	func (r *repo) Copy(dest string) (Repo, error) {
    	cmd := exec.Command("cp", "-rf", r.dir, dest)
    	out, err := cmd.CombinedOutput()
    	if err != nil {
    	return nil, formatCommandError(err, out)
    	}
    	return &repo{
    	dir: dest,
    	gitPath: r.gitPath,
    	remote: r.remote,
    	clonedBranch: r.clonedBranch,
    	}, nil
    	}

So it happens when starting cp during git gc and finishing git gc during cp.

[ffjlabo]

The temporary solution is to execute git gc in the foreground.
To do so, we can execute the command on the environment (on the container, on the VMs).

git config --global gc.autoDetach false

ref: https://git-scm.com/docs/git-gc#Documentation/git-gc.txt-gcautoDetach

This enables the execute git gc synchronously after executing some commands which execute git gc automatically.

[ffjlabo]

Also, we should plan the essential solution. This problem may happen with each kind of application.
The point is to enable piped to execute git gc in the foreground.

There are some solutions.

1. allow users to use .gitconfig on piped
2. reflect git config gc.autoDetach false when cloning the source repo on piped

It might be good to choose the second one to minimize the scope of the effect.