[Bug][GitHub] GitExtractor fails on empty repository due to shallow clone

[gustavobini]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When syncing GitHub, a few repositories had errors as they were cloned with --bare --shallow-since=2024-03-09T13:24:51Z and the last commit on these repositories was over 3 months before the date.

What do you expect to happen

I expected a warning to be shown and not an error, as it's expected for the extractor to not do anything if there's no repository.

How to reproduce

Set the sync policy to only include the last day and sync a GitHub repository with changes before that date.

Anything else

time="2024-05-14 00:04:15" level=info msg=" [pipeline service] [pipeline #773] [task #60301] executing subtask Clone Git Repo"
time="2024-05-14 00:04:15" level=debug msg=" [pipeline service] [pipeline #773] [task #60301] [Clone Git Repo] [gitcli] git [clone https://git:<redacted>@github.com/org/repo.git /tmp/gitextractor1800277984 --bare --progress --shallow-since=2024-03-09T13:24:51Z]"
time="2024-05-14 00:04:16" level=error msg=" [pipeline service] [pipeline #773] [task #60301] [Clone Git Repo] [gitcli] git exited with error\nstderr: Cloning into bare repository '/tmp/gitextractor1800277984'...\nstderr: fatal: error processing shallow info: 4\n\n\tcaused by: exit status 128"
time="2024-05-14 00:04:16" level=info msg=" [pipeline service] [pipeline #773] [task #60301] finished step: 1 / 4"
time="2024-05-14 00:04:16" level=info msg=" [pipeline service] [pipeline #773] [task #60301] executing subtask Collect Commits"
time="2024-05-14 00:04:16" level=error msg=" [pipeline service] [pipeline #773] [task #60301] run task failed with panic\n\tcaused by: run task failed with panic (github.com/apache/incubator-devlake/plugins/gitextractor/tasks.getGitRepo:87)\n\tWraps: (2) git repo is empty, please check subtask: clone repo\n\tError types: (1) *hintdetail.withDetail (2) *errors.errorString"

Version

v1.0.0-beta7

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klesh]

Thanks for reporting the problem, will look into it ASAP

[gustavobini]

@klesh FYI the fix on #7475 causes this in some repos:

time="2024-05-16 11:57:30" level=debug msg=" [pipeline service] [pipeline #797] [task #61700] [Clone Git Repo] [gitcli] git [clone https://git:<token>@github.com/<org>/<repo>.git /tmp/gitextractor1830545665 --bare --progress --shallow-since=2024-05-16T00:02:53Z]"
time="2024-05-16 11:57:38" level=error msg=" [pipeline service] [pipeline #797] [task #61700] subtask Clone Git Repo ended unexpectedly\n\tWraps: (2) failed to deepen the cloned repo\n\tWraps: (3) exit status 128\n\tError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *exec.ExitError"
time="2024-05-16 11:57:38" level=error msg=" [pipeline service] [pipeline #797] [task #61700] error writing subtask Clone Git Repo status to DB\n\tcaused by: failed to encode args[0]: unable to encode 1 into binary format for bool (OID 16): cannot find encode plan (500)\n\tWraps: (2) failed to encode args[0]: unable to encode 1 into binary format for bool (OID 16): cannot find encode plan\n\tError types: (1) *hintdetail.withDetail (2) *errors.errorString"

Running with this image release-v1.0_240516_0721_3ae23bf (I was testing the Jenkins fix I did). I increased the time range but no luck.

[klesh]

@gustavobini Thanks for reporting the problem.

[klesh]

@gustavobini Can you share some of them if possible? I am having trouble to reproduce the problem.

[gustavobini]

@gustavobini Can you share some of them if possible? I am having trouble to reproduce the problem.

@klesh they're private org repos. Let me see what I can do.

[gustavobini]

@klesh I cloned the repo locally and executed the same commands Devlake does, and my issue is exactly the same as this one https://stackoverflow.com/questions/63878612/git-fatal-error-in-object-unshallow-sha-1 and also got resolved by git repack -d

[klesh]

@gustavobini Thanks for the information. Will submit another PR to fix the problem even though I do not fully understand why... 🤣