[SEC] potential security issue related to GitHub CI servers?

[fkiraly]

There have been multiple strange occurrences around GitHub CI servers that may be security related:

• sporadically, logs showing unexpected package checksums, e.g., here: [ENH] Rework of base series annotator API #6265 (comment)

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    unknown package:
        Expected sha256 55b380cff92d493e465b23e96a4bce8eec1a7b02db92c7718bb0d883f244c2b9
             Got        f8a96f8553d80288955fd9fa54bb4f9e6aec5df2028074a8cc65a11510af611a

These also do not always seem to be the same packages, or same sha.

• internet security flagging seemingly trustworthy microsoft URLs with GitHub logs as compromised by malware: [MNT] GitHub log server on AVG internet security blocklist skpro#304

I also wonder whether the two are correlated.

What is going on here? I really hope Microsoft don't have hackers in there...

[fkiraly]

GitHub support reply on the second issue (blocklist)

Hi Franz,

Thanks for that (and apologies about the upload.) I can see the screenshots via that issue you created.

The windows.net domain points to an Azure blob storage resource. On the surface this shouldn't indicate anything suspicious, but if the resource is ephemeral and/or not recognised as a reputable domain then I can see why AVG might be inclined to flag it.

In any case, thanks again for bringing this up. I'll raise this with the rest of the Actions team and close this out, but feel free to reopen if you have any other questions.

No clear "next step" or instructions how to resolve were given so far.

[achieveordie]

Related: pypa/pip#12680

It doesn't seem like something we should be worried about.

[achieveordie]

On second thoughts, we should keep an eye out, just in case (I'm looking at you xz)

[fkiraly]

Related: pypa/pip#12680

It doesn't seem like something we should be worried about.

I think our problem is not an instance of this - the hashes in the error logs are very different, not just lower/uppercase variants.

[fkiraly]

Reply from GitHub support:

Please rest assured that this has not been a widespread issue impacting GitHub services or resources. Nevertheless, while this has not been a widespread issue, we have communicated this with AVG directly. But with respect to next steps regarding the blacklisted log URL, the only other thing we can advise right now is that you send your feedback to AVG to notify them of the false positive too.

And on the topic of the more recent issue regarding the package SHAs:

This doesn't appear to be related to AVG's blacklisting; instead, it's likely related to some kind of build, deployment, or CICD workflow process expecting one version of a dependency that's pinned to a particular commit, but receiving a different one.

Do you happen to be using a requirements.txt file (or something similar)?

The reason I ask is because this error is typically seen when using Python's pip package manager with a requirements file that includes hashes for each package. The hash is a way to verify the integrity of the downloaded package, ensuring it has not been tampered with. The error message is suggests that the hash of the downloaded package does not match the expected hash listed in the requirements file. This could be due to:

• The package version has been updated, and the new version has a different hash. In this case, you should update the hashes in your requirements file.
• The downloaded package has been tampered with or corrupted during download. In this case, you should delete the package and try downloading it again.

To update the hashes in your requirements file, you can use the pip freeze command to get the current versions and hashes of your installed packages, and then update your requirements file with this information. But please also note that this will overwrite your current requirements file, including any packages you've installed that are not in your requirements file.

[fkiraly]

My analysis of this is:

• they consider the AVG blocklisting as a false positive, although no reasoning is given here. I wonder if this is jumping to conclusions, or if there is an argument they are not sharing.
• regarding package SHA, we are not using requirements.txt or anything similar that specifies SHA explicitly, afaik. This should be clear from inspecting this GitHub repository? Or is there sth that I am missing?
• in particular, we are using pyproject with ranges rather than pins, and GHA VMs are set up dynamically. The advice seems to be geared at a "end user project" rather than a package that one would expect to be consumed by end users?

I will follow up.

[fkiraly]

My reply to GH support:

Thanks for your update.

• regarding AVG blacklisting, can you confirm that you consider this a false positive? If so, can you explain how you know that it is a false positive, i.e., not actually flagging malicious behaviour?

• regarding requirements: we do not pin SHA or similar, sktime's dependencies are specified by version ranges in the pyproject.toml, the entire repository is public and you can inspect it: https://github.com/sktime/sktime

• as a consequence of this, I would conclude that your first suspicion does not apply, since we do not pin SHAs. Hence, by exclusion, your second one becomes more likely, i.e., corruption or tampering. It has become very frequent now, over hte last months, so I think it warrants a proper investigation?

[fkiraly]

Response by GitHub support, indicates the log access is legit false positive.
No comment on the SHA issue.

Hi Franz,

Thanks for that. So we have concluded our investigation and I have the following updates for you.

For the blacklisted URLs you brought to our attention:

• https://productionresultssa12.blob.core.windows.net/actions-results/a2211f51-1ffb-4a77-b0d1-4947a7771142/workflow-job-run-c60e60fd-a3d3-53b7-c41f-646dcc3872e7/logs/steps/step-logs-074b3269-d3b7-5acf-0eef-735521b5e2da.txt?rsct=text%2Fplain&se=2024-05-08T15%3A2
• https://productionresultssa8.blob.core.windows.net/actions-results/49b504b2-c8e2-4425-9528-81ddc6322d03/workflow-job-run-7aac5dea-4f7a-5f9c-4c64-d86e3469d277/logs/steps/step-logs-3a1f0620-1485-5dd2-a4b0-c64e6c11e846.txt?rsct=text%2Fplain&se=2024-05-06Т12%3A14

These were false positive flags by AVG, and everything in these flagged URLs matches the data we have in the backend. The URLs are generated from Azure Blob storage and we have no evidence of a security breach. Even if this was a man-in-the-middle attack, we would have TLS failures making it evident. In short, all the details of the URL match the backend data we have and there's no indication of foul play here.

We have brought this up with our counterparts at AVG; however, as I'm sure you can understand, we cannot share anything about the nature of the communication, or give any guarantees about what actions might be taken from their side.

We also reviewed the workflow runs themselves and found no tampering or issues with them. If and where possible, I advise contacting AVG to mark this as a false positive, as we have no control over what AVG says is a threat. It's likely these URLs are hitting some common rule that their software uses by default.

In addition to the URLs above, we can confirm that we don't have any indication that there are any security threats from accessing any of our blob storage URLs:

     "productionresultssa0.blob.core.windows.net",
      "productionresultssa1.blob.core.windows.net",
      "productionresultssa2.blob.core.windows.net",
      "productionresultssa3.blob.core.windows.net",
      "productionresultssa4.blob.core.windows.net",
      "productionresultssa5.blob.core.windows.net",
      "productionresultssa6.blob.core.windows.net",
      "productionresultssa7.blob.core.windows.net",
      "productionresultssa8.blob.core.windows.net",
      "productionresultssa9.blob.core.windows.net",
      "productionresultssa10.blob.core.windows.net",
      "productionresultssa11.blob.core.windows.net",
      "productionresultssa12.blob.core.windows.net",
      "productionresultssa13.blob.core.windows.net",
      "productionresultssa14.blob.core.windows.net",
      "productionresultssa15.blob.core.windows.net",
      "productionresultssa16.blob.core.windows.net",
      "productionresultssa17.blob.core.windows.net",
      "productionresultssa18.blob.core.windows.net",
      "productionresultssa19.blob.core.windows.net",

We want to reinforce that these are all owned and managed by GitHub Actions and there are no indications of security threats. For a full list of URLs and IPs, see: https://api.github.com/meta.

Thanks again for reporting this and for your concern about the security of GitHub's products.

Kind regards,

Dan | GitHub Support Engineer

[fkiraly]

New reply from GitHub support:

Hi Franz,

Yes - sorry about that. So this is tricky, as it's not directly related to an issue with GitHub Actions itself, so may be pushing outside of the scope of what we can help you with. Nevertheless, I'll do my best to help.

We haven't seen anything that suggests this is a security concern. The "THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE" error you're encountering is relates to pip's hash-checking mode, and one reason for this is indeed to ensure that the packages haven't been tampered with, but it could also simply be related to a network issue during the download of your packages. In short, this error gets thrown when the hash of the downloaded package doesn't match the projects requirements, and this could be for a number of reasons.

I appreciate we've touched on it before, and you may have your reasons for this, but I can see that there is no requirements.txt file in sktime/sktime. This may be related and could prevent the error, even if it only comes up intermittently, as the file would let you specify exactly which dependencies and which versions are needed for your project.

Other reasons for this error could be:

Network issues: There might be problems with your network that are causing the downloaded file to be incomplete or corrupted, leading to a different - or missing - hash.
Cache issues: If you're using a cache for your dependencies, it might be serving a corrupted or outdated version of the package.
Package changes: This doesn't necessarily mean corruption or tampering, but instead the package maintainers might simply have re-uploaded the package and/or changed a version number, thereby changing the hash.

Here are a few general steps that might help with troubleshooting.

Clear your pip cache: Pip caches downloaded packages to speed up future installations. Clearing this cache can help if it's serving a corrupted version of the package.
Check your network: If you're on a flaky network connection, this problem will be more common.
Contact the package maintainers: If possible, contact the maintainers to confirm whether any changes were made.
Disable hash-checking: As a last resort, you can disable hash-checking; however, this might make your installations less secure, so it's not recommended unless you're sure it's safe to do so.

Something else that may help: We also have a setup-python Action that can be used for caching dependencies in Python projects. The action creates and restores a cache identified by a unique key, and can also speed up your workflows.

Hope that helps!

Kind regards,

Dan | GitHub Support Engineer

[fkiraly]

Analysis of the response:

• suggests is not malicious, but without giving any reasoning why it's not (e.g., what they checked). "nothing we've seen suggests" is not a very concrete statement if there's no clear mention of checks done or diagnostics. But perhaps they did their checks and do not want to disclose anything, which would also be strange.
• suggestion that lack of requirements.txt causes the issue, and that sktime should have requirements.txt. That also feels odd, sktime does have pyproject.toml, and it would be unusual for packages that other people use to have requirements.txt and not pyproject.toml, or both, compare, e.g., numpy or sklearn
• suggestion to enable pip cache. Maybe this helps, though I also don't understand the reasoning why it would. We can try it of course, perhaps it also helps speeding up the tests: [MNT] add pip cache to setup-python actions #6513
• suggestions for "other reasons" are given but none of them seem plausible:
  • "network issues", "check your network" - we do not control or maintain the network, this is GitHub's own CI.
  • "cache issues" - we are not using a package cache currently, afaik.
  • "package changes" - we have checked this, this does not match pypi release history, so can be ruled out as a reason. No need to "contact the package maintainers", because we are installing from pypi, and know there were no changes.