Project:Infrastructure/Incident reports/2018-06-28 Github - Gentoo wiki (original) (raw)

Incident metadata

• Status: Resolved
• Incident Commander: Robbat2
• Incident Backups: MGorny, Antarus
• Communications: Antarus, Dilfridge,
• Private Incident Page: (for infra only): https://infrawiki.gentoo.org/github-2018-06-28

Incident summary

An unknown entity gained control of an admin account for the Gentoo GitHub Organization and removed all access to the organization (and its repositories) from Gentoo developers. They then proceeded to make various changes to content. Gentoo Developers & Infrastructure escalated to GitHub support and the Gentoo Organization was frozen by GitHub staff. Gentoo has regained control of the Gentoo GitHub Organization and has reverted the bad commits and defaced content.

Impact

• Approximately 5 days of GitHub being unavailable for Gentoo use.
• Pull request CI was down; only master was being tested for issues.
• The Gentoo Proxy Maintainers Project was impacted as many proxy-maint contributors use GitHub to submit PRs.
• All past pull requests were apparently disconnected from their original commits and closed. GitHub can't fix it for us, so users have to open new pull requests.
• The entity attempted to wipe user content by adding "rm -rf" to various repositories; however this code was unlikely to be executed by end users due to various technical guards in place.
• Development not related to proxy-maint continued as normal; over 700 commits were made during the incident.

Malicious content available

Initial clones of these repositories during these time intervals will have malicious content. Gentoo recommends recreating these from a new clone if you cloned during this period.

• gentoo/gentoo: (2018-06-28 20:38 - 2018-06-29 06:58)
• gentoo/musl: (2018-06-28 20:56 - 2018-06-29 06:59)
• gentoo/systemd: (2018-06-28 21:07 - 2018-06-29 06:57)

Root cause

The attacker gained access to a password of an organization administrator. Evidence collected suggests a password scheme where disclosure on one site made it easy to guess passwords for unrelated webpages.

Background

Gentoo's use of GitHub

The main Gentoo repositories are kept on Gentoo hosted infrastructure and Gentoo mirrors to GitHub in order to "be where the contributors are." We do not believe the private keys of the account impacted were at risk, and so the gentoo-hosted infrastructure was not impacted by this incident.

Lessons

What went well

• Gentoo responded quickly to reports of problems.
• GitHub responded quickly and had a function to hide the impacted organization.
• Gentoo quickly removed account access once the entry point was located.
• GitHub provided audit logs that helped map out the incident.

What went badly

• Initial communications were unclear and lacking detail in two areas.
  • How can users verify their tree to be sure they had a clean copy?
  • Clearer guidelines that even if users got bad copies of data with malicious commits, that the malicious commits would not execute.
• Communications had three avenues (www.gentoo.org, infra-status.gentoo.org, and email lists.) Later we added a wiki page (this page) and were inconsistent on where to get updates.
• GitHub failed to block access to the repositories via git, resulting in the malicious commits being externally accessible. Gentoo had to force-push over them as soon as this was discovered that.
• Credential revocation procedures were incomplete.
• We did not have a backup copy of the Gentoo GitHub Organization detail.
• The systemd repo is not mirrored from Gentoo, but is stored directly on GitHub.

Lucky items

• Numerous Gentoo Developers have personal contacts at GitHub, and in the security industry and these contacts proved valuable throughout the incident response.
• The attack was loud; removing all developers caused everyone to get emailed. Given the credential taken, its likely a quieter attack would have provided a longer opportunity window.
• The method by which the attackers pushed commits (force pushing their commits) made downstream consumption more conspicuous; this would have blocked git from silently pulling in new content to existing checkouts on 'git pull'.

Action items

• action-item: make frequent offline backup of GH settings
• action-item: stream GH audit log into gentoo infra
  • https://developer.github.com/v3/activity/events/types/#organizationevent here as well.
• action-item: review 2FA requirements for GitHub org
  • Done: Gentoo GitHub Organization currently requires 2FA to join.
• action-item: reduce number of people with GitHub owner power
  • https://www.terraform.io/docs/providers/github/r/team_repository.html: Consider complete automation of org, from git with terraform as actuation with service account.
• action-item: be more proactive in retiring inactive people from infra
  • In-Progress: Bugs open to retire inactive infra members.
• action-item: see if there is a way we can receive email for everybody as they are added to the org
  • https://developer.github.com/v3/activity/events/types/#organizationevent; a hook is under development to keep an off-GitHub log.
• action-item: validate infra member retirement procedures separately from undertaker procedures.
  • Note: During the access revocation, some hosts were missed.
• action-item: prod gentoo-infra members to start using local password managers (pass, gopass, etc.)
  • In-Progress: Draft guidelines are being reviewed.
• action-item: Apply 2fa protection to gentoo services generally for all users.
• action-item: Document an incident plan for communications.
  • Done: Incident plan is in place.
• action-item: sponsor potential for hardware based 2FA for Gentoo's devs? (C.f. nitrokey / Linux Foundation)
  • Done: Gentoo Foundation partnered with NitroKey to equip all Gentoo Developers with USB keys.
• action-item: Publish clear password policy for the org, including recommendations for password management.
  • In-Progress: Draft guidelines are being reviewed.
• action-item: mirror systemd and musl repos on git.gentoo.org
  • In-Progress
• action-item: Audit gentoo system logins for 90d to verify no unexpected activity.
• action-item: Audit logs for compromised account.
  • This item is complete.
• action-item: Rotate credentials for compromised account.
  • This item is complete.

Timeline

These times are in UTC and are compiled from IRC and activity logs.

2018-04-08 - 2018-06-27

Logs indicate that various GitHub accounts were probed looking for vulnerable accounts.

2018-06-28

• 20:05 2nd to last known legimate commit to gentoo/gentoo. Matches git.gentoo.org/repo/gentoo.git
  • Auto-pushed by mirror bot.
  • Commit ID 38281f4252f89e3ef9cbae54dfc1ad553d296979
• 20:08 Last known legimate commit to gentoo/musl. matches git.gentoo.org/proj/musl.git.
  • Commit ID 60461ca1385809bacf6a114a7f1ecfe22f6da47f
• 20:19 Attacker tries a bad password on the account.
• 20:19 Attacker successfully gains administrative access
• 20:25 Attacker invites a dummy account to the org
• 20:25 Attacker creates a dummy account with administrative access.
• 20:25 Last known legimate commit to gentoo/gentoo. Matches git.gentoo.org/repo/gentoo.git
  • Auto-pushed by mirror bot.
  • Commit ID 73b724093b9c2a8756b8c35d3e09793342fa9ca9
  • Does NOT appear in the GitHub audit log for the org.
• 20:25 Attacker starts removing valid users
• 20:26 Earliest email timestamp of someone being removed from the organization.
• 20:29 First person notices that something is going on with the GitHub organization
• 20:30 Attacker invites a second malicious user.
• 20:32 Attacker adds second malicious user with admin privileges.
• 20:34 Malicious commit to gentoo/gentoo, 73b72409->fdd8da2e
  • adds readme.md file with racist text.
• 20:36 First report to Infra that something is going on with the GitHub organization.
• 20:38 Malicious commit to gentoo/gentoo, fdd8da2e->49464b73.
  • adds rm -rf /*& at the top of skel.ebuild
• 20:39 Attacker changes billing email, the first time.
• 20:45 Malicious commit 49464b73 is first noticed
• 20:48 Attacker changes billing email, the second time
• 20:49 First abuse report to GitHub support
• 20:50 Malicious commit to gentoo/gentoo, 49464b73->afcdc03b.
  • adds rm -rf /* at the top of every ebuild.
• 20:51 Infra's informal contact to GitHub via multiple personal channels
• 20:53 Second abuse report to GitHub
• 20:55 Malicious commit to gentoo/gentoo, _afcdc03b_->e6db0eb4, force-push.
  • Squash of entire history as of afcdc03b (rm -rf /* in ebuilds)
• 20:56 Malicious commit to gentoo/musl, _60461ca1_->e6db0eb4. Force-push.
  • Same history as gentoo/gentoo in a squashed commit.
• 21:00 (approx) GitHub informal report that they are starting to look
• 21:05 Infra's formal ticket to GitHub Support
• 21:07 Malicious commit to gentoo/systemd, _bf0e0a4d_->50e3544d.
  • Payload: slightly obfuscated rm -rf $HOME ~/ at the top of the configure script.
• 21:11 Malicious commit to gentoo/systemd, _50e3544d_->c46d8bbf. Force-push.
  • Revert of previous commit bf0e0a4d squashed with commit 50e3544d.
• 21:28 GitHub support responds; Gentoo GitHub org frozen.
• 22:14 Gentoo emails GitHub requesting activity logs.
• 22:45 GitHub locks suspected entry point
  • GitHub does not disclose this to Gentoo, it's found in an audit log of the compromised user's account on 2018-06-29T14:30:18Z
• 22:47 GitHub responds, assuring Gentoo that the audit is ongoing and logs will be produced soon.
• 23:35 GitHub provides limited access to the org to Gentoo.
• 23:40 Gentoo determines which account was the entry point. Gentoo Infra preemptively removes all access for that account from primary Gentoo properties (git repos, bugs, email, etc.)
• 23:47 GitHub formally responds with audit logs and security recommendations (e.g. 2FA)

2018-06-29

• 00:00 Gentoo reviews activity of compromised account to see if it was used on other services.
• 00:05 Gentoo emails GitHub, requesting the org be hidden again while Gentoo affects repairs. Of particular concern are the PRs and how we can audit them.
• 00:25 GitHub responds saying they have re-hidden the org, and will wait for confirmation before un-hiding it again, but are concerned about proper operation for organization members
• 01:20 Gentoo responds to GitHub saying keeping it hidden during the investigation is preferable to keeping it operable (but potentially unreliable / malicious.
• 01:38 GitHub and Gentoo discuss various cleanup strategies over email.
• 02:33 Formal request from Gentoo to GitHub for an audit log for the affected compromised account (with explicit consent from the compromised user.)
• 05:27 Gentoo Infra restores billing email.
• 06:39 Gentoo emails GitHub request incident commander handoff (robbat2 => mgorny) and we trade contact information.
• 06:57 Gentoo Infra does force-push on gentoo/systemd to restore state. c46d8bbf->bf0e0a4d.
• 06:58 Gentoo Infra does force-push on gentoo/gentoo to restore state. e6db0eb4->73b72409.
  • Push takes several minutes due to size.
• 06:59 Gentoo Infra does force-push on gentoo/musl to restore state. e6db0eb4->60461ca1.
• 13:05 GitHub unlocks the compromised account and resets the password.
• 14:14 Compromised account holder regains account access, produces a security log from that account.
• 14:29 Gentoo does another incident handoff (mgorny => robbat2) and Gentoo requests an update on the remediation plan for open PR requests.
• 20:07 GitHub responds with a statement that they are still working on a remediation plan.
• 20:46 Gentoo emails GitHub asking for an ETA for remediation.
• 23:06 GitHub responds with an ETA for the remediation. Internal discussion ensues regarding whether to wait, or try to unlock the org over the weekend.

2018-06-30

• 01:47 Attacker probes compromised account to see if the stolen credential still functions; but this attempt fails.
• 17:34 Gentoo emails GitHub asking for clarification on remediation actions and what security logs Gentoo thinks are still required.

2018-07-01

• 07:41 Gentoo requests status update from GitHub.

2018-07-03

• 03:51 GitHub responds with the result of their investigation and described remediation actions they took on their side.
• 10:10 Gentoo responds to GitHub and asks that the organization be made public so Gentoo can conclude repairs.
• 11:46 GitHub responds and unlocks the Gentoo GitHub Organization, making it publicly visible once again,

Appendix

Known malicious content

The following commits were known to be introduced by the unknown entities. They were only present on the GitHub-hosted repositories, and never present on the Gentoo-hosted master repositories.

• gentoo/gentoo, master branch:
  • e6db0eb4 (force-push)
  • afcdc03b
  • 49464b73
  • fdd8da2e
• gentoo/musl, master branch:
  • e6db0eb4 (force-push)
• gentoo/systemd
  • c46d8bbf (force-push)
  • 50e3544d

Once Gentoo regained access to the GitHub repositories, we forced-pushed over these malicious repos.