production SAL - SAL (original) (raw)

2026-05-15 §

21:03

<jforrester@deploy1003>

Finished scap sync-world: Backport for [[gerrit:1287940|Revert "Enable wgTrackMediaRequestProvenance on remaining Wikipedias" (T425580)]] (duration: 07m 43s)

[production]

20:59

<jforrester@deploy1003>

jforrester, seddon: Continuing with deployment

[production]

20:57

<jforrester@deploy1003>

jforrester, seddon: Backport for [[gerrit:1287940|Revert "Enable wgTrackMediaRequestProvenance on remaining Wikipedias" (T425580)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

[production]

20:55

<jforrester@deploy1003>

Started scap sync-world: Backport for [[gerrit:1287940|Revert "Enable wgTrackMediaRequestProvenance on remaining Wikipedias" (T425580)]]

[production]

20:13

<vriley@cumin1003>

END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1290.eqiad.wmnet with OS bookworm

[production]

20:12

<vriley@cumin1003>

END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"

[production]

20:09

<vriley@cumin1003>

START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"

[production]

19:53

<vriley@cumin1003>

END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1290.eqiad.wmnet with reason: host reimage

[production]

19:47

<vriley@cumin1003>

START - Cookbook sre.hosts.downtime for 2:00:00 on db1290.eqiad.wmnet with reason: host reimage

[production]

19:32

<vriley@cumin1003>

START - Cookbook sre.hosts.reimage for host db1290.eqiad.wmnet with OS bookworm

[production]

19:30

<vriley@cumin1003>

END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED

[production]

19:23

<vriley@cumin1003>

START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED

[production]

19:22

<vriley@cumin1003>

END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1290

[production]

19:21

<vriley@cumin1003>

START - Cookbook sre.network.configure-switch-interfaces for host db1290

[production]

19:21

<vriley@cumin1003>

END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

[production]

19:18

<vriley@cumin1003>

START - Cookbook sre.dns.netbox

[production]

16:53

<btullis@deploy1003>

helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply

[production]

16:53

<btullis@deploy1003>

helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply

[production]

16:02

<dancy@deploy1003>

Installation of scap version "4.265.1" completed for 2 hosts

[production]

16:00

<dancy@deploy1003>

Installing scap version "4.265.1" for 2 host(s)

[production]

12:18

<cmooney@cumin1003>

END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

[production]

12:18

<cmooney@cumin1003>

END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for ulsfo cr links from dns - cmooney@cumin1003"

[production]

12:18

<cmooney@cumin1003>

START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for ulsfo cr links from dns - cmooney@cumin1003"

[production]

12:02

<mvernon@cumin2002>

END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2009.codfw.wmnet

[production]

11:59

depool / restart swift / repool on ms-fe2010 ms-fe2012

[production]

11:58

<mvernon@cumin2002>

START - Cookbook sre.hosts.reboot-single for host ms-fe2009.codfw.wmnet

[production]

11:34

<atsuko@deploy1003>

helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply

[production]

11:34

<atsuko@deploy1003>

helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply

[production]

11:24

<mvernon@cumin2002>

END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2065.codfw.wmnet with OS bullseye

[production]

11:14

<cmooney@cumin1003>

START - Cookbook sre.dns.netbox

[production]

11:10

<aokoth@cumin1003>

END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T426298

[production]

11:04

<mvernon@cumin2002>

END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage

[production]

10:59

<mvernon@cumin2002>

START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage

[production]

10:55

<mvernon@cumin2002>

END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2064.codfw.wmnet with OS bullseye

[production]

10:52

<atsuko@deploy1003>

helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply

[production]

10:52

<atsuko@deploy1003>

helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply

[production]

10:46

<elukey@cumin1003>

END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie

[production]

10:43

<elukey@cumin1003>

START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie

[production]

10:42

<elukey@cumin1003>

END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie

[production]

10:41

<mvernon@cumin2002>

END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2065

[production]

10:41

<mvernon@cumin2002>

END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2065

[production]

10:40

<mvernon@cumin2002>

START - Cookbook sre.network.configure-switch-interfaces for host ms-be2065

[production]

10:40

<mvernon@cumin2002>

END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2065.codfw.wmnet 167.48.192.10.in-addr.arpa 7.6.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors

[production]

10:40

<mvernon@cumin2002>

START - Cookbook sre.dns.wipe-cache ms-be2065.codfw.wmnet 167.48.192.10.in-addr.arpa 7.6.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors

[production]

10:40

<mvernon@cumin2002>

END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

[production]

10:40

<mvernon@cumin2002>

END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2065 - mvernon@cumin2002"

[production]

10:40

<mvernon@cumin2002>

START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2065 - mvernon@cumin2002"

[production]

10:36

<mvernon@cumin2002>

START - Cookbook sre.dns.netbox

[production]

10:36

<mvernon@cumin2002>

START - Cookbook sre.hosts.move-vlan for host ms-be2065

[production]

10:35

<mvernon@cumin2002>

START - Cookbook sre.hosts.reimage for host ms-be2065.codfw.wmnet with OS bullseye

[production]