2026-03-01
Incident Post-Mortem: Summer 2026 courses accidently released
Summary
The 2026 summer course catalog was accidently released. Users signed up for trackers in the time before it was detected.
Timeline
~08:00- uploader was run with summer 2026 course data, connected to `the prod db21:05- incident was noticed, thread started21:06- spring 2026 catalog accidently deleted while trying to remove `summer 202621:09- spring 2026 catalog restored (new upload of cache)21:12- summer 2026 trackers deleted by concensus21:15- summer 2026 term deleted in prod
Root Cause
The PL had a production env var set in the scraper package. Before running
the uploader, it was checked that the db package had the correct env var,
but not the scraper package. These split env vars allowed for the cache to
upload, with the PL thinking it was a fluke upload.
What Went Well
- Checking all the env vars when things stopped working found the cause
- Communication in Slack to practice incident management process
- Post-mortem to follow up
What Could Improve
- Production environment variables should have never been on an IC's computer
- DB migrations require local env vars
- Trackers should have been managed better (users should have been notified)
Action Items
- Remove all local production env vars
- Develop better DB migration system
- Have development variables / systems to ensure production variables remain safe