Posts By Rachel Fong

Postmortem: RDS Clogs & Cache-Refresh Crash Loops

On Thursday, October 4, we experienced a partial API outage from 21:02-21:56 UTC (14:02-14:56 PDT). Despite some remediation work, we saw a similar (though less serious) incident again on Thursday October 11 from 15:00-16:02 UTC (8:00-9:02PDT). To implement a more permanent fix, we scheduled an emergency maintenance window which completely interrupted service on Friday Oct 12 for approximately two minutes, from 4:38-4:40 UTC (Thursday Oct 11, 21:38-21:40 PDT).

New Custom Regex Log Ingestion

Unstructured text logs are so last decade, but sometimes you have to deal with them because they aren’t actually all that prehistoric in human years…or you just need us to ingest formats we don’t…