So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.
I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
What’s extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn’t live in it for no discernible reason.
Job security
He does charge a consulting fee to “fix” these issues
Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.