• 1 Post
  • 40 Comments
Joined 1 year ago
cake
Cake day: June 19th, 2023

help-circle
  • Certs are a waste of time tbh. If you have 8 years of experience, you should have more than enough to fill out a resume already.

    An AWS cert is almost certainly even more useless for you specifically unless you wanted to get into devops/sre and do systems design. I have been in sre for a very long time and have never even heard of anyone writing tooling in Java. That section of the industry is entirely dominated by go, python, and (more often than anything else) bash for really quick automation.




  • As a counter balance to that though, interviewers need to understand what they are hiring for and tailor the questions asked to those requirements.

    For example, there is genuinely very little coding required of an SRE these days but EVERY job interview wants you to do some leetcode style algorithm design… Since containers took over, the times I have used anything beyond relatively unremarkable bash scripts is exceptionally small. It’s extremely unlikely that I will be responsible for a task that is so dependent on performance that I need to design a perfect O(1) algorithm. On terraform though, I’m a fucking surgeon.

    SRE specifically should HEAVILY focus on system design and almost all other things should have much much less priority… I’ve failed plenty of skill assessments just because of the code though.




  • Most resistance I have seen mostly comes down to a misunderstanding in the benefits that kubernetes offers. The assumption is that kube is used for autoscaling and that, if the inbound traffic is predictable then the added complexity is unnecessary. When that happens the “kube isn’t right for all situations” turns into “kube isn’t right for any situation” whether the person in question would ever admit that or not…

    All of this ignores the MASSIVE reliability enhancement kube delivers and the huge amount of effort currently going into modern tool development surrounding the kube ecosystem.


  • Real talk, you don’t have the luxury of being an idealist right out of university. Your goal is to get a job. When you’re in that job you will likely not have the luxury of being an idealist either.

    When you have enough experience making practical, reasoned decisions, then you can stand on principals.

    For context, I have been in this business for nearly 20 years. The people I have personally worked with who have resisted things on philosophical grounds ALWAYS get left behind. I’ve seen it with systemd, the cloud, and now I’m seeing it again with kubernetes. You cannot escape the collective inertia of an entire industry.

    Obviously there are still thresholds… I would never work for someone like Raytheon. You have to draw lines somewhere but saying you aren’t going to work for a company that does user behavior tracking is short sighted and impractical.



  • We had a service that compiles a dataset once per quarter. The total size is ~30gb. We were starting a container, storing it on an EFS volume, and mounting like any other disk.

    Every time a pod started it would need to read this data into memory so we would get quick initial start-up time but the time to be ready for traffic still took a while.

    Since we didn’t need to update it very often, we decided to just package the compiled dataset into the container and skip the EFS volume. We updated the image pull policy to ifNotPresent so it cut egress traffic pricing from EFS to zero. Now there is a cost to pull the image from ECR but that’s only if the pod is being scheduled onto a node it hasn’t been run on before. There was no noticable change in behavior or performance and we saved a bunch on cost.

    Sometimes the big, dumb option is the right choice.



  • We focus a lot more on production than the average developer. It’s our job to make sure whatever devs build is run quickly, efficiently, safely, and scalably.

    You will work with a lot of kubernetes, Argo, terraform, Prometheus, grafana. You’ll design build pipelines and software rollout strategies. You plan for zero downtime migrations and upgrades, database maintenance… You’ll have your hands in everything from capacity planning to security to cost optimization to developer support… User permissions, infrastructure, networking, observability… You will write RFCs and setup POCs for new tools. You define and track error budgets and figure out how to keep your org under those projections. When there is an outage you will be involved in writing post mortems.

    The days are so varied and unpredictable that it keeps things interesting. The landscape changes so often you’re never really stuck doing the same thing over and over.

    I genuinely love it.

    EDIT: The SRE Podcast from Google is actually really great for learning about this world. The first season talks about what you’ll be doing and why (based around the SRE O’Riley book). The second season talks about what to expect in different stages of your career progression.





  • It's completely overkill for pretty much everyone but I have been thinking about building a kubernetes native client for months now.

    Like the torrent should be treated as a normal resource with a Torrent CRD. It should be scheduled onto whichever node has available capacity and rescheduled onto a different node if it goes down. If allowed by the tracker, multiple instances could be run. You could set resource limits programmatically, easily configure block storage, build dashboards, export logs/metrics… It would be open ended enough that you could have interfaces built as browser extensions, web ui, mobile app, tui, cli and be unopinionated so much that the method for torrent ingestions could be left up to the used. HTTP request, watch directory, rss client, download manager… You could even do stuff like throw magnet links into a queue… etc, etc…

    I keep thinking it would be a great project but I just do not have the spare time to dedicate to it… I imagine it could be used for large scale deployments for something like the Internet archive or whatever.