• 0 Posts
  • 33 Comments
Joined 1 year ago
cake
Cake day: June 10th, 2023

help-circle




  • Could use Polars, afaik it supports streaming from CSVs too, and frankly the syntax is so much nicer than pandas coming from spark land.

    Do you need to persist? What are you doing with them? A really common pattern for analytics is landing those in something like Parquet, Delta, less frequently seen Avro or ORC and then working right off that. If they don’t change, it’s an option. 100 gigs of CSVs will take some time to write to a database depending on resources, tools, db flavour, tbf writing into a compressed format takes time too, but saves you managing databases (unless you want to, just presenting some alternates)

    Could look at a document db, again, will take time to ingest and index, but definitely another tool, I’ve touched elastic and stood up mongo before, but Solr is around and built on top of lucene which I knew elastic was but apparently so is mongo.

    Edit: searchable? I’d look into a document db, it’s quite literally what they’re meant for, all of those I mentioned are used for enterprise search.




  • I was originally going to to go the docker route but honestly just ended up going the binary route and leaving it using sqlite as it’s good enough for now. It’s pretty well documented and a chunk of the prereqs I already had, like the git user creation.

    Did have SSH auth issues though, probably becauae I didn’t fully cleanup after uninstalling gitlab (oops), had them in parallel for a bit to migrate the repos, gitlab had it trying to use gitlab-shell which didn’t exist anymore. Probably a better/proper solution but what worked was changing the git user’s home directory back to /home/git as gitlab had it using a gitlab config directory. I welcome anyone giving me a better/cleaner solution for this, on my to do list to do some more cleanup.



  • The only way I ever played StarCraft was StarCraft 64, the split screen multiplayer was cool but absolutely hands down the worst way to play it. I don’t know how they managed to make that work at all honestly, I know there woukd have had to have been a lot of concessions to fit it on the cart, but still, kinda impressive to me. I realised just how bad it was to play after playing wc3 a few years later.

    Sunshine+Moonlight has taken over most of my console use, there’s so much less screwing around with games needed these days, if you’re not modding they tend to run well out of the box in my experience, seeing so many games with native controller support + local multiplayer is fantastic, steam input fills the gap on a lot of the others. That said though nothing really beats the pick up and go of a console, my GameCube still runs perfectly after 20 something years, I can emulate them (and do for some games, metroid prime trilogy is better on m+k, but that’s the Wii version of the trilogy) but I don’t feel the need to tweak things endlessly on the native hardware.



  • Heck, there are already ISO language standards, and there’s ISO Software Lifecycle standards, it’s absolutely not a leap to move into standards adhering processes. It’s not like there’s no desire to do it either, code standards alone, how many times have you had discussions about style guides and coding standards company wide? It makes things more consistent and easier for different developers to maintain.

    Semi related, I see a lot of non-iso standard SQL that’s a pain if you do migrations or refactors, often even just sucks to read through (old school oracle joins look really strange and aren’t clear compared to iso standard joins). I really wish people would adhere to the standards as much as possible.


  • I realised you meant this over lunch, I’m a mech eng who changed disciplines into software (data and systems mainly) over my career, I 100% feel you, I have seen enough colleagues do things that wouldn’t fly in other disciplines, it’s definitely put me off a number of times. I’m personally for rubber stamping by a PEng and the responsibility that comes with that. There’s enough regulatory and ethical considerations just in data usage that warrants an engineering review, systems designed for compliance should be stamped too.

    Really bothers me sometimes how wildwest things are.


  • Edit: see my response, realised the comment was about engineering accountability which I 100% agree with, leaving my original post untouched aside from a typo that’s annoying me.

    I respectfully disagree coming from a reliability POV, you won’t address culture or processes that enable a person to make a mistake. With the exception of malice or negligence, no one does something like this in a vacuum; insufficient or incorrect training, unreasonable pressure, poorly designed processes, a culture that enables actions that lead to failure.

    Example I recall from when I worked manufacturing, operator runs a piece of equipment that joins pieces together in manual rather than automatic, failed to return it to a ready flag and caused a line stop. Yeah, operator did something outside of process and caused an issue, clear cut right? Send them home? That was a symptom, not a cause, the operator ran in manual because the auto cycle time was borderline causing linestops, especially on the material being run. The operator was also using manual as there were some location sensors that had issues with that material and there was incoming quality issues, so running manually, while not standard procedure, was a work around to handle processing issues, we also found that culturally, a lot of the operators did not trust the auto cycles and would often override. The operator was unlucky, if we just put all the “accountability” on them we’d never have started projects to improve reliability at that location and change the automation to flick over that flag the operator forgot about if conditions were met regardless.

    Accountability is important, but it needs to be applied where appropriate, if someone is being negligent or malicious, yeah there’s consequences, but it’s limiting to focus on that only. You can implement what you suggest that the devs get accountability for any failure so they’re “empowered”, but if your culture doesn’t enable them to say no or make them feel comfortable to do so, you’re not doing anything that will actually prevent an issue in the future.

    Besides, I’d almost consider it a PPE control and those are on the bottom of the controls hierarchy with administrative just above it, yes I’m applying oh&s to software because risk is risk conceptually, automated tests, multi phase approvals etc. All of those are better controls than relying on a single developer saying no.





  • 100% a title that would struggle with full controller, for me it was cities skylines and rimworld. Also played a lot of warframe and spec ops:the line with mine, being able to have actions trigger at different points of the trigger pull was interesting, had a profile I grabbed for shooters that’d enable gyro aiming at the last bit of your trigger pull for fine adjustment and seriously, it works extremely well once you get used to it. The pads also supported osd rotary menus for hotkeys which was probably what the left pad got the most use out of, had the ability to set different behaviour too using mod buttons are by touching the rim of the pad. Also the haptic feedback on the pads was interesting, did a lot to make them feel more real, seriously had a really powerful piece of hardware with the og steam controller.



  • I really liked the wavebird for the gamecube, unfortunately mine went into the aether on my last move, got bluetooth adapters to pair modern controllers with it but the wavebird was really cool at the time, was really amazing to not have to be tethered to the console and it being first party, though at the time the madcatz stuff was decent.

    For recent controllers, I’ve been using a knockoff 360 controller for moonlight recently and after a lot of back and forth I really think MS nailed the controller setup back then (OG Xbox being decent but not a preference, I hated the duke, s controller was solid though), I like the xbone controllers as well, but IMO they’re just iterations on the 360 controller, easily my preference as an all rounder controller layout.

    I have a steam controller, used it for a while but it’s been some time now, had some really great ideas, I’d totally go for an updated steamdeck style layout on that, probably a second for me.

    I’ve had so much drift issues with ds4s that I personally don’t reach for a ds4 or dualsense for non playstation games, I like being able to swap batteries and the Xbox/Steam controllers all seem to have way better battery life in general, I keep a stock of rechargeables around so not generating piles of waste.