• 4 Posts
  • 18 Comments
Joined 6 months ago
cake
Cake day: January 3rd, 2024

help-circle


  • It was a mixture of factors.

    Data was to be dumped into a S3 bucket (minio), this created an event and anouther team had built an orchestrator which would do a couple of things but eventually supply an endpoint a reference to a plain/txt file for analysis.

    For the Java devs they had to [modify the example camel docs.](https://camel.apache.org/manual/rest-dsl.html) and use the built in jackson library to convert the incoming object to a class. This used the default AWS S3 api to create a stream handle which fed into the OpenNLP docs. .

    The Python project first hit a wall in setting up Flask. They followed the instructions and it didn’t work from setup tools. The Java team had just created a new maven project from the Intelij but the same approach didn’t work for the Python team using pycharm. It lost them a couple days, I helped them overcome it.

    Then they hit a wall with Boto3, the team expected to stream data but Boto3 only supports downloading, there was also a complexity issue the AWS SDK in Java waa about 20 lines to setup and a single line to call, it was about 50 lines in Python. On the positive side I got to explain what all the config meant in S3.

    This caused the team anouther few days of delay because the team knew I used a 350MiB Samsung TV guide to test the robustness and had to go learn about Docker volume mounting and they thought they needed a stateful kubernetes service and I had to explain why that was wrong.

    Basically Python threw up a lot of additional complexity and the docs weren’t as helpful as they could have been.



  • You do, but considering the scales they process data I suspect Google would be better building Go tooling (or whatever the dominate internal language is).

    A few years back I was trying to teach some graduates the importance of looking at a programming language ecosystem and selecting it based on that.

    One of my comparison projects was Apache OpenNLP/Camel vs Flask/Spacy.

    Spacy is the go to for NLP, I expected it to be either quicker to develop, easier to use, better results or just less resources.

    I assigned Grads with Java experience Spacy and Python experience OpenNLP.

    The OpenNLP guys were done first, they raved about being able to stream data into the model and how much simplier it made life.

    When compared with the same corpus (Books, Team emails, corporate sharepoint, dev docs, etc…) OpenNLP would complete on 4GiB of RAM in less than a second on 0.5vCPU. Spacy needed 12GiB and was taking ~2 seconds with 2vCPU. They identified the same results…

    Me and a few others ended up spending a day reading the python and trying to optimise it, clearly the juniors had done something daft, they had not.

    It rather undermined my point.


  • My expectation is whatever the solution it needs to dockerise and be really easy to deploy via docker compose or Kubernetes so people can quickly and easily set up their own.

    The front end is effectively static files so I would probably choose Apache or Express (whichever gives me a smaller docker image)…

    For the backend I would choose Java for Spring Boot. An Alpine image with OpenJDK and the app is tiny. Spring has a library for every kind of interface making them trivial to implement but the main reason is hibernate.

    Hibernate (now Spring Data) was the first library for being able to switch out databases without having to change code (its all config). A lot of mastodon instances struggle with the resource requirements of elastic search so letting small instances use something like postgres would seem ideal.

    I have noticed Go/Rust still expect you to write or manage a lot of stuff Spring gives away for free. Python is ok if your backend is really tiny but there is a lot of boilerplate in how Python libraries work so complex projects get hard to manage and I assume interacting with the fediverse will add complexity.


  • I agree Grian shenanigans are fantastic.

    I am currently working my way through Mumbo’s S6 and the best episodes so far have Iskall and Grian in them.

    The thing I have noticed it Mumbo isn’t really pressuring himself on a build. He has just kept focus on the storage system because he is finding it fun.

    I thought Grian was at his best with the barge, his episodes had a flow where he would resource, stock the bardge, do a “small” build/project and then shenanigans.

    I am not sure that flow is sustainable but he was clearly having fun.

    At the moment it just seems he is increasingly beating himself up on being a builder and needing to build the most epic base ever.


  • I have listened and as the seasons go on I get increasingly worried about him. He is increasingly showing signs of burn out so clearly his flow isn’t working for him.

    Listen to season 7 as he talks about the back of the mansion. He is feeling self pressure to complete the back and he is getting fustrated at himself.

    Its the same with the Ally, you can hear how excited he is at the start and as the season progresses anything to do with the ally he talks about like an unpleasant chore.

    With the stones other hermits try to intervene to help him, his reaction there really doesn’t seem a bit. It sounded a lot like the panic caused by burn out.

    He clearly loves being on the SMP and his shenanigans and ideas are fantastic and clearly a lot of fun for him. I want him to stick around and think unless he learns how to break up work so he can find joy in all his builds he will loose that joy






  • While there is nothing wrong with trying something new, the point of using a franchise is to leverage the existing fanbase.

    If you can’t get the fanbase enthusiastic you have a problem. Since you aren’t leveraging the existing fan base and the franchise will alienate some of your new target fan base.

    Replying to every comment that expresses an ambivalent or pessimistic view about a new show doesn’t change that. It just makes this space seem hostile to discussion.


  • stevecrox@kbin.runtoToday I Learned@lemmy.world...
    link
    fedilink
    arrow-up
    4
    arrow-down
    4
    ·
    4 months ago

    I wouldn’t get massively excited.

    Python is a scripting language, its shines when you want to write a stand alone file which takes an input and performs a task. Scripting languages are great to learn as a first language and so python is wonderful for non developers.

    The issue you hit is the build management solutions for Python are kind of broken and these help support and encourage good development practice so a lot of Python projects end up a collection of scripts rather than a mature project. You can have good projects but…

    In raw benchmarks Java has 90% of the performance of C/C++, but in reality Java is more performant because developers get bogged down in memory management on C/C++ and they get more time to optomise in Java as a result. I’m not sure where Rust will come out to be honest.

    Python benchmarks at 50% the performance of Java, in reality I’ve found code ends up slightly worse because Python is procedural, library support and streaming is poorly supported.

    Take library support, Spring really rose to prominese because of ‘hibernate’ which was a way to abstract talking to different databases through objects, you could switch from PostgreSQL to Oracle through config. Spring data has dumbed this down so I define a plain old Java object and Spring will generate everything I need.

    Python expects you to hand craft SQL statements and every database extends SQL slightly differently, so i need to write SQL for every operation and manage/own it. So the win in being able to quickly read/write to a database (since you don’t have to learn anything about Spring) is quickly ruined because of the all the boilerplate and error handling you now have to write.


  • stevecrox@kbin.runtoToday I Learned@lemmy.world...
    link
    fedilink
    arrow-up
    20
    arrow-down
    2
    ·
    4 months ago

    Every programming languages has communities built around them.

    Its becoming clear Rust solves a lot of C/C++ type problems and the embedded communities are definitely shifting over.

    Apache is the primary community for Java, a quick look at their project list shows it’s entirely web servers, data engineering and clustered projects for distributed computing.

    Personally if you asked me to solve this problem I would use Spring Boot with various Spring libraries for talking to the caddy, user control, etc… Looking at the project, its exactly what they have done


  • See its the opposite in Linux land.

    AMD open sourced their drivers so everything just works, while Nvidia drivers have to be built against your system and Nvidia refused to supply proper desktop drivers for years (EGLStreams vs GBM).

    The downside of AMD’s approach is it has to trickle down which depending on what distribution you use can take weeks to a year and it normally takes a couple iterations to get everything working nicely. Which basically expect the 6800 XT to work brilliantly but the 7300 to be flakey for a bit.

    My favourite bit is I owned a few Athlon 5300 APU and 5 years after they were released AMD were still adding performance improvements to them.



  • stevecrox@kbin.runtoFediverse@lemmy.worldWhat's going on with kbin.social?
    link
    fedilink
    arrow-up
    62
    arrow-down
    1
    ·
    edit-2
    5 months ago

    The developer behind KBin seems to have issues delegating/accepting contributors.

    If you look at the pull requests, most have been unreviewed for months and he tends to regularly push his branches once complete and just merge them in.

    That behaviour drove the MBin fork, where 4-5 people were really keen to contribute but were frustrated.

    To some extent that would be ok, its his project and if he doesn’t want to encourage contributions that is his decision but…

    KBin.social has gotten to the size where it really should have multiple admins (or a paid full time person). Which it doesn’t have.

    The developer has also told us he has gone through a divorce, moved into his own place, gotten a full time job and now had surgery.

    Thats a lot for any normal person and he is going through that while trying to wear 2 hats (dev & ops) each of which would consume most of your free time.

    Personally I moved to kbin.run which is run by one of the MBin devs



  • Technical Leads are not rational beings and lots of software is developed from an emotional stand point.

    Engineering is trade offs, every technical decision you make has a pro/con.

    What you should do is write out the core requirements/constraints.Then you weigh the choices to select the option that best meets it.

    What actually happens is someone really likes X framework, Y programming language or Z methodology and so decides the solution and then looks for reasons to justify it.

    Currently the obvious tell is if they pitch Rust. I am not saying Rust is bad, but you’ll notice they will extoll the memory safety or performance and forget about the actual requirements of the project.