Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

  • curbstickle@anarchist.nexusMEnglish
    41·
    2 months ago

    Yep.

    Ollama + about 8 different models at the moment, hosted on a mac mini with open webui as a front end.

    Predominantly for transcription, translation, an extra round of security checks on code, a more context friendly home assistant interface, and a daily run of context evaluation on property I’m looking for with a lot of specific needs (acreage, min elevation change, soil type, area, etc).

      • curbstickle@anarchist.nexusMEnglish
        21·
        2 months ago

        On the list but haven’t gotten to it yet, but I know I should. I could probably get a bit more out of that box with it, expand the context windows a bit…

      • curbstickle@anarchist.nexusMEnglish
        21·
        2 months ago

        Just an m2 w/ 16gb I repurposed.

        Can’t really do a lot at once, and the context is limited, but it does the trick. I’d buy a few more if I saw them at the right price.

        • async_amuro@lemmy.zipEnglish
          21·
          2 months ago

          Nice, I’ve got a Mac Studio M1 Max with 32GB of RAM that I use with Ollama and then I host OpenWebUI and OpenCode on my Arch Server. I use the Mac as a primary workstation, so it’s a little rough when I start running a model. I’m sure I could probably do and learn more about Ollama to improve my experience, but for now it works for certain tasks.

          • curbstickle@anarchist.nexusMEnglish
            21·
            2 months ago

            I got mine a few years back for some iOS builds, don’t need to do them that often so it became the model host for me

      • curbstickle@anarchist.nexusMEnglish
        3·
        2 months ago

        Apple silicon is pretty good at it as long as you’ve got the ram for it. I wouldn’t do less than 16GB.

        A few seconds for most of the tasks