Local AI instances

Is this a real need or something to play with? I.e., does having the AI offer you real benefit?

Insofar as possible, I will be trying to create identical environments, save for the hardware. The point being to identify/quantify the impact of the hardware on performance. (otherwise, there's too many variables to try to evaluate)

I am hoping to just build an environment and then pull the disk and move it to another "very similar" machine (changing memory, MIPS, GPU in/out, etc.)

I've got gobs of RAM so even some of the larger LLMs weights would easily fit in them.

Yes, that's the point. If *designing* with an AI, then you want it to be performant as you are likely twiddling your thumbs waiting to critique it's latest offering.

If, OTOH, you are using it to perform a particular *job*, you can walk away and return when the results are available.

Consider the parallel to folks who "need" fast compiles because they're waiting to see how their latest patches perform vs. doing a "make world"

I have boxes with half a terabyte of RAM so assume the real problem will be the slower speed (and narrower pipe) that system RAM provides vs the GPUs access to VRAM.

OK.

Exactly. But, having an online AI to act as a reference to validate the output of the local instance would be reassuring.

This is the heart of the problem I'm facing. I don't see any real use for AIs *in* my workflow, unless as bug hunters. And, I expect little value, there, as the sources of the "tough" bugs almost always lie in concurrency issues. (who writes single-threaded code in the 21st century?)

Edward's image creator would be entertaining and likely produce repeatable results (input doesn't change so why should the output?). But, would be hard to correlate to any real "design" effort.

I'd never put my IP out to train something -- any more than I would train an employee for a client/employer. Nor can I see any of my colleagues giving away the farm.

OTOH, if there was an activity (of value!) that could be performed using something *locally*...

(Some of the speech machines have been of use to me but that's more of a niche need)

Apparently, Oracle:

One has to wonder what other tools could have found those faults, possibly BEFORE the advent of LLMs.

This is for reading things in languages I don't speak. If want to read a book in eg German, first I need to get it into a digital form. Then I need to translate it, and re-apply the translations to the images (so diagrams make sense)

Translation apps like Google Translate will translate one image at a time, but if the book has hundreds of pages that is awkward to manage. You really want a pipeline that does everything automatically. No doubt that can be done via the APIs of the big translation services, but that starts getting expensive in terms of API costs. Each book has a lot of pages, and there are a lot of books...

I expect that's going to be annoying with GPU drivers.

That assumes you aren't going to need multiple roundtrips. ie it's a simple mechanical process that you run exactly once. If you need to close the loop, ie tweak some of the inputs and repeat, then the latency matters as well as the throughput.

Even if you aren't tweaking the model you may need to tweak the inputs.

I don't know how repeatable these flows are expected to be. ie can you put the same input into the same model run on different hardware and expect to get identical outputs? If you put the same prompt twice into ChatGPT will you get identical output? Or are there sources of divergence (either numerical or context)?

For simple stuff run locally I expect more determinism (better control of context, same hardware etc) but you don't know what is being used behind an online API.

Theo

Theo <theom+ snipped-for-privacy@Chiark.greenEnd.org.UK> wrote: |----------------------------------------------------------------| |"This is for reading things in languages I don't speak. [. . .]"| |----------------------------------------------------------------|

O brave new World!

Beware that software is even worse at translations than professional translations persons are! (S.