Is this a real need or something to play with? I.e., does having the AI offer you real benefit?
Insofar as possible, I will be trying to create identical environments, save for the hardware. The point being to identify/quantify the impact of the hardware on performance. (otherwise, there's too many variables to try to evaluate)
I am hoping to just build an environment and then pull the disk and move it to another "very similar" machine (changing memory, MIPS, GPU in/out, etc.)
I've got gobs of RAM so even some of the larger LLMs weights would easily fit in them.
Yes, that's the point. If *designing* with an AI, then you want it to be performant as you are likely twiddling your thumbs waiting to critique it's latest offering.
If, OTOH, you are using it to perform a particular *job*, you can walk away and return when the results are available.
Consider the parallel to folks who "need" fast compiles because they're waiting to see how their latest patches perform vs. doing a "make world"
I have boxes with half a terabyte of RAM so assume the real problem will be the slower speed (and narrower pipe) that system RAM provides vs the GPUs access to VRAM.
OK.
Exactly. But, having an online AI to act as a reference to validate the output of the local instance would be reassuring.
This is the heart of the problem I'm facing. I don't see any real use for AIs *in* my workflow, unless as bug hunters. And, I expect little value, there, as the sources of the "tough" bugs almost always lie in concurrency issues. (who writes single-threaded code in the 21st century?)
Edward's image creator would be entertaining and likely produce repeatable results (input doesn't change so why should the output?). But, would be hard to correlate to any real "design" effort.
I'd never put my IP out to train something -- any more than I would train an employee for a client/employer. Nor can I see any of my colleagues giving away the farm.
OTOH, if there was an activity (of value!) that could be performed using something *locally*...
(Some of the speech machines have been of use to me but that's more of a niche need)