Hi! So I’m considering…maybe having an NPU or something similar to be hooked to my proxmox server, which runs in a mini PC. It’s a EliteDesk 800 micro form factor. It has a Core i5 8500 CPU, which at the moment of purchase was good enough for live encoding HEVC video on Jellyfin…that was my main concern back then. But I’d like to consider the possibility of hooking maybe some docker instances or other containers to some local-only AI acceleration. Is there any NPU or cheap GPU I could hook on USB to this proxmox server to run? Has it been done before?
Thanks!
if you don’t have oculink, you can get a nvme to oculink adapter (if you have 2nd nvme port).
oculink
Thanks…I didn’t know about this. But this doesn’t resolve the initial question. What hardware could I use to have accelerated ollama, diffussion etc?
USB is not good as its a huge bottleneck, and most external accelerators are (was?) passive without any ram onboard. Google Coral was/is ‘passive’ in that it resends all data all the time over the interface - Npu<->system ram.
I were going to recommend these: https://shop.geniatech.com/product/m2-ai-inference-acceleration-module/
40tops, ARM + Npu + 16gb ddr4 - a whole little Inference computer on a NVME interface. Kinara (Ara240) is a homegrown Chinese chip. While they are usually selling b2b, you can ask anyway. YMMW atmo. Also note that some NPU’s are less efficient at llm’s vs vision.
…but I see that they also rose almost 4* in price since I asked for, and were offered the price of 179$ ~7M ago, which is already a long time in this space. Not sure how the current ~650$ stacks up to the rest of the offers out there at the moment, but these small active AI systems on NVME are a great way of upgrading a piss-old server, enhance a new cheap 4-8port NVME mini-NAS or similar, and there’s no clear bottleneck in the interface.
Look for something like this instead of Nvidia Coral and other ‘passive’ sticks, that are all - imho - overpriced/underperforming.
Awesome, I have more or less easy access to Chinese products! I’m going to take a look at these! I might get a free m.2 slot if I remove the Wifi card. I need to check.
EDIT: Yeah no…the M.2 from the wifi card…of course it’s 2230 so it won’t fit those 2280 :(
Do you want to run TensorFlow Lite / LiteRT models? PyTorch Mobile? TensorRT? onnx? YOLO? vLLM? Something else? The recommendations will vary based on your use case.
Google Coral was decent for TensorFlow Lite, but it’s EOL (end of life) now. I’ve got the dual TPU Mini PCIe version in my home server, via a PCIe adapter board. I use it for object detection with Blue Iris + CodeProject AI and it works pretty well for that use case.
Hailo-8 is supposed to be like a more powerful version of the Coral, but I don’t have experience with it. It supports a bunch of frameworks: TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX. I’d be interested in hearing other people’s thoughts on it.
I don’t know if any of these work over USB though. They’re usually internal devices. Google marketed the Coral USB as being for development and testing only, pointing people to the M.2 and PCIe versions for production usage.
As for something totally different… There’s the Nvidia Jetson single board computer which supports TensorRT, but I don’t have experience with it either. I also think it’s a bit older too. You could also consider getting a newer mini PC with a AMD Ryzen AI processor in it, or an Nvidia DGX Spark.
Google’s latest TPUs are only available in their cloud - they’re not selling the hardware to end users any more.
Thanks! I was considering using it perhaps to have a diffusion model? Or running ollama or similar without a RAM hit on my laptop or phone. Also whisper comes to mind, for Bazarr or other tools to use.
Any NPU/TPU you can buy is going to be essentially useless for either image diffusion or LLMs. The onboard RAM is both far too small and far too slow (LLM text generation speed relies on RAM speed first and foremost, and both LLMs and image models tend to be, you know, big), and USB isn’t nearly fast enough to help with that, not to mention that software support is pretty much nonexistent. You’d be better off upgrading the GPU to a 3060ti 12gb or something.
P.S. A word of advice, consider using something other than Ollama. Llama.cpp in router mode or llama-swap support pretty much all of the functionality that Ollama does without being crap. Ik_llama.cpp is also nice if you have a CPU/Nvidia setup.
If you wanna make the most out of what you’ve got now, the LFM2.5 series of LLMs are quite good for the small size and fast inference speeds with sizes ranging from 0.2 billion to 8 billion parameters, though their low parameter count means that you’ll probably wanna hook them up to some sort of web search or similar since they won’t have a ton of general knowledge.
If you have at least 32GB of RAM, Qwen3.6 35B is quite a good general-purpose model that runs faster than its parameter count would suggest.
Aren’t diffusion models and LLMs (ollama) too big for an external NPU? As far as I know something like a Coral runs specific models only. And it’s limited to the 1 or maybe 2GB of memory on it. It’d do tasks like voice recognition, or image classification. But not generate images or text.
If you want to run arbitrary AI models and generative AI, I think you should be looking for a graphics card?!
Agreed. OP should probably upgrade to a bigger case and discrete graphics card.
google coral is what you need, there’s also an m.2 pcie version, also intel igpu also has limited AI capabilities through openvino, i use it to accelerate AI features in Immich (face and object recognition) on my N100 based NAS, but it’s a bit too slow for Frigate realtime object recognition for example
A little warning about the google coral, it seems not maintained anymore by google and there can be a few issues in the future. I have it for Frigate (7 cameras) and I still manage to have an inference speed < 10ms (but it detects people, not objects)
You should probably have a look at the software you intend to run and see what kind of external NPU they support. I’m aware of Frigate for example which leverages hardware acceleration to classify camera surveillance footage. Guess their community knows which stick to get. Or you’d have a look at the frameworks which package machine learning models for inference on some NPU and have a look at their table of supported hardware.
Inference from GPU, even QuickSync, is better now than what TensorFlow was able to do on a Google Coral TPU.
Plus, the Coral drivers are untouched for the last 4 years.
I think you mean SyCL or Vulkan, OpenCL or oneDNN… QuickSync is just video de- and encoding.
QuickSync is the GPU hardware used to infer from vino models.
https://docs.frigate.video/configuration/hardware_acceleration_video/#via-quicksync
Thx! That’s the video decoding page, though. The thing that decodes the video stream and makes individual images from the stream, for the actual processing. Seems the thing that does inference / object detection is called OpenVINO.
https://docs.frigate.video/configuration/hardware_acceleration_enrichments
