Rendered at 05:04:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jacquesm 1 days ago [-]
What is really impressive about this project and llama.cpp as well is that they keep finding new ways to speed up the code on the same hardware. Every time I upgrade (once a month or so) there is usually a 5 to 10% speed boost.
edit: Just to verify: I had planned to upgrade today, ran my usual benchmark prior and after and the runtime went from 4:22 to 3:59. That's a really long prompt (50K tokens) so prefill is a good chunk of that and the rest is spent on token generation.
dust42 21 hours ago [-]
While I think that llama.cpp is an outstanding project - a one-stop-shop to get almost any model to run on almost any hardware, you get a lot more speed from specialised inference engines with kernels made for your hardware.
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is one of the best ones to go to.
jacquesm 21 hours ago [-]
> you get a lot more speed from specialised inference engines with kernels made for your hardware.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.
barrkel 18 hours ago [-]
I run vllm and llama.cpp; llama.cpp is pretty hackable by a local agent and is fast to start up, while vllm takes and age to build its optimized cuda graph.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
xrd 12 hours ago [-]
What hardware is this, may I ask?
barrkel 12 hours ago [-]
9995wx 768GB system, RTX Pro 6000 GPU
eMPee584 15 hours ago [-]
what happens if you swap the setup? i.e., what's the perf ( and UX) diff just from the inference engine?
barrkel 12 hours ago [-]
vllm has limited support for quantized models, whereas ggufs are the native format for llama.cpp so it was always an easy choice. Also I never need to do batch GLM inference, whereas for e.g. photo captioning or embedding for semantic search, batch Qwen makes more sense.
35 tok/s for GLM 5.3 flash q4 quant
100 to 200 tok/s for Qwen3 VL 8b ; but it's usually short context, single image, captioning that kind of thing. It's generally sub-second for captioning a single image.
This is hybrid GPU/CPU inference for GLM (9995wx + RTX Pro 6000); and RTX Pro 6000 for Qwen3 VL.
I can't directly swap the setup because GLM 5.3 flash is huge, even q4 quant, it doesn't fit in 96GB. But it does fit in my 768GB of system RAM.
thibsrn 19 hours ago [-]
For this project it's more about achieving the 'train once, run everywhere' paradigm that Java had.
macNchz 16 hours ago [-]
Totally, I’ve been fiddling with local models on my machine (16GB AMD GPU + 64GB DDR4) for a few years and it’s remarkable the pace of improvement in what I can run. It has gone from complete novelty/useless to legitimately capable models running fast enough and with a big enough context to be useful running in a coding harness on a real codebase. A lot of that is down to the models themselves and the people making quants, but it feels like every time I pull llama.cpp changes and rebuild I get more performance and capability.
titaniumtown 19 hours ago [-]
What hardware are you running inference on?
I have an Intel Arc Pro B70. Serves me well! I use Qwen 3.8 27B
moffkalast 19 hours ago [-]
Running it with Vulkan or SYCL?
epistasis 12 hours ago [-]
That's fantastic to hear. Reminds me that software improvements over the years have been roughly as effective at speeding computation as the hardware improvements have been.
gabucino 23 hours ago [-]
[dead]
evilelectron 1 days ago [-]
Awesome! ggml and llama.cpp are the Linux of the AI world. Driving local inference one bit at a time.
Wishing you unlimited power and may you never see a day of low battery. Respect and Love. Keep it up.
kodoman 1 days ago [-]
Long time llama.cpp user (and user of other ggml based AI programs) running on my crappy AMD card that only supports vulkan and it's usable thanks to GGML and llama.cpp, great program great project.
but this post is pointing out that they've hit v1.0.0 (a major milestone) by linking to the v1.0.0 release notes
thisisauserid 18 hours ago [-]
If you load an MoE model that slightly exceeds your GPU memory budget, llama.cpp and ggml v1.0.0 can cache less-frequently used experts in host memory rather than failing to load.
hizlikovboy27 23 hours ago [-]
In my opinion, this is a genuine engineering product.
LLM.cpp, the savior of our difficult times...
trilogic 12 hours ago [-]
Who the heck deleted my comments, I didnt know that was a thing in Hacker news, OFC I will repost them:
-1 points by trilogic 9 hours ago | parent | context | next [–] | on: Ggml v1.0.0
> We like simplicity and aim to keep the codebase as small and as simple as possible
Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
reply
gabucino 7 hours ago | parent | next [–]
They were already doing that by making the build process phone home to Huggingface and also download huge swathes of indecipherable JavaScript (3 of those have been backdoored a month ago and only version pinning saved downstream).
reply
*
1 point by trilogic 4 hours ago | root | parent | next [–]
Yes there is much more I forgot to mention (you nailed some of it), but as I already said, everyone has a price. Although it remains a piece of engineering that empowered users certainly. Anyway we managed to eliminate all unpleasantries and have a clean CLI. I leave the server to whom may like it. Hit me up if you want to test it.
reply
gabucino 4 hours ago [dead] | root | parent | next [–]
Thanks, I've also maintained my own "sanity" patchset for llama.cpp (and sent PRs), but eventually gave up in the deluge of bugs, ignored PRs (except one that I deliberately made unignoreable, but that simply got closed: https://github.com/ggml-org/llama.cpp/issues/24410) incompatible changes, and instead rolled my own engine, that's more futureproof: https://github.com/whoreson/picolm
gabucino 2 hours ago | unvote | parent | prev | next [–]
Those replies were made in a thread that is hidden by default. I think it makes sense to repost them if they want to contribute to the conversation.
nullsanity 8 hours ago [-]
[dead]
igravious 16 hours ago [-]
for context:
"""
Description
The main goal of ggml is to be a simple, portable, and efficient tensor library for machine learning with minimal setup.
Plain C/C++ implementation without any dependencies
Cross-platform - x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly
SIMD-optimized kernels for x86, ARM, and RISC-V
Broad backend support - CPU, GPU, NPU, and browser
2- to 8-bit integer quantization, plus MXFP4 and NVFP4 microscaling formats
Zero memory allocations during runtime
"""
retwakm4 14 hours ago [-]
As a mainframer, love to see the s390x support and zdnn (C library with all the instructions for our on chip accelerators) support. I like that IBM and Broadcom et al. went with something open source and local that could be tested on our Linux machines so that it wouldn't be a big pain to implement on z/OS or zLinux so to speak.
I've had the opportunity to do some fun playing around with zdnn on a z17. I've not tried the Spyre cards, but the on chip accelerator for Telum II is surprisingly capable with smaller models. I initially thought you would basically NEED the Spyre cards to get anything significant running but I was wrong.
snvzz 23 hours ago [-]
Not just legacy hardware, but also ready for the immediate future, with RISC-V support.
Congrats for 1.0.0.
perdy 22 hours ago [-]
Congratulations on 1.0, and thanks to everyone who's worked on ggml. It's impressive how much it keeps improving on the same hardware. I'm curious whether 1.0 also comes with a stability commitment for the public C API. I've been looking at Python bindings to embed llama.cpp in-process, and knowing the API is stable until a 2.0 would make building on top of it much easier.
no1guangming 24 hours ago [-]
[flagged]
owlmem 21 hours ago [-]
I've been running llama.cpp on a plain CPU-only box for local work for over a year now, and the thing that keeps surprising me is how often an upgrade is just free performance. Same model, same hardware, noticeably faster prompt processing — jacquesm's 4:22 to 3:59 matches what I've seen on my own machine. The other underrated part is that the codebase is small enough to actually read. When something is slow I can usually find the hot loop myself instead of treating the inference engine as a black box, which is not something I can say about most of the GPU stacks I've used.
oochernyshev 19 hours ago [-]
[flagged]
happybox2016 17 hours ago [-]
[flagged]
om8 1 days ago [-]
Imagine naming a library after yourself.
> GGML is a C library for machine learning (ML) - the "GG" refers to the initials of its originator (Georgi Gerganov).
Every single .exe file begins with the two letters MZ because Mark Zbikowski needed a "magic number" to begin the PE format.
21 hours ago [-]
cldellow 1 days ago [-]
There's ample precedent for this. When you're building something to scratch an itch, the name is secondary. You cast about for one. Ah, you have a name. That'll do.
For example, DJ Delorie's djgpp project, built in 1989, was how I got into "real" programming on MS DOS.
magicalhippo 1 days ago [-]
> DJ Delorie's djgpp project
Fun story time. I started programming on Turbo Pascal. I had read some C code from those demo scene guides, so felt I understood it fairly well by the time I got my hands on a copy of djgpp. I decided to port a relatively simple program I had written in Turbo Pascal, about 300 LOC, line-by-line. It was a small demo just doing a simple rotating, texture-mapped triangle.
I carefully converted it, and went over it several time one line at a time, fixing it until I had read over it three times and was happy.
I then ran the djgpp compiler...
Now, what I hadn't realized at the time was the difference between C and C++. The examples I had seen was C, and I didn't really know what C++ was.
The compiler churned on my code and spit out about 3500 errors and warnings (it counted for me).
I was shocked. How did my 300 line C program manage to produce over 10 errors and warnings per line, on average?!
I spent a long time trying to figure this one out. The first error was inside the first function, and the error made no sense to me.
After several days of carefully going over each line and banging my head against the table, I suddenly saw it. There, at the very start of the program, before the line the first error or warning pointed to.
In Turbo Pascal I had used const to define things like screen width and height. So I'd have
I removed the two semi-colons and the program compiled and ran just fine...
I ended up staying in Turbo Pascal, I preferred a compiler that gave me sensible errors and quickly. I did pick up C++ much later, and I guess ironically came to appreciate its const correctness.
feffe 18 hours ago [-]
Also dlmalloc, phkmalloc, jemalloc for a few more examples
loeg 1 days ago [-]
For example, Debian. AWK. PKZIP. ARJ. Korn shell. Bourne shell. jemalloc, dlmalloc. djbdns.
satvikpendem 24 hours ago [-]
I don't have to imagine.
_kb 21 hours ago [-]
"Linux".
maximilianburke 12 hours ago [-]
dlmalloc, jemalloc
arcanemachiner 1 days ago [-]
Hey, it worked for Carnegie!
musicale 24 hours ago [-]
This is the winning response.
And we can only hope that our modern robber barons will use their ill-gotten gains to create something as beneficial as Carnegie libraries.
nullsanity 8 hours ago [-]
[dead]
jacquesm 1 days ago [-]
I'm biased (see: pianojacq.com)
Imagine trying to name a file format for yourself even if it wasn't yours to begin with...
cherioo 1 days ago [-]
Par for course for being Linux of AI
om8 1 days ago [-]
Historically, Linus didn't name linux. Other people did it for him.
> During the start of his work on the system, he stored the files under the name "Freax" for about half of a year. Torvalds had already considered the name "Linux", but initially dismissed it as too egotistical.
They wrote a large portion (and all of the initial version I think), they can name it whatever tf they want.
24 hours ago [-]
CodingJeebus 1 days ago [-]
What do you think Linux is named after?
om8 1 days ago [-]
Answered in another thread
monster_truck 23 hours ago [-]
jealous?
layer8 10 hours ago [-]
Imagine if his name had been Sergi Gerganov.
toolssogood 1 days ago [-]
[flagged]
minimaxa 23 hours ago [-]
[flagged]
nvme0n1p1 23 hours ago [-]
> The ggml way
> Minimal
> We like simplicity and aim to keep the codebase as small and as simple as possible
I'm glad their philosophy extends to the website as well. Genuinely refreshing.
trilogic 21 hours ago [-]
> We like simplicity and aim to keep the codebase as small and as simple as possible
Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
gabucino 19 hours ago [-]
They were already doing that by making the build process phone home to Huggingface and also download huge swathes of indecipherable JavaScript (3 of those have been backdoored a month ago and only version pinning saved downstream).
trilogic 16 hours ago [-]
Yes there is much more I forgot to mention (you nailed some of it), but as I already said, everyone has a price. Although it remains a piece of engineering that empowered users certainly.
Anyway we managed to eliminate all unpleasantries and have a clean CLI. I leave the server to whom may like it. Hit me up if you want to test it.
good thing i'm never touching that repo ever again, or only to steal implementations
am17an 11 hours ago [-]
Oh, not cos you’re banned because of your uncouth language? Good riddance
gabucino 8 hours ago [-]
i am? not that i know of, since i don't even try to interact with such an unsavory bunch as you sloppingface spyware maintainers
anyway you'll address me as saar
_kb 21 hours ago [-]
Loads instantly as a single 1.33 KB request. No excess crud. Readable, accessible, and provides exact information on what the project is about.
That seems borderline perfect to me.
18 hours ago [-]
sureglymop 20 hours ago [-]
I disagree. This landing page is perfectly simple and minimal. What your landing page has to look like just depends on your target audience.
Besides, some fancy AI generated landing page would signal even lower effort.
dataking 23 hours ago [-]
I think the minimalism is the point. Compare with https://ssi.inc
ducktective 23 hours ago [-]
> Claude, design a landing page with beige background color in a gradient, 3 fonts in UI, a blinking led at top right, and enough animation and javascript that makes laptops built before 2025 stutter in scrolling.
nvme0n1p1 23 hours ago [-]
And don't give any clues about what the project actually does, Just repeat how amazing and fast and modern it is, rephrased in 10 different ways.
phkx 22 hours ago [-]
Why is this getting downvoted? That’s my grief with most landing pages, including popular ones which I simply get in contact with for the first time. Most of them somehow expect that you know the domain already. And even then, it wouldn’t hurt to simply state once in which context you want to provide what kind of solution.
tbh I much prefer this than another claude slop landing page. Not saying it couldn't be better but its kinda refreshing, the whole thing fits on my phone screen
edit: Just to verify: I had planned to upgrade today, ran my usual benchmark prior and after and the runtime went from 4:22 to 3:59. That's a really long prompt (50K tokens) so prefill is a good chunk of that and the rest is spent on token generation.
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is one of the best ones to go to.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
35 tok/s for GLM 5.3 flash q4 quant
100 to 200 tok/s for Qwen3 VL 8b ; but it's usually short context, single image, captioning that kind of thing. It's generally sub-second for captioning a single image.
This is hybrid GPU/CPU inference for GLM (9995wx + RTX Pro 6000); and RTX Pro 6000 for Qwen3 VL.
I can't directly swap the setup because GLM 5.3 flash is huge, even q4 quant, it doesn't fit in 96GB. But it does fit in my 768GB of system RAM.
I have an Intel Arc Pro B70. Serves me well! I use Qwen 3.8 27B
Wishing you unlimited power and may you never see a day of low battery. Respect and Love. Keep it up.
-1 points by trilogic 9 hours ago | parent | context | next [–] | on: Ggml v1.0.0
> We like simplicity and aim to keep the codebase as small and as simple as possible Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
reply
gabucino 7 hours ago | parent | next [–]
They were already doing that by making the build process phone home to Huggingface and also download huge swathes of indecipherable JavaScript (3 of those have been backdoored a month ago and only version pinning saved downstream). reply
*
1 point by trilogic 4 hours ago | root | parent | next [–]
Yes there is much more I forgot to mention (you nailed some of it), but as I already said, everyone has a price. Although it remains a piece of engineering that empowered users certainly. Anyway we managed to eliminate all unpleasantries and have a clean CLI. I leave the server to whom may like it. Hit me up if you want to test it. reply
gabucino 4 hours ago [dead] | root | parent | next [–]
Thanks, I've also maintained my own "sanity" patchset for llama.cpp (and sent PRs), but eventually gave up in the deluge of bugs, ignored PRs (except one that I deliberately made unignoreable, but that simply got closed: https://github.com/ggml-org/llama.cpp/issues/24410) incompatible changes, and instead rolled my own engine, that's more futureproof: https://github.com/whoreson/picolm
gabucino 2 hours ago | unvote | parent | prev | next [–]
>Spawning the cli to server holy fucking shit I just checked and they really did this LOL (https://github.com/ggml-org/llama.cpp/pull/17824)
good thing i'm never touching that repo ever again, or only to steal implementations
reply
I've had the opportunity to do some fun playing around with zdnn on a z17. I've not tried the Spyre cards, but the on chip accelerator for Telum II is surprisingly capable with smaller models. I initially thought you would basically NEED the Spyre cards to get anything significant running but I was wrong.
Congrats for 1.0.0.
> GGML is a C library for machine learning (ML) - the "GG" refers to the initials of its originator (Georgi Gerganov).
https://github.com/rustformers/llm/blob/main/crates/ggml/REA...
For example, DJ Delorie's djgpp project, built in 1989, was how I got into "real" programming on MS DOS.
Fun story time. I started programming on Turbo Pascal. I had read some C code from those demo scene guides, so felt I understood it fairly well by the time I got my hands on a copy of djgpp. I decided to port a relatively simple program I had written in Turbo Pascal, about 300 LOC, line-by-line. It was a small demo just doing a simple rotating, texture-mapped triangle.
I carefully converted it, and went over it several time one line at a time, fixing it until I had read over it three times and was happy.
I then ran the djgpp compiler...
Now, what I hadn't realized at the time was the difference between C and C++. The examples I had seen was C, and I didn't really know what C++ was.
The compiler churned on my code and spit out about 3500 errors and warnings (it counted for me).
I was shocked. How did my 300 line C program manage to produce over 10 errors and warnings per line, on average?!
I spent a long time trying to figure this one out. The first error was inside the first function, and the error made no sense to me.
After several days of carefully going over each line and banging my head against the table, I suddenly saw it. There, at the very start of the program, before the line the first error or warning pointed to.
In Turbo Pascal I had used const to define things like screen width and height. So I'd have
Well, C doesn't have const, but I had picked up it had #define. However, my Turbo Pascal habits had kicked in, and so I had written I removed the two semi-colons and the program compiled and ran just fine...I ended up staying in Turbo Pascal, I preferred a compiler that gave me sensible errors and quickly. I did pick up C++ much later, and I guess ironically came to appreciate its const correctness.
And we can only hope that our modern robber barons will use their ill-gotten gains to create something as beneficial as Carnegie libraries.
Imagine trying to name a file format for yourself even if it wasn't yours to begin with...
> During the start of his work on the system, he stored the files under the name "Freax" for about half of a year. Torvalds had already considered the name "Linux", but initially dismissed it as too egotistical.
https://en.wikipedia.org/wiki/History_of_Linux#Naming
> Minimal
> We like simplicity and aim to keep the codebase as small and as simple as possible
I'm glad their philosophy extends to the website as well. Genuinely refreshing.
Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
holy fucking shit I just checked and they really did this LOL (https://github.com/ggml-org/llama.cpp/pull/17824)
good thing i'm never touching that repo ever again, or only to steal implementations
anyway you'll address me as saar
That seems borderline perfect to me.
Besides, some fancy AI generated landing page would signal even lower effort.