ggml (original) (raw)

This is the first step towards full GPU and custom hardware inference support (see ggml-org/llama.cpp#915)

The idea is to be able to export the ggml computation graphs (ggml_cgraph) into standalone .ggml files.
These files can be later imported by a separate application and evaluated based on the available hardware / framework (CUDA, Metal, WebGPU, etc.). The computation graph contains everything necessary to perform the inference:

model weights
operations
work buffers
sizes + offsets
memory layout

As an example, we export the MNIST computation graph from the mnist example into the file mnist.ggml:

$ ./bin/mnist ./models/mnist/ggml-model-f32.bin ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

Next, using the mnist-cpu tool, we load the graph and re-evaluate it on the CPU using ggml_graph_compute():

./bin/mnist-cpu ./mnist.ggml ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

Or we can run it on the Apple Silicon GPU using Metal:

./bin/mnist-mtl ./mnist.ggml ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

Here is a sample run:

$ ./bin/mnist ./models/mnist/ggml-model-f32.bin ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

mnist_model_load: loading model from './models/mnist/ggml-model-f32.bin' mnist_model_load: ggml ctx size = 1.52 MB main: loaded model in 1.02 ms

_ _ _ _ _ _ _ _ _ * * * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * * _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * * * _ _ _ _ _ _ _ _ _

ggml_graph_dump_dot: dot -Tpng mnist.dot -o mnist.dot.png && open mnist.dot.png

magic 67676d6c version 1 leafs 5 nodes 6 eval 6120

TYPE OP NDIMS NE0 NE1 NE2 NE3 NB0 NB1 NB2 NB3 DATA NAME f32 NONE 2 500 10 1 1 4 2000 20000 20000 0x1201877d0 fc2_weight f32 NONE 2 784 500 1 1 4 3136 1568000 1568000 0x120008100 fc1_weight f32 NONE 1 784 1 1 1 4 3136 3136 3136 0x11e809f00 input f32 NONE 1 500 1 1 1 4 2000 2000 2000 0x120186f00 fc1_bias f32 NONE 1 10 1 1 1 4 40 40 40 0x12018c6f0 fc2_bias

ARG TYPE OP NDIMS NE0 NE1 NE2 NE3 NB0 NB1 NB2 NB3 NTASKS DATA NAME DST f32 MUL_MAT 1 500 1 1 1 4 2000 2000 2000 1 0x11e80ac40 node_0 SRC0 f32 NONE 2 784 500 1 1 4 3136 1568000 1568000 0 0x120008100 fc1_weight SRC1 f32 NONE 1 784 1 1 1 4 3136 3136 3136 0 0x11e809f00 input

DST f32 ADD 1 500 1 1 1 4 2000 2000 2000 1 0x11e80b510 node_1 SRC0 f32 MUL_MAT 1 500 1 1 1 4 2000 2000 2000 1 0x11e80ac40 node_0 SRC1 f32 NONE 1 500 1 1 1 4 2000 2000 2000 0 0x120186f00 fc1_bias

DST f32 RELU 1 500 1 1 1 4 2000 2000 2000 1 0x11e80bde0 node_2 SRC0 f32 ADD 1 500 1 1 1 4 2000 2000 2000 1 0x11e80b510 node_1

DST f32 MUL_MAT 1 10 1 1 1 4 40 40 40 1 0x11e80c6b0 node_3 SRC0 f32 NONE 2 500 10 1 1 4 2000 20000 20000 0 0x1201877d0 fc2_weight SRC1 f32 RELU 1 500 1 1 1 4 2000 2000 2000 1 0x11e80bde0 node_2

DST f32 ADD 1 10 1 1 1 4 40 40 40 1 0x11e80c7e0 node_4 SRC0 f32 MUL_MAT 1 10 1 1 1 4 40 40 40 1 0x11e80c6b0 node_3 SRC1 f32 NONE 1 10 1 1 1 4 40 40 40 0 0x12018c6f0 fc2_bias

DST f32 SOFT_MAX 1 10 1 1 1 4 40 40 40 1 0x11e80c910 probs SRC0 f32 ADD 1 10 1 1 1 4 40 40 40 1 0x11e80c7e0 node_4

mnist_eval: exported compute graph to 'mnist.ggml' main: predicted digit is 3

$ dot -Tpng mnist.dot -o mnist.dot.png && open mnist.dot.png

CPU (via ggml)

$ ./bin/mnist-cpu ./mnist.ggml ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ * * * * * _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ * * _ _ _ * * _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ * _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ * _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * * _ _ _ * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * * * * _ _ _ _ _ _ _

ggml_graph_import: loaded leaf 0: ' fc2_weight', 2 dims, 20000 bytes ggml_graph_import: loaded leaf 1: ' fc1_weight', 2 dims, 1568000 bytes ggml_graph_import: loaded leaf 2: ' input', 1 dims, 3136 bytes ggml_graph_import: loaded leaf 3: ' fc1_bias', 1 dims, 2000 bytes ggml_graph_import: loaded leaf 4: ' fc2_bias', 1 dims, 40 bytes ggml_graph_import: loaded node 0: ' node_0', 1 dims, 2000 bytes ggml_graph_import: loaded node 1: ' node_1', 1 dims, 2000 bytes ggml_graph_import: loaded node 2: ' node_2', 1 dims, 2000 bytes ggml_graph_import: loaded node 3: ' node_3', 1 dims, 40 bytes ggml_graph_import: loaded node 4: ' node_4', 1 dims, 40 bytes ggml_graph_import: loaded node 5: ' probs', 1 dims, 40 bytes main: predicted digit is 6

Metal

$ ./bin/mnist-mtl ./mnist.ggml ../examples/mnist/models/mnist/t10k-images.idx3-ubyte

_ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ * * * _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * * * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * _ _ _ _ _ _ _ _ _ _ _

ggml_cgraph_import: loaded leaf 0: ' ggml_cgraph_import: loaded leaf 1: ' ggml_cgraph_import: loaded leaf 2: ' ggml_cgraph_import: loaded leaf 3: ' ggml_cgraph_import: loaded leaf 4: ' ggml_cgraph_import: loaded node 0: ' ggml_cgraph_import: loaded node 1: ' ggml_cgraph_import: loaded node 2: ' ggml_cgraph_import: loaded node 3: ' ggml_cgraph_import: loaded node 4: ' ggml_cgraph_import: loaded node 5: ' mnist_mtl_init: allocating mnist_mtl_init: using MPS mnist_mtl_init: allocated mnist_mtl_init: allocated mnist_mtl_init: allocated mnist_mtl_eval: evaluating mnist_mtl_eval: encoding node mnist_mtl_get_buffer: data tensor ' mnist_mtl_get_buffer: data tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: encoding node 1, op = mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: data tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: encoding node 2, op = mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: encoding node mnist_mtl_get_buffer: data tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: encoding node 4, op = mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: data tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: encoding node mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_get_buffer: eval tensor ' mnist_mtl_eval: time elapsed = 0.001637 mnist_mtl_eval: probs[ 0] = 0.000000 mnist_mtl_eval: probs[ 1] = 0.000000 mnist_mtl_eval: probs[ 2] = 0.000000 mnist_mtl_eval: probs[ 3] = 0.000000 mnist_mtl_eval: probs[ 4] = 0.000000 mnist_mtl_eval: probs[ 5] = 0.000000 mnist_mtl_eval: probs[ 6] = 0.000000 mnist_mtl_eval: probs[ 7] = 0.000000 mnist_mtl_eval: probs[ 8] = 0.000000 mnist_mtl_eval: probs[ 9] = 1.000000 mnist_mtl_free: deallocating main: predicted digit is 9 fc2_weight', 2 dims, 20000 bytes fc1_weight', 2 dims, 1568000 bytes input', 1 dims, 3136 bytes fc1_bias', 1 dims, 2000 bytes fc2_bias', 1 dims, 40 bytes node_0', 1 dims, 2000 bytes node_1', 1 dims, 2000 bytes node_2', 1 dims, 2000 bytes node_3', 1 dims, 40 bytes node_4', 1 dims, 40 bytes probs', 1 dims, 40 bytes data buffer, size = 1594896 eval buffer, size = 9120 results buffer, size = 40 0, op = MUL_MAT fc1_weight', offs = 20512, size = 1568000 input', offs = 1588628, size = 3136 node_0', offs = 1536, size = 2000 ADD node_0', offs = 1536, size = 2000 fc1_bias', offs = 1591880, size = 2000 node_1', offs = 3792, size = 2000 RELU node_1', offs = 3792, size = 2000 node_2', offs = 6048, size = 2000 3, op = MUL_MAT fc2_weight', offs = 396, size = 20000 node_2', offs = 6048, size = 2000 node_3', offs = 8304, size = 40 ADD node_3', offs = 8304, size = 40 fc2_bias', offs = 1593996, size = 40 node_4', offs = 8608, size = 40 5, op = SOFT_MAX node_4', offs = 8608, size = 40 probs', offs = 8912, size = 40 probs', offs = 8912, size = 40