ThinkColorful

Running Microsoft’s 1-Bit BitNet LLM on My Dell R7910 – A Self-Hosting Adventure

June 11, 2025 Uncategorized

So Microsoft dropped this 1-bit LLM called BitNet, and I couldn’t resist trying to get it running on my new homelab server. Spoiler alert: it actually works incredibly well, and now I have a pretty capable AI assistant running entirely on CPU power! For the demo click on the plus button on the bottom right ;)

My Setup (And Why This Matters)

I’m running this on my Dell Precision Rack 7910 – yeah, it’s basically a workstation crammed into a rack case, but hey, it works! Here’s what I’m working with:

My Dell R7910:

  • Dual Xeon E5-2690V4 processors (28 cores total)
  • 64GB ECC RAM
  • Running Proxmox VE
  • Already hosting Nextcloud, Jellyfin, and WordPress

The cool thing about BitNet is that it doesn’t need fancy GPU hardware. While I’m running it on dual Xeons, you could probably get away with much less.

Minimum specs you’d probably want:

  • Any modern 4+ core CPU
  • 8GB RAM (though 16GB+ is better)
  • 50GB storage space
  • That’s literally it – no GPU required!

What the Heck is BitNet Anyway?

Before we dive in, let me explain why I got excited about this. Most AI models use 32-bit or 16-bit numbers for their “weights” (basically the model’s learned knowledge). BitNet uses just three values: -1, 0, and +1.

Sounds crazy, right? But somehow it works! The 2 billion parameter BitNet model:

  • Uses only ~400MB of RAM (my Llama models use 4-8GB+)
  • Runs 2-6x faster than similar models
  • Uses way less power
  • Still gives pretty decent responses

I mean, when I first heard “1-bit AI,” I thought it would be terrible, but Microsoft’s research team clearly knew what they were doing.

The Journey: Setting This Thing Up

Step 1: Creating a Container for BitNet

Since I’m already running a bunch of services running on Proxmox on my R7910, I decided to give BitNet its own LXC container. This keeps things clean and prevents it from messing with my other stuff.

In Proxmox, I created a new container with these specs:

  • Template: Ubuntu 22.04 LTS
  • CPU: 16 cores (leaving 12 for my other services)
  • Memory: 32GB (plenty of headroom)
  • Storage: 80GB

Important: You need to edit the container config file to add these lines, or the build will fail:

# Edit /etc/pve/lxc/[YOUR_CONTAINER_ID].conf
features: nesting=1
lxc.apparmor.profile: unconfined

Trust me, I learned this the hard way after wondering why cmake was throwing mysterious errors!

Step 2: Getting the Environment Ready

First things first – we need the right tools. BitNet is picky about its build environment:

# Basic stuff
apt update && apt upgrade -y
apt install -y curl wget git build-essential cmake clang clang++ libomp-dev

Now here’s where I made my first mistake – I tried to use Python’s venv initially, but BitNet’s instructions specifically mention conda, and there’s a good reason for that. Just install Miniconda:

cd /tmp
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc

Step 3: The BitNet Installation Saga

This is where things got interesting. The GitHub instructions look straightforward, but there are some gotchas:

mkdir -p /opt/bitnet && cd /opt/bitnet

# This --recursive flag is CRUCIAL - don't skip it!
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

# Create the conda environment
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp
pip install -r requirements.txt
pip install huggingface_hub

Now for the fun part – downloading the model and building everything:

# Download the official Microsoft model
mkdir -p models
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T

# Build the whole thing
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

Pro tip: This build step takes a while. On my dual Xeons, it was about 10 minutes of heavy CPU usage. Grab a coffee – it’s compiling a ton of optimized C++ code.

Step 4: Testing My New AI

Once the build finished, I had to try it out:

python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "Hello, how are you?" -cnv

And it worked! The responses weren’t GPT-4 quality, but they were coherent and surprisingly good for something running entirely on CPU. My partner thought I was connected to a service like openAI because the responses were so fast and the resource usage was so low ^_^

But I didn’t want to just run it in a terminal. I wanted to integrate it with AnythingLLM that I already had running.

Step 5: Making BitNet Play Nice with AnythingLLM

Here’s the cool part – BitNet comes with a built-in API server:

python run_inference_server.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p 8080 --host 0.0.0.0

Then in AnythingLLM, I just added it as a “Generic OpenAI” provider:

  • API Endpoint: http://[my_container_ip]:8080
  • Model Name: BitNet-b1.58-2B-4T
  • Token Context Window: 4096 (can be adjusted)
  • Max Tokens: 1024 (can be adjusted)

And boom – I had BitNet responding to queries through AnythingLLM’s nice web interface!

Step 6: Making It Actually Reliable

Running things manually is fun for testing, but I wanted this to be a proper service. So I created a systemd service:

sudo nano /etc/systemd/system/bitnet.service

[Unit]
Description=BitNet LLM API Server
After=network.target
Wants=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/bitnet/BitNet
Environment=PATH=/root/miniconda3/envs/bitnet-cpp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/root/miniconda3/envs/bitnet-cpp/bin/python run_inference_server.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p 8080 --host 0.0.0.0
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable bitnet.service
sudo systemctl start bitnet.service

Now BitNet automatically starts when the container boots, and if it crashes, systemd brings it right back up.

The Results: How Does It Actually Perform?

I’ve been running this setup for a while now, and I’m honestly impressed. On my R7910:

  • Response time: Usually 1-3 seconds for short responses when running in the container itself.
    • When connected to anythingllm given the context windows and max tokens I put in it averaged about 7-8 seconds before responding but then would output that response relatively quickly
  • Memory usage: Steady ~400MB as advertised
  • CPU usage: Spikes during inference, then drops to almost nothing
  • Quality: Good enough for basic tasks, coding help, and general questions

It’s not going to replace GPT-4 for complex reasoning, but for a lot of everyday AI tasks, it’s surprisingly capable. And the fact that it’s running entirely on my own hardware with no API calls or subscriptions? That’s pretty sweet.

Demo!!

Click the plus button on the bottom right!

Lessons Learned and Gotchas

Things that tripped me up:

  1. Forgetting the --recursive flag when cloning – this downloads necessary submodules
  2. Not installing clang – the error messages weren’t super clear about this
  3. Trying to use venv instead of conda – just follow their instructions!
  4. Container permissions – those LXC config additions are crucial

Performance tips:

  • Give it plenty of CPU cores if you can
  • 32GB RAM is probably overkill, but it’s nice to have headroom
  • The i2_s quantization seems to be the sweet spot for quality vs speed

What’s Next?

I’m planning to experiment with:

  • Different quantization types (tl1 vs i2_s)
  • Running multiple model variants simultaneously
  • Maybe trying some fine-tuning if Microsoft releases tools for that

The self-hosting AI space is moving fast, and BitNet feels like a real game-changer for those of us who want capable AI without needing a mortgage-sized GPU budget.

Wrapping Up

Setting up BitNet on my Dell R7910 turned out to be way more straightforward than I expected, once I figured out the few gotchas. If you’ve got a decent CPU and some spare RAM, I’d definitely recommend giving it a shot.

Having a capable AI assistant running entirely on your own hardware is pretty liberating. No API keys, no usage limits, no privacy concerns about your data leaving your network. Just pure, self-hosted AI goodness.

Plus, there’s something satisfying about telling people your AI assistant is running on a 1-bit model that uses less RAM than Chrome with a few tabs open!