Project Summary

Offline AI Brain is an experimental system implementing a local-first autonomous agent. It coordinates a lightweight local LLM (e.g., Llama-3-8B-Instruct via GGUF/Ollama), a sqlite-vector similarity memory layer, and a sandboxed subprocess pipeline to execute commands and scripts locally, providing total privacy and offline resilience.

Overview

Most modern AI assistant frameworks depend on cloud APIs, presenting privacy concerns and requiring persistent web connections. Offline AI Brain explores the limits of what can run locally on an individual desktop computer or laptop.

The system is built on a modular loop: perceive (input and context building), plan (hierarchical task breakdown), retrieve (local vector memory query), execute (sandboxed local python runner), and reflect (evaluating outputs and committing new knowledge to memory).

Key Features

Local Inference Engine: Configured to use llama.cpp or Ollama as a backend, utilizing quantized weights to fit comfortably into consumer VRAM.
Vector Database in SQLite: Implements a simple flat-index cosine similarity search using NumPy embeddings or sqlite-vec extension, storing contextual knowledge and task histories locally.
Sandboxed Code Execution: Safely runs generated Python scripts inside virtual environments to query local files, system status, or process datasets.
Dynamic Context Assembly: Dynamically injects local metadata (time, file lists, operating system details, CPU load) into the system prompt before each run.

System Architecture

The architecture separates cognitive reasoning from state management and tool execution. Below is a simplified representation of the loop:

+------------------+     +-----------------------+     +-----------------------+
|  User Request /  | --> | Dynamic Context       | --> | Local LLM             |
|  Cron Trigger    |     | Builder (SQLite Vec)  |     | (Reasoning / Planning)|
+------------------+     +-----------------------+     +-----------------------+
                                                                   |
                                                                   v
+------------------+     +-----------------------+     +-----------------------+
| Local Database   | <-- | Reflect & Learn       | <-- | Execution Sandbox     |
| (State & Memory) |     | (Result Evaluation)   |     | (Command execution)   |
+------------------+     +-----------------------+     +-----------------------+

Code Implementation

1. Local SQLite Vector Retrieval

Rather than pulling in heavy cloud vector database libraries, we utilize a lightweight local similarity search using Python and NumPy, or standard SQLite storage:

import numpy as np
import sqlite3

def query_local_memory(conn, query_embedding, limit=5):
    cursor = conn.cursor()
    cursor.execute("SELECT id, content, embedding_blob FROM memory_store")
    results = []
    
    for row_id, content, emb_blob in cursor.fetchall():
        embedding = np.frombuffer(emb_blob, dtype=np.float32)
        # Cosine similarity calculation
        dot_prod = np.dot(query_embedding, embedding)
        norm_q = np.linalg.norm(query_embedding)
        norm_e = np.linalg.norm(embedding)
        similarity = dot_prod / (norm_q * norm_e) if norm_q and norm_e else 0
        results.append((row_id, content, similarity))
        
    # Sort by similarity descending
    results.sort(key=lambda x: x[2], reverse=True)
    return results[:limit]

2. Sandboxed Subprocess Runner

Executing code locally is dangerous. To mitigate risk, we invoke commands using isolated environments with restrictive permissions, timeouts, and resource monitoring:

import subprocess
import os

def run_isolated_script(script_path, timeout_seconds=15):
    env = os.environ.copy()
    # Remove write/execution access to sensitive paths in copy env
    env["PATH"] = "/usr/bin:/bin"  # Example for Unix-like systems
    
    try:
        proc = subprocess.run(
            ["python", script_path],
            env=env,
            capture_output=True,
            text=True,
            timeout=timeout_seconds,
            check=True
        )
        return {"success": True, "stdout": proc.stdout, "stderr": proc.stderr}
    except subprocess.TimeoutExpired as e:
        return {"success": False, "error": "Execution timed out", "stdout": e.stdout, "stderr": e.stderr}
    except subprocess.CalledProcessError as e:
        return {"success": False, "error": f"Exit code {e.returncode}", "stdout": e.stdout, "stderr": e.stderr}

Performance & Trade-offs

Operating a fully local system requires balancing accuracy, speed, and hardware constraints:

Model Size	Precision (Quantization)	Inference Speed (Tokens/s)	VRAM Usage	Task Success Rate
Llama-3-8B	Q4_K_M	~42 t/s	5.8 GB	78.4%
Llama-3-8B	Q8_0	~31 t/s	8.9 GB	84.1%
Phi-3-Medium (14B)	Q4_K_M	~22 t/s	9.2 GB	81.9%
Qwen-2.5-7B	Q4_K_M	~45 t/s	5.2 GB	83.6%

Challenges & Solutions

Context Window Exhaustion: As the history grows, local LLMs hit strict context boundaries (typically 8k tokens in GGUF setups). Solution: Implemented a rolling context window where the agent condenses older dialogue history into bullet-point summaries and commits detailed interactions to vector memory.

Hallucinating Tool Syntax: Smaller models frequently output invalid JSON or hallucinate parameters in system commands. Solution: Wrapped all LLM outputs in strict JSON schema validation, feeding parser errors back to the model iteratively until formatting is corrected.

Future Roadmap

Integrate a native web crawler tool (running locally via headless chromium) for offline caching of sites.
Add full support for local multimodal models (like LLaVA) to enable vision tasks.
Build an electron interface for easier system tray integration and notifications.

Offline AI Brain