在物流中本地运行 LLM：Ollama + VPN + 隔离部署✎ Edit

I've burned more GPU budget on "private cloud" demos than I care to admit. At AINNA, where we run NeuralOps infrastructure and a growing fleet of 分离式系统, one of the hardest problems we solve is putting 大 语言 Models inside logistics environments that cannot - and should not - phone home to OpenAI. This article is the technical blueprint I wish I had three years ago: Ollama + VPN + air-gapped deployment, built for real 3PL warehouses, freight brokers, and Malaysian SMEs that handle customer manifests, customs forms, and supplier contracts.

1. Why 物流 Needs 本地 LLMs

物流 data is messy, regulated, and high velocity. A typical 3PL in 马六甲 processes thousands of airway bills, packing lists, and customs declarations every day. Most of those documents contain:

• Customer names, IC numbers, and phone numbers (PDPA-sensitive).
• 供应商 pricing and contract terms (commercially sensitive).
• Customs HS codes, declared values, and origin data (legally sensitive).

Sending that to a public API - even with promises of "zero retention" - is a non-starter for many compliance officers I've worked with. 本地 inference removes that risk. But it also introduces 新 ones: model management, GPU failures, context-window crashes, and VPN latency that can make a chatbot feel like it's typing on a typewriter.

We originally tried a hybrid approach: sensitive extraction on local models, summarization on Groq. It worked until a customs broker asked, "Where exactly did my Bill of Lading go?" That single audit question killed the hybrid 设计. Now our default architecture at AINNA is VPN-first, local-first. 云 LLMs are an explicit opt-in, not a default.

2. The 技术栈: Ollama, WireGuard, and Air-Gapped vLLM

2.1 Ollama as the 边缘 推理 引擎

Ollama is not the fastest inference engine. It is not the most feature-complete. What it is, in my experience, is the most reliable way to get a model running on an edge box in under ten minutes. For logistics clients who need to deploy in a warehouse with no dedicated ML engineer on site, that matters more than raw throughput.

Our standard edge node is a workstation with an NVIDIA RTX A6000 (48 GB VRAM), 128 GB RAM, and a 2 TB NVMe. That box lives in the warehouse rack, behind the client's firewall, reachable only over the AINNA-managed WireGuard VPN. Ollama runs inside Docker with a restricted Modelfile tuned for 结构化提取:

# /models/qwen-logistics/Modelfile
FROM qwen2.5-coder:14b
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 1024
PARAMETER top_p 0.3
PARAMETER repeat_penalty 1.1
SYSTEM "You are a logistics document parser. Extract only the fields requested. Respond in valid JSON. Do not add explanations."

Why qwen2.5-coder:14b? In our benchmarks it beats llama3.1:8b on structured JSON extraction from mixed 英语/马来语/Bahasa documents, and it fits comfortably in 48 GB with headroom for the KV cache. The 14B parameter size is the sweet spot for edge: good enough accuracy, small enough latency, cheap enough hardware.

2.2 VPN-首先, Not 云-首先

Every AINNA deployment starts with a WireGuard 中心-and-spoke topology. The 中心 is a hardened VPS or the client's own NeuralOps 云 托管 node. The spokes are the warehouse edge boxes, 财务 office desktops, and any detached inventory servers. No spoke has a public IP. No spoke can reach the internet directly. All control-plane traffic - model pulls, telemetry, agent orchestration - flows through the 中心.

# /etc/wireguard/wg0.conf on a warehouse spoke
[界面]
PrivateKey = 
Address = 10.200.0.5/32
ListenPort = 51820
DNS = 10.200.0.1

[Peer]
PublicKey = <中心-public-key>
AllowedIPs = 10.200.0.1/32, 10.201.0.0/24
Endpoint = neuralops-中心.ainna.bond:51820
PersistentKeepalive = 25

The AllowedIPs line is the critical security control. It means the warehouse Ollama node can talk to the NeuralOps 中心 and the internal analytics subnet, but not to Google, GitHub, or a rogue mirror. 时间 we need to update a model, we stage the GGUF or safetensors on the 中心, sign it, and push it down through the VPN. The edge node never downloads from the public internet.

2.3 Air-Gapped vLLM Cluster for 重型 Workloads

Ollama is great for single-box inference. For a centralized cluster serving multiple warehouses - what we call the AI 智能体 & LLM 服务器 Layer in NeuralOps - we use vLLM. A typical 中心 cluster is two A100 80GB GPUs in tensor-parallel mode, running llama-3.3-70b AWQ-quantized:

python -m vllm.entrypoints.openai.api_server \
  --model /models/Llama-3.3-70B-Instruct-AWQ-INT4 \
  --served-model-name llama-3.3-70b-logistics \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 32768 \
  --max-num-batched-令牌 8192 \
  --port 8000 \
  --api-key ${VLLM_API_KEY}

This cluster has no outbound NAT. Model weights 实时 on an encrypted NFS mount. 日志 ship to a local Loki instance, never to a SaaS. Backups go to a tape rotation. It is, by 设计, boring infrastructure - and boring infrastructure is what keeps auditors happy.

3. Model 选择 for 物流 Workloads

We do not deploy one model and hope. We run a model router in the AI 智能体 Layer that picks the cheapest capable model for each task. Here is what actually works in production:

3.1 Coding and Agentic 任务

• kimi-k2.7-code:cloud: We use this for complex agent reasoning when the client permits cloud usage. It is expensive but excellent at multi-step planning - e.g., "Reschedule this container shipment around a port strike and update the inventory reservations." On a cloud endpoint we see roughly $0.02 / 1K 令牌 depending on provider. We only route high-value decisions here.
• qwen2.5-coder: Our default on-premise code and extraction model. Handles JSON, SQL generation for the 库存 系统, and light report automation. Runs at ~120 ms/token on the A6000 edge node.

3.2 推理 and 文档 Parsing

• deepseek-coder: Strong for long-context document understanding. We use it for customs declaration checks where the prompt includes 20+ pages of HS-code reference tables. 监视 out: the 236B variant needs serious VRAM. We usually run the 33B distilled version locally.
• llama-3.3-70b: The "safe choice" for mixed 英语/Bahasa conversations in the AI 智能体 仪表盘. Quantized to AWQ INT4, it fits on two A100s and gives acceptable quality for chat and classification. Not the cheapest to run locally - power and cooling are real costs - but cheaper than leaking data.

成本 reality check: A 1,000-token call to a cloud API might cost $0.02 on Groq or $0.03+ on OpenAI. The same call on local Ollama costs $0 in API fees - but you already paid RM 50,000 for the server, RM 800/月 for power, and my team's time to keep it patched. 本地 is not free. It is a capex-and-compliance trade, not a price hack.

4. AINNA's VPN-首先, Hub-and-Spoke vLLM 架构

Here is the pattern we repeat across Malaysian logistics deployments. I call it "中心-and-spoke vLLM."

1. Hub: One or more GPU servers in the client's data center or a dedicated NeuralOps 云 AI/代理 node. Runs vLLM, the model registry, the agent orchestrator, and the central vector database.
2. Spokes: 数据仓库 edge nodes running Ollama for low-latency extraction and offline fallback. Each spoke connects to the 中心 via WireGuard. Spokes can operate autonomously if the VPN drops.
3. 控制 plane: AINNA NeuralOps manages WireGuard keys, Docker image updates, database backups, and 24/7 monitoring. We do not manage the client's network - we secure the overlay.

Client applications talk to the models through a thin OpenAI-compatible client. The base URL points to a VPN address, not a public endpoint:

import openai
import os

client = openai.OpenAI(
    base_url="http://10.201.0.5:11434/v1",
    api_key="ollama",
    timeout=30.0
)

resp = client.chat.completions.create(
    model="qwen2.5-coder:14b",
    messages=[
        {"role": "系统", "content": "Extract consignee, HS code, and declared value as JSON."},
        {"role": "user", "content": doc_text[:6000]}
    ],
    max_tokens=512,
    temperature=0.0,
    response_format={"type": "json_object"}
)
print(resp.choices[0].message.content)

That base_url is reachable only inside the WireGuard tunnel. If an attacker compromises the warehouse LAN, they still cannot reach the model server unless they also have the WireGuard private key and the 中心 authorizes their IP. Defense in depth is not a slogan; it is the only reason I sleep on Friday nights.

5. Token 优化 and 延迟 Tuning

本地 inference saves money only if you are not wasting 令牌. I have seen warehouses burn 30% of their GPU capacity on bloated prompts and oversized context windows.

5.1 上下文 Window Budgeting

Do not set num_ctx = 128k because the model supports it. KV cache scales quadratically with context length in attention, and on consumer GPUs that translates directly to slower generation and higher VRAM use. For airway bill extraction, we cap context at 8,192 令牌. For long-form customs analysis, we chunk documents into 4,000-token windows and run parallel inference, then merge results.

5.2 Batching and KV Cache

vLLM's PagedAttention is the reason we use it for the 中心. We configure max-num-batched-令牌 and max-num-seqs based on observed traffic, not defaults. For a 3PL with 40 concurrent agents, we typically run:

--max-num-seqs 64
--max-num-batched-令牌 16384
--gpu-memory-utilization 0.90

That pushes the A100s to ~85% utilization without OOM crashes. Ollama does not support continuous batching the same way, so for edge nodes we 限制 concurrent requests in the reverse proxy (usually Nginx or Caddy) and queue the rest.

5.3 Quantization 交易-offs

We quantize almost everything that runs locally. AWQ INT4 and Q4_K_M are standard. The quality drop is measurable on coding tasks but rarely on logistics extraction, where the output schema constrains the model heavily. My rule of thumb: if the task is classification or 结构化提取, quantize aggressively. If the task is open-ended reasoning or customer-facing chat, keep more bits.

6. 生产 Failures and What Actually Works

I will not pretend this has been smooth. Here are three failures that shaped our current architecture.

6.1 The 32K 上下文 Crash

We deployed llama-3.3-70b with max-model-len 32768 and told the 财务 team they could dump entire annual ledgers into the 财务 仪表盘. It worked in testing. In production, the first user pasted a 28,000-token spreadsheet and the vLLM process was killed by the OOM killer. Root cause: long context plus high batch size exceeded VRAM. Fix: cap per-request context to 16K, add input length validation in the API gateway, and expose a token counter in the UI so users see the cost before they submit.

6.2 VPN 延迟 Surprises

A client in Johor routed all traffic through a 中心 in Kuala Lumpur because "the cloud is in KL." The round-trip added 35 ms. That does not sound like much, but LLM inference is a sequential token stream. For a 500-token response, 35 ms per token becomes 17.5 seconds of perceived latency. We moved the 中心 closer - same city, same ISP - and latency dropped below 4 ms per token. Lesson: place inference where the users are, not where the cloud provider is.

6.3 Model Drift on Air-Gapped Nodes

边缘 nodes that never touch the internet also never get silent security patches. We once had a spoke running an outdated Ollama version with a known container escape. Because it was air-gapped, the CVE was theoretical - until a technician plugged in a USB drive. Now every detached 系统 has an offline patch schedule: signed update packages staged on the 中心, pushed over VPN, installed during maintenance windows, and 已验证 by AINNA's monitoring layer before the node rejoins the mesh.

7. 安全, PDPA, and Malaysian SME Reality

马来西亚's Personal 数据保护 Act 2010 (PDPA) does not explicitly ban cloud AI, but it requires clear purpose limitation, consent, and security safeguards. In practice, that means a DPO asking uncomfortable questions about data residency. 本地 LLMs give a clean answer: data never leaves the premises.

We layer the usual NeuralOps controls on top:

• SSL/TLS 加密 inside the VPN, because encryption should not stop at the tunnel.
• Docker 隔离 for Ollama and the AI 智能体 仪表盘.
• 全面日志记录 to a local SIEM, not a foreign SaaS.
• Automated 备份 of model configs, fine-tunes, and vector stores.
• 24/7 监控 with alerts for GPU temperature, OOM events, and unauthorized API calls.

For a 马六甲 SME running a single 3PL warehouse, the total cost of a local Ollama deployment - server, switch, UPS, VPN setup, and managed support - is roughly RM 45,000–65,000 first year. That is not cheap. But a single PDPA fine or a leaked supplier contract can cost far more. We do not sell local LLMs as a cost saver. We sell them as a risk reducer.

8. Bottom 产线

运行中 LLMs locally in logistics is not a hobbyist project. It is an infrastructure decision with real trade-offs in cost, latency, security, and model quality. My production-tested recipe is:

• Use Ollama on warehouse edge nodes for fast, offline, 结构化提取.
• Use vLLM on a central GPU 中心 for heavy reasoning and multi-tenant agents.
• Connect everything with a WireGuard VPN 中心-and-spoke; never expose model APIs to the public internet.
• 保留 the cluster air-gapped by default, with signed, staged updates.
• Pick models pragmatically: qwen2.5-coder for edge coding/extraction, deepseek-coder for long documents, llama-3.3-70b for general chat, and cloud models like kimi-k2.7-code:cloud only when the business case justifies the risk.
• Optimize 令牌, context windows, and batch sizes like you optimize SQL queries - because wasted 令牌 are wasted GPU dollars.

At AINNA, this pattern powers our 分离式系统, 电子商务 系统, 库存 系统, and the AI 智能体 Dashboards we ship to logistics clients across 马来西亚. It is not the flashiest architecture. It is the one that still works at 2 AM when a customs officer asks for an 审计追踪 and the internet is down.

That is why I build local-first.