{"id":679998,"date":"2026-11-10T01:16:15","date_gmt":"2026-11-10T01:16:15","guid":{"rendered":"https:\/\/www.newsbeep.com\/ie\/679998\/"},"modified":"2026-11-10T01:16:15","modified_gmt":"2026-11-10T01:16:15","slug":"lawyer-builds-private-ai-server-with-eight-used-nvidia-v100-gpus-up-to-215-tokens-per-second-to-keep-legal-files-offline-costing-about-2400-per-four-gpu-board","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ie\/679998\/","title":{"rendered":"Lawyer builds private AI server with eight used Nvidia V100 GPUs \u2014 up to 215 tokens per second to keep legal files offline, costing about $2,400 per four-GPU board"},"content":{"rendered":"

Reddit<\/a> about their private AI server built from eight used 2018-era Nvidia Tesla V100 data center GPUs<\/a> so case files never have to leave the office. With two 4-card SXM2 NVLink boards, four PEX switches, and SlimSAS cables everywhere \u2014 plus one burnt power connector \u2014 the builder now runs the open DeepSeek V4.1 Flash model with every expert in GPU memory, at up to 215 tokens per second. The four-GPU boards cost about $2,400 apiece when the builder bought them.<\/p>\n

Lawyer’s V100 homelab, update: two 4-card SXM2 NVLink boards, four PEX switches, SlimSAS everywhere, one burnt power connector, and a 250 GB model running entirely in VRAM at 200+ tok\/s<\/a>\u00a0from\u00a0r\/homelab<\/a><\/p>\n

The eight Tesla V100-SXM2-32GB cards, each with 32GB of HBM2 at 900 GB\/s, plug into SXM2, Nvidia\u2019s mezzanine socket for data center GPUs, rather than PCIe slots, and are split across two baseboards. Inside each board, all four cards are NVLinked \u2014 NVLink is Nvidia\u2019s direct GPU-to-GPU link \u2014 for 128GB per board, but the two boards only communicate over PCIe. Each board connects to the host through two PEX (PLX) switch cards over SlimSAS cables, four switches in all, with each switch handling two GPUs at PCIe 3.0 x8 speeds over its own x16 uplink.<\/p>\n

<\/p>\n

The given prices are already in the past, though: used V100 SXM2 32GB cards are now listed on eBay for about $900, more than double the roughly $400 apiece from the post, and listings suggest prices are still climbing. The system also already has some other necessary components: a Gigabyte Threadripper Pro motherboard with a 12-core 3945WX CPU to match, 8x 32GB DDR4 RDIMM, one NUMA node, and two PSUs, 2,800 W in total, are on separate regular office circuits. This is a real-world case for used data center hardware as a cheap route to getting lots of fast VRAM together in one place.<\/p>\n

Latest Videos FromTom’s Hardware<\/p>\n

The server is currently in daily use for first-pass work on case files: it reads a folder of records, pulls out facts, dates, and numbers, summarizes them, and takes a first swing at routine drafting. \u201cNothing goes out the door without a real review,\u201d which is a good human-in-the-loop policy in general. It also helps ensure privacy for cases where client files can\u2019t be sent to a cloud or frontier model. RAG databases and tool calls are wired in; the operator can drive the system through a Discord-style interface over SSH, in this case from a Windows PC. For now, it handles only one request at a time.<\/p>\n

DeepSeek V4.1 Flash is compressed to mostly 2 bits here, and its weights still total 248.8GB. The model\u2019s \u201cexperts,\u201d the parts that do most of the work for each word, live entirely in GPU memory, while about 110GB of lookup tables, called Engram tables, stay in the system\u2019s regular RAM. Fetching from RAM costs about 0.12 milliseconds per step, which the builder says barely affects the speed numbers.<\/p>\n

You may like<\/p>\n

The first half of the model\u2019s layers (0\u201319) runs on the first SXM2 board and the second half (20\u201339) on the second. Each board handles its heavy traffic over its own NVLink, so only one hand-off per step has to cross the slower PCIe link between the boards. In an earlier post, the builder found that splitting every layer across all eight cards made speeds \u201cfall off a cliff\u201d because of that PCIe hop.<\/p>\n

Performance is surprisingly good for a model this size. Code comes out fastest, at up to 215 tokens per second, because a small draft model built into DeepSeek guesses five tokens ahead, and code is easy to guess. Legal writing, which is what the server is for, runs at up to 118 tokens per second.<\/p>\n

Get Tom’s Hardware’s best news and in-depth reviews, straight to your inbox.<\/p>\n

Long documents take longer to get going: the model reads prompts at about 2,200 tokens per second, so at 128K tokens, the first token arrives after about a minute. It can accept up to 524K tokens of context. Three days before the post, the same model on four of these cards managed just 36\u201338 tokens per second. The jump came with the second board, which could presumably double performance under ideal circumstances, and the builder credits custom kernels and prefill work for the rest. See the table for the full breakdown.<\/p>\n

Swipe to scroll horizontally<\/p>\n

DeepSeek V4.1 Flash on eight Tesla V100s, by prompt length<\/p>\n

Prompt length<\/p>\n

Wait for first token<\/p>\n

Code (tokens\/s)<\/p>\n

Legal writing (tokens\/s)<\/p>\n

1K tokens<\/p>\n

1.2 seconds<\/p>\n

215<\/p>\n

118<\/p>\n

32K tokens<\/p>\n

17.2 seconds<\/p>\n

200<\/p>\n

117<\/p>\n

128K tokens<\/p>\n

59.2 seconds<\/p>\n

183<\/p>\n

111<\/p>\n

256K tokens<\/p>\n

118.7 seconds<\/p>\n

160<\/p>\n

103<\/p>\n

Builder\u2019s measurements, Nov. 5; greedy decoding, 1,024 tokens out, mean of two runs.<\/p>\n

It wasn\u2019t as simple as hooking everything up and turning on the power. The host reset itself twice with a fatal PCIe error on the link to one GPU board, both times with a higher power limit and uncapped GPU clocks. In this state, the cards drew 192\u2013232 W, past the 180 W limit. The best guess, although \u201cnot proven,\u201d is that a board stops responding during load spikes.<\/p>\n

What to read next<\/p>\n

The resets stopped after the builder capped the clocks and removed one board whose power connector had a burn mark; \u201cCheck your connectors, people.\u201d Each card is now capped at 150 W, half of its 300 W rating, with the graphics clock capped at 1522 MHz. A 200 W cap brought no appreciable performance gains, according to tests.<\/p>\n

The system memory hit uncorrected ECC errors, too, which went away after the builder dropped the RAM from DDR4-3200 to DDR4-2666. One GPU was connected at x4 instead of x8 at last check, and the builder plans to reseat it; one commenter traced the same kind of problem on their own V100 to a bad cable.<\/p>\n

The pain was worth the payoff as an estimated 50\u201360% of the lawyer\u2019s existing LLM workflows can now run on the box instead of frontier models. Helping bridge the gap is a second DeepSeek server on two DGX Sparks<\/a>, where the two systems hand work off to each other. The setup does appear to save money, at least when compared to API pricing<\/a>, although the builder says it\u2019s not really faster. However, it will protect client data \u201conce it is self contained, which is the goal.\u201d<\/p>\n

The builder added that the V100 box might also get a 240 V circuit. Beyond power, the system is also on borrowed time with the Nvidia R580 driver branch, which is the last to support Volta<\/a>. For now, the lawyer admits that it\u2019s mostly a hobby, but in the future it may be a business idea. The next step is to handle two or three requests at once, something built and checked on a test system but not yet on the server.<\/p>\n

<\/p>\n

\n

\"Google\n<\/p>\n

<\/a><\/p>\n

Follow Tom\u2019s Hardware on Google News<\/a>, or add us as a preferred source<\/a>, to get our latest news, analysis, & reviews in your feeds.<\/p>\n","protected":false},"excerpt":{"rendered":"Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter A lawyer posted…\n","protected":false},"author":2,"featured_media":679999,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[220,218,219,61,60,80],"class_list":["post-679998","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-ie","tag-ireland","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/679998","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/comments?post=679998"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/679998\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media\/679999"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media?parent=679998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/categories?post=679998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/tags?post=679998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}