Category Archives: Linux

Experiences with Linux, primarily Debian and Ubuntu.

Digoo IP Camera Teardown and Links

Posted on by
Reply

I was just looking at unfinished posts and noticed that I’d taken, but not published, a bunch of notes I’d made earlier this year in hopes of hacking better firmware onto the Digoo BB-M2 WiFi PTZ Security Camera.I gave up on the quest, but here are my notes, with minimal editing.

Someone mentioned that the Chinese language page for Netcam360 has a link to the IPC-SDK. When I downloaded it and looked inside, I saw client-side code, but there was also a self-extracting archive called “HSmartLink Win32 SDK” and I remembered the PCB marking started with “HSL.” Searching for HSmartLink brings up hsmartlink.com, which, among other things, has IP webcams! The i9812 looks like a good match for my camera!

Unfortunately, no sign of any firmware updates. I checked the .cn version of the site too. It doesn’t seem as up to date on products (i9812 isn’t listed), and while there is more info in support section, it is still quite sparse. Page that looks like it is intended to link to downloads hasn’t been updated since 2015

Company is “Shenzhen Hsmartlink Technology Co. Ltd”

FCC Database listing for company

So what is the relationship to NetCam360 (check whois & IP ) and they mysterious APKLink?

Nmap Pobe

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-06 18:38 PST
Nmap scan report for 10.31.1.124
Host is up (0.0043s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
23/tcp open telnet
81/tcp open hosts2-ns
MAC Address: E0:B9:4D:8F:61:6C (Shenzhen Bilian Electronicltd)

Nmap done: 1 IP address (1 host up) scanned in 0.49 seconds

Teardown Inventory

Photos

Ingenic

Module with Mediatek MT7601UN

Unknown

  • 15UDN8WY, ULN2803AG 18Pin in 2-row SMD
  • Darlington Transistor Array: http://www.ti.com/lit/ds/symlink/uln2803a.pdf
  • Probably used for driving PT motors

Atmel 542

  • 24C02N, SU27D
  • AT24C02
  • Two-wire serial EEPROM (2K)

Vertical PCB

  • SF1810-002, www.sufeitech.com. PCB antenna?

oosilicon or dosilicon?

Other

Berries, Bananas, Oranges and C.H.I.P.s

Posted on by
Reply

I’m in the midst of finding a low-cost embedded linux platform to standardize on for IoT/network device experiments. I’ve been using Debian on Marvel Kirkwood ARM devices as inexpensive, low power servers and network appliances for a number of years now. I’ve also used OpenWRT for my home network and other network appliances.

A few years ago though, I grew tired with the difficulty of maintaining/upgrading Debian on my ARM devices and replaced them with a small AMD based x86 box for my main home server. There have still been upgrade hassles, but overall, I think it reduced the fussyness and frustration enough to be worthwhile.

At about the same time, I started growing interested in using OpenWRT more widely, and I was encouraged when the LEDE Project forked off for reasons that included a desire to fix some of the things that frustrated me about OpenWRT. I liked the simplicity vs a full linux distribution. And the minimal system requirements meant OpenWRT could run on small, inexpensive, relatively low-power devices. Last year, I started work on trying to get OpenWRT running on some cheap linux-based network music servers. Before going further, I paused to think about what’s really important.

I realized that, for my purposes the fact that OpenWRT/LEDE runs on a wide variety of compact, inexpensive devices was as much a liability as an asset. It meant more of the communities effort was invested in supporting the hardware, which meant that a lot of the supported hardware and some of the higher-level software wasn’t well supported, or supported at all. Furthermore, the constrained resources on the various hardware made it harder to compile, package, and deploy software packages that were well supported on full linux distributions. These tradeoffs between cost and ease of development/deployment might be acceptable if I was working on a high-volume device, but I’m not. I’m working on one-offs for my own projects. At most, I might make/deploy a dozen similar devices. How much hassle was I inflicting on myself in exchange for saving $20, or $200 vs an easier to use alternative?

As for the alternative, figuring that out is really the point of this post. As I see it, there are three major options.

  • RaspberryPi
  • Chinese ARM (Allwinner, Rockchip, Amlogic, etc)
  • NextThing

And my current impressions on each option:

RaspberryPi

Anyone who has read this far is probably well aware of the success of the RaspberryPi and its offspring, like the RPi Zero and RaspberryPi v3. Community support is surely the strongest out of any of the inexpensive ARM boards. Unfortunately, I don’t find the boards a great fit for most of my projects.

The Raspberry Pi v3 is quite capable in terms of processing and video. Of those the processing power is probably the most relevant to me, but the projects that call for that also call for better network and disk IO, rather than the shared USB2 on the v3. Also, at $35-40 it’s more expensive than I’d like. It’s a bit on the large size too, and power consumption can be a bit high.

The RPi Zero is cheap ~$5 but availability has been poor, and it still seems hard to get. It’s small, and power consumption is better than the v3. Unfortunately, the Zero seems targeted at people who want to do more hardware hacking than I do. IO is minimal. There is no network, and no included support for USB peripherals (it can be a USB peripheral though).

It’s been almost a year since these two devices were released, so there is a good chance we’ll be seeing upgrades soon. Still, I’m not holding my breath.

Chinese ARM Boards ( with SoCs from Allwinner, Rockchip, etc ).

If I remember right, ARM SoCs from Allwinner and Rockchip first appeared at about the same time as the first RaspberryPi, in android TV boxes and sticks. The first developer board I remember was the Allwinner-based Cubieboard. Since then, there have been many variants from a half-dozen board designers like FriendlyARM, (SinoVOIP) Bannana Pi, (Xunlong) Orange Pi.

Unfortunately Allwinner, and most/all of the other SoC makers aren’t great about providing documentation or source code for their linux variants, and the board makers don’t/can’t exert much leverage. Allwinner chips seem to have a decent community around them, but there are so many SoC variants and board variants that a lot of effort is burned trying to keep up with them. The linux-sunixi (Allwinner) community is making a concerted effort at getting support into the mainline linux kernel, which should both make ongoing support easier and broaden the base of potential contributors.

Next Thing C.H.I.P.

Next Thing grabbed a lot of attention in late 2015 for their promise of a $9 computer called the C.H.I.P. After some challenges, they delivered on their Kickstarter and went on to produce more of the boards, as well as the PocketCHIP and upcoming CHIP Pro.

The C.H.I.P. includes 4GB of fast onboard flash, 2.4GHz 802.11n WiFi, Bluetooth 4.0,  512MB RAM and multi-format video out (built-in composite, or VGA & HDMI with adapters).

NextThing used yet another Allwinner ARM SoC (R8), and by that criteria, belong in the previous category. What sets them apart from other people spinning boards with cheap ARM SoCs is that they have invested time and money into software and documentation — including a full datasheet for the GR8 (R8 + 256MG RAM in package) used on their C.H.I.P. Pro.

They are currently out of stock on the CHIP and all they can say about backorders is that they are expected to ship in Q1 2017.

Conclusions

I already have an RPi v3, which I’ll find some semi-permenant use for. the C.H.I.P. is attractive because of it’s manufacturer support, despite being out of stock, and having an older Cortex-A8 CPU core. I’m not in a huge hurry, so I think I’ll order a couple now and hope they arrive before I wish they were here sooner.

I’m also tempted to buy an Orange Pi Zero, or one of the FriendlyARM NanoPi variants, for comparison.

AirMobi iReceiver Teardown

Posted on by
1

I’ve ended up with five small, inexpensive ($7-15 each) routers, running OpenWrt and only really need two of them, so I’ve been thinking of ways to use the others. One of my ideas was to get an external USB DAC, install Shairport-Sync, and use it as an AirPlay receiver for my car stereo, eliminating the need to connect an audio cable to my phone, and avoiding the mediocre sound quality of Bluetooth audio. It hasn’t quite worked out that way though…

While looking for an inexpensive (>$20), compact USB DAC with reasonable quality, I discovered there were integrated commercial products that already do what I planned to do. I already knew there were Apple-approved MFi-certified devices, but they tended to be expensive. I discovered there were cheaper devices using Shairport, but they tended to start at $30+.

Damaged while trying to open the case.

Damaged while trying to open the case.

With a little more digging though, I found a device called the iReceiver, from AirMobi that sells for as little as $12!!!. According to the scant marketing materials, it has a 24-bit Wolfson DAC. I was surprised I couldn’t find anyone who’d opened one up to see what was inside. I did find an Amazon review from someone complaining that the usb power connector had broken off on theirs, and the included photo showed it had a Ralink RT5350F WiFi SoC, which gave me hope that it would be hackable. So, I bought one.

Before opening it up, I tried it out. It works as promised. It defaults to broadcasting an unsecured WiFi network. Once connected, it shows up as an AirPlay receiver in iTunes, etc. From there, you can connect it to some powered speakers, select it and start playing music. The audio quality doesn’t suck (no obvious noise, clipping, or distortion), and in my limited use, there were fewer dropouts that I’m used to with Bluetooth.

Beyond that, there are various configuration options available through a browser based interface. There are no audio-related settings at all. Most of the settings are networking related. You can rename and secure the WiFi network with a password (good), WPS (bad) and by limited connections to specific devices by MAC address (meh). You can also connect to an existing network (good), and, optionally, extend it (meh). This seems like a good point to mention that it also works as a DNLA “renderer” (DNLA is a more open standard than AirPlay, making this useful to Windows and Linux devices, and Android phones with an appropriate app)

Of course, I didn’t buy it to use it with the stock firmware, so after trying it out, I opened it up to take a look inside. In the process, I managed to tear the translucent plastic that was affixed to the top of the case with adhesive. With the trim removed, it was easy to pry off the top, revealing the single PCB inside.

Version 2

As I expected, it is based on the obsolete but inexpensive and popular Ralink RT5350F WiFi SoC which includes a CPU and 802.11n WiFi.

  • Marked “RT5350F, TP08P40609, 1408STA”
  • 360MHz MIPS 24KEc CPU
  • 802.11n 1T/1R (1×1:1) 2.4 GHz 150Mbps MAC/BB/PA/RF
  • 5-port 10/100 Mbps Ethernet switch w/ 5 10/100 PHYs (unused)
  • USB 2.0 host/client (unused)

This is complimented by a modest, but sufficient 32MB of RAM and 8MB of flash memory to hold the firmware.

  • RAM
    • Marked: “EtronTech EM63A165TS-6G”
    • 255Mbit 16Mx16 5, 6, 7ns 166MHz SDRAM
  • Flash
    • Marked: “MXIC MX, 25L6406E, M2I-12G, 30392500, K141983”
    • Macronix MX25L6406E
    • 64Mbit NOR Flash
    • 4KB sector, 64KB block, 2.7-3.6v, H/W Hold
    • 1 or 2 bit bus, 86MHz x1 bus, 80MHz x2

The other major component is a Wolfson WM8960 CODEC to provide the audio output. This chip debuted in 2006, and includes 24-bit stereo DAC and ADC converters supporting sample rates up to 48Khz, a 40mW headphone driver, and a 1W Class D speaker driver.

Despite being a 24-bit DAC, the specified SnR of 98dBS matches that of the 16-bit TI/Burr Brown PCM2705 DAC used in the original AirportExpress, rather than of a modern, premium 24-bit DAC used in more recent AirportExpress’s. Oh well. Good enough for my purposes. Most of what I’m playing is compressed AAC files derived from 16-bit sources, and, AirPlay only passes 16-bit anyway. Beyond that, the design of the rest of the circuitry matters, and I’m not qualified to analyze it, nor am I equipped or inclined to try and measure it.

Beyond that, I see two inductors on the board (one of which is cracked). My guess is that these are part of some small switch mode power supplies, perhaps one for the digital section, and the other for the analog. There are two small LEDs to indicate device status and two momentary switches, one to reset the device, and the other to trigger WPS. It looks like it uses a single ceramic chip antenna for the WiFi.

There are a few unused pads for components, eight test points (half seemingly to do with power) and four unused holes for pin headers that I suspect provide a serial console.

That’s really it for the hardware. I’ve already started poking more deeply into the software and investigating the suspected serial console, and I hope to have another post soon documenting what I found.

IMG_9679

iReceiver Elsewhere

ASRock AM1B-ITX + AMD Kabini Sempron 3850 Linux Notes

Posted on by
4

Earlier this summer I built a new home server using an ASRock AM1B-ITX motherboard and a AMD Kabini Sempron 3850 CPU.

To make a long story short, this motherboard doesn’t work well for my intended use as a headless Linux server. The problems are manifold and interconnected:

  • If I boot headless, it decides the integrated GPU isn’t being used.
  • Once it decides the integrated GPU isn’t being used, it tries to use a PCI Express GPU, which it doesn’t find.
  • At some point, it also reactivates compatibility mode.
  • With compatibility mode activated it is, ironically, incompatible with my combination of hardware.
  • The combination of all of the above means that it won’t boot headless.

 

These issues weren’t immediately obvious. The storage issues showed up early on, once I added the extra drives, but others took longer to show their face because, while I’ve been using it for its intended purpose for a couple of weeks now, I only just finally got around to moving it off the corner of my desk and into its final position on a shelf in a closet. I assumed this move would be relatively uneventful. It wasn’t, it was frustrating and tedious.

By way of context, I thought I’d give a few more details on my installation.

The system drive is a 256gb Crucial MX100 SSD. The root volume is relatively small, like 8GB or so. There is a small swap partition, an EFI partition, a good chunk of unused space  as a lazy sort of SSD over-provisioning for longer life, but the bulk of the drive is set aside as for SSD caching of various volumes using Bcache. The root volume is un-exotic though, straight ext4. I’d intially set the system up to boot using legacy BIOS, but after some backflips, managed to convert it to use gpt partitions, and UEFI booting.

The SSD is connected to the main SATA3 controller on the Kabini SoC, as is a 3TB Western Digital Red drive. There are two other motherboard SATA3 ports provided by an ASMedia chip. These are attatched to  3TB and 1TB WD Green drives. None of this is very exotic.

The CPU/Motherboard has integrated video, which I had attached over DVI to an external monitor. The machine is intended to run headless, but I want to run some OpenCL stuff on the GPU, so I had to install video card drivers. By default, the system installed the open source radeon driver, but, from what I could tell, this doesn’t yet have OpenCL support, so I switched to the proprietary binary flgrx driver.

With that background out of the way, I’ll detail the many annoyances I’ve had with this system.

First off, I found that it would often boot slowly, or hang all together, and this tended to involve drives connected to the ASMedia SATA controller. Sometimes it would hang or take forever to detect connected drives. Other times, it would hang on the main BIOS screen, while lighting an activity light on one of those drives. After some trial and error, I figured out it worked much better if I disabled “Compatibility Support Mode” (CSM) in the boot section of the BIOS setup.

The next problem came when I shut the machine down, detached it, and moved the machine to its final location. When I rebooted it, it emitted 5 sharp beeps and then didn’t seem to do much of anything else, except light up the activity light on one of the drives connected to the ASMedia controller. I tried leaving it for a while, to see if it proceeded to boot, but finally gave up and tried resetting it. That didn’t work either, no beeps this time, but it still seemed to hang with the drive light activated. I moved it back to the desk, hooked up the monitor and tried to figure out what had gone wrong.

I found that the BIOS seemed to have reverted back to compatibility mode, moreover, the primary GPU was listed as being PCI Express, rather than integrated. A little digging and I learned that the 5 beeps meant “without vga card.” I mucked around a bit more, trying different things, before reaching the conclusion that this board has major problems, at least for my application.

I’m not sure what I’m going to do next. I realize it might be worth disabling the boot recovery mode, because that may be part of the reason it is falling back to a problematic BIOS configuration. My guess is that I may still have trouble with the internal video, but I might be able to address that with an explicit kernel option (assuming that the boot process still continues). Another option is to see if I can hook something to one of video ports that tricks it into thinking a monitor is connected.

My not-so-inexpensive mini-ITX home storage server build

Posted on by
2

IMG_5779I wrote about the tortuous path I took to building a new home server in an earlier post. Now I’m actually going to take the opportunity to document the build.

First off, the parts list:

My goal was to have an inexpensive, capable, efficient, compact machine to run as a home server as an upgrade to some older Marvel Kirkwood based servers I have. Intel’s BayTrail Celeron CPUs were an interesting option, but I chose to go with an AMD Kabini APU, for a few reasons. Formost among them, it was available as part of a $100 bundle with a case and a motherboard with four 6Gbps SATA channels, rather than the two common on BayTrail boards.

Norco S4-ITX case (top) compared to Coolermaster Elite 130

Unfortunately, the Cooler Master Elite 130 case in the bundle was bigger than I really wanted, so I ended up spending another $100 to get a Norco ITX-S4.

Unfortunately, the ITX-S4 required further upgrades and part-swapping.  The stock fan was amazingly loud at full speed, and because it was a 3-pin fan, couldn’t be regulated by my motherboard, so it was loud all the time. I ended up replacing it was a Arctic F8 PWM fan. The new fan is quieter, at full speed, and I set up a custom fan profile in the motherboard BIOS so it spends most of its time turning even more slowly and quietly.

IMG_5578 IMG_5580A bigger issue is that there wasn’t enough clearance between the stock CPU fan and the bottom of the hard drive cage, which made it impossible to install the motherboard.

I was able to fit it in by removing the hard drive cage, but this resulted in about 1mm of clearance between the top of the can and the bottom of the drive cage.

I considered many options. The AM1 platform used by the socketed Kabini chips like the Sempron I used is still relatively new, and its a low cost platform, so 3rd parties haven’t been rushing to produce aftermarket heatsinks. I looked at the possibility of modifying a heatsink intended for another application, and considered grinding 5mm off the stock heatsink.

IMG_5769I finally realized that the stock fan was 15mm thick, and so I fitted a 10mm fan instead. Cheap and easy, so I did it. This seems to work pretty well, but I’m considering either cutting a hole in the bottom of the drive cage to improve airflow, a wider, lower-profile heatsink with a larger fan, or making some sort of baffle to improve airflow so I can keep the chip cool at lower quieter fan speeds.

Overall, I’m going to have to give mixed marks to the ITX-S4 case. I like the size, and the looks, but there are a number of downsides. In addition to the noisy stock fan, the build-quality is kind of mixed. The sides bow a bit, but the biggest issue is the drive trays. The latching levers feel a little flimsy. I worry I’m going to break something when I insert or remove them.

IMG_5752Even worse, the latches that are supposed to hold the levers closed keep slipping free of the levers. This has happened at least once to each of the four drive trays. If it happens again I’m going to need to resort to some superglue.  This is completely unacceptable for a $100 case!

 

 

A few more notes on things that didn’t work out as expected. I purchased an IO Crest 2 Port SATA III PCI-Express x1 Card so that I had an extra SATA channel available to take advantage of an internal SSD bracket, saving the drive bays for 3.5″ drives. Unfortunately, the machine resets during the linux boot process, shortly after identifying devices connected to the card, so I’ve set it aside for now.

I also tried Rosewill 19.7″ Serial ATA III Blue Round Cables, thinking that the round cables would be easier to route. They weren’t. First off, I missed the fact that the ones I bought have 90° connectors on one end, which doesn’t work given the configuration of ports in the drive cage.  In addition, the 180° connectors were too thick for the close spaced SATA ports on the the motherboard. Moreover, the cables were still quite thick and stiff. I found it easier to use standard flat SATA cables. Some of the ones I had on hand were a tight fit because of the length of the molded connectors, so I picked up some matching cables with shorter connectors.

IMG_5732IMG_5740

 

Overall, I’m pretty happy with the way the project turned out so far. With a 3.5″ Western Digital Green HDD, two surplus laptop drives (one 7,200 RPM the other 5,400), and the SSD, it idles at about 25W. With cpuminer and sgminer mining litecoins on the CPU and GPU, respectively, it uses ~45W. At full-load on a warm day the fans are a little noisy, in particular, the CPU fan has to spin at nearly full speed, (4,500 RPM), to keep it under 60°C, otherwise the CPUs thermal throttling will kick in, which seems to shave ~5W off the load by dropping the CPU and GPU clocks by 20% or so.

Next step is to install Debian and migrate over data and services from my old server. In the longer run, I may revise this build with a BayTrail build for lower power consumption and noise.

 

My costly mistakes on the path to a somewhat inexpensive home server

Posted on by
1

For the last few years, I’ve been using a couple of ZyXEL NSA320 NASs with Debian in place of the stock firmware as home servers for files, backup, and misc applications. Recently though, I figured out that one of them has a flakey SATA channel, plus, I’ve been wanting better performance for Time Machine backups, AND I’ve had the urge to build a new PC.  So, I decided to address all three issues at once.

In retrospect, the sensible thing would have been to buy an Gen 7 HP Microserver, but when I started out, I didn’t have as much information as I do now. I was drawn to a $95 bundle on NewEgg that included a Cooler Master Elite 130 Case, a ASRock AM1B-ITX mini-ITX motherboard, and an AMD Sempron 3850 quad-core Kabini APU. I thought I’d have a system ready for my existing drives for just $150.

I was shocked by the price of RAM, and irritated with the limited options for low-wattage, efficient power supplies. After much dithering, I realized that the hours I was wasting dithering over my sub-optimal choices was more valuable than just spending an extra ~$50. So I found the best price I could on 4GB of DDR3-1600 RAM, added a 90w pico-PSU + an efficient 80w, 12v AC-DC power brick and got on with the build.

The parts arrived in short order, but once I saw the size of the case, I realized I was going to need to rethink things. The Cooler Master Elite 130 is well made and quite compact for a case that can take a full-sized video card, optical drive, ATX PSU and a CPU with a big heatsink/fan combination on top. Its pretty huge though for a machine with an external power brick and two hard drives. I started looking for other options and wasn’t too happy with what I found.

As far as I can tell, cases with just enough room for an mini-ITX motherboard and two 3.5″ HDDs don’t really exist, or if they do, they are bigger than I’d like and cost more than I wanted to spend. I started to consider cases with room for more drives. Most of them were about as big as the Elite 130, which could hold 3 full sized drives along with some 2.5″ drives, but it was still bigger than I wanted.

I also considered building my own case, and quickly rejected the idea, but kept returning to it over and over. Again, I decided to loosen the purse-strings to save time and energy, but there just weren’t many choices. I didn’t need hot-swap drive bays, but most of the small cases came with them. I finally settled on the Norco ITX-S4 case for $100.

At about this time I got a notification about a special price on the HP Microservers. The cost was ~$270. For that price, I got the whole machine, including 2GB of ECC protected RAM, ready for my hard drives. It was too late though, I was already in for $200. I bit the bullet and ordered the Norco case. Since I had the room for it in the case, I decided to move up my plan to use an SSD as a cache drive to speed up performance, and bought a a 256GB Crucial MX100 for $110.

Soon thereafter, the Microserver went on special at least two more times, but now I was in for $300 (not including the SSD). The ECC memory would have been nice though, and I seriously considered setting aside the  CPU, motherboard and RAM I already had and buying something that supported ECC. After spending, again, much too much time determining my options, I decided against it, because it would have cost another $300 or so, and my limit was $200.

I wasn’t done yet though. Once I assembled the system in the new Norco case, I realized that I had another problem. The heatsink/fan combination that came with my CPU was too tall, and the space for it was too short — the top of the fan was pretty much flush with the bottom of the hard disk cage. Only now, as I’m writing this, do I realize that it would probably have been perfectly acceptable to Dremel cut a hole for the fan in the bottom of the HDD cage. Instead, I went looking for alternative CPU coolers that would fit better. There weren’t any, at least not any made for the AM1 platform. I took measurements and went looking for options that might fit, or that I could easily modify. I didn’t find much. I considered grinding down the top of the heat sink and reducing its height and cooling area by 10-20%, since it is a low-power CPU in a low-power application.

I was leaning towards ordering a generic heat-sink that would work with minimal modification when I realized a better solution. The stock fan was 15mm-thick. A 10mm-thick fan would open enough of a gap and solve my problem for less than $10 and 10 minutes work. Somehow though, the first fan I managed to order was also 15mm-thick…

The case had another problem too, the 80mm exhaust fan it shipped with was LOUD at full speed, and it ran at full speed ALL THE DAMN TIME because it only had 3 pins, and my motherboard required a 4-pin chassis fan for speed control (it does speed control on a 3 pin CPU fan though). This time though, I ordered the right thing, first time.

I did make another mistake though, I didn’t like my jumble of SATA cables and found them hard to route, so I decided to order some nice blue ones with round cables. Unfortunately, I missed the fact that they had 90° connectors on one end, which wouldn’t work with the positioning of the sockets on either the SATA backplane, or the motherboard. What’s more, the plastic molded on the ends of the cables was too thick, so 180° connectors wouldn’t have worked either. I ended up ordering more SATA cables!

Ah, but thats not the end of it. I decided to order a PCIe SATA card so I could mount the SATA drive on an internal bracket and save all the hot-swap bays for 3.5″ drives. I was happy to find an inexpensive card that used the same chip as the motherboard. Unfortunately, if I hook any devices up to it, Linux crashes on boot, shortly after detecting the device on the external card.

This post is long enough. I’ll document the actual build in a separate post after I have dinner.