I use Crashplan for backup on OSX. Tonight I happened to notice that the OSX windowserver process was consistently burning ~25% of a CPU core on my Mac for no reason. I also noticed that Crashplan was backing up my computer at the time.
I decided to try switching the preferences for the Crashplan menu bar icon to disable animation, windowserver’s CPU utilization dropped. I turned animation back on it jumped back up.
Someone should change the default.
I installed iOS on both my iPad Retina (3rd gen) and my iPhone 4s. I had some misgivings, since they are both at or near the bottom of the barrel in terms of supported hardware for this release. After some research though, it sounded like the iPad would do reasonably well. As for the iPhone, well, I should have a newer model in a week or two if it doesn’t work out.
My report is that both of them seem to work just fine. There may be a few more rough spots and hitches than before, but if there are, they are rare and subtle (for the most part), and they are counterbalanced by new features and refinements.
One issue on the iPhone 4s is that all the other models supported by iOS have taller displays, and so some UI is more suited to larger phones. The only place where this really stands out is when composing an email. Between the predictive text selections above the keyboard (which can be hidden, if desired) and the new UI affordance for easily switching between the email you are drafting and other messages, there isn’t much room to read the text you are writing.
There are definitely signs of memory pressure though. Tabs in Safari need to reload more (too) often and switching between some apps can take a little too long.
My advice for iPhone 4s users is to wait until they release a round of bug fixes, unless you absolutely need to try the latest and greatest.
As for the iPad, I don’t notice many differences in interaction experience from iOS 7. Safari may need to reload tabs a bit more often, but no where near as often as on the iPhone.
My advice for anyone with a 3rd generation iPad, or better is: If you want it, go for it.
If you want to know why you’d want to upgrade in the first place, there are lots of reviews of what’s new in iOS to choose from. Its worth noting though, that some features aren’t available on all devices. It looks like my devices will be left out of some of the seamless handoff as you move from a Mac to your iPad, or vice versa. One thing that just worked though, incoming phone calls show up on my iPad, and I can choose to answer them there, if I want. Not a huge deal, but convenient in some situations.
I recently decided to buy my first oscilloscope to help with the process of learning more about electronics. The process of selecting an oscilloscope has been longer than expected. I started by looking at the scopes that Sparkfun and Adafruit offered, which lead me to wonder what other options were out there. I was a bit overwhelmed at the variety of models and manufactures, but managed to cut through a lot of the noise.
In this post, I’ll talk more about how I narrowed things down further, and what I ended up choosing.
To start, its worth covering why I wanted an oscilloscope in the first place. It all stems from a growing interest in the declining costs of capable “systems on a chip,” low power communication technologies like Bluetooth LE, and even WiFi, and the accessibility of the Chinese electronics manufacturing supply chain. Taken together, it seemed to me that I should learn enough about electronics to have a sense of the possibilities and limitations, and to be better able to collaborate with people with deeper expertise. This approach served me well in the past when I worked on software projects; I wasn’t a programmer, but I understood enough to be a good collaborator.
In looking around for learning projects, I decided to explore the world of power sources for these devices with a new site called Power Cartel. Part of what I’m doing on Power Cartel is doing teardowns of battery packs and chargers, with the goal of collaboratively creating useful opensource designs. In order to dive deeper in my teardowns, and support my own design efforts, I need to be able to start exploring what is happening inside the circuits and understanding how the different components interact.
There are a lot of test instruments. I already have a basic multimeter, but I felt like I needed a way to look at signals over time. There are two instruments that fit that job description. A data logger or chart recorder is useful for looking at signals over long periods of time (ie hours), where as an oscilloscope is good for looking at signals over much shorter periods, seconds down to micro, or even nanoseconds. I actually want to look at signals over both time periods. I want to look at current and voltage for charging and discharging batteries over the period of hours or days, but I also want to look at sub-second changes in signals.
I decided to start with an oscilloscope, rather than a data logger, because the sub-second changes are more fundamental. With a better understanding of such things, I could build my own data loggers. Moreover, most modern digital storage oscilloscopes can actually record signal changes over longer periods of time, and they can be connected to a computer for control and data recording.
I originally thought that some sort of oscilloscope module that I could connect to my computer or iPad would be a good place to start because it would save me money and space. I quickly learned there were problems with that approach. Most electrical engineers and technicians are trained on traditional stand-alone oscilloscopes, and training aside, traditional stand-alone oscillosopes are often easier/faster to work with because they have dials and buttons arranged in a user interface that has seen steady improvement for close to a century. Connected scopes are newer, and more of a niche item, so their software is less refined. More importantly, because they are a niche item, there is less competition and scale to push prices down, so any savings that might come from omitting a screen and controls is offset. As a result, USB scopes are not any cheaper than stand-alone scopes with otherwise similar specifications. The space savings was still attractive, but I decided I was going to purchase a stand-alone scope.
With that decision made, I had to figure the other key specifications for my work. For any scope, whether an older analog scope, or a more modern digital storage scope, bandwidth is a key consideration. Bandwidth determines the range of signal frequencies you can measure with the scope. Adafruit sells scopes with 50MHz and 100MHz bandwidth. Sparkfun’s stand-alone scope offering is 100MHz. From a little reading, the switch mode power supplies I’m going to be working with typically operate in the range of 200KHz to 2MHz, and the microcontrollers I’m working with operate at 4-16MHz, or perhaps 50-70MHz. Some of the wired communications protocols I’m using may be 5MHz. It would seem then that a 50-100MHz scope would cover almost anything I’m likely to use it for in the near future.
Closely related to bandwidth is the sampling rate. Sampling rate is the number of times per second a signal is read. For a pure sine wave, an accurate estimate of frequency and amplitude requires a sampling rate that is at least two times higher than that of the signal being measured. One of the big reasons for having an oscilloscope though is to look at signals that are not pure sine waves. A lot of digital communications use square wave signals, and a lot of signals, whether square, sign, step, pulse, or something else, can get distorted by characteristics of the circuits they travel in. A rule of thumb I’ve come across is that the sample rate should be 4x the frequency of the signal you are observing.
Almost all the scopes in my price-range ($400-650) have a 1GHz max sample rate. That may sound like overkill for a 100MHz scope, but in this segment of the market, that sample rate is across all the channels of the scope. So, if you have a scope with two channels, and you are using both of them, thats a 500MHz rate per channel, which is closer to the rule of thumb.
The next thing I considered was the number of channels. Each signal you measure requires a channel. Often you are comparing two signals against each other some how, and so, not surprisingly, most scopes have at least two channels, and, indeed, in this entry level segment, most don’t have more than two channels. I found a few with four channels, though. Most were out of my price range, but one, the Rigol DS1074Z was available for under $600. A two channel scope would probably do everything I reasonably needed, but the option of having four channels was intriguing. Four channels would allow me to look at voltage and current at the same time for both the input and output of a power supply. Even better, I could use some of the channels as a basic logic analyzer, and look at how analog signals changed in relationship to specific digital signals.
The consideration of the number of channels also turned my attention to scopes with a logic analyzer option to look at even more digital signals. Oscilloscopes with this feature are generally called Mixed Signal Oscilloscopes, which add 8 or 16 logic channels. The added digital channels come with an added cost. There is a $600 mixed-signal scope from Rigol (DS1052D), but it only has two analog channels, 50MHz bandwidth, and 1Mpts of memory. The Rigol MSO10747Z, which is the mixed signal version of the DS1074Z is $250 more expensive, and out of my price range. Since people seemed less bothered by USB logic analyzers, and since a cheap one could be had or $20 or so, I decided that I didn’t need a full logic analyzer.
Another consideration is the amount of memory available to hold samples. The Rigol DS1074Z has memory for twelve million sample points (12Mpts). Some of the alternatives I considered, like the Siglent SDS1102CML only has 2Mpts of memory, and the Siglent SDS1074CFL only has 24Kpts.
By now, you’ve probably figured out the DS1074Z is the scope I’m leaning towards. I took repeated looks at the other options from Rigol and Siglent, but kept coming back to the DS1074Z.
- Siglent SDS1072CML for $319: +lower price -2 channels -memory
- Rigol DS1052E for $329 +lower price -50MHz -2ch -smaller screen – older
- Siglent SDS1102CML & SDS1102CNL ~$360 +lower price +100MHz -2ch -smaller memory -can’t tell how they differ from each other, other than memory
- Gatten GA1102CAL $400 +lower price +100MHz -2ch -smaller memory
I would have liked spending less, but I wasn’t willing to go with a last generation Rigol, or forgo 2 channels and a bunch of sample memory just to save $150 or so
- Siglent SDS1202CNL+ $546 +200MHz, 2Gigasamples/s -2ch -memory
- Rigol DS1052D $610 +16 channel logic analyzer, -2ch -memory
The higher bandwidth and sample rate of the Siglent just didn’t seem that compelling, and nor did spending another $60 for the 16-channel logic analyzer on a last generation instrument.
- Siglent SDS1074CFL $723 -higher price +2GSa/s, =4ch, – memory
- Siglent SDS2072 $805 -higher price, +2GSa/s +memory -2ch +larger screen
- Rigol DS1074Z-S $818 +signal generator
- Rigol DS1104Z $830 +100MHz bandwidth
- Rigol MSO1074Z $835 +16 channel logic analyzer
- Rigol DS2072A $839 -higher price -2ch +2GSa/s
The price was really enough to knock all of these out. The Siglent SDS1074CFL was tempting since it had 4 channels and a higher sample rate, but the higher sample rate isn’t that important to me at this point, and so not worth the extra $175 or so. The only other one I gave serious consideration to was the MSO1074Z, but the integrated logic analyzer just didn’t seem compelling enough to drop another $280 or so.
In the end, the Rigol DS1074Z won me over with its combination of price, four channels and deep memory, along with the fact that Rigol seems to be a well-understood quantity at this point. This model has been out for almost a year, most of the bugs are known, and many have already been address. There are lots of good in depth reviews and tutorials for Rigol scopes. Certainly more than I saw for Siglent/Atten/Gatten or Owon.
The Rigol has another thing in its favor. It apparently shares the same hardware as the DS1104Z, and people have figured out a way to unlock the higher bandwidth. They’ve also figured out how to unlock some otherwise extra cost after market options, most of which don’t interest me, but its nice to have the option.
In which I sit in my office in Seattle, watch the Apple live stream and comment on it:
- 10:04am PDT
- I still see the test pattern and schedule. This never would have happened when Steve Jobs was Alive.
- Time Cook is on, and then the bars come back…
- Ok, its working again…and fail.
- It is working again, for now…
- Bigger phones, um, hooray? I guess.
- Phil is talking about % more pixels. Lame…
- Now he’s finally talking about why that matters: two up view.
- Thinner is good though, I guess, as long as it feels solid.
- What is with the translation track?
- And the stream breaks again…
- Something about apps taking advantage of larger displays.
- Desktop-class scaler?
- 10:24, it is working again…
- A8+METAL = FAST GRAPHICS
- BETTER BATTERY LIFE
- New M8 co-processor.
- Distance and elevation estimation.
- Ok, so, I do want one, just to be clear. My iPhone 4s seems a little too old now.
- Hmm. Focus pixels?… Phase detection autofocus!!! Fast autofocus! YES I MUST HAVE THIS!
- Something about the macro mode that I didn’t catch.
- Apple broke the internet.
- 120 or 240fps video. Sweet
- Continuous focus for video…
- More stream breakage
- 64GB for the mid-range for $299 on contract! Yay
- $100 for thoe the 6plus
- Steady incremental improvement can really add up. I was pretty “meh” leading up to this event. I felt the same way when it started, but everything adds up.
- The payment stuff seems pretty important, especially the API…
- One more thing…
- It would be really really awsome if it isn’t a watch.
- Its a new planet!
- And the space ship that will get you there
- No, it really is a watch…
- I must say, it looks really cool…
- …for a watch.
- I’m so over watches…
- At least I thought I was…
- For 10-15 years…
- At least it isn’t called iWatch.
- The personal touch stuff is really interesting…
- I wonder if 3rd party developers will be able to create “watch face” apps.
- I look forward to the bio-mimetic bands for the next version.
- They will graft themselves into your skin.
- Or rather, your skin will incorporate them.
- This is a long-ass video.
- Why does Kevin Lynch seem familiar…
- He’s the former Adobe Flash Evangelist
- Gruber called him a bad hire for Apple…
- Coldplay? What a fucking bozo.
- 3rd party dev support sounds pretty good.
- I’m running out of steam here…
- I wonder how the Apple watch works when you don’t have your phone with you?
- The live stream was working so well, until, another fail.
- So, if Apple Pay is built into Apple watch, and Apple watch works with many versions of iPhone, plus all the accessories, Apple has multiple revenue streams for the Apple watch. I wonder if the pricing will be subsidized.
Earlier I wrote a long blog post that condensed down the path I’ve taken to buying my first oscilloscope. I hadn’t decided what to buy though., so this post will continue with more of the decision making process.
But first, a short review of the ground I covered in my earlier post. I explained that I needed an oscilloscope to help me better understand why circuits work, or don’t work. I’m not just doing this for fun, my goal is to start designing and building some useful circuits of my own. I covered how I started figuring out what my options were for an entry-level scope, in order to better understand what I should be thinking about.
At the end, I’d concluded that buying a cheap compact oscilloscope like the DSO Quad wasn’t a good idea (not capable enough to be useful for long, and not cheap enough given its limitations), and why a USB scope wasn’t going to save me any money (the meager savings from omitting a screen and controls are offset by the higher prices that come with limited volumes).
I started by considering Rigol scopes like the dual-channel Rigol scopes sold by Adafruit, or the dual-channel. Gratten scope sold by Sparkfun. At this point though, I’ve all but ruled out those options. The truth is, at $390-460, they are all within my budget, and with 50-100MHz of bandwidth, they can all do what I think I’m going to need them to, which is to deal with ~1-2MHz signals from switch mode power supplies. The problem is, while I don’t know much about oscilloscopes, I’ve already figured out that I have better options.
A hint of this is already obvious among the three scopes on Adafruit and Sparkfun. Both the Rigol scopes have 5.7″ 320x24o displays, while the Gratten has a 7″ 800×480 display. Of course, even I know that while a large screen is better, screen size probably shouldn’t be the first, or even fifth thing to judge a scope on. Things like bandwidth, sample rate and memory size, on the other hand, are much more important, and here the Gratten scope matches or beats the Rigol offerings sold by Adafruit. For $400, it offers 100MHz bandwidth and it does so for just $10 more than the 50MHz Rigol scope and $50 less than the 100MHz Rigol. That might be enough to convince them to buy the Atten scope from Sparkfun, me, on the otherhand, I get curious about what else is out there.
What I found is daunting. There are so many options, I’m tempted to buy the Gratten GR1102CAL from Sparkfun and get on with things, but that wouldn’t be like me. I press on.
One of the things I notice in my research is that a lot of these scopes have similar model names. This isn’t exactly a surprise though, because often the numerical part of the name represents some fundamental characteristics. Taking the Sparkfun and Adafruit offerings as an example:
The last/right-most digit on all three scopes is the number two (2). If one were to check out more models of scope from Rigol and Atten (among others), you’d probably recognize that this is the number of input channels. Two channels is common among entry level scopes. Higher end scopes sometimes have 4 channels.
With enough perspective, it becomes obvious that the middle two digits often indicate the bandwidth offered by the scopes: 100MHz or 050MHz in the case of these examples.
The meaning of the first digit seems to indicate the model line, and the letters at the end often indicate key options, like memory size.
The similarities don’t end with just the numbers though. Model names between manufacturers seem very similar. Sometimes even the names seem similar. For example, there is the Gratten GR1102CAL sold by Sparkfun, and there is another manufacturer called Atten, which has a model SDS1102CAL; the specs are pretty much identical between the two. In fact, the case is too, down to the position of the front USB port. Oh, and then there is the Siglent SDS1102CAL, also with remarkably similar specs and appearance.
There is an explanation though, which is that some manufactures sell the same or similar equipment under multiple brands and/or sell equipment which other brands sell under their own label. This simplifies things somewhat, if you can keep everything straight.
I learned something else too, which is that while the Rigol scopes Adafruit sells were revolutionary revolutionary for the value they offered when they first came out, they are a bit long in the tooth now. Last year Rigol released the DS1000Z series, which occupies the same price point the DS1000 series originally did when it came out.
I’ve learned a lot more about the options available for good entry level scopes. Now I’m really sure that I don’t know what to buy. Narrowing down my options will have to be the subject for a new post
My new project is going to involve testing and designing electronics, and so I find myself in need of a bunch of tools I don’t have. Foremost among them, is an oscilloscope, so I can probe circuits in devices to try and understand how they work, and why they don’t work.
My overall electronics aptitude is pretty mediocre. I had a job soldering connectors to serial cables when I was a kid, but my soldering skill isn’t very high and most of the electronic theory I learned in my introductory physics course in college is long forgotten. I have used an oscilloscope though. I used one when I learned about electrophysiology techniques for looking at the activity of individual cells, and I also used one to tune the video system of a cell-florescence microscope to take full advantage of its dynamic range.
More important than what I’ve done with oscilloscopes is that I know what I’d use one for now. I think it will come in handy for looking at the operation of DC-DC power supplies, and just generally for probing circuits. Beyond that though, I’m a novice, and so I went looking a place to get advice. Actually, I didn’t have to look, because I’d already found the EEVBlog Forum, which even has a special section where people are supposed to go easy on beginners for asking dumb questions. Still, they ask that you provide plenty of information when you ask a question, so I had to do some shopping on my own, so I could get a sense of the questions I should ask, and the background I should give.
My first stop was to see what Sparkfun and Adafruit, two online stores that specialize in electronics for hobbyists, were offering. Adafruit has two options, one a 50MHz, two-channel Rigol DS1052E digital storage oscilloscope, the other, a Rigol DS1102, is very similar, but has 100MHz bandwidth. Sparkfun offers the Gratten GA1102CAL, which is also a 100MHz, 2-channel digital scope. They also have two USB scopes for use with your computer, the MSO-19, and MSO-28 from link instruments. Sparkfun and Adafruit also both carry the small, inexpensive DSO Nano and/or DSO Quad.
I’d been considering a DSO Nano or Quad because of their low price and compact size, but people who seem to know better hate them, first because their user interfaces are difficult to use compared to a real oscilloscope, but also because they can’t actually do much. Of course, my needs, as I understand them, aren’t that demanding. I think the signals I’ll be looking at are in the 1-2MHz range, which should be within the capabilities of the DSO Quad, but it doesn’t leave a lot of headroom, and at $200 bucks, its a bit too expensive given that you could get a more capable used analog scope for less, or a new digital scope for another $50-100 more.
I also considered a USB scope. They are more compact, and I also assumed they’d be cheaper, since they didn’t have to include a screen or UI, and I also thought it might make it easier to work with the data on a computer. I was wrong on multiple counts. USB Scopes are a bit of a niche item when compared to a benchtop scope, because regular O’scope users find physical knobs and buttons more productive and easier to use than the often poor software that ships with USB scopes. As a result, volumes are lower, and there is less competition to drive prices down. Moreover, screens are pretty cheap these days, so eliminating one isn’t a huge cost savings. Put the two together, and a decent USB scope is about as expensive as a dedicated scope with similar specs, and you can hook most dedicated scopes up over USB if you need to anyway.
Exploring the existing options helped me think through what I needed and why. I was ready to ask the experts over on EEVBlog. I’ll cover that in my next blog post.
An english datasheet for an intriguing ~$5 WiFi module with an microprocessor core… ESP8266_Specifications_English.pdf – Google Drive.
The same base urges that led me to upgrade my home server, have had me itching to upgrade my home network. The server build sated that hunger for a while, but now that it is done, I’ve been feeling the pull, again. I’ve held out, but yesterday, I thought I was going to HAVE to upgrade because one of my routers started acting up. Happily/sadly, successful troubleshooting has stimied such rationalizations.
The current network arrangement consists of a Netgear WNDR3800 running OpenWRT as the main router/firewall. It has direct connections a couple of small ARM servers running Debian, which run various services. It also provides WiFi access. At the other side of the house, there is another router, a Netgear WNDR3700, also running OpenWRT that is bridged to the main router over 5GHz WiFi. It provides wired connectivity for a printer, a server, and a Acu-Link bridge that connects a weather station to the Internet. It also provides WiFi access in the 2.4GHz band and guarantees decent connectivity from our back yard.
I’ve been considering various options for upgrading, with the goals of getting a higher-speed link between the two routers, and a speed bump for my laptop, which has a 3-stream radio, whereas the routers are limited to two streams. I wasn’t entirely happy with any of the options though, which is one of the reasons I’d been able to resist the upgrade urge.
Yesterday though, as I’ve already mentioned, things started getting flakey. I noticed that the second router seemed to be rebooting every ten to fifteen minutes, interrupting connectivity for connected devices for a good 60-90 seconds. I could have used this as an excuse to upgrade, but I tried to troubleshoot the problem first.
I’d recently moved some equipment around, so the first thing I checked was that the power cord hadn’t been partially unplugged. It didn’t seem like it had, but I made sure it was well plugged in and waited to see if the problem continued. It did.
As part of the equipment moves, I’d swapped in a higher-capacity UPS. The UPS had been functioning just fine, in its previous location, but I wondered if it was adding noise that the routers power supply was having trouble contending with, so I swapped in another 12V DC wall wart. At first, that seemed to solve the problem, no reboots in over 20 minutes, but before another 20 minutes were over, the router rebooted again.
I wondered if it was a temperature issue. It was a warm day, though no warmer than other days this summer. I decided to take the router apart to see if there was any dust slowing down convection from the case. Once I got the case open though, I realized that I’d already opened and cleaned out the router a few weeks before. Back to the drawing board.
I decided to open up an SSH session so I could watch the system log (OpenWRT logs to a ring buffer in memory, so you have to use “logread -f.” in the hopes that I’d see something useful before the device rebooted itself. Once I had that running, I took my dog for a ~60 minute walk. When I returned, the router was still running, still logging to my screen. I scrolled back over the logs, and didn’t see anything unusual. I started to wonder if it was a “heisenbug.” Perhaps I was going to have to upgrade after all.
The log filled with information about connections and disconnections from WiFi devices, and IPv6 related housekeeping. A few minutes later though, something different flashed past on the console. I scrolled back and saw that the kernel reported that it detected a low memory condition and killed off a maintenance script to keep from running out of memory. I thought it strange, but I didn’t see what would have changed to make low-memory conditions commonplace, or how they’d lead to a reboot. It was, however, my only lead, and as I sat there thinking it over, I saw another low memory warning.
It was starting to make sense, I imagined how the router could get in a situation where it killed off an essential process and ended up rebooting. It might even be that the router hardware had a watchdog function, which would reboot the device if a process didn’t reset the watchdog timer on a regular interval. Killing that process could lead to a reboot.
Now that my only lead was starting to seem plausible, I dug deeper. I first checked the amount of memory in use with the “free” command. It reported there were less than 2MB free, which surprised me since I remembered it was typically many times that. Next I used “ps” to see what processes were running, and which were using the most memory. None of them were obviously huge, but a few of the larger processes didn’t look quite right.
I have an Acu-Rite weatherstation and an Acu-Link bridge. The bridge relays readings from the weatherstation to Acu-Rite’s servers. I have a system in place to capture the data as it is being transmitted, and feed it to Weewx on one of my servers. Weewx keeps its own record, and also updates Weather Underground. To accomplish this, I have a startup script on the secondary router that uses “ncat” to wait from an incoming connection from the driver I wrote for WeeWX. Once the connection comes in, ngrep sniffs for packets from the Acu-Link bridge relaying data from my weather station to the internet service. That data forwarded overmy network to the waiting weewx driver on my server.
The problem was that there were 3-4 copies of these processes running on the router. Ordinarily, there should only be one. After a connection is lost temporarily and weewx has reconnected, there might be two running before the old copy times out. I tried reducing the max number of connections from four to two, which helped a bit, but it was still cycling the connection much to quickly and for no obvious reason.
I tried restarting weewx, but that didn’t help, it was still connecting and then disconnecting too frequently. To debug things further, I decided to simulate the server connection and see if the data was being captured and forwarded properly. It wasn’t!
From here, I wanted to connect to the diagnostic webserver running on the Acu-Link to confirm that it had a connection to the weather station. I looked at the main router to find the Acu-Link’s IP address, and in the process I realized that it had changed. I hadn’t created a DHCP reservation for the Acu-Link device because DNSMasq, which provides DHCP service on the router, generally provides a stable IP address to devices. Something had happened though, probably when I was moving around hardware, and it had assigned a new IP address.
So, to recap:
- My script was depending on a result of the default behavior of DNSMasq for the IP address of the Acu-Link bridge.
- As a result of reconfiguring my network, something changed, and with it the result of that default behavior, leading to the IP address of the Acu-Link bridge changing.
- As a result, of the changed IP address, my script for sniffing weather data failed to collect that data and forward it to weewx running on my server.
- Because it wasn’t getting data as expected, weewx tried to reconnect to the sniffer script.
- Because weewx was reconnecting as frequently as it was, excess copies of ncat, ngrep and the ash shell accumulated on the router, eating up memory.
- Because of the low memory condition, the kernels out of memory killer (OOM killer) started killing off processes.
- Because some key process was killed by the OOM killer, the router rebooted, continuing the cycle.
- Because the router was rebooting frequently, I decided to troubleshoot it.
- Because I succeeded in troubleshooting the root cause, I have removed a reason to buy a new router.
Sometimes being awesome has to be its own reward, I guess.
Since I couldn’t in good conscience buy a new router, I did a few easy things to fix the problem. I configured the DHCP server to assign a predefined IP to the Acu-Link bridge, and update the script to look for packets from that IP.
There additional mediations to this issue that I probably won’t undertake, including:
- Updating my WeeWX driver to make sure it properly cleans up sockets when reconnecting. This might lead to quicker cleanup of the old processes on the router.
- Updating my WeeWX driver to throttle the rate at which it retries connections.
- Trying to automatically detect the IP of the Acu-Link bridge and then use that IP for the ongoing packet sniffing.
- Adding some code to the sniffer script that will be more aggressive about cleaning up unused connections.
To make a long story short, this motherboard doesn’t work well for my intended use as a headless Linux server. The problems are manifold and interconnected:
- If I boot headless, it decides the integrated GPU isn’t being used.
- Once it decides the integrated GPU isn’t being used, it tries to use a PCI Express GPU, which it doesn’t find.
- At some point, it also reactivates compatibility mode.
- With compatibility mode activated it is, ironically, incompatible with my combination of hardware.
- The combination of all of the above means that it won’t boot headless.
These issues weren’t immediately obvious. The storage issues showed up early on, once I added the extra drives, but others took longer to show their face because, while I’ve been using it for its intended purpose for a couple of weeks now, I only just finally got around to moving it off the corner of my desk and into its final position on a shelf in a closet. I assumed this move would be relatively uneventful. It wasn’t, it was frustrating and tedious.
By way of context, I thought I’d give a few more details on my installation.
The system drive is a 256gb Crucial MX100 SSD. The root volume is relatively small, like 8GB or so. There is a small swap partition, an EFI partition, a good chunk of unused space as a lazy sort of SSD over-provisioning for longer life, but the bulk of the drive is set aside as for SSD caching of various volumes using Bcache. The root volume is un-exotic though, straight ext4. I’d intially set the system up to boot using legacy BIOS, but after some backflips, managed to convert it to use gpt partitions, and UEFI booting.
The SSD is connected to the main SATA3 controller on the Kabini SoC, as is a 3TB Western Digital Red drive. There are two other motherboard SATA3 ports provided by an ASMedia chip. These are attatched to 3TB and 1TB WD Green drives. None of this is very exotic.
The CPU/Motherboard has integrated video, which I had attached over DVI to an external monitor. The machine is intended to run headless, but I want to run some OpenCL stuff on the GPU, so I had to install video card drivers. By default, the system installed the open source radeon driver, but, from what I could tell, this doesn’t yet have OpenCL support, so I switched to the proprietary binary flgrx driver.
With that background out of the way, I’ll detail the many annoyances I’ve had with this system.
First off, I found that it would often boot slowly, or hang all together, and this tended to involve drives connected to the ASMedia SATA controller. Sometimes it would hang or take forever to detect connected drives. Other times, it would hang on the main BIOS screen, while lighting an activity light on one of those drives. After some trial and error, I figured out it worked much better if I disabled “Compatibility Support Mode” (CSM) in the boot section of the BIOS setup.
The next problem came when I shut the machine down, detached it, and moved the machine to its final location. When I rebooted it, it emitted 5 sharp beeps and then didn’t seem to do much of anything else, except light up the activity light on one of the drives connected to the ASMedia controller. I tried leaving it for a while, to see if it proceeded to boot, but finally gave up and tried resetting it. That didn’t work either, no beeps this time, but it still seemed to hang with the drive light activated. I moved it back to the desk, hooked up the monitor and tried to figure out what had gone wrong.
I found that the BIOS seemed to have reverted back to compatibility mode, moreover, the primary GPU was listed as being PCI Express, rather than integrated. A little digging and I learned that the 5 beeps meant “without vga card.” I mucked around a bit more, trying different things, before reaching the conclusion that this board has major problems, at least for my application.
I’m not sure what I’m going to do next. I realize it might be worth disabling the boot recovery mode, because that may be part of the reason it is falling back to a problematic BIOS configuration. My guess is that I may still have trouble with the internal video, but I might be able to address that with an explicit kernel option (assuming that the boot process still continues). Another option is to see if I can hook something to one of video ports that tricks it into thinking a monitor is connected.