If you know me, you know that one of my favorite events each year is the Eagle Cap Extreme Sled Dog Race. No, I’m not a big fan of dogs, or sleds, but when you put the two together, you get a really fun annual event in the wilderness of Eastern Oregon. The race runs 200 miles through the Wallowa Mountains near Joseph, OR, and is far from any commercial communications infrastructure. Each year, I go to great effort and expense to travel to the other side of the state with lots of gear and help the other hams provide excellent communications facilities in the woods for a few days, where there would otherwise be none.
The communications team is headed up by an excellent guy with finely-honed organizational skills suitable for running a group responsible for life-safety operations like this. Last year, we discussed a way to make things better, by logging all events and personnel in an electronic system. This would provide a digital record of the entire race, as well as a way to display more in-depth information to the administrative folks at HQ, and potentially to the spectating public. The net control and administrative folks run the race from the community center in Joseph, OR, which has commercial power, heat, and an internet connection, so an electronic system like this is possible, as long as it doesn’t become a liability.
We settled on a drupal-based system, which could be made to provide almost all of what we needed out of the box with things like Views and CCK. Being web-based meant that it was easy to access from multiple devices, and to collaborate on the design and implementation ahead of time. The only non-standard thing we really needed was facilitated by a small module I wrote to provide some additional fields to a few Views queries.
We smoke-tested this system over the summer at the Hells Canyon Relay Race, a shorter and slightly less complicated event, but with many of the same challenges and requirements of ECX. The goal was to have the system accessible in two ways:
- The net control folks had to have everything local in the building. This was Eastern Oregon, not Manhattan, and internet access reliable enough to depend on for something like this was not available. To provide this, we ran the MySQL and Apache/PHP servers on a Linux laptop, with a local web browser. This allowed someone to sit at the laptop and operate as an island, if necessary, but also for other laptops in the room to connect to the system as well.
- Anyone outside the room that needed access to the system connected to one of my colocated servers to do so. This machine received replication updates from the master copy of the database on the laptop server to keep it in sync, and was marked read-only to avoid anyone inserting something that net control wouldn’t be able to see.
This provided a reasonably robust setup, avoiding the need for external folks to come into the system over the temporary internet connection to the net control building. I used a persistent SSH tunnel to the external server from the laptop, which allowed MySQL traffic in both directions if necessary.
Of course, during the HCR event, the net control station’s internet connection never went down, but the folks working there wouldn’t have even noticed since they appeared to be working only on a local copy of the data at all times. Organizationally, the system was a big success, and things looked good for its use in ECX this year. There was only one concern: what if the net control building got hit by a missile and someone external needed to take over net control responsibilities and start to modify the data on the external server? Since that copy was marked read-only, this wouldn’t be possible.
For ECX, I’ve now set things up with MySQL multi-master replication. This allows both copies to be writable in both places, effectively allowing either to become an island if necessary. As long as the two systems can see each other, anything added to one is also added to the other. If they become separated for a period of time, they’re still functional, and they sync back up as soon as they’re able to talk again. While this is rather nightmarish for a bank or stock market, it’s actually exactly how we want the system to behave in our scenario.
ECX 2013 is only a few weeks away, so we’ll be running this at full scale pretty soon!